The expression "brown study" is a fourteenth century British term that started out meaning a dark melancholy mood, but later came to mean a state of thinking deeply about something. Before the two words were ever commonly used together in language, brown was used to describe something dark and sober while being in a study meant daydreaming. The saying is outdated in today's language.
However, the term "browned off" is related to brown study and is commonly used in Britain today. Like "brown study," browned off also used to be more associated with sadness or depression in its earlier usage, but then the meaning changed. Today, to be browned off means to be annoyed or fed up of someone or something. For example, browned off could be used to describe the attitude of protesters or workers on strike.
The term brown study is used in 19th century British literature fairly frequently. For example, Grace S. Richmond's book A Brown Study, published in 1919, even has a chapter entitled "Brown's Brown Study," which is written about a character named Donald Brown, who realizes, as a result of his brown study, that he cannot afford to waste even one of his "happy hours" worrying as he did not have that many hours in his life of being happy to waste. Richmond's use of this term is closer to its original meaning as unhappiness is mentioned in relation to it and not just deep thought.
Contrastingly, Arthur Conan Doyle in The Adventure of the Cardboard Box published in 1893 uses the later meaning of brown study even though the book was published long before Richmond's. For example, Watson tells the reader "leaning back into my chair, I fell into a brown study." Holmes proves to Watson he could tell what Watson's "train of thought" had been. Holmes surprises Watson by correctly guessing that Watson's "reverie" had been about placing a picture on a wall. Doyle's use of the term as thinking fits in with the later meaning of being deep in thought, yet he also uses the older idea of the word study meaning daydreaming.