'brown bread' is actually cockney rhyming slang for 'dead', but you still got the right meaning by thinking it was like saying they're toast. I actually looked up where the phrase 'you're toast' came from to see if it was anything to do with brown bread, but apparently not, and it was only first used like that by Bill Murray's character in Ghostbusters (1984).