The Multiple Comparisons Problem 

Thanks! As I'm trying to get this channel going, feel free to make topic requests. I'd be curious to know what else is on your boards. Appreciate the support!

It was replaced for the newer videos!

Genetics research has a couple benefits over other domains in part because you usually have enough data to do split half analyses. Helps to cross validate your findings!

Thanks for the encouragement! It's sure tough to get to them. Not really any monetary incentive unfortunately. I'd love to do something more in the future, but don't have nearly as much time for the hobby as I used to. I really appreciate the appreciation though! I hope I can find some time soon/eventually.

@@SprightlyPedagogue Thanks for the reply. I understand your point. Won't mind your explanations in form of blogposts as well. Your lucid explanation in one of the replies to illustrate the difference between Fischerian and Bayesian Statistics was also a joy to read. Glad to come across your channel. Wishing you the best in your future endeavors.

At least 2 tests are significant

Yeah, I'll do my best! Warning, this gets into the weeds a bit. In traditional (Fisherian) statistics, we use p-values to determine whether or not an effect is significant. Each of those decisions is a dichotomous, all-or-nothing, discrete choice. The question is whether those choices reflect the real world. Multiple comparisons becomes an issue because random things happen. Rarely, yes. But they do happen. So if we run a bunch of tests, eventually one of them will be labeled as significant... because... random things happen randomly. That's Type I error btw. So, the multiple comparisons problem is a result of our making these significance decisions. It exists because we use these threshold-style decisions as our basis for determining whether a pattern in our data is important/different/rare. Bayesian stats don't operate like that. In Bayes, you aren't trying to determine whether there is a significant effect. You're just estimating how big the effect size is (oversimplifying a bit). It is arguably (though I say convincingly) a better approach for many reasons, but this is one of them. There are no significance decisions to inflate. Take an independent samples t-test for example. Fisherian stats (as a gross overgeneralization) are trying to determine whether the differences in means (relative to the standard error) are big enough to care about. Thus, Fisherian statisticians will try to answer the question: "Is this difference big enough to matter?" Bayesian stats are trying to use those same bits of information (sample size, standard deviation, observed mean difference, as well as our prior beliefs about the expected differences) to answer the following question: "What is our best guess as to *how* different these two groups are? The answer to the Fisherian question might be "yes" or "no". The answer to the Bayesian question might be, for example, "5" or "8." As you get more information and run more tests, you're more likely to get more "yes" answers... randomly. You aren't necessarily more likely to get more "5" answers if the data keeps being consistent with "8." It is the yes/no decision that leads to the multiple comparisons problem. If we're talking about whether there were *any* differences then we have inadvertently created an *OR* probability situation. (Quick primer on OR probability if you don't know it; skip this if you do) Imagine that there four coins in front of you, A penny, nickle, dime and quarter. You pick one up at random. Question 1: What is the probability that you picked up a quarter? That's the question we act like we're asking when we make significance decisions using p-values. If we only did it once, what is the probability that some event (like a big enough difference between two groups) happened? Question 2: What is the probability that you picked up a quarter assuming you get to draw four times? (with replacement). That's a much higher value. If you can pick a coin up, put it back, draw another, put it back... and so on.... you're much more likely to draw a quarter at some point. If you're allowed to check for a significant difference between Var 1 and Var 2 OR between Var 1 and Var 3 OR between Var 2 and Var 3, it's much more likely that you will find a difference somewhere at some point. Again, Bayesian's bypass all of this by refusing to play that game. I'm being a little unfair to Fisherians. Using significance values isn't universal in Fisherian stats. Many good Fisherian statisticians know about this problem and take steps to address it (including refusing to use or report p-values). However, using these p-values in this way is common. Fisherian statisticians *often* do this. They don't *only* do this. They are the *only* ones who do this. Bayesian statisticians don't use these threshold decisions in their work. So there isn't an "OR" probability to inflate. If I didn't answer your exact question or if some of that seems mildly inaccurate, confusing or incomplete, feel free to ask more and I'll try to clarify. There could be a whole video series on this and similar topics.

History of the United States of America account details for your reference I have attached my resume thanks for the night

When you're thinking about independent or non-independent events, you're really talking about the possibility of a specific outcome occurring in a one-off event. If I draw marbles from a bag, what result will I get? The answer depends on what marbles are in the bag. The multiple comparisons problem comes into play only when you keep looking for that single specific outcome but you keep drawing many times. It's the repeated drawing, rolling, and flipping that causes the problem. So each dice roll is independent from each other dice roll. But, YOU keep looking for a four to be rolled. You are not dependent or independent. You're just trying to game the system and get a result you prefer.

edit: your example sucks

Those dice rolls are independent of one another, so they are functionally the same statistical statement.

Rerolling the same die six times and recording the results is no different from rolling six dice and recording the results. In both cases each roll has a probability to result in a 1 of 1/6. The problem of multiple comparisons arises when (implicitly or explicitly) a conclusion about a more general hypothesis is drawn from a number of tests. A "more general hypothesis" could mean a hypothesis like "there are differences between groups somewhere" in a case with more than 2 groups. With every additional group the number of (possible) group comparisons increases. Since each comparison has a chance of resulting in a false rejection of the null hypothesis multiple group comparisons result in an increase of chance to falsely reject the null hypothesis of that "more general hypothesis" (that there are differences somewhere). The type I error rate of such a "more general hypothesis" is sometimes called the family-wise error rate. The more comparisons you look at the higher the chance that you end up with at least one significant comparison just by chance (as a result of random sampling) even though there are no differences in the population. That said, I don't like the analogy of the kid that is rerolling the dice for this either. The kid rolls the dice again if (and only if) it does not like the first result. It does that until it likes the result and then uses that. An equivalent to this exists in data analysis: An analysts performs additional tests until they find one that is significant and then report that one (it is like making up new hypotheses on the go). This behavior is problematic as it leads to spurious results that are essentially worthless. However, this does not describe the problem of multiple comparisons. This behavior is known as fishing (for effects) in the literature. The analogy with the kid that is rerolling the dice suggests that adjusting for multiple comparisons solves the problem of fishing, which it does not. The analogy also suggests that the problem with multiple comparisons only exists when one makes up new hypotheses on the go. This is not the case either. For me the analogy of the kid rerolling the dice is a great analogy for fishing. But aside from the fact that both cases involve multiple comparisons, the multiple comparisons problem that one tries to resolve via a p-value adjustment is independent from fishing. Do you agree?

Great question! The data doesn't change and the actual reality of the true relationships between the variables doesn't change. But, we don't have an exact measurement of these variables. From classical test theory each datum is made up of some proportion of 'true score' and some 'error'. If you look at enough error, you'll eventually see patterns in it. The more you torture your data, the more likely you are to accidentally see patterns in error and misinterpret it as meaning something that it doesn't.

Recommend checking out the book The Signal and the Noise.

Yes. It's annoying. But possible. Usually you're dealing with low effect size. Or low power.

I'll admit to not being able to solve that one based on the info provided. It's been a while since I've dug into BH adjustments. Email if you're totally stuck and I can dig in with you, potentially.