NOTE: This StatQuest was brought to you, in part, by a generous donation from TRIPLE BAM!!! members: M. Scola, N. Thomson, X. Liu, J. Lombana, A. Doss, A. Takeh, J. Butt. Thank you!!!! Support StatQuest by buying my book The StatQuest Illustrated Guide to Machine Learning or a Study Guide or Merch!!! statquest.org/statquest-store/
@@lcoandrade He has no answer to your question because p values are really one big hack that don't prove anything at all. Search for: "Cohen - Things I have learned so far."
I studied Industrial engineering, however, engineering tends to have a way too "practical" approach in Mexico, so my statistics teachers were more focused on "formula -> result -> verdict" rather than a true explanation of why we do things. This series has helped me a lot in understanding the whole movement behind some of the everyday uses i give to my statistics knowledge. That's truly worthy of a triple BAM! Thank you for sharing your knowledge.
I am a great fan of your work. Here is an idea/need to enhance learning: It would be helpful that after a video or a set of videos you give us problems (or homework) to learn better. In further videos, you could give us the answers. Thank you
Thank you for the video! I have a question though. Since we are testing different drugs, why do we need to consider the p-values of other drugs for False Discovery Rate? I thought these were independent events.
I believe this is a intrinsic problem the way "expediction experiments" works. Even if the drugs are independent, given the multiple testing there is the problem with inflation of type I errors. The same would occur by measuring 100 uncorrelated variables from a homogenous two sample (null). You'd expect false discoveries even if the variables are themselves independent and equall between the groups. Also, the are procedures for FDR based on independent and dependent scenarios, eg., the BH and BY methods.
LOVE YOUR VIDEOS!!! After discovering one of your videos, I started to review my statistics from the very basic. Here is a little question. I can't figure out how to calculate the p-value from the statistical test that compares the means of two sets of samples, based on the things I learned from the last one 'How to Calculate p-values'. In 'How to Calculate p-values' I understand calculating the p-value of one event can see whether the occurrence of one event is that special (
It sounds like you are asking how to calculate a p-value for a t-test. There are two ways to do it. The "normal way" (and I don't have a video for that), and the "more flexible linear model" way. I have a video for that. Here's the playlist: ru-vid.com/group/PLblh5JKOoLUIzaEkCLIUxQFjPIlapw8nU
This video starts out tough: "imagine a virus" Damn, can't you come up with something less hard for me to imagine, such as Santa Claus visiting people's houses all on the same night?
Good contents indeed. Sometimes students find it easy to use critical value method but feel kind of difficulty with p value. Your content explains clearly!
Congratulations and thank you for the videos! I appreciate the clarity, simplicity, and humor in your content. While you probably don't need more compliments, I wanted to express how much I enjoy it and how I find it to be a brilliant way to convey key concepts in statistics. BAM!
So tempted to send this video to my former PhD director... 🙄 Thank you for your immensely valuable content!!! Learning more on your channel than in class 😅
I almost understand the first example about drug A to Z, except for one difficulty. If we assume that the distributions for each drug are independent, from the perspective of drug Z, there’s only one test. Then this will not have the multiple testing problem.
If something has a 5% chance of happening, then 5% of the time we do it, we'll get a false positive. So now imagine we did 100 tests for all 27 drugs. That means 5 of the tests for drug A are false positives, 5 of the tests for drug B are false positives, etc. So we have 100 rows of results, with mixtures of true negatives and false positives. When we test each drug 1 time, we get one of those rows, and it's very likely that it will include at least one false positive.
The P-hacking culture is so endemic that sometimes I seriously get so demotivated that I wonder leaving the field entirely. It's depressing. Thanks, Josh. Sorry about the vent.
If each drug trial was a different separate study though, wouldn't it mean the p-value false positive would still stand? Because the authors are acting independently? So you get different results depending on if one person trialled or the drugs or if one person trialled each drug.
The way the p-value threshold is specified to be 0.05, we expect 5% false positives - and, overall, this is a good trade off of cost/benefit/risk/reward. So, if different people are doing tests, sure they will will get false positives from time to time. But the goal is to limit them within our own study, so we adjust the p-values with FDR.
Hi Statquest great video! I watched this video and your power analysis video and I have one quick question. If you already collected preliminary data, can you perform a power analysis on that data or would that be considered p-hacking as well? Thanks
In the first example going through drugs A...Z you dealt with rejecting the null hypothesis that theres no difference between drug Z and not taking a drug. The next example deals with taking samples from a distribution of people who did not take a drug. At 4:30 what does it mean the null hypothesis is the two groups came from the same distribution? What does it mean that one would come from a different distribution?... Would it correspond to a different distribution of people who took medication instead? Im doing my best to follow along but this one has me confused. P.s thank you for the wonderful videos!
It might be helpful to first watch my video on how to calculate p-values: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-JQc3yx0-Q9E.html and my video on the null hypothesis: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-0oc49DyA3hU.html However, to answer your questions, we're trying to decide if there is a statistically significant difference in the two samples (note: there will always be differences in the two samples due to little random things that happen, so just seeing small differences isn't enough to decide that there is a profound difference between the two groups). If the difference is statistically significant, that suggests there is a true difference - and maybe one drug works better than another. However, the statistical test is not perfect. Sometimes the statistical test will say there is a difference, even when we know there isn't one (because both groups of measurements came from the same "distribution" of people - for example, people that all took the same drug, or people that didn't take any drug at all). When the test fails, it suggests that there is a profound difference in the groups, even though we know there isn't one.
I did not get one part: I understand why if I add 1 more measurement for each group I would be p-hacking. But does that mean that if I get more measurements than the one previously estipulated sample size by the power analysis I would be p-hacking? so if my power analysis says i need 100 samples, but by the time i did the test i already had 500, what then? What happens if instead of adding 1 more measurement for each group, im adding 100 more? Is this p-hacking as well?
Hi I still don’t understand at 1:47 section, why is it p-hacking for rejecting the null hypothesis for drug Z? They aren’t like the latter example where they’re taking the samples from the same population? I presumed we call it a different population for each drug the samples are put on
Say like we have 100 drugs and none of them are effective - they are all variations on a sugar pill. Then we test all of them like in this example. Well, due to random sampling of people taking the pill, there's a good chance that in one of those tests, a bunch of health people take one pill and a bunch of sick people take the other. This will make it look like there is a difference between the two pills even though it's not the pill, it's just the people that take the pill. Thus, this is p-hacking because it looks like the pills are different.
P-hacking is how you get a high volume of papers published and cited. It's incentivized in all academic fields. Just try asking a researcher if you could take a look at their data and they will ghost you.
How do you calculate a p-value based on two means (from two samples) - isn't a p-value calculated between a number and a distribution? what is the exact process of using the two means parameters to calculate a p-value on a distribution?
We can calculate p-values for two means using something called a t-test. For details, see: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-nk2CQITm_eo.html and then ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-NF5_btOaCig.html
I don't see how the first example with the different drugs is problematic, given that the drugs actually are different and that we draw a new sample every time.
What if i get a p value of 0.06 using a sample of 29 and do a power analysis AFTER that and power analysis says that i need a sample size of 30 so i add one more observation to my data. Would this be okay to do?
Can you nuance what is meant by one more measurement and adding more data. I assume they are the same in this context regarding " a subtle form of p-hacking' segment
Question: Why the drug-free experiments are repeated? Why not reuse one set of drug-free results, or even better, aggregate all the drug-free results into one single set?
Even if you re-used the drug-free results, the results would be the same - there's a chance that, due to random noise, you'll get results that look very extreme.
"...and compare these two means and get a p-value = 0.86." Wait, how do you compare two means to get a p-value? On the Statistics Fundamentals playlist I'm working through, it's only been explained so far how to determine the p-value of a certain event happening (like a Brazilian woman being a certain height). It hasn't yet explained how I can understand the statistical significance of two sample means being a given distance apart from each other if they're hypothesized to belong to the same overall population.
Sorry about that. In this case, I was just assuming that knowing the concept of a p-value would be enough to understand what was going on. However, if you'd like to know more about how to compare means, see: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-nk2CQITm_eo.html and then ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-NF5_btOaCig.html
what if i repeat all of the experiments with other samples a number of times, e.g. i do 20 experiments for each drug total, and then i try to look if anything changed is that real bad (cause it isnt guaranteed that i dont just get 20 false positives out of nowhere) or might that be useful in some obscure scenario
@@statquest I am not trying to justify the correctness of the number. Just wondering how to calculate the p-value in cases like the example. Do not recall you mentioned in previous videos. Thanks.
@@chendong2197 With this type of data we would use a t-test. I explain these in my playlist on linear models(which sound much scarier than they really are): ru-vid.com/group/PLblh5JKOoLUIzaEkCLIUxQFjPIlapw8nU
@@statquest In that case of course it would, but I find applying this logic to the case of truly different drugs confusing as that is surely not p-hacking if the tested drugs are not the same.
When we compute a p-value, we select a "null distribution", which is usually "no difference". If the null hypothesis is correct, and there is no difference, then a p-value threshold of significance of 0.05 means that there is a 5% chance we'll get a false positive.
@@statquest so can i understand in this way: say the p-value is 0.05, when our observed data lays on the 5% of null distribution, there is also 5% of chance we incorrectly reject the null hypothesis if we decide the rejection, but we are still 95% confident to reject the null hypothesis.
@@alexlee3511 The p-value = 0.05 means that there's a 5% chance that the null distribution (no difference) can generate the observed data or something more extreme. To be honest, I'm not super comfortable interpreting 1 - 0.05 (alpha) outside of the context of a confidence interval.
One thing that confuses me, if the threshold 0.05 means there will be 5% false positives (bogus tests in this example), then how to link p-value (say we got 0.02) to false-positive? Is there anything like 2% false positive involved with p-value? I think that's not the case. I watched all of your p-value videos and it's been made clear. But the definition of "threshold" confuses me. I hope p-value has nothing to do with false-positives. Correct me if I am wrong.
The p-value tell us the probability that the null hypothesis ( i.e.. that there is no difference in the drugs ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-0oc49DyA3hU.html ) will give us a differences as extreme or more extreme than what we observed. When we choose a p-value threshold to make a decision, like all p-values < 0.05 will cause us to reject the null hypothesis that there is no difference in the drugs, then there is a 5% chance that the null hypothesis could generate a result as extreme or more extreme as what we observed, and thus, there is a 5% that we will incorrectly reject the null hypothesis and conclude that there is a difference between the two drugs when there is no difference.
Great video! I have a question about drug testing. Is it better to perform an experiment that tests all of the drugs at once instead of testing each candidate one by one? Besides, if I want to test 6 candidates versus control, each of them has three tech replicates, and I performed the same experiment three times. Should I use the mean of the three tech replicates from the three times of experiments to calculate the p-value? Or are there better solutions for the experiential design? Look forward to your reply!
If you do all the tests at the same time, that's called an ANOVA, and you can do that - it will tell you if something is different or not. However, it won't tell you which test was different. In order to determine which test was different, you'll still have to do all the pairwise tests, so it might not be better. So it depends on what you want to do.
@@statquest Many thanks for your reply! What about the technical replicates? If I have 6 technical replicates for each biological replicate, should I use the mean of technical replicate to stand for each biological replicate?
@@yijingwang7308 If you have technical replicates, then you probably want to use a "general linear model." Don't worry, I've got you covered here: ru-vid.com/group/PLblh5JKOoLUIzaEkCLIUxQFjPIlapw8nU
Thank you so much for explaining probability concepts with such good pedagogy (and positive energy). These lessons are usually soporiphic. Question: why do we need to re-test new samples of recovery WITHOUT any drug every time? Can't we just compare the results of each drug to the results of "no drug" ? As in, in practice, we would only have 1 placebo group against which we compare the results of a given drug, wouldn't we?
Sure, if you did all the tests at the same time, you could use the same placebo group for all the tests, however, that doesn't change the possibility of false positives.
If you get p-value of less than 0.05 and you set sample size beforehand, are there ways of further interrogating the data to ensure you haven't just got that unlikely but possible '1 in 20' result without accidentally reverse p-hacking (not sure of the right term but taking more samples, or sets of samples, so now the p-value is greater than 0.05)?
Can you please explain Why increasing the sample size after measuring the p-value would increase the likelihood of false error? I would think it would be the opposite. If you're adding 2 observations to each set, it's more likely that each of these observations are closer to the distribution mean than far away from them. This would imply that the sample means of both sets are more likely to come closer (p-value would increase) than for them to move apart (p-value would decrease)
Unfortunately that is not correct. Even if the new points are closer to the mean, there is a 25% chance that the new point for the observations with higher values will be on the high side of the true mean AND the new point for the observations with the lower values will be on the low side of the true mean.
in the experiment 0:29 - 2:18, what if the drug z was actually a better drug? p hacking would happen if we test the results of drug z itself multiple times until we get a false positive. In example mentioned above we just tested 1 drug relative to other drugs, which gave a p value less than 0.05, so why would that be a false positive?
If Drug Z was actually a better drug, then the times when we failed to detect a difference would be false negatives and the time when we detected a difference would have been a true positive. So that means it is possible to make both types of mistakes - if Drug Z is not different, then we can have false positives and if Drug Z is different, then we can have false negatives. The good news, is that we can prevent false negatives by doing a power analysis. For details, see: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-Rsc5znwR5FA.html and ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-VX_M3tIyiYk.html
Hey Josh, but what if my real difference is smaller than what I considered when calculating the experiment Power? Can't I just keep the experiment running to reach a higher sample size, then reach a good experiment Power for the smaller difference and test it? Would that be p-hacking?
That would be p-hacking. Instead, use your data as "preliminary data" to estimate the population parameters and use that for a power analysis: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-VX_M3tIyiYk.html
Let's say I am developing a new method of analysis and it gives me a p-value. Is it ok to keep changing my method to get a better p-value? Would this be p-hacking as well? Thank you for your videos! It helps me a lot!
I don't think so. For example, if you ran the wrong test on your data, then realized your mistake and ran the correct one, I don't think you should be penalized for that. However, in your example, you have to make sure that when you modify your test, you're not modifying it just to get a small p-value. Instead you are modifying it in ways that are statistically justifiable in a broad sense.
If example 1 is p-hacking because the drug z result relies on only one test, how do you view all the social experiments that rely on one test (because it would be too costly to reproduce them)?
If I'm doing an experiment then a lot of other people must have already done experiments and didn't get a useful result - so eventually the spare budget falls to me making my test dependent just the same as if I'd done all of them. Does that mean we must increase our sample size based on the area under the curve of the economic capacity for investigating the problem?
No. It is inevitable that some of our results will be false positives. So we just need to focus on our own experiments and do what we can. That said, we can reduce the number of false positives further by using complementary experiments that essentially show the same thing. Like a criminal trial that lots uses pieces of evidence used to convince us that someone is guilty or innocent, our final conclusions should be based on multiple pieces of evidence.
Thank you A LOT for this video!!! Please, answer my question: which correction method for p-values should I use, if I make several (say, 30) comparisons, but each 2 groups of observations come from different distributions? To be specific, I compare two variants of 5 different enhancer sequences, each in 6 cell lines, using luciferase reporter technique.
I should have clarified that it doesn't matter if you mix and match distributions. When we do multiple tests, regardless of the distribution, we run the risk of getting false positives. So I would recommend adjusting your p-values with FDR (false discovery rate).
Hi, thank you for your great work ! However, there is something that I have a hard time processing. I understand the last example of p-hacking, but we are expected to do the same experiments 3 independent times. How do we analyze and represent these results without p-hacking ?
Is is really true that you are expected to do the same experiment 3 times? Or just that you are supposed to have at least 3 biological replicates within a single experiment? If you really are doing the exact same experiment 3 separate times, then you could avoid p-hacking by just requiring all 3 to result in p-values < whatever threshold for significance you are using.
Amazing! I have a question about Benjamin-Hochberg method. Is this method only applicable to parameter tests such as t test and chi-square test, or is it also applicable to non-parametric tests such as Wilcoxon Signed Rank Test. Thanks a lot.
@@statquest just discovered your channel. Do you cover different distributions. What I'm finding is explainions of them in isolation but nothing comparing them, when their used, how to test for them.
You can do it with something called a 't-test'. There are two ways to do t-tests, the traditional way that is very limited, or you can use linear regression and it opens up all kinds of cool possibilities. If you want to learn how to do it with linear regression, see: ru-vid.com/group/PLblh5JKOoLUIzaEkCLIUxQFjPIlapw8nU
I had expected that adding more data would have made it less likely to get such false positives. Why does the p value decrease as we add more data in the 2nd example?
I did some simulations with a normal distribution and when the p-value was between 0.05 and 0.1, adding more observations resulted in a 30% probability of a false positive.
@@statquest Excellent video. Thank you! So basically you say that observations in the power analysis can not be included in the final analysis? I have just worked through the book "Medical statistics at a glance". It says that it is OK , and calls it an "intetnal pilot study"?
I am really confused what do u mean by exact same distribution I mean only the people we are testing are the same the drugs are different therefore if we get different result then why do we assume it as a false positive.
I am really confused what different and same distributions mean here yes the people we are testing on are same but the drug is different in each scenario right so it can have different effects
@@shivverma1459 This example starts at 2:38, when I say that we are measuring how long it took people to recover and these people did not take any drugs. So there are no drugs to compare - we just collected two groups of 3 people each and compared their recovery times. When we see a significant difference, this is because of a false positive.
Awesome video as always! Would love to hear your thoughts on moving the standard alpha to 0.005 or some alternatives to p-value reporting (surprisals/s-values) in a future video too!
When we talk about Bayesian stuff, we'll talk about alternatives to the p-value. As for changing the "standard" threshold for significance. That's always been a cost/benefit/risk balance and ends up being field specific. For example, in a lot of medical science, the threshold can be as high as 0.1 because it makes sense in terms of cost/benefit/risk.
We are testing some drugs to see if they change the recovery period distribution, right? Why do you repeat that they are from the same distribution? How we can be sure that the drug didn’t change the recovery period beforehand?
StatQuest with Josh Starmer thank you for your reply. So if I’m not wrong we generated those stochastic numbers with the same distribution as a tool to teach this stuff. But in reality we can’t be sure if they are form the same distribution or not. So we will test them and power analysis will help us to find the right amount of data we need to have a reliable statistical result. Little bam or not a bam at all? :-D
I'm sorry - my previous response was intended for someone else - I have no idea how that got mixed up. Anyway, here's the deal... The drugs can be from any distribution. However, the worst case scenario is when they both come from the same distribution. This means that any small p-value is a false positive (whereas, if they come from other distributions, then any small p-value is a true positive). So we assume the worst to determine how bad things can really get. Does that make sense?
Big fan of StatQuest, really appreciate the work and humor you put into this. Just a question on the approach, why do we need to have different control groups for each of the drug for comparison? Can we have one control group that doesn't take any drugs, and have it to compared with all the treatment groups that take different types of drugs. Is there a pros and cons of doing this in comparison to what's done in the video?
@@statquest Thanks for the prompt response. Yeh I agree that's by any mean not the solution to the p-hack problem or even attempt to mitigate it. I was wondering what is the pros and cons of having different control group for each drug vs having one control group for all. Under what scenario should i pick one approach over another?
@@alsonyang230 It all just depends. You want to control for all things other than the drug or "treatment" or whatever you are testing. So, if you can do everything all at the same time, you could just collect one control sample. But if there are changes (like time of year, or location), then you need to get extra controls.
@@statquest The intros and the way they are represented grows on you as u progress through more content. Absolutely loving it. :) Statistically speaking there is no significant evidence to reject the null hypothesis: " Intros are awesome :)"
