I'm astonished ... you explain the intuition behind this in 11 minutes while my teacher spent 3 hours trying to explain this ! Glad my sis recommended you ! ^^
@@statquest Hi Josh, great stuff, though I am still a bit confused. :) So that I can get the fact straight ... Question 1: "The threshold of 0.05 means if A and B are the same, there is only 5% of the tests will exhibit p-values that are less than 0.05". This statement is then equivalent to "If A and B are different, 95% of the tests will exhibit p-values that are great or equal than 0.05". If so, how can you come to a conclusion in the video at 8:48 that A and B are different just because they have ONE p-value which is less than 0.05? Question 2: When you say if some one wants to be very strict, he can set the threshold to be 0.0001. In this case, what will the corresponding p-value be, also 0.0001?
@@jtbauer3789 Let me put this in a different way. Suppose that you are the guy trying to develop this new drug A. To be 100% sure that your drug really works and that isn't just luck or placebo that are curing people you would need to test this new drug in every person in the world, or even in the universe. For obvious reasons, this is not possible. To still be able to prove that your drug is curing people you can run this statistical test with a portion of people. We can say that this portion of people is a sample of your universe of people. For this statistical test, you need to establish a hypothesis that nullifies what you are actually trying to prove (this is the tricky part). In this case, this null hypothesis could be that your drug A is no different from drug B that we previously know that doesn't work. The origin of this knowledge about drug B it's not relevant for this test, you can just assume this previous knowledge. You also need to choose a threshold. Now comes the practical part of the test as explained in the video. You create two groups of people from your sample of people and each group is treated with only one of the drugs. You count, for each group, how many people got cured and how many were not cured. With these values, you calculate the p-value. The calculation of p-value needs an apart explanation. You just need to know that the p-value is the probability that the null hypothesis is true. Then if you get a p-value under the chosen threshold, this means a very low probability, it's very unlikely that the values that you observed in this one experiment happened only by luck (placebo or whatever). In conclusion, you can deny the null hypothesis and you can be pretty confident that your drug A does work.
@@jtbauer3789 the equivalent statement should be that: if A and B are different, 95% of the the tests will exhibit p-values that are less than 0.05 since it will be more likely that the test will report significant differences thus producing smaller p-values most of the time. or if A and B are different, 5% of the tests will exhibit p-values that are greater or equal to 0.05(only 5% of the test should suggest that A and B are the same)
@@JohnYoga The p-value is related to the assumption that the null hypothesis is correct. If it is correct, then our data should have a relatively high probability, and thus, we get a relatively high p-value.
I was struggling in my stats class lately but your videos SAVED me, thank you so much. I'll definitely be watching more of your videos. I wish my prof was as incredible as you!
I really appreciate the slowed down, enunciated explanation as though you were explaining to a 5 year old. As someone with ADD, it's super hard for me to keep track of information especially if its even a little bit faster paced than this, but you explained everything perfectly as though it was made for me. Thank you !
This is the reason I have started my channel here to help others understand mathematics like you. This is very good approach in this field ,DAVIKA Academy which is my channel am working on both pure, applied and statistical mathematics. Once more keep it up let us transform the educational sector .
My brain has been so enlightened, thus I feel so much better about p-values as well :D How many times did I read about this trying to understand it?! But this video concisely elucidated the concept so well, I'll return here if I need a refresh. Thank you Josh and Statquest! Double Bam!!
I spent like 2 days read about NULL HYPOTHESIS and when I came here you just said that to prove whether the drugs are the same or not.... Sir you are intelligent indeed!
Loved it... was struggling to understand how the p value actually determine the differences in effect. This video explained it all. Thanks a lot StatQuest
Thanks, for such creative interpretation of p value. Sometimes it appears to me that i have to cram a lot of formula in stats and that is quite irking for me.But your video is simple, adorable and short in which you have used set induction and then illustrated through various distinct examples. Lovely!
Thank you so much! I'm a bit confused about why the closer a p-value is to 0, the more confidence we have that Drug A and B are different as if the threshold is 0.05 which means if A and B are same and if we did this exact same experiment a bunch of times, then only 5% of those experiments would result in the wrong decision. Then if the p-value is 0.03. It means only 3% of those experiments would result in the wrong decision. Why it doesn't mean it is more likely that A and B are same?
The p-value tells us how different the observed data are from what we would expect if the data were the result of random chance. The smaller the p-value, the less likely that the data were the result of random chance.
@OKUBO SELESTINE OPIYO It's not exactly "the the p-value, meaning there was less random chance influencing the results", it is more that "if it were random chance, then the event we saw would be very rare".
If all the teachers were this fun and cleared concept with such lucid explanation, there would be many bright students who would be interested in the subject rather than cramming up definitions!
Great video! The thing is, you did not mention how the P Value is computed?...my mistake, it was on your other video🤣. Thanks for making good stat videos.
when i learning these statistics / probability things for machine learning, i feel most of time im not learning math, im learning machineLearning Lingo! especially when english is my second language, it becomes English-machineLearningLingo! double bam!
Hi josh big fan of your work.. All these videos helped me a lot in my career.. Just one thing which is not yet clear to me that is for a logistic regression classification or Regression problem we see p values for each independent variable.. Just want to learn exactly how it is calculated.. Hope someday you make video about this maybe with a small example.. All d best fr upcoming series.. Thanks a lot man..
I explain this in my series on linear models. Specifically, these videos: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-nk2CQITm_eo.html ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-zITIFTsivN8.html and ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-hokALdIst8k.html In short, we calculate the F-statistic for the model with and without each parameter.
Hey, first of all thank you for your content, it's extremely helpful. I have a question: When calculating p-values for the same drug A at 06:00, ju referred to Fisher's exact test. In your video the example is based on summing up the probabilities of things rarer or equaly rare to taking 7 blue and 1 red. I understand this concept easily. Coming back to to samples of drug A's, how exactly count the values by summing the probabilities of rarer and equaly rare events? I'd be grateful for the explanation 🙏 😊 All the best!
That's a really important point, especially in relation to political polls. I would love to see a video explaining in "bam terms" how to critically interpret the results of such polls. Josh: keep up the good work. Much appreciated.
@@jacksonmacd Thanks! I have notes for a StatQuest on polling statistics, thinking it would be a simple topic to cover, but it's got an exception to every single rule, making it very complicated. One day when I have a lot of time I'll go over the details.
Thank you very much♥️, but I want to know how you knew the p_value + Does the p_value decrease with the presence of more population in the test and vice versa + when can we say that the difference was due to chance? Please answer these three questions for me. I have an exam, and so far I don't differentiate anymore😢.
I'm not sure I understand your first question. The second one is answered in this video: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-Rsc5znwR5FA.html and the third one is answered in this video.
@@statquest Thank you for your answer🤍، I would like to know at what minute the answer to the third question was mentioned? because I watched the entire video and did not notice that.
I appreciate your videos, especially the one about how p-value is calculated. But I'm still confused (probably, because other sources are also so obtuse about these things). So, and tell me if I'm wrong, there's Type 1 and Type 2 errors with their probabilities (with Type 1 being a significance level Alpha, which we set ourselves, say, Alpha = 0.05). Then there is p-value, which shows a probability that if H0 is true this is a probability of getting this data set or more extreme one. So, in the end, we have probability Alpha for Type 1 errors, probability Beta for Type 2 errors and p-value, which we calculate for the selected H0. And the reason we use Alpha as a threshold for p-value is only because of convenience, because we don't want to bother with selecting two distinct significance levels In any case, thank you for the content! Stay awesome! P.S. Probably confused myself, trying to equate Type 1 or Type 2 to p-value. I don't know anymore
If you asked me, the terminology "Type 1" and "Type 2" errors is super confusing. What I don't like about those terms is that the terms themselves do not give us a hint about what they referred to. I wish we could say "false positive error" rather than "type 1 error" and "false negative error" instead of "type 2" error. Anyway, these errors have different uses. The "false negative error" (or type 2 error) is the probability that we will fail to reject the null hypothesis when we should reject this. We only need to deal with the "false negative error" when we do a power analysis: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-VX_M3tIyiYk.html . However, once we have done the power analysis and have determined the correct sample size for our experiment, we are no longer interested in beta and the false negative error. In contrast, alpha determines the false positive error and is useful once we have done the statistical test. So "beta" is useful before a test, "alpha" is useful after the test.
@@statquest Thank you! I'll definitely watch power analysis video later! Just to be sure, "False Negative error" for alpha (and we want it smaller), ""False Positive" for beta (and we do power analysis to make it smaller) and p is also for "False Negative" but we want it bigger, not smaller, unlike alpha?
You have it backwards, a false positive error = type 1 error and is related to alpha. a false negative error = type 2 error and is related to beta. Beta is related to power. If you see the video on power you should be good to go (by the way, you may have to watch a prerequisite or two. The links are in the description).
@@statquest my bad. Thank you! Took my awhile, but after rewatching some bits of this video and watching the video about False Discovery Rates and how equally distributed p-values are on a histogram, it really clicked in my head. May be not perfectly, but I finally got a sense, what is the relation between the p-values and False Positives errors (and Alpha). Not sure if I can explain it yet, but I feel like I got it, somewhat. Hope I make sense here xD Feel really stupid, for it took an eternity, tbh, and I probably still need time to absorb it properly, but it's a real progress, and I'm really grateful that I found your channel. Thank you!
I was really really needing the "What the p-value does tell us is..." at the end of the video, right after what the p-value doesn't tells. Sorry, is just that my native language is not english i think. But really i needed it.
@@statquest Yes, exactly! To be very specific I would have LOVED you repeating just that at the end. Thanks a lot for the fast response, channel is awesome!
This video is very helpful. Can we say with 100% certainty that Drug A is different from Drug B using only p-value? eg if the p-value is less than 0.05
Unfortunately, we can never say with 100% that Drug A is different from Drug B, because there is always some possible combination of random events, not related to the drug itself, that results in the observed data. However, the p-value, when it is very small, gives us confidence that they are different and it would be a vary rare combination of random events that resulted in the same observations or something more extreme.
Thanks a lot for your videos! They are all very precious source of knowledge in Statistics! Related to the last section of this video ("p-values do not measure effect size") whate do you could do to measure the effect size?
R-squared is a great place to start. See: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-2AQKmw14mHM.html and ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-nk2CQITm_eo.html
Hi Josh, I watched your Fisher's exact test video, but still not get 6:03 why p=0.9? Can you please enlighten here how do you calc p-value for the two drugs distinguishing? Thank you!
Great video, many thanks. Could you make a video about Neyman-Pearson lemma and explain the difference between p-value and alpha value, they confuse me 😅
I'll keep that in mind. However, for alpha vs p-values, alpha is just a threshold for making decision. In this video, we use alpha = 0.05 at 7:06. We then talk about alternative values for alpha (the threshold for significance) at 7:37.
Thanks for explanation! Everything is clear, but you reached the most important point and you didn't explain it with details. If the difference is so large but the p-value is bigger than 0,05 can we consider that the drug A Is different than drug B since the difference is large?
If your threshold for significance is 0.05, then we would fail to reject the hypothesis that there is no difference. For example, if only one person took drug A and failed to get better and only one person took drug B and they got better, then the difference would be huge, but that difference could be for a lot of random reasons, like maybe the person who took drug A was allergic to it. So, in that case, we would get a large p-value, which suggests that we don't have a lot of confidence that the difference in the drugs is not due to random things.
Hey Josh really enjoyed your teaching. I just want to clarify a doubt is the p-value mentioned at 8:57 right? cuz I calculated using prop.test() in R and it gave me 0.022 🤷🏽
You can compare the means of the two groups or you can compare the odds or the log(odds): ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-ARfXDSkQf1Y.html
@@statquest there u have mentioned about mnm which I understood. But here you are taking 2 different populations and calculating p value. It is much different than what is mentioned in that video.
OK, let me summarize my thoughts. The p-value tells us the credibility of an experiment/the degree of conformity with our hypothesis, the closer it is to 0, the more credible it is/the more it conforms to our hypothesis, and we can set a threshold for the p-value of all the experiments to indicate that we can accept a false positive/error, e.g. 0.05 is the threshold that we can accept that 5 percent or less of all experiments have a p- value less than 0.05.
p-values aways refer to the assumption that the null hypothesis is true. The null hypothesis is that there is no difference. To learn more about these nuances, check out this video: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-0oc49DyA3hU.html
@@statquest thank you a lot sir for answering me, I did some further research and I want to make some statements just for clarity, if there is any mistakes it would be really appreciated if you can point them out. 1. p - value is a possibility, only applies on null hypothesis. 2. p - value tells you that if null hypothesis is true, then the data from the experiment‘s result has that much of possibility to occur, for example in the video, the null hypothesis is drug A and drug B has no differences, if the p - value of the experiment is 0.01, then it means that if Drug A and Drug B has no difference/null hypothesis is true, such data from the experiment's result has a possibility of 0.01 to occur, and that means there is a low possibility that drug A and Drug B has no difference, hence the null hypothesis can be rejected. 3. A false positive refers to incorrectly rejecting the null hypothesis when it is true. The p-value itself is not an error, but a tool we use to decide whether to reject the null hypothesis. 4. The p-value threshold refers to the proportion of experiments we can accept that result in a false positive among all experiments.
@@jxaskcijiaxhsic9943 1) correct 2) p-value is the probability of the observed data (assuming the null is correct) or something more extreme. 3) correct 4) correct
the most primitive video I have ever watched on this subject , YET was the only one where I was like " Oh my God I GET IT now!" lol thank you I may now pass my capstone class..
The p-values in this video we calculated using Fisher's Exact Test, which I explain in this video: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-udyAvvaMjfM.html
@@statquest i watched it 3 times. well... I think I am stupid and that's why i don't understand how can calculating the sum probabilities equal to the data or rarer can be enough. I mean how did we get 0.9 at 6:19, the p value of what? The smallest possibility is to get cured about 35 % and 37 in both data how can I get p value from 2 percentages.
@@antoniovivaldi6053 The p-values are not calculated from the percentages. In this case, they are calculated using fisher's exact test (as linked before). However, before you go through that video again, you might want to see my video that gives more details on how p-values are calculated in general. See: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-JQc3yx0-Q9E.html
Definitely! And even within academics there are differences. It really depends on how much you can control variation. In physics, they can sometimes have crazy control over the variation and, as a result, require very strict p-value thresholds (like 0.00001). In medicine, where there is very little control, the p-value thresholds are more like 0.5.
Do you have a video on sample size versus experiment replication? To combat the possibility of observing a result that is randomly below the threshold, would it be better in some instances to split your total sample into multiple experiments? Is there a trade off between statistical power and replication?
If small P values don't tell us how much the difference is, how much confidence is there in using it to reject the Null Hypothesis really? The example you gave to demonstrate this was a P-value of 0.04 but the difference in the samples was 1 %. From your explanation on testing the Alternative hypothesis, if we compared the difference between distances about a single mean and separate means, could we get results that contradict the P-value? That is find that the differences when we compare the distances about the means are not significant for us to reject the Null Hypothesis and yet the P-Value is below 0.05?
The p-value should never be used, on it's own, to determine if a "statistically significant difference" is actually meaningful in a practical sense. You have to combine it with other metrics that tell you how different things are. For example, you could combine it with a measure of the distance between the means, or with a measure of how well a line fits the data (like R^2) etc.
Helpful and well done, but one thing you might want to check if you want to adjust. Starting at 3:57 you refer to 73 people cured out of 125 as being 37% of them cured, but that might actually be 58%. Minor thing, but it threw me off for a minute and I had to rewind to stop thinking about the math and just follow the concept. Great video, much easier to follow than several other sources I looked at. Thank you.
The math in the video is actually correct. There were 73 + 125 = 198 people that took drug A. Of those 198 people, only 73, or 37%, were cured. Does that make sense?
Hi Josh, thanks again! Is my interpretation correct: In this drug A and B example, our hypothesis test would be as follows: H0: both drugs are the same H1: they are different Result: p=0.04 means the probability of both being the same is 4% while probability they are different is 96%. If our rejection value is p
The result, p=0.04, means that, if there were no differences, the probability of getting the observed data, or something more extreme is 0.04. For details, see: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-JQc3yx0-Q9E.html NOTE: This is different from the probability that the drugs are the same, so, unfortunately, your interpretation is not correct.
