Actually, there is a small misconception in this video (the table in the beginning). When we reject the alternative hypothesis that does not mean that the the null hypothesis is true. It simply means that we do not have statistical evidence to reject the null hypothesis - we cannot say with 100 % statistical certainty that Ho is true. Andy Fields writes in 'Discovering Statistics in SPSS': "If the p-value is greater than 0,05 you can decide to reject the alternative hypothesis but that is not the same as the null hypothesis being true" (Fields, 2018: 76).
This crash course is great. The level is fairly high (I believe that most people with a PhD could even learn a thing or two), and it's still comprehensible. Great job!
During my Ba and MA I certainly did feel the strong push to "just try this or that" so the p-values would be a little bit more acceptable. I never really got a straight and clear answer how that wasn't similar to cheating. But everyone seemed to think it was quite okay - That was a weird experience. Gaslighting, that's called right?
If you aren't publishing (and as a student, you probably weren't expected to publish) then p-hacking isn't a concern. If you're learning the "art" of statistics, you need to experiment with the way different analyses work.
Hi! I really like your videos, I'll eventually go through all of them. Please make a video on multi-variate analysis. I' beginning to understand them but would really like for you to explain them (PCA, CVA)
All of that and you didn't mention the common terms "fishing" and "exploratory analyses", nor how such approaches can be used ethically to generate new hypotheses, or be methodologically accommodated, such as with screening and hold-out samples (cross-validation).
I just want to say, when you calculate that with twenty tests all looking for p < 0.05, there is a 65% chance that we will get a false positive, you said that this "might be higher than you would expect". My statistically challenged brain sees it the other way, though: if each test has a 5% chance of giving a false positive and we run 20 tests, then we get a 100% chance of a false positive because 20 * 5% = 100%!! (Of course I understand why this is wrong, and you've explained it very well over the course of this series--I just wanted to point out what seemed to me to be the obvious mistake to make!)
You can think of it like "what's the chances that I'll get NO false positives in 20 tests of p = 0.05", which is ( 1 - 0.05 ) ^ 20 = 0.358 or 36%, so on the other hand, 1 - 0.358 = 0.642, or 64.2% chance of getting more than 1 false positive.
I was a bit confused about the Bonferroni correction, because surely, for a total p-value of 0.05 and 5 tests, the p-value per test should be p < 1 - 5th.root(0.95) But apparently, the simple division always leads to a smaller value than this calculation, making it basically close enough, and an even stricter limit. Although it surprises me that you didn't point this out, as that quotient comes kind of out of nowhere.
Yes it's a conservative approximation (that's close enough if you don't have a huge number of tests) that almost everyone uses without knowing that it is always (a little) too conservative (I guess that's why she doesn't mention it). The accurate correction (for independent tests) which you pointed out is called the Sidak correction :)
Yes, she also didn't talk about other solutions ie cross validation. Bringing up only Bonferroni (and also not talking about how conservative it is) and not talking about the issue more fully can be a bit misleading /:
I use the Bonferroni correction quite often. There are more than 20 thousand human genes in our DNA. Sometimes I need to look for genes which **significantly** change under a treatment. Without the Bonferroni correction, I would be doing over 20 thousand different p-values, which of course I would find something that appears "significant" with a p < 0.05.
WIth such a number of tests though maybe you could use another correction. Bonferonni is always a little too conservative, you could use the more acurrate Sidak correction. Or even Holm's or Hochberg's procedure...
how to get p-hacking out of "for profit" research? 1-make only "no profit" & massively perreviewed research significant enough for laws. 2-implement UBI and/or have a "national scientific research fund" that allows for scientists to follow through theier research within ethical conditions for themselfs and their results.
So, in order to believe any statistic we see, and even then the statistic might itself be a chance result, we need Ph.D.'s in statistical analysis, an army of scientist employees, and sophisticated lab and computer equipment to verify claims. Oh, and access to the journals said statistic was published in, each of which will cost a pretty penny. Not just CC, but in general, the media and experts do a fine job of outlining the problem but rarely give solutions and even when they do, the solutions are impractical or theoretical. The same media and experts, due to shenanigans like p-hacking, have broken the public's trust in what they have to offer. My solution: mandatory statistics classes beginning in Kindergarten. This will take forty years for any critical mass of trust to return to the societal influencers because in forty years, today's kindergarteners will have the power to influence and the old guard of prevaricating and unethical doyens will have died or be too old to dictate the direction of society in any meaningful way. Or, wait for the Matrix plugin seats. They're coming!
Michael Crichton was talking about this 20 years ago. He suggested that the government fund a return to basic reseach and that researchers make their data available so that others can replicate their studies. His senate testimony is available online.
How to get a tiny p-value every time? It's very easy conceptually. Increase your sample size. In fact, in areas of science where very large sample sizes are the norm, the p-values are almost always less than 0.05. Hmm.....so if with very large studies p-values are kind of pointless....does this mean that p-values are kinda dumb in general. Yes, and many, many people think that and write about it. So how did p-value type analyses become so popular? Easy. It provides a veneer of objectivity when scientific decision making is not totally objective. You can't have a machine spit out a yes or no, usually. Also, with moderate sample sizes, a low p-value indicates a relatively strong strength of effect. Hmm...so why not just look at strength of effect directly and lay everything out so that individuals can decide upon it? Well, that's a better approach. Cue first year math students, people that have taken one stats course, or a semi-competent stats teacher/researcher to call me crazy. Research this question about null hypothesis and significance testing and the controversy around it. There are hundreds of resources.
Yeah, the p-value is just a number you can use to argue your results. You should still be able to adapt your argument to your experiment/methods. Picking .05 as a cutoff (as it was said on other CC:Statistics videos) is just arbitrary and means nothing without context. The problem is that statistics isn't the most prestigous field of mathematics, but it is one of the most important one in ALL fields of science. So there are few really good Statisticians, that are good at teaching new generations.
You still need to have an effect (however small) to get a tiny p-value no matter how large the sample size. The way p-hacking is done is with lots of tests on lots of variables, not lots of samples.
I still don’t understand. What does p-value mean, and how do we choose the p-value? Does low p-value mean significant affect, and high p-value means no affect? Is it just cherry picking? Can someone please explain I’m so confused😭
I liked this video, but it was basically a drawn out way of saying that if you don't take everything into account or intentionally leave some things out you can get bad results.
Regards Adrian, I am planning to carry out an empirical study with p hacking being the subject at hand, could you please a case study that could be worked upon. Thanks in advance !!!!!!
But that information useless in that it does not offer any details about which variables (jelly beans) are significant or even how many! By adjusting your p-values you've CHANGED your null hypothesis. There are better ways of doing this!