The scandal that shook psychology to its core 

My brothers phd was originally replicating quite a well known study about gut microbiome of rats affecting their brains. He went to great efforts to make sure all his rats were exactly the same strain as the original study, and actually found a higher quality source for the feed with less nutrient variability, and was super careful with all his methods. And he found absolutely no results whatsoever. He was quite upset and anxious for a while that his whole PhD had been a failure and no one would publish his study because it contradicted this famous one and everyone would think he’d just screwed up somehow

Instead of “publish or perish,” psychology needs a culture of “replicate or retire”.

The problem is not the Science, the problem is the system. Publishers hold too much power and researchers depend too much on grants that do not consider much more than number of publications. It is vicious cycle, that can spiral downwards quickly: no positive results-> manuscript not accepted in any relevant journal -> no grants -> no money to do more research -> no more results -> no more publications, etc. Many researchers depend on grants not only for their projects, but also for their salary. So you can easily see the point of failure, if your way of life depends on positive results, and you don’t have them, there is only one way out.

We need more people brave enough to publish ‘non results’.

I saw a lot of this behavior while working on my PhD. There is such strong pressure to get positive, confirmatory results, researchers will twist the data any way necessary.

In the Navy we used to say, "You don't get what you expect, you get what you inspect." If you reward publications, you get publications; if you reward research you get research.

Professor in Statistics here. Just two comments: firstly, one would expect researchers to involve statisticians in their statistical anslysis but that is rarely the case, and secondly, I dare to say, research in empirical economics is generally much more erratic than in psychology.

I firmly believe that the first year of a PhD, or at least for master's capstone projects, students should be spending their time validating other people's experiments.

I feel like we need a catalogue of the null hypothesis, where scientists can publish like a two to three pages report on Failed or inconclusive studies. Make it searchable and that'd be an amazing resource for future scientists to see what pitfalls to avoid. And it would probably take less resources to create and run than Wikipedia or other such websites.

Math is excellent at revealing the truth. It's even better at hiding it.

I remember studying statistics for psychology and thinking "Wow, so I can say my findings either disprove the null or don't disprove the null according to how I slide this p-value between .05 and .01!" It was a WTF moment.

In my Psych 101 course, when asked, our professor admitted that the field attracts people with mental problems, but stated that exposure to the field during their studies often gave students sufficient insight to drop out. This is no longer true. Over many decades, the requirement for personal therapy to attain a degree in Psychology has dropped from 40 hours to 24 hours to 8 hours to, at many institutions, "highly recommended," i.e., zero. At the same time, tuition costs have gone up by a factor of ten or more. Fewer students drop out and kiss goodbye to tens of thousands of dollars already invested in their education. The field is in deep trouble, and the consequences in education, research, and clinical psychology are outrageous.

I remember in a Linguistics class I took as an undergrad, our final project was a research project based on language data and my hypothesis was completely unable to be supported one way or the other based on the data we had to work with. My professor was SO THRILLED when I stood up there, described my hypothesis and research design, all the effort I took towards proving it, and then went "and I still have no idea." It was honestly a great learning experience.

It's worse when doctors are then advised to treat patients based off of flawed studies.

Former PhD candidate here (with a background in psychology). Publish or perish, p-hacking and even p-value itself were extensively discussed within the field while I was doing my PhD (in cognitive neuroscience, or specifically neuroeconomics). Personally, I preferred using Bayesian inferences when it comes to statistics. But there’s many problems why scientists face the replication crises: 1. The sugar daddy of science: in theory, you don’t need to be a good scientist. You need to be a good marketer and a good salesperson to get the grant. Yes, and I was a part of a EU grant with billions of €€ but bullshit research plans. 2. Academic politics: following what I stated above, when the higher up is focused on using the €€ to buy fancy new tools, the lower-ranked PhD researchers as as myself (previously) became tools as well. We had no say in anything, we were just means to the end. 3. No one cares: with €€, power, and politics involved. Who’ll care if anyone is doing good science? It’s about publishing results right? So who cares about the PhD student, assign a shitty supervisor who doesn’t care, write bullshit things in the reports. P.s. the general rule for my PhD was to publish 3 papers in 3 years. And the two studies I was involved in did not have any “positive” results so to speak.

This is so weird to me. In medicine, in my experience, people publish null results all the time. “Is this treatment better than this other one?” “There is no diference” “ok, publish in the new england journal of medicine”

I love the obnoxious hubris of humanity, we assume that we know so much and that we’ve figured it all out until one guy tries to prove psychic phenomena and all hell breaks loose.

Its so important to spread this topic beyond psychology and academia so people are not swayed by glamorous and exciting sounding results. I'll be starting a PhD in personality psychology soon and part of the reason I've chosen my lab to work in was their commitment to open science.

My x wife is a biological research scientist (meaning she designs drugs and medical treatments) and from her I can tell you that the problems found in psychology research are actually pretty common across all medical research. The process of handling and interpreting the data is often manipulated to give the desired result to ensure funding.

Many years ago I submitted a research paper- my first major research project. It was rejected outright. A few months later, I ran into a senior colleague at a research meeting who told me " I rejected your paper because I didn't think you could actually do the analyses. And I thought" why would he think that?" Now I know he was projecting his own dishonesty. I left academia after that.

It's not just psychology. This issue has been discovered in other disciplines as well...

"No significant results, negative results etc", are also results and should be published. Those findings are important too. If we are really capable of learning, those findings at the very least, can show researchers what not to do next time. Just a thought 🤔.

There's nothing wrong with studies that aren't replicated. It just means there was a flaw before. The real problem is studies where the replication hasn't been attempted are put forward as knowledge, and used to make decisions.

Im a 1st year psych student. Here in UK they have been talking a lot about how journals should encourage people differently. Some journals are trying to publish research, no matter the p value will come out, by reviewing the methodology and accepting it before research is completed. I am excited about this as it seems to be a step to the right direction.

There is absolutely no way to replicate EVERY variable. If we were smart enough to not EVER say ANYTHING was an absolute truth, we'd begin on the path of honest science.

They should publish the no-result studies. Doesn't have to take up much space - just the titlle, abstract, result summary and a link to the data.

As a member of the public. We are often told/taught that science is strong because it is replicable and that what the data shows is the result. Now I knew you could abuse the data to get what result you want. But I didn't realize how wide spread all this is.

As a PhD candidate who had formerly undertaken statistical research in a non academic context and actually studied statistics at a formal level before which involved understanding the concepts, I found that the research training for PhD candidates left a lot to be desired, particularly given the predominance of computer packages such as SPSS. Younger students simply did not have the formal understanding to know what they were doing in my opinion. As you say you can bludgeon almost anything into significance. What is most important in understanding research results from a critical perspective is not what they say, but how the conclusions were drawn and above all of most significance (in the non mathematical sense) is a sound methodology and thorough logical consideration of all the possible confounding factors. You know it is an old trope that most psychology is based upon white middle class college kids, who form an availability heuristic of willing research subjects.

1:09 "ESP, the ability to sense the future". That's not what ESP means; ESP is an acronym standing for "Extra Sensory Perception" and is an umbrella term for everything involving a sixth sense. The alleged ability to specifically sense the future is called Precognition.

“I thought I had something there but it wasn’t significant” 😆😆🤣 that was the wittiest p-value joke of them all.

I remember when I was back in university we used to make fun of psychology and medical science to be "easier" than math and the harder sciences. We were never convinced of many of the results coming out of psychology because of the lack of rigour and lack of quantitative literacy among those who studied the softer sciences. We were always made to seem like assholes for pointing these things out and maybe we could have framed things more nicely, but I'm glad to see more gradual mainstream recognition of the failure within the softer science fields. One thing we were completely wrong about, though, is that the softer sciences were "easier" than the hard sciences. The complete opposite is what is true. The hard sciences are what are easier since we can arrive at more well-defined answers in those fields. The amount of complexity involved in the softer sciences is mind-numbing to say the least. I have a lot of respect for anyone who ventures in search of the truth and I have no doubt that, as the softer science fields mature and develop more tools and methods to aid in their study, we will arrive at some amazing insights. I'm grateful to be witnessing this inflection point in these fields and I'm excited to see what we uncover. The other possibility is that we are approaching the limits of what we can understand with science and mathematical modelling and that makes me feel differently, but still grants me a sense of awe at the wonders of the universe. Maybe we will need something completely different from traditional science and math to gain further understanding of the world. We can never know everything there is to know and that's hard to accept, but not unfathomable. We already have examples of things we will never know from studies in Cosmology.

My issue with psychology is that it's fully arbitrary when it comes to find a difference between health and illness. Potentially any behaviour society doesn't like can be considered a disease. Example is homosexuality which was a illness in the past and now no longer since society accepted it. Some pacifists were accused of being insane for not wanting millions of people killing each other in WW1

What I HATE is the system that connects being a good researcher with being a good teacher. One of the many things in a long list of things that are broken in American Universities.

I was a psych major around 1998-2002. I started learning about unscrupulous studies from pharmaceutical companies, specifically about the SSRIs I had taken for 20+ years. Great essay, I learned and marveled. Edit: I took Statistics twice (2 different schools) and barely passed with Cs both times. If I'd had to produce a whole study it might have been all over 😂

Same problem exists in other fields, remember going to our bioinformatics department and asking which clustering algorithms to use for a certain type of data and I was asked how many populations I want to have...

There's only two options for Psychology. It either has to step up it's game or go to oblivion. A student of psychology here !

As an outpatient social worker I also was Intrigued about the need for positive results and ignoring unfounded or negative results because when you’re working with a client there’s wisdom and anything and everything they say. Meeting goals give info and there’s reasons why a client doesn’t meet their treatment goals and there’s information in that as well and it’s ““ unsuccessful“ but there’s still a Why

Some issues that I and others I've met have had is how inherently reductionist research is required to be. How can we possibly factor in cultural differences and beliefs, feelings of safety, perception of self, social identity, community activities, social status, location, housing insecurity, political oppression, disability, minority stress, perceptions of trauma etc all at once? Lived Experience has so many factors and psychology's perception of "problem is with the self" only helps to a degree. Maybe someone doesn't need to get better and it's their environment that does need to get better. It also assumes that a "normal" and "unwell" exists, but this is cultural and has led to many within the Lived Experience movement to see psychology as just another tool of oppression and medical abuse. It also assumes that "statistical anomalies" are at fault and treated as "too complex" because we can't find a cure or make them normal, and so, they are then dehumanised by medical staff and therapists. There are 7 billion unique understandings and experiences with entirely different and unique understandings and ways of thinking. Is it realistic to think that we'll ever find something shared by them all? There has been a very distinct lack of Lived Experience within research and it has led many within the field to not hold trust in it.

Great and detailed video - two thoughts to add from me... (1) You opened with Daryl Bem's "Feeling the Future" study, but it was never mentioned again?! After hearing he rigorously replicated it himself 9 times, I was expecting to hear how he either did it "wrong" (and if so, how), or did it "right" (and if so, what a can of worms that opens up). Dean Radin and others are consistently shunned for investigating psi phenomena, despite taking rigorous research approaches. I'd love to hear more to tie this all together. (2) I believe researchers should be required to make ALL their data open and public. You argued that this might not be a good idea because researchers might not have the ability to properly organise their data for public release... I appreciate you followed that up joking it's not too much to expect, and I totally agree! Why can't every study require its data to be released publicly, regardless of how nicely formatted it might be? Just adding the requirement would encourage researchers to ensure their data is well-organised (thus improving its accuracy), but also allow for anyone to make their own analyses after the fact. There should be NO expectation on academic journals to conduct their own replications on the data - leave this for the public or other researchers to do after-the-fact. Journals should still demand the same rigorous standards, just ensuring that data is made open as a requirement too. When unbelievable studies are published based on meeting publication requirements, other researchers can then attempt to replicate them with access to the same underlying data too. This is imperative to computer science, where open-source research has advanced the field exponentially in recent years.

it would be nice if there was a place people could post all the negative results. Even if it wasn’t officially peer reviewed, there would still be feedback from comments by other researchers and there would probably be minimal intentional fraud.

God damn can Netflix give this channel a call. This is just golden content.

"the p-value sounds like incontinence medicine" had me laughing real hard

While delivering a lecture in class discussing the chapter Standard continuous probability distributions, an assertion was made by someone, 'But, the numbers don't lie as we all know'. In reply, she told, 'The data must be fed to the system of equations appropriately'. Now I see that it also applies elsewhere in all disciplines other than raw hardcore convoluted pure mathematics and statistics. Truly amazing!

If you had two books, one of every provable thing we really know about psychology and the other about all that they think they know about psychology you would end up with the first book about the size of a typical comic book and the second with about as many pages as the local library has in its books.

I think the problem begins in grade school and high school learning that is over focused on right and wrong answers that fit testing methods instead of teaching people to be broad minded and see learning as exploration.

It kinda feels like journals should add a special stamp/seal/mark on studies that replicate or have been replicated by other studies, so that everyone can know to take those studies more seriously, compared to studies without that stamp/seal/mark.

Out of my experience as a coach for PhD students, the statistical bs that's going on in computer science or economics is 🤥. I had more than one student wanting to quit science bc the professor was engaging them to "make the effects better for pubclicating" 😅. Usually those professors have had a tremendous list of different "A" - journal entries. To a point where I am suspicious now towards those "unbelievable successful" Professors. 😮 I had a hard time with telling those students , not to quit science but change the professor. One is a prof now himself 😊 and a good, honest one. But organizational wise, my hands are tied until today to expose those "successful" professors questionable research practices... Another annoying thing is how the sample is conducted, either it's Amazon Turk or students (usually first semester), hence not representative at all...

As someone who was diagnosed with MDD at 42, ptsd at age 55 , and ASD at 57 i can say with certainty that psychology and psychiatry are broken. I had ptsd and asd at age 42. I was hospitalized for 6 weeks, had meltdowns, severe sensory issues, couldn't eat the foid, and no one tried to connect the dots. I could have been diagnosed with aspergers at the time, but they couldn't be bothered to look deeper than the 15 minute evaluation the original psychiatrist did.

The video discusses the "replication crisis" in psychology, where many influential studies have failed to replicate or have much smaller effect sizes when replication is attempted. This calls into question what research in psychology we can actually rely on. One major contributing factor is publication bias - journals favor publishing positive, exciting results rather than null or negative results. This skews the literature. There are also issues with researchers engaging in "p-hacking" - manipulating data or analyses until significant results are obtained. This leads to false positives. Questionable research practices are very common, often unintentionally. Things like flexibility in data analysis, lack of transparency, and small sample sizes contribute to non-reproducibility. Solutions involve focusing more on study preregistration, sharing data, placing less emphasis on positive results, and changing incentives in academia to value replication more. Overall more rigor, transparency, and replication is needed. The host argues psychology has made a lot of progress in the last decade on these issues, and this "existential crisis" may ultimately improve the field by leading to more robust and reproducible research.

Notes on Main Ideas 2011- Pyscologist Daryl Ben discovered its possible to sense the future. He sent the results to The Journal of Personality and Social Pyscology, which resulting in a reckoning in pyscology. 2015-Reproducibility Project Pyscology: 27 scientists discovered only 36% of studies had replicated results 2017- Nature and Science; 62% had replicated results

This is why the only people who take the social sciences seriously are people *in* the social sciences.

Idk why, but the idea that everything we might know about psychology is wrong, sorta excites me, as someone who is studying psychological science one of the things I love about experiments is the ability to replicate them and see if anything is different, and if you find something different yyou can go on a trip to find out why or predict a phenomenon that no one had looked at yet. There are so many possibilities on what you can learn and from a different perspective

The fact is that scientific advance and knowledge is a national security issue. The best science is kept under wraps until it can't be held secret anymore. Lesser science struggles for funding and bad results are allowed to stand because even though it doesn't benefit us, neither does it benefit our adversaries.

I had a professor back in the old days who said the way you tackle the p-hacking problem is you only allow the testing of hypotheses which have an a priori reason to justify further investigation. I know, too vague to apply in practice.

My degree was in experimental psychology at a prestigious red brick university. The first big practical project was to replicate a fairly famous game-theory equation. Nobody got the predicted result. Most students forged their results to comply with the established algorithm: I stuck to my guns and instead proposed a very simple revision - introducing an almost self-evident variable to denote incentive - which then fit the data perfectly. Guess who got the lowest marks? When I protested I was simply told to get the results I was told to get. That wasn't an isolated incident either - it was systemic. Let's not pretend that our "scientists" can be trusted, it isn't true.

I'm so happy I discovered your channel. What a brilliantly put together video! Thank you for this, I think you're spot on with everything.

I remembered there is a journal/website on flopped experiments. As also many biology and chemistry (?) studies fail because of replication issues. It kan help phd students also to figure out which methods to gain certain protein structures don't actually work.

I have a biologist friend and he told me from the time at university, he found the studies in the psychology department shockingly amateurish in their methodology. He had no respect for the psychology department and it's people at all, finding them very unscientific.

I’m currently doing honours in psychology and the Australian university I go to teaches about the replication crisis as part of learning about systematic reviews. It is possible to detect publication bias through statistical analysis in the review process, but obviously that doesn’t solve the problem of it happening in the first place. Fudging data to get published, and journals selectively publishing need to be called out more quickly.

I remember starting my job and having total imposter syndrome for the first 2 months. I was stressing every single day, losing sleep at night in a perpetual state of fear each day might be my last there. I was essentially just waiting for the day I got called into my bosses office to be fired, but when I finally got what I thought was THAT call, it ended up being a call to pickup a 200$ gift card. I got employee of the month because I had the best numbers in my department that month lol. They also gave me a +2$/hour raise. I was FOR SURE underestimating my performance because I had nothing to compare it to nor any sort of metric to serve as a standard.

40 years ago I wrote an unpublished undergrad psych paper covering most of the issues you so eloquently cover here. My research methods professor pulled me into his office and soundly chewed me out. I forced him to accept the paper, and of course he gave me a poor grade, which I took before the academic dean. After the academic committee reviewed both sides, the paper was returned the second time with a perfect score. I’m sure smarter people than I who were post-doc raised similar issues back then, so the harder question is how do we as a society demand academia to get their shit together. Pretty tough in a society which seems increasingly to only value profit and wealth. Those are ultimately the pressures behind a great deal of this. We can’t even get our congress to cooperate on agreeing on bathroom protocol (tongue thoroughly planted in cheek). Sigh.

Most branches of 'scientific' psychology are complete bullshit. Personality/Social takes the cake though. Even when I was studying as an undergrad I was being sent to earn credits on PhD studies that involved 5 point questionnaires to harvest most of their data. I really enjoy the more analytic strains like Comparative and Developmental, but there's always room for debate there. The desire for positive results even came down to the undergraduate level. They weren't interested in studies that had 'problems', it had to prove an effect. Most successful ones usually involved making people do tasks while under the effect of coffee. It didn't help that we had a media famous Psychology dept that got interest through stunts and shows.

I think a study measuring the effect of psychological experiments on a psychologists psychology should be undertaken

Problem with using statistics is that the underlying mathematical model must be good.

Tenure should not even be a thing. I can’t think of any professional I would hire whose performance would be better if they couldn’t be fired.

It was never addressed if the 2011 research on Esp ended up being replicable or not, I was kinda hoping there would be a resolution like his study was unable to be replicated or the Professor was found to be p-hacking etc. but it was never brought up again.

Journals should publish at LEAST one replication for every novel study

Every PhD student should be given the tast to replicate older studies, never try to prove their own new hypotheses. That should give them a proper respect for the scientific method, and be a good way for scientific research to sanitize itself.

So I'm guessing the guy who "proved" ESP exists was p-hacking hard

Well psychology is a pretty murky area of study to start out with. Something like this was just going to happen, in fact I foresaw it in 1970 in a per-cognitive vision.

I lost all confidence in psychology as a science when this stuff first came to light. To this day, I am deeply skeptical of the field as a whole.

My new favorite RU-vid channel! Why did it take so long to find you? Thanks for being brave enough to talk honestly and encourage critical thinking about subjects that matter.

5:45 + You are an absolutely brilliant researcher and producer, thank you for working on & making this for us.

The people that exposed this are heroes, and so are you! Extremely admirable to rank about a period of your life that you’re not necessarily proud of! We all have those times and more of us need to see a Rockstar that can admit to mistakes!

When I was in college in the seventies an anthro. teacher told the class it was the most common thing in the world for a grad student to check the records of a famous scientist and find his famous findings had been faked.

in 6 months ill be doing my masters. Ive already floated some ideas about doing a replication masters but my handler is arguing that it will hinder my carrier possibilities. in other news ill be doing a replication masters in education.

I am way too late to this video but this was incredible. Very informative and very funny. This is fantastic content.

Am I the only one who expected there to be a hidden text or maybe context or something at 12:32 only to read "Media offline. Anyways, great video and an important topic, psychology is plagued the worst, but many things like publication bias and p hacking to straigh up making data up happens in every field and can affect actual human being if conclusion for clinical diagnosis and treatment and made based on that. Even the pressure on academics and the publish or perish mentality has more or less directly lead to loss of some great scientist, the statistics on suicide rate back it up.

Very informative, thank you for your work. Cheers!

" Let's move on, it wasn't significant " Good stuff.

Being as a lot of attitudes and public policy are based on findings from psychological studies this is rather disturbing.

What a wonderful video. Looking into neuroscience as a career path. I’m glad I watched this.

How incentives are structured is a major issue. As long as there is an economic/career incentive to publish, there will be manipulation.

This is a systemic issue across all sciences, not just psychology. But it is worrisome, regardless. I side-eye everything now.

I didn't even realise that making a hypothesis based on the findings is unethical until now, was thinking of doing that for my small research project (more of a passion project) because I thought it was acceptable but now I won't do that. Thank you letting me know.

It's absolutely a good thing this all is being brought to light and continues to be, and I'm not just saying that because I didn't do well in my psych stats class. :P There is a large disconnect between research/lab and clinical work too.

Really appreciate your calm clarity.

Thank you for this in-depth discussion!

Maybe the emphasis should be on falsification, not on replication. This is what Karl Popper said is the proper scientific method. In other words, design an experiment to disprove the candidate hypothesis, don't just leave it hanging in the air as not significant. Ok, not many hypotheses will yield concrete results if treated this way, but you never hear of any experiments designed to falsify.

My favorite story in this vein comes from the measurement of the electron charge - Feynman discusses this beautifully

"You can use facts to prove anything that's even remotely true."

As an undergrad in Sociology, it was easy to see how this could be possible. It soured my feelings about the field...

You summed up my advanced research methods psychology class really well done

As for the "paranormal" studies, the problem, for me, is the lack of scientific interdisciplinarity. If a psychologist concludes something unbelievable, his peers should demand a review from someone of another area, like a biologist, a neuroscientist, a physicist, etc. A psychologist should only publish his allegedly remarkable study, if he can purpose a multidisciplinary theory for his alleged results.

I did a study in my research methods class and the p value was really tiny, something like 0.00002 or so (can't remember exactly but there were many zeroes). This invited much scrutiny from several professors in the psych department but after I gave them my data and they looked everything over, they agreed I had done it properly. I ended up being invited to present my paper at a large APA conference but I declined because I was only taking psych for something to do and wasn't serious about persuing it much further.

Great topic! Thank you!

Another awesome video!

I've always considered psychology to be a bogus "science." I find the idea that human personality can be "studied" with the same level of confidence as other sciences like chemistry or geology or even biology to be ridiculous. There are too many smoky glasses to barely see through, too many fuzzy, uncategorizable factors, changes and variables to be able to reach any kind of reliable finding. As far as I'm concerned, all the major theories are no more than speculation masquerading as science. They hover around ideas and concepts that might be closer to the real truth than to falsehood, but that's about the best one can say for them. They aren't science. And I've had this feeling ever since "psychology" and what it means was first made known to me all the way back in elementary school. In university courses I never encountered any reason to change my mind. This is all just another example of how our culture has been so impressed with the results of using the scientific method that we have very, very wrongfully assumed that it could be applied to all aspects of reality -- and if reality doesn't cooperate, well reality can just go pound sand. Sorry. Some areas are just murky. They don't lend themselves the level of certainty we demand. And if we try to do it anyway, well. We end up in the situation this video is describing. And a lot of people are revealed to have been naive at best and dishonest and disrespectful of truth at worst. Know what? After listening to this whole video, here's what I think the answer to all this is: humility.

Current Psych Undergrad (and postmodern degenerate in my spare time) Undergraduate courses are really only beginning to address this. I’m taking a class specifically in techniques developed to address the replication crisis. Yet in basically all my other classes it seems like the department is moving in business as usual. Another thing I’ve always wondered about the replication crisis is that psychology is affected by where you are in history. It could be that some of the psychological effects they described were cultural affects that shifted.

There's a very good reason for the saying (quote?) 'Lies, damn lies and statistics'. Certainly the data should be accessible for ALL published studies so that the results and statistical analysis can also be queried. I found it quite surprising when doing my research methods courses that not all peer reviewers review the datasets. We should also know who the reviewers are (some areas of science are quite cliquey) and their reports (and, perhaps, the reply) should be published. Psychology does get a bad rap for dodgy science (perhaps because of the type of people attracted to it?) but there are numerous cases in other sciences. Great video though!