Most preregistered research in psychology have undisclosed deviations. As we already know from a couple of Cochrane reviews of clinical trials, preregistration is no safe guard against questionable research practices. Do people wittingly cheat or do they just not know that they’re cheating? So, so tired of crappy research. Can’t people just shape up or change profession… 
1 Like
My experience with preregistration has been very smooth, largely because I followed a clear and structured protocol developed by the Center for Open Science. My project was also suited for preregistration: it was confirmatory, with directional hypothesis stemming from a large body of literature. However, not all projects are like this, and that’s where the process becomes tricky. Add time constraints, lack of formal training, and no rewards/incentives, and the chances of “suboptimal” prereg protocols increase dramatically.
1 Like
I totally get your disappointment. I get frustrated too when I see crappy research. Still, I have hope that it’s just the beginning, people are still learning how to do preregistration and report results and deviations. Looking at the table with number of deviations reported and non-reported in the article you posted, I assume that there would be many more “deviations” (or researcher degrees of freedom, whatever we would call it without a preregistration), if the same studies had not been preregistered. Even if some people are going to cheat simply for getting badges or other strategic reasons, I still think that preregistration will help make psychology more reproducible.
1 Like
But are we really at the beginning? Mandatory preregistration of clinical trials was launched 2005; 14 years ago. At least two Cochrane reviews have been published since then investigating how well preregistration has worked. Both find the same results: unexplained discrepancies between preregistration (aka protocol) and report are common.
Dwan et al (2008).pdf (1.3 MB)
Dwan et al (2011) COCHRANE(1).pdf (564.1 KB)
In addition has it been investigated how well Cochrane’s use of preregistration (aka protocol) works (all systematic reviews conducted in the name of Cochrane must be preregistered; that’s probably an important reason why Cochrane reviews are considered to be the gold standard in medicine). The results: again, lots of unexplained discrepancies.
Kirkham et al (2010) Publication bias in Cochrane reviews.pdf (166.0 KB)
The current preprint (https://psyarxiv.com/d8wex) from psychology again shows the same thing.
I hypothesise that we in 10 years from now will find pretty much the same results in psychology as well as in medicine. And that these non-differences will remain also 20 years later. (But I do really hope I will be proved wrong.)
Yes, it’s true that preregistration in clinical trials started quite a while ago. Still, we are at the beginning in the field of psychology, many people still haven’t done or are only doing their first preregistration. Anyways, as you say things will probably turn out similarly in psychology as in medicine. But even if these discrepancies will continue to occur, do you think it could be argued that things at least moved in the right direction, i.e. that preregistration reduced researcher degrees of freedom to a considerable degree and thus makes research a bit more reproducible than it was before?
In 10 years from now, most good research will be preregistered - which indeed is a win for science because it will be easier for everyone to know that the good research is truly good but also for purposes of replication and reviews.
However, in 10 years from now, most preregistered research will still not be good. I think this is a seriously weak spot of the current revolution and that open practices will not fix it. I do have another idea of what might help, though, and that is a quite radical solution: Change how the recruitment of graduate students and academics is made; let’s not accept people into academia if they do not join for the right reasons (= truth seeking).
In addition, how to make it more interesting for the right people to stay in or join academia? I believe the following things could help in that: location independence (i.e. allow people to work from wherever they want to be in the world), completely flexible working hours, no mandatory administration or teaching.
How to make it less interesting for the wrong people to stay in or join academia? This is more difficult and I’m not sure it’s even possible unless using e.g. personality screening tools. I think it was Wicherts et al who reported in a paper < 5 years ago that the most important difference they could find between researchers who used QRPs and researchers who didn’t use QRPs was in personality: the former were higher on narcissism and psychopathy (note that I don’t remember exactly which traits they measured and how, I recall it as being these two traits but it might have been something different albeit similar).
In short, I don’t any longer think open science is enough if we want to make a real and lasting change. But it is indeed an important development for those of us who want to do good science, and it might turn a few who were already on the border between open and closed science.
I agree pretty much, letting only the “right” truth seeking people into academia is a radical idea, but would definitely make science better. I also must admit location independence (and the other aspects you mention) would be very motivating factors to stay in academia for me. And open science will help, but not solve the problem. Thanks for sharing your valuable thoughts and ideas!
1 Like