Blog: P-hacking and optional stopping have been judged violations of scientific integrity

@Lakens published a very interesting blog post two days ago:

On July 28, 2020, the first Dutch academic has been judged to have violated the code of conduct for research integrity for p -hacking and optional stopping with the aim of improving the chances of obtaining a statistically significant result. I think this is a noteworthy event that marks a turning point in the way the scientific research community interprets research practices that up to a decade ago were widely practiced.

Some people, in particular a senior researcher, were very upset when I during a seminar at the OPLS 2019 described my own questionable research practices as “scientific misconduct”. So upset that s/he convinced a colleague of us both, an ECR who had decided to join IGDORE (of which I’m the founder), to withdraw his/her application. This ECR colleague had got very worried during OPLS when hearing my descriptions of p-hacking, HARKing, etc, realising that methods like that had been employed in her/his own published research and thus decided to join IGDORE. However, after the seminar, the senior researcher convinced her/him that questionable research practices were not at all very questionable - and definitely not misconduct. I later received an email from the ECR withdrawing her/his application to join IGDORE, saying that “I don’t want to be part of personal conflicts between you and your former supervisors”. Apparently, that’s how my critique and my descriptions of the scientific misconduct (i.e. questionable research practices) I’ve engaged in is explained by (former) colleagues in the small sub-field I used to belong to: a personal conflict. I usually avoid publicly getting into details on the relationship between me and my former colleagues but I want to be very clear on this: there have never been any personal conflicts whatsoever between me and my supervisors, only professional/methodological disagreements.

Misleading ECRs this way can have severe consequences on the ECR’s future career, as shown by this new Dutch case described by Lakens.

As norms change, what we deemed a misdemeanor before, is now simply classified as a violation of academic integrity. […] The Dutch National Body for Scientific Integrity has classified p -hacking and optional stopping as violations of research integrity.

To describe the changed views on questionable research practices as mere misdemeanour to the more modern view as misconduct Lakens tell the story of society’s view on smoking. Denny Boorsboom did this as well 7 years ago in this brilliant post:

If you had told me that any of this would happen back in 2005, I would have laughed you away, just as I would have laughed you away in 1990, had you told me that the future would involve such bizarre elements as smoke-free airplanes.

The moral order of research in psychology has changed. It has changed for the better, and I hope it has changed for good.

3 Likes

Can’t wait for more defined violations!

As I tweeted about this, it’s a great development, but the test will come when we see somebody being “indicted” (or whatever the academic term is) for research misconduct based solely on QRPs. In the case discussed here, it seems that they had a list of other, ostensibly (by past standards anyway) more severe, including taking blood from people without informed consent.

Put another way: We are still prosecuting (to the extent that it happens at all) the gangsters of science for their obvious first-order crimes, the equivalent of bumping off opponents or robbing banks. But the Feds (famously) ended up putting Al Capone in jail for tax evasion.

2 Likes

@sTeamTraen Wasn’t that the case with Jens Förster at University of Amsterdam?

I wouldn’t say so. The results that Förster published were clearly fabricated (the lines were all far too straight to be possible). IIRC he more or less admitted as much, but then blamed a succession of grad students, none of whose names he could remember.

I suppose you could collect data and eliminate 95% of points until you ended up with a straight line through the rest and call it “extreme p-hacking”, but that wasn’t what happened here AFAIK.

You were one of the people investigating his work, right? Fabricated in what way, you think?

You were one of the people investigating his work, right? No, I was just a spectator, looking on in a mixture of horror and amusement at the unfolding car crash.

The results were clearly manipulated. Whether there was a real experiment and JF just changed a lot of the numbers, or whether no study ever took place at all, I have no idea (and there were multiple papers).

1 Like