A fantastic paper which I found on ResearchGate that seems highly relevant to all of us:
https://doi.org/10.1038/s44271-023-00003-2
the-replication-crisis-has-led-to-positive-structural-procedural-and-community-changes.pdf (1.1 MB)
I have not read the entire paper, but I believe it remains to be seen if replicability is improved by these measures.
Are there any general statistics indicating the extent to which findings in studies published in major journals have proved replicable in other studies? It would be very interesting to develop a measure for this and then keep track of it to see if there are any noteworthy trends over time.
Otherwise, the suggestion that the replication crisis is met with useful strategies that improve the credibility of research is just an un-tested hypothesis.
1 Like
Are there any general statistics indicating the extent to which findings in studies published in major journals have proved replicable in other studies?
From the article:
In one of the most impactful replication initiative of the last decade, the Open Science Collaboration5 sampled studies from three prominent journals representing different sub-fields of psychology to estimate the replicability of psychological research. Out of 100 independently performed replications, only 39% were subjectively labelled as successful replications, and on average, the effects were roughly half the original size. Putting these results into a wider context, a minimum replicability rate of 89% should have been expected if all of the original effects were true (and not false positives; ref. 6). Pooling the Open Science Collaboration5 replications with 207 other replications from recent years resulted in a higher estimate; 64% of effects successfully replicated with effect sizes being 32% smaller than the original effects7. While estimations of replicability may vary, they nevertheless appear to be sub-optimal—an issue that is not exclusive to psychology and found across many other disciplines (e.g., animal behaviour8,9,10; cancer biology11; economics12), and symptomatic of persistent issues within the research environment13,14.
It would be very interesting to develop a measure for this and then keep track of it to see if there are any noteworthy trends over time.
Good point, I don’t know if any studies that have tested whether the interventions described in the article have actually improved replicability. This reminds me the metascience learning loop described in A Vision of Metascience. Unfortunately, the whole loop probably takes a while to generate a meausrable change in replicability.
From the end of the article:
While developments within the credibility revolution were originally fuelled by failed replications, these in themselves are not the only issue of discussion within the credibility revolution. Furthermore, replication rates alone may not be the best measure of research quality. Instead of focusing purely on replicability, we should strive to maximize transparency, rigour, and quality in all aspects of research18,200
I view the credibility revolution very positively, but this does seems like shifting the goal posts from a hard to measure output of the research process (replicability) to proxies that are potentially easier to measure (transparency, rigour, and quality). I just skimmed the article, so I may have missed it, but it would be nice to see some evidence that the proxies are at least correlated with replicability.
1 Like
I’m a new member here, so this may have been mentioned before, but in the field of cancer biology a reproducibility study was conducted and published in 2021. Here is the link https://elifesciences.org/articles/67995
1 Like
Another paper that fits the context here:
www.doi.org/10.1038/s41559-017-0160
lowndes2017.pdf (329.9 KB)