# The replication crisis has led to positive structural, procedural, and community changes

**URL:** <https://onscienceandacademia.org/t/the-replication-crisis-has-led-to-positive-structural-procedural-and-community-changes/1959>\
**Category:** Open and replicable science\
**Tags:** open-science, replicability, new-paper, replicability-crisis, replicable-science\
**Created:** [August 26, 2023, 12:39am UTC](https://onscienceandacademia.org/t/the-replication-crisis-has-led-to-positive-structural-procedural-and-community-changes/1959 "2023-08-26T00:39:23Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![keithtse](https://sea2.discourse-cdn.com/flex020/user_avatar/onscienceandacademia.org/keithtse/32/224_2.png) [@keithtse](https://onscienceandacademia.org/u/keithtse)\
**Post date:** [August 26, 2023, 12:39am UTC](https://onscienceandacademia.org/t/the-replication-crisis-has-led-to-positive-structural-procedural-and-community-changes/1959/1 "2023-08-26T00:39:23Z")

</div>

A fantastic paper which I found on ResearchGate that seems highly relevant to all of us:

[https://doi.org/10.1038/s44271-023-00003-2](https://doi.org/10.1038/s44271-023-00003-2)

[the-replication-crisis-has-led-to-positive-structural-procedural-and-community-changes.pdf](https://onscienceandacademia.org/uploads/short-url/92kw19IzBaOJWKawoh2YkhrMUGL.pdf) (1.1 MB)

---

<div class="post-metadata">

**Author:** ![Olov](https://sea2.discourse-cdn.com/flex020/user_avatar/onscienceandacademia.org/olov/32/785_2.png) [@Olov](https://onscienceandacademia.org/u/Olov)\
**Post date:** [September 1, 2023, 1:23pm UTC](https://onscienceandacademia.org/t/the-replication-crisis-has-led-to-positive-structural-procedural-and-community-changes/1959/2 "2023-09-01T13:23:02Z")

</div>

I have not read the entire paper, but I believe it remains to be seen if replicability is improved by these measures.

Are there any general statistics indicating the extent to which findings in studies published in major journals have proved replicable in other studies? It would be very interesting to develop a measure for this and then keep track of it to see if there are any noteworthy trends over time.

Otherwise, the suggestion that the replication crisis is met with useful strategies that improve the credibility of research is just an un-tested hypothesis.

---

<div class="post-metadata">

**Author:** ![Gavin](https://avatars.discourse-cdn.com/v4/letter/g/439d5e/32.png) [@Gavin](https://onscienceandacademia.org/u/Gavin)\
**Post date:** [September 3, 2023, 7:58pm UTC](https://onscienceandacademia.org/t/the-replication-crisis-has-led-to-positive-structural-procedural-and-community-changes/1959/3 "2023-09-03T19:58:17Z")

</div>

> Are there any general statistics indicating the extent to which findings in studies published in major journals have proved replicable in other studies?

From the article:

> In one of the most impactful replication initiative of the last decade, the Open Science Collaboration[5](https://www.nature.com/articles/s44271-023-00003-2#ref-CR5) sampled studies from three prominent journals representing different sub-fields of psychology to estimate the replicability of psychological research. Out of 100 independently performed replications, only 39% were subjectively labelled as successful replications, and on average, the effects were roughly half the original size. Putting these results into a wider context, a minimum replicability rate of 89% should have been expected if all of the original effects were true (and not false positives; ref. [6](https://www.nature.com/articles/s44271-023-00003-2#ref-CR6)). Pooling the Open Science Collaboration[5](https://www.nature.com/articles/s44271-023-00003-2#ref-CR5) replications with 207 other replications from recent years resulted in a higher estimate; 64% of effects successfully replicated with effect sizes being 32% smaller than the original effects[7](https://www.nature.com/articles/s44271-023-00003-2#ref-CR7). While estimations of replicability may vary, they nevertheless appear to be sub-optimal—an issue that is not exclusive to psychology and found across many other disciplines (e.g., animal behaviour[8](https://www.nature.com/articles/s44271-023-00003-2#ref-CR8),[9](https://www.nature.com/articles/s44271-023-00003-2#ref-CR9),[10](https://www.nature.com/articles/s44271-023-00003-2#ref-CR10); cancer biology[11](https://www.nature.com/articles/s44271-023-00003-2#ref-CR11); economics[12](https://www.nature.com/articles/s44271-023-00003-2#ref-CR12)), and symptomatic of persistent issues within the research environment[13](https://www.nature.com/articles/s44271-023-00003-2#ref-CR13),[14](https://www.nature.com/articles/s44271-023-00003-2#ref-CR14).

> It would be very interesting to develop a measure for this and then keep track of it to see if there are any noteworthy trends over time.

Good point, I don’t know if any studies that have tested whether the interventions described in the article have actually improved replicability. This reminds me the metascience learning loop described in [A Vision of Metascience](https://scienceplusplus.org/metascience/#can-we-use-decisive-metascientific-results-to-drive-improvement-in-the-social-processes-of-science). Unfortunately, the whole loop probably takes a while to generate a meausrable change in replicability.

From the end of the article:

> While developments within the credibility revolution were originally fuelled by failed replications, these in themselves are not the only issue of discussion within the credibility revolution. Furthermore, replication rates alone may not be the best measure of research quality. Instead of focusing purely on replicability, we should strive to maximize transparency, rigour, and quality in all aspects of research[18](https://www.nature.com/articles/s44271-023-00003-2#ref-CR18),[200](https://www.nature.com/articles/s44271-023-00003-2#ref-CR200)

I view the credibility revolution very positively, but this does seems like shifting the goal posts from a hard to measure output of the research process (replicability) to proxies that are potentially easier to measure (transparency, rigour, and quality). I just skimmed the article, so I may have missed it, but it would be nice to see some evidence that the proxies are at least correlated with replicability.

---

<div class="post-metadata">

**Author:** ![Gavin\_McStay](https://sea2.discourse-cdn.com/flex020/user_avatar/onscienceandacademia.org/gavin_mcstay/32/795_2.png) [@Gavin\_McStay](https://onscienceandacademia.org/u/Gavin_McStay)\
**Post date:** [September 3, 2023, 8:35pm UTC](https://onscienceandacademia.org/t/the-replication-crisis-has-led-to-positive-structural-procedural-and-community-changes/1959/4 "2023-09-03T20:35:07Z")

</div>

I’m a new member here, so this may have been mentioned before, but in the field of cancer biology a reproducibility study was conducted and published in 2021. Here is the link [https://elifesciences.org/articles/67995](https://elifesciences.org/articles/67995)

---

<div class="post-metadata">

**Author:** ![keithtse](https://sea2.discourse-cdn.com/flex020/user_avatar/onscienceandacademia.org/keithtse/32/224_2.png) [@keithtse](https://onscienceandacademia.org/u/keithtse)\
**Post date:** [September 4, 2023, 1:52am UTC](https://onscienceandacademia.org/t/the-replication-crisis-has-led-to-positive-structural-procedural-and-community-changes/1959/5 "2023-09-04T01:52:18Z")

</div>

Another paper that fits the context here:

[www.doi.org/10.1038/s41559-017-0160](http://www.doi.org/10.1038/s41559-017-0160)

[lowndes2017.pdf](https://onscienceandacademia.org/uploads/short-url/lzquxb3ZDXW9EXqeB1l3Cpz540N.pdf) (329.9 KB)

---

<div class="post-metadata">

**Author:** ![Gavin](https://avatars.discourse-cdn.com/v4/letter/g/439d5e/32.png) [@Gavin](https://onscienceandacademia.org/u/Gavin)\
**Post date:** [November 15, 2023, 12:46pm UTC](https://onscienceandacademia.org/t/the-replication-crisis-has-led-to-positive-structural-procedural-and-community-changes/1959/6 "2023-11-15T12:46:02Z")

</div>

Hot of the press, it looks like these interventions do work after all:

> This paper reports an investigation by four coordinated laboratories of the prospective replicability of 16 novel experimental findings using rigour-enhancing practices: confirmatory tests, large sample sizes, preregistration and methodological transparency. In contrast to past systematic replication efforts that reported replication rates averaging 50%, replication attempts here produced the expected effects with significance testing (_P_ \< 0.05) in 86% of attempts, slightly exceeding the maximum expected replicability based on observed effect sizes and sample sizes. When one lab attempted to replicate an effect discovered by another lab, the effect size in the replications was 97% that in the original study. This high replication rate justifies confidence in rigour-enhancing methods to increase the replicability of new discoveries.

[High replicability of newly discovered social-behavioural findings is achievable](https://www.nature.com/articles/s41562-023-01749-9)

See these commentaries as well:

- Nature: [What reproducibility crisis? New research protocol yields ultra-high replication rate](https://www.nature.com/articles/d41586-023-03486-5)
- Science: [Preregistering, transparency, and large samples boost psychology studies’ replication rate to nearly 90%](https://www.science.org/content/article/preregistering-transparency-and-large-samples-boost-psychology-studies-replication-rate)

---

<div class="post-metadata">

**Author:** ![Gavin](https://avatars.discourse-cdn.com/v4/letter/g/439d5e/32.png) [@Gavin](https://onscienceandacademia.org/u/Gavin)\
**Post date:** [September 30, 2024, 7:37pm UTC](https://onscienceandacademia.org/t/the-replication-crisis-has-led-to-positive-structural-procedural-and-community-changes/1959/7 "2024-09-30T19:37:37Z")

</div>

Hmm… apparently it was too good to be true.

[What’s the story behind that paper by the Center for Open Science team that just got retracted?](https://statmodeling.stat.columbia.edu/2024/09/26/whats-the-story-behind-that-paper-by-the-center-for-open-science-team-that-just-got-retracted/)