# Monthly Reading Recommendations **URL:** \ **Category:** Open and replicable science\ **Tags:** metascience, igdore, philosophy\ **Created:** [June 25, 2021, 5:40pm UTC](https://onscienceandacademia.org/t/monthly-reading-recommendations/1412 "2021-06-25T17:40:55Z")\ **Posts on this page:** 1\ **Showing post:** 24
**Author:** ![Gavin](https://avatars.discourse-cdn.com/v4/letter/g/439d5e/32.png) [@Gavin](https://onscienceandacademia.org/u/Gavin)\ **Post date:** [October 31, 2022, 4:01pm UTC](https://onscienceandacademia.org/t/monthly-reading-recommendations/1412/24 "2022-10-31T16:01:53Z")
This month’s article is [Reimagining peer review as an expert elicitation process](https://link.springer.com/article/10.1186/s13104-022-06016-0) by Marcoci et al. I came across it when interacting with several of the authors on another [peer review project](https://onscienceandacademia.org/t/unjournal-call-for-participants-and-research/1731) and thought the idea of using structured expert elicitation as a peer review method very interesting. Indeed, it seems to go well beyond how structured reporting, cross-reviewer commenting and collaborative reviews are described in previous research on peer review innovation conducted by the RoRI (which I recommended [here](https://onscienceandacademia.org/t/monthly-reading-recommendations/1412/19) a few months ago), and may provide a more robust extension of the ‘discussion during review’ model being used at several journals (see [Horbach and Halffman 2018](https://researchintegrityjournal.biomedcentral.com/articles/10.1186/s41073-018-0051-5#:~:text=New%20openness%3A%20discussion%20during%20review)). A structured expert elicitation process '_can demonstrably improve the quality of expert judgements, especially in the context of critical decisions’_. The authors base their recommendations on their ‘_collective experience developing and implementing the IDEA protocol (Investigate—Discuss—Estimate—Aggregate) for structured expert elicitation in diverse settings including conservation, intelligence analysis, biosecurity, and, most recently, for the collaborative evaluation of research replicability and credibility_’. The latter setting refers to the well known [repliCATS project](https://replicats.research.unimelb.edu.au/), in which the IDEA protocol ‘_has been shown to facilitate accurate predictions about which research findings will replicate by prompting experts to investigate and discuss the transparency and robustness of the findings in a structured manner._’ A summary of the basic steps of the IDEA protocol is ([from Hemming et al 2017](https://besjournals.onlinelibrary.wiley.com/doi/full/10.1111/2041-210X.12857#:~:text=important%20practical%20gap.-,2%20THE%20IDEA%20PROTOCOL,-The%20acronym%20IDEA)): > A diverse group of experts is recruited to answer questions with probabilistic or quantitative responses. The experts are asked to first _Investigate_ the questions and to clarify their meanings, and then to provide their private, individual best guess point estimates and associated credible intervals (Round 1). The experts receive feedback on their estimates in relation to other experts. With assistance of a facilitator, the experts are encouraged to _Discuss_ the results, resolve different interpretations of the questions, cross-examine reasoning and evidence, and then provide a second and final private _Estimate_ (Round 2). Notably, the purpose of discussion in the IDEA protocol is not to reach consensus but to resolve linguistic ambiguity, promote critical thinking, and to share evidence. This is based on evidence that incorporating a single discussion stage within a standard Delphi process generates improvements in response accuracy. The individual estimates are then combined using mathematical _Aggregation_. The present article ‘_outline[s] five recommendations focusing on individual and group characteristics that contribute to higher quality judgements, and on ways of structuring elicitation protocols that promote constructive discussion to enable editorial decisions that represent a transparent aggregation of diverse opinions_’. These are: - _Elicit diverse opinions_: Leverage the wisdom of the crowd by incorporating reviewers with diverse backgrounds and perspectives - _Challenge conventional definitions of expertise_: The judgement of individual or small groups of experts isn’t always very good, but aggregating the feedback of larger groups of reviewers, drawn from outside traditional expert reviewer pools, may provide more accurate decisions - _Provide structure_: Quantitative estimates of research quality can be aggregated mathematically and quantifies uncertainty in the reviewer judgements - _Encourage and facilitate interaction_: Group discussion often identifies errors and leads to novel ideas that individuals wouldn’t reach by themselves. - _Anonymise judgements_: Social influences can undermine the wisdom of the crowd (i.e. group think) While the IDEA protocol has been able to increase the collective accuracy of expert judgements in a variety of settings, ‘_[t]o what extent similar effects can be achieved in peer review is an empirical question that remains unaddressed_.’ I would certainly be excited to hear about a journal experimenting with peer review based on the IDEA protocol, although, as the article concludes, it ‘_will require some editorial bravery_’! --- _[View the full topic](https://onscienceandacademia.org/t/monthly-reading-recommendations/1412)._