This month’s article is Reimagining peer review as an expert elicitation process by Marcoci et al. I came across it when interacting with several of the authors on another peer review project and thought the idea of using structured expert elicitation as a peer review method very interesting. Indeed, it seems to go well beyond how structured reporting, cross-reviewer commenting and collaborative reviews are described in previous research on peer review innovation conducted by the RoRI (which I recommended here a few months ago), and may provide a more robust extension of the ‘discussion during review’ model being used at several journals (see Horbach and Halffman 2018).
A structured expert elicitation process 'can demonstrably improve the quality of expert judgements, especially in the context of critical decisions’. The authors base their recommendations on their ‘collective experience developing and implementing the IDEA protocol (Investigate—Discuss—Estimate—Aggregate) for structured expert elicitation in diverse settings including conservation, intelligence analysis, biosecurity, and, most recently, for the collaborative evaluation of research replicability and credibility’. The latter setting refers to the well known repliCATS project, in which the IDEA protocol ‘has been shown to facilitate accurate predictions about which research findings will replicate by prompting experts to investigate and discuss the transparency and robustness of the findings in a structured manner.’
A summary of the basic steps of the IDEA protocol is (from Hemming et al 2017):
A diverse group of experts is recruited to answer questions with probabilistic or quantitative responses. The experts are asked to first Investigate the questions and to clarify their meanings, and then to provide their private, individual best guess point estimates and associated credible intervals (Round 1). The experts receive feedback on their estimates in relation to other experts. With assistance of a facilitator, the experts are encouraged to Discuss the results, resolve different interpretations of the questions, cross-examine reasoning and evidence, and then provide a second and final private Estimate (Round 2). Notably, the purpose of discussion in the IDEA protocol is not to reach consensus but to resolve linguistic ambiguity, promote critical thinking, and to share evidence. This is based on evidence that incorporating a single discussion stage within a standard Delphi process generates improvements in response accuracy. The individual estimates are then combined using mathematical Aggregation.
The present article ‘outline[s] five recommendations focusing on individual and group characteristics that contribute to higher quality judgements, and on ways of structuring elicitation protocols that promote constructive discussion to enable editorial decisions that represent a transparent aggregation of diverse opinions’. These are:
- Elicit diverse opinions: Leverage the wisdom of the crowd by incorporating reviewers with diverse backgrounds and perspectives
- Challenge conventional definitions of expertise: The judgement of individual or small groups of experts isn’t always very good, but aggregating the feedback of larger groups of reviewers, drawn from outside traditional expert reviewer pools, may provide more accurate decisions
- Provide structure: Quantitative estimates of research quality can be aggregated mathematically and quantifies uncertainty in the reviewer judgements
- Encourage and facilitate interaction: Group discussion often identifies errors and leads to novel ideas that individuals wouldn’t reach by themselves.
- Anonymise judgements: Social influences can undermine the wisdom of the crowd (i.e. group think)
While the IDEA protocol has been able to increase the collective accuracy of expert judgements in a variety of settings, ‘[t]o what extent similar effects can be achieved in peer review is an empirical question that remains unaddressed.’ I would certainly be excited to hear about a journal experimenting with peer review based on the IDEA protocol, although, as the article concludes, it ‘will require some editorial bravery’!