This month there is a double recommendation! Innovations in peer review in scholarly publishing: a meta-summary and Innovating peer review, reconfiguring scholarly communication: An analytical overview of ongoing peer review innovation activities are two complementary preprints, both from the Peer Review project at the Research on Research Institute (RORI). The first article provides a ‘review of reviews’ of six previous literature reviews (including two from @jon_tennant) on innovations in peer review, while the second article reports an inductively developed taxonomy of peer review innovation based on the responses to a survey sent to scholarly communication organisations. Together, the two articles provide an excellent overview of the state of peer review innovation and include far more material than I can summarise here, so I will simply present the frameworks in which the articles present their results and then comment on their conclusions.
The meta-summary places peer review innovations in the context of three categories and a variety of subcategories
-
Approaches to peer review
-
Open/masked peer review
-
Pre/post publication review
-
Collaboration and decoupling
-
Focussed and specialised review
-
-
Review focused incentives
-
Reviewer incentives
-
Reviewer support
-
-
Technology to support peer review
-
Current uses
-
Potential models
-
An interesting point in the meta-summary is the description of the peer review innovation evaluations covered in Bruce et al (2016): ‘The authors found, based on these outcome measures, that compared with standard peer review, reviewer training was not successful in improving the quality of the peer review report and use of checklists by peer reviewers to check the quality of a manuscript did not improve the quality of the final article. However, the addition of a specialised statistical reviewer did improve the quality of the final article and open peer review was also successful in improving the quality of review reports. It did not affect the time reviewers spent on their report. Open peer review also decreased the number of papers rejected. Finally, blinded peer review did not affect the quality of review reports or rejection rates.’
The survey overview led to a slightly more complex taxonomy with five main elements:
An interesting point from the overview is about the participation of patients in reviews at biomedical journals: ‘Our sample contains two examples that involve patients as reviewers for journals. Both obviously focus on biomedical research with a particular emphasis on its practical or clinical relevance - one is the BMJ, and the other Research Involvement and Engagement (published by BMC). Whereas in the former, patient reviewers are invited on a selective basis depending on the submission, the latter journal foresees the regular use of two patient reviewers and two academic reviewers for all manuscripts.’
The overview notes that the diverse range of peer review innovations they have catalogued pull in mutually opposed directions. These oppositions include:
- Efficiency vs. rigour: ‘numerous innovations in the categories “role of reviewers” and “transparency of review” aim to increase the efficiency of peer review, which can to some extent be seen as a remedy to the growing amount and cost of review work’ ← → ‘Our data firstly suggests that many innovations in the categories “objects” and “nature of review” amount to promoting more rigorous quality control, namely by multiplying the objects of and occasions for review.‘
- Singular vs. pluralistic considerations of quality: ‘Registered reports assume a specific understanding of how the research process should be organized, based on an epistemological ideal of particular forms of experimental science.’ ← → ‘Innovations that remove social and disciplinary boundaries to reviewing … [and] deanonymize the review process … encourage a deliberative approach where potentially opposed speakers can explicitly address each other, even though there is also a risk that reviewers may not feel comfortable giving their frank opinion in a deanonymized setting.’
- Transparency vs. objectivity: ‘making review reports and reviewer identities transparent is now a widely offered possibility … assuming that disclosing identities of authors and reviewers is useful for accountability in peer review’ ← → ‘there are also some signs of a trend towards abandoning mandatory disclosure of reviewer identities (BMC) and towards double-blind peer review (IOP Publishing) … presupposing that objectivity of peer review requires anonymity of authors and reviewers’
While the overview mediates the opposing directions of peer review development by calling for coordination between scholarly communication innovators (an activity that the RORI would be well-positioned to facilitate), the meta-summary also identified a strong conclusion that there is the need ‘for a wider reflection on the peer review process as a research community, with both Barroga (2020) and Tennant (2018) underscoring the need to consider what different stakeholders bring to the peer review process and which role they inhabit.’ It seems that ‘coordination’ should not just be limited to scholarly communication organisations but extend into consultation with peer review stakeholders in the broader academic community. Furthermore, the meta-summary also reports ‘Three reviews (Bruce et al, 2016; Horbach & Halffman, 2018; Tennant et al, 2017) conclude that there is a lack of empirical evidence to assess the effectiveness of innovations in peer review.’ This lack of empirical evidence provides little basis for assessing trade-offs made by moving along the opposing directions of innovation, and empirical studies should be prioritised if the evidence base can keep up with the rapid innovation.
Despite the lack of empirical evidence and opposing directions of development, it seems to be a promising time for peer review innovation as the meta-summary notes: ‘The fact that there are enough review articles to warrant a review of reviews, indicates the growing maturity of the field of peer review research.’