Monthly Reading Recommendations

Openness Movements
, ,
21

The title of this month’s article says a lot: Correction of scientific literature: Too little, too late! The article is short and well worth reading through, so I’ll also keep this recommendation short and to the point.

Essentially, the COVID-19 pandemic lead to a lot of fast and high-profile science getting published, some of which cut corners in terms of quality control and transparency. Yet, the article notes that the traditional response to poor quality of fraudulent research are too little: ‘Nowadays, preprints and peer-reviewed research papers are rapidly shared on online platforms among millions of readers within days of being published. A paper can impact worldwide health and well-being in a few weeks online; that it may be retracted at some point months in the future does not undo any harm caused in the meantime. Even if a paper is removed entirely from the publication record, it will never be removed from the digital space and is still likely to be cited by researchers and laypeople alike as evidence. Often, its removal contributes to its mystique.’ And because papers are shared so fast, the traditional responses usually come too late: ‘Identifying flaws in a paper may only take hours, but even the most basic formal correction can take months of mutual correspondence between scientific journal editors, authors, and critics. Even when authors try to correct their own published manuscripts, they can face strenuous challenges that prompt many to give up.’

A key point highlighted by the article is that scientific critics are rarely rewarded, and often penalized or stigmatized, for their work to correct scientific errors. Indeed, the authors of this article speak from personal experience: ‘[We] have all been involved in error detection in this manner. For our voluntary work, we have received both legal and physical threats and been defamed by senior academics and internet trolls.’ I agree with their position that ‘Public, open, and moderated review on PubPeer [8] and similar websites that expose serious concerns should be rewarded with praise rather than scorn, personal attacks, or threats’

The article provides several recommendations to facilitate faster and more visible scientific correction, and importantly three more aimed at destigmatizing the work of error correctors:

  • Rewarding scientific error correction during assessments for hiring, promotion and funding.
  • Train scientists to recognize mistakes and scientific institutions and funders to value error-checking.
  • Provide legal protection for scientific critics who raise concerns in a professional and non-defamatory manner.

Is this enough? it seems like the least the scientific community could do…

22

This month’s recommendation is for the light-hearted Night Science series of editorials written by Yanai and Lercher. They started by introducing the concept of Night Science, the unstructured and apparently haphazard search for possible hypothesis and theories, as the counterpart to Day Science, where hypotheses are rigorously tested through experimentation. Later articles in the series have explored:

I have thoroughly enjoyed reading all the Night Science series so far, but will limit myself to sharing some thoughts on Yanai and Lercher’s latest article, What puzzle are you in? (I’d love to discuss them all, but I realised I’d committed to more than I wanted to last time I tried to do this recommendation for two articles, let alone a series!)

The authors start by presenting several examples of artificial puzzles and dimensions along which they can be classified which can also be useful for natural problems solved by researchers:

Despite their complexity, nature’s puzzles can be classified in the same way as puzzles humans invented for entertainment: jigsaw puzzles [Class I], logical puzzles [Class II], puzzles where we need to find connections to phenomena outside the problem description [Class III], and puzzles that require us to think outside the box [Class IV], often by identifying and dropping implicit assumptions. These archetypes can be distinguished along with two dimensions: whether they are closed-world or open-world and whether the solutions require either making connections or deeper insights into the problem structure.

Some specific examples were given for scientific discoveries that match these puzzles are:

  1. Jigsaw puzzles - genome assembly and protein crystallography
  2. Logical puzzles - comma-free coding of codons into proteins
  3. Outside connections - Gödel’s incompleteness theorems and natural selection
  4. Out of the box - CRISPR

But the article makes an important observation about the real practice of science: ‘When we actively work on a scientific problem, we have no way to be certain what kind of a puzzle we are in, or if the puzzle as we see it even has a solution. Solving research puzzles is a hierarchical problem. You not only have to find the solution to a puzzle that belongs to one of the four classes. You also have to solve the meta-puzzle of discovering what class of puzzle you are in.’ And frustratingly (illustrated with examples of the authors’ own genomics research): ‘At any instance, the puzzle may switch, making you realize that you are in a different kind of puzzle than you originally thought.’

The authors also speculate that there may be common patterns for how research problems switch between classes of puzzles, which seems like it could be a useful approach to explore further for systematising scientific problem-solving. Treating research like puzzles might ultimately be both an enjoyable and productive strategy to approach the ambiguity inherent in it:

Adopting the mindset of a puzzle solver may help us to reframe this uncertainty—we may view it as part of a playful process, allowing us to have an open mind and to not stick rigidly to the project’s original framing. Without this playful, puzzle-solving attitude, we may not only limit the joy of doing science. We may also miss out on quite a few insights, big or small.

The dimensions of puzzle classification prompted me to think about how this related to deductive and inductive reasoning. I initially expected these modes of reasoning would correspond to closed and open-world puzzles, respectively. Indeed, solving closed-world puzzles clearly seems to be deductive, while reframing open-world puzzles does seem inductive (as would be the meta-puzzle of determining which puzzle class you are working with). Yet, (at least to me) the examples provided for finding connections in open-world puzzles appear to be split between both types of reasoning: relabelling the pot and Gödel’s proofs both still seem to be applications of deduction, while Darwin’s theory is more obviously induction. I feel it is in the spirit of the series to suggest that looking at additional examples of this class of puzzles could be a useful exercise to refine the framing of the puzzle classification dimensions. (or maybe I have just done a bad job of classifying reasoning required for the Class III puzzle examples!)

I’d also like to add a shoutout to a class of puzzles I enjoy (and struggle with…): Bongard problems. These are games of inductive reasoning popular in computer science, and there is a large collection of problems available here if the reader would like to try some. (David Chapman also has a nice post relating Bongard problems to his idea of meta-rationality, which treats some ideas that are similar to the discussion of the meta-puzzle of puzzle class identification in this article)

If you are too excited about Night Science to wait for the next article, then I also recommend the Night Science podcast that the authors host.

2 Likes
23

This month article is Ten simple rules for implementing open and reproducible research practices after attending a training course preprint from Heise et al. Many people attend training courses about robust research and open science at conferences or other events, but learning the material is just the first step—implementing the practices in your research often leads to unexpected challenges to overcome (particularly if your colleagues aren’t as enthusiastic about making changes as you are!) This article presents ten clear and concise rules to help everybody make the most of their robust research training:

  1. Join a robust research community to access expertise and support (this forum and IGDORE are two possibilities)
  2. Shortlist the practice you’d like to try implementing first in a project you are currently working on
  3. Discuss the changes you want to make with your research team
  4. Prepare for concerns your colleagues may have and address them constructively
  5. Set up an implementation plan after your team has reached agreement
  6. Compromise if needed and stay patient while working towards long-term improvements
  7. Make your changes sustainable by creating documentation and peer support structures
  8. Continue developing your competencies and seek recognition for doing so
  9. Practice self-care and avoid burnout
  10. Find future employers/colleagues who share your values and will utilize your robust research skills!

As a bonus, the article also provides 10 tips to help course organizers prepare their participants for the challenges of implementation:

  1. Consider the background of your participants when designing course material
  2. Cover a range of topics so different participants all find something of interest
  3. Talk about how the course content relates to institutional and funder policies
  4. Train participants in ‘soft-skills’ they can use to encourage behavioral change in their research team
  5. Allow time for participants to start implementing the practices they are being taught
  6. Avoid overwhelming your audience by breaking the training up into a series of short sessions
  7. Make the teaching material and resources reusable, so participants can host their own training events
  8. Create communities and networks for alumni to stay in touch with each other and the course organizers
  9. Organize times for participants to focus on implementing specific practices
  10. Plan to host follow-up events to keep the momentum going!

Check the article for more details on all the points above.

2 Likes
24

This month’s article is Reimagining peer review as an expert elicitation process by Marcoci et al. I came across it when interacting with several of the authors on another peer review project and thought the idea of using structured expert elicitation as a peer review method very interesting. Indeed, it seems to go well beyond how structured reporting, cross-reviewer commenting and collaborative reviews are described in previous research on peer review innovation conducted by the RoRI (which I recommended here a few months ago), and may provide a more robust extension of the ‘discussion during review’ model being used at several journals (see Horbach and Halffman 2018).

A structured expert elicitation process 'can demonstrably improve the quality of expert judgements, especially in the context of critical decisions’. The authors base their recommendations on their ‘collective experience developing and implementing the IDEA protocol (Investigate—Discuss—Estimate—Aggregate) for structured expert elicitation in diverse settings including conservation, intelligence analysis, biosecurity, and, most recently, for the collaborative evaluation of research replicability and credibility’. The latter setting refers to the well known repliCATS project, in which the IDEA protocol ‘has been shown to facilitate accurate predictions about which research findings will replicate by prompting experts to investigate and discuss the transparency and robustness of the findings in a structured manner.’

A summary of the basic steps of the IDEA protocol is (from Hemming et al 2017):

A diverse group of experts is recruited to answer questions with probabilistic or quantitative responses. The experts are asked to first Investigate the questions and to clarify their meanings, and then to provide their private, individual best guess point estimates and associated credible intervals (Round 1). The experts receive feedback on their estimates in relation to other experts. With assistance of a facilitator, the experts are encouraged to Discuss the results, resolve different interpretations of the questions, cross-examine reasoning and evidence, and then provide a second and final private Estimate (Round 2). Notably, the purpose of discussion in the IDEA protocol is not to reach consensus but to resolve linguistic ambiguity, promote critical thinking, and to share evidence. This is based on evidence that incorporating a single discussion stage within a standard Delphi process generates improvements in response accuracy. The individual estimates are then combined using mathematical Aggregation.

The present article ‘outline[s] five recommendations focusing on individual and group characteristics that contribute to higher quality judgements, and on ways of structuring elicitation protocols that promote constructive discussion to enable editorial decisions that represent a transparent aggregation of diverse opinions’. These are:

  • Elicit diverse opinions: Leverage the wisdom of the crowd by incorporating reviewers with diverse backgrounds and perspectives
  • Challenge conventional definitions of expertise: The judgement of individual or small groups of experts isn’t always very good, but aggregating the feedback of larger groups of reviewers, drawn from outside traditional expert reviewer pools, may provide more accurate decisions
  • Provide structure: Quantitative estimates of research quality can be aggregated mathematically and quantifies uncertainty in the reviewer judgements
  • Encourage and facilitate interaction: Group discussion often identifies errors and leads to novel ideas that individuals wouldn’t reach by themselves.
  • Anonymise judgements: Social influences can undermine the wisdom of the crowd (i.e. group think)

While the IDEA protocol has been able to increase the collective accuracy of expert judgements in a variety of settings, ‘[t]o what extent similar effects can be achieved in peer review is an empirical question that remains unaddressed.’ I would certainly be excited to hear about a journal experimenting with peer review based on the IDEA protocol, although, as the article concludes, it ‘will require some editorial bravery’!

1 Like
25

This month we will be looking at Open science and public trust in science: Results from two studies by Rosman et al. The article reports the results of two empirical studies on how open science practices influence public trust in science, and also provides an excellent introduction that thoroughly covers this topic.

Many open science advocates would say that open science makes research more trustworthy, but ‘despite increasing adherence to open science practices (OSPs) in the scientific community, little is known about the public’s expectations of such practices and their effects on the perceived trustworthiness of science.’ Indeed, ‘the few experimental studies on the relationship between OSPs and trust in science have yielded rather inconclusive results’, with one finding open science badges increased trust in scientists, two other studies providing inconclusive results and, finally, a study found the informing participants about the replicability crisis in psychology (including proposed reforms) reduced their trust in future research.

The current article built on the previous work by, firstly, replicating survey results on the beneficial effects of OSPs on trust and, secondly, using an experimental study using a vignette-based manipulation to test if OSPs were causally related to trust. Additionally, the research extended on prior research by addressing the field specificity of the relationship between OSP practices and trust (between science as a whole, psychology, and medicine), and the influence of whether research was publicly or privately funded (including whether OSPs buffered the trust-damaging effect of private funding). To their credit, the authors clearly practice the OSPs they study, and the article contains links to their preregistration, materials, data, and code.

The survey found:

  • ‘An overwhelming majority of our sample found it important that researchers make their findings openly accessible and that they implement OSPs’
  • ‘a large proportion of participants indicated that their trust in a scientific study would increase if they saw that researchers made their materials, data, and code openly accessible.’

However, the experimental study was less conclusive:

  • ‘there are some indications in our data that the use of OSPs may increase trust—although it should also be noted that the corresponding effect sizes were rather small.’
  • ‘analyses yielded evidence for the effects of a study’s funding type on trust, such that publicly funded studies are trusted more than privately funded ones.’
  • ‘the trust-damaging effects of research being privately funded may be buffered by OSPs … this hypothesis was clearly not supported by our data.’

After placing these finding in the context of the other existing experimental studies, the two studies ‘imply that people may well recognize open science as a trust-increasing factor, especially when directly asked about it, but that other factors such as communication strategies may play a comparatively stronger role in the development of trust in science.’ As well as focusing on communication, the discussion also notes that ‘combining increased transparency with such participatory approaches may thus be even more promising to increase trust in science compared to transparency alone.’

While the authors conclude their ‘results suggest that OSPs may well contribute to increasing trust in science and scientists’ and ‘recognize the potential role of OSPs in attenuating the negative effects of the replication and trust crisis’, reassuringly they findings also show that the public’s trust in science is actually already rather high.

1 Like