Eight problems with literature reviews and how to fix them

Philosophy & History Metascience
1

Some ways to improve your systematic reviews. Written by environmental researchers but I think the scope is general.

https://www.nature.com/articles/s41559-020-01295-x

Traditional approaches to reviewing literature may be susceptible to bias and result in incorrect decisions. This is of particular concern when reviews address policy- and practice-relevant questions. Systematic reviews have been introduced as a more rigorous approach to synthesizing evidence across studies; they rely on a suite of evidence-based methods aimed at maximizing rigour and minimizing susceptibility to bias. Despite the increasing popularity of systematic reviews in the environmental field, evidence synthesis methods continue to be poorly applied in practice, resulting in the publication of syntheses that are highly susceptible to bias. Recognizing the constraints that researchers can sometimes feel when attempting to plan, conduct and publish rigorous and comprehensive evidence syntheses, we aim here to identify major pitfalls in the conduct and reporting of systematic reviews, making use of recent examples from across the field. Adopting a ‘critical friend’ role in supporting would-be systematic reviews and avoiding individual responses to police use of the ‘systematic review’ label, we go on to identify methodological solutions to mitigate these pitfalls. We then highlight existing support available to avoid these issues and call on the entire community, including systematic review specialists, to work towards better evidence syntheses for better evidence and better decisions.

The 8 problems addressed are:

  1. Lack of relevance (limited stakeholder engagement).
  2. Mission creep and lack of a protocol.
  3. Lack of transparency/replicability (inability to repeat the study)
  4. Selection bias and a lack of comprehensiveness (inappropriate search methods and strategy).
  5. Publication bias (exclusion of grey literature and failure to test for evidence of publication bias).
  6. Lack of appropriate critical appraisal (treating all evidence as equally valid)
  7. Inappropriate synthesis (using vote-counting and inappropriate statistics)
  8. A lack of consistency and error checking (working individually).

Box 1 (Recommended actions for authors wishing to conduct more rigorous literature reviews) contains some good tips as well. Check this pay-wall free link

3 Likes