Literature Review

Common Pitfalls

1. Starting the search before finalizing eligibility criteria: Running database searches before defining inclusion/exclusion criteria leads to circular logic — researchers unconsciously adjust criteria based on what papers they found.

   • How to avoid: Write and finalize the eligibility criteria (including PICO, study design, language, date range, and publication type restrictions) before running any database search. Register in PROSPERO to lock in the protocol.

2. Searching only PubMed and calling it systematic: PubMed covers MEDLINE but misses significant literature indexed only in Embase, PsycINFO, CINAHL, or regional databases. A single-database search fails the comprehensiveness criterion for a systematic review.

   • How to avoid: Use at least 3 databases for clinical topics; document all databases searched, date of search, and search strings in the Methods section.

3. Updating search results during screening without documenting the update: Literature is published continuously; some researchers run updated searches mid-screening and add results without documenting. This invalidates the flow diagram and makes the review non-reproducible.

   • How to avoid: Define a database lock date before screening begins. If an update is needed, document it as a separate search with its own date and numbers. Consider running a final "top-up" search immediately before submission.

4. Applying inclusion criteria inconsistently between screeners: Without a piloting phase, Screener A may interpret "adult" as ≥18 and Screener B as ≥21, or one screener may include conference abstracts while the other excludes them.

   • How to avoid: Pilot screen on 50–100 records, calculate kappa, discuss every disagreement to align interpretations, and update the eligibility criteria document with clarifying examples before full screening.

5. Conducting meta-analysis when studies are too heterogeneous: Pooling estimates from studies with very different populations, interventions, outcomes, or follow-up durations produces a misleading "average" that may not apply to any real clinical scenario. I² > 75% typically signals unacceptably high statistical heterogeneity.

   • How to avoid: Pre-specify a heterogeneity threshold (e.g., I² < 50% required for pooling) in the PROSPERO protocol. When heterogeneity is high, report a narrative synthesis with subgroup analysis, not a single pooled estimate.

6. Omitting quality/risk of bias assessment: Without appraising study quality, a systematic review cannot assess confidence in the evidence. Reviewers routinely reject systematic reviews that compile evidence without evaluating methodological rigor.

   • How to avoid: Select the appropriate risk of bias tool before screening: RoB 2 (randomized trials), ROBINS-I (non-randomized interventions), QUADAS-2 (diagnostic accuracy), NOS (cohort/case-control). Apply it to all included studies and include results in the synthesis.

7. Writing the discussion as if a narrative review: After conducting a rigorous systematic review, authors sometimes revert to selective, opinion-driven discussion that ignores the quality appraisal results and treats all evidence as equivalent.

   • How to avoid: Anchor each discussion paragraph to specific evidence quality assessments. Use GRADE language: "moderate-certainty evidence suggests..."; "low-certainty evidence from a single small RCT...". Distinguish what the evidence shows from what the authors believe.

Workflow

1. Question formulation and scoping

   • Define the research question using PICO/PECS framework
   • Conduct a preliminary search (PubMed, Google Scholar) to confirm the topic is not already covered by a recent systematic review
   • Decide on review type (systematic, scoping, narrative)
   • Write the protocol; register in PROSPERO (systematic reviews)

2. Search strategy development

   • Identify index terms (MeSH for PubMed, Emtree for Embase) + free-text synonyms for each PICO component
   • Combine within components using OR; combine components using AND
   • Have search peer-reviewed using PRESS checklist
   • Run searches across all databases; record date, database, and total hits

3. Deduplication and screening

   • Import all results into reference manager or screening tool (Rayyan, Covidence)
   • Remove duplicates (automated + manual)
   • Stage 1: title/abstract screening by ≥2 independent reviewers; resolve conflicts
   • Stage 2: full-text screening by ≥2 independent reviewers; document reasons for exclusion
   • Calculate and report inter-rater agreement (Cohen's kappa)

4. Data extraction and quality appraisal

   • Extract data using pre-designed structured forms
   • Assess risk of bias using appropriate tool (RoB 2, ROBINS-I, QUADAS-2, NOS)
   • Contact study authors for missing data if needed

5. Synthesis

   • Narrative synthesis: tabulate studies, describe patterns, explore heterogeneity qualitatively
   • Meta-analysis (if appropriate): calculate pooled effect estimates (OR, RR, MD, SMD) with 95% CI; test heterogeneity (I², Cochran Q); assess publication bias (funnel plot, Egger's test)
   • Rate certainty of evidence using GRADE

6. Reporting and submission

   • Draft manuscript following PRISMA 2020 checklist
   • Create PRISMA flow diagram
   • Submit PRISMA checklist as supplementary material
   • Share search strategies, data extraction forms, and risk of bias assessments as supplementary data or OSF repository

Further Reading

• PRISMA 2020 statement and checklist — the standard reporting guideline for systematic reviews and meta-analyses
• Cochrane Handbook for Systematic Reviews of Interventions — comprehensive methodological reference; freely available
• PROSPERO registration — international register for systematic review protocols
• Rayyan systematic review screening tool — free web-based title/abstract screening platform
• GRADE Working Group — evidence quality assessment framework used in clinical systematic reviews

Related Skills

• citation-management — reference managers for collecting and organizing search results before and during review
• statistical-analysis — statistical methods for meta-analysis (pooled effects, heterogeneity, forest plots)
• scientific-critical-thinking — evaluating individual study quality and interpreting effect sizes in the context of a review