Peer Review Methodology

Common Pitfalls

1. Reviewing what the paper should be instead of what it is: Judging the paper against your preferred experimental approach rather than evaluating whether the authors' approach is valid. How to avoid: Focus on whether the methods answer the stated research question, not on whether you would have used different methods.

2. Conflating statistical significance with scientific importance: A paper with p=0.001 can be trivial; a paper with p=0.06 can be groundbreaking. How to avoid: Evaluate effect sizes and practical significance separately from p-values. Ask "Does this matter?" not just "Is this significant?"

3. Scope creep in revision requests: Requesting additional experiments that constitute a new study rather than validation of the current one. How to avoid: Before writing a revision request, ask "Is this necessary to support the paper's current conclusions?" If it addresses a new question, suggest it as future work instead.

4. Overlooking positive controls: Checking for negative controls while ignoring whether the experiment demonstrates it CAN detect the claimed effect. How to avoid: For every experiment, check both "What would happen if the effect doesn't exist?" (negative control) AND "What demonstrates the assay actually works?" (positive control).

5. Copy-paste reviewing: Using the same boilerplate concerns across different manuscripts without engaging with the specific content. How to avoid: Every comment should reference a specific section, figure, or claim in THIS manuscript. Generic advice ("improve writing quality") is unhelpful.

6. Ignoring supplementary materials: Many critical methods details and validation data are in supplements. How to avoid: Always read supplementary materials, especially methods and figures referenced from the main text.

7. Asymmetric skepticism: Being more critical of findings that contradict your expectations while accepting confirmatory findings uncritically. How to avoid: Apply the same methodological scrutiny to all results regardless of whether you agree with the conclusions.

Workflow

7-Stage Peer Review Process

1. Initial assessment (read full manuscript without annotation): Identify central question, main findings, overall soundness, venue appropriateness. Write 2-3 sentence summary capturing the manuscript's essence

2. Section-by-section review: Evaluate each section (Abstract, Introduction, Methods, Results, Discussion, References) against specific criteria:

   • Abstract: accuracy, completeness, clarity
   • Introduction: context, rationale, novelty, literature coverage
   • Methods: reproducibility, rigor, detail, statistical justification
   • Results: presentation logic, completeness, objectivity
   • Discussion: interpretation support, limitations, context, speculation

3. Statistical and methodological rigor: Check statistical assumptions, effect sizes, multiple testing correction, sample size justification, appropriate test selection, control adequacy, randomization, blinding

4. Reproducibility and transparency: Verify data availability (repositories, accession numbers), code availability, reporting standards compliance (CONSORT/STROBE/PRISMA/ARRIVE), protocol detail

5. Figure and data integrity: Check resolution, labeling, error bar definitions, color accessibility, signs of image manipulation, scale bars, data type appropriateness

6. Ethical considerations: Verify IRB/IACUC approval, informed consent, 3Rs compliance, conflicts of interest disclosure, funding disclosure, data privacy

7. Writing quality: Assess organization, language clarity, jargon use, narrative flow, accessibility to broad audience

Review Report Structure

After completing all stages, organize feedback into:

• Summary statement (1-2 paragraphs): Synopsis, overall recommendation, 2-3 strengths, 2-3 weaknesses
• Major comments (numbered): Critical validity/interpretability issues with specific suggestions
• Minor comments (numbered): Clarity/completeness improvements with locations
• Questions for authors: Specific clarification requests
• Recommendation: Accept / Minor revision / Major revision / Reject (consistent with comments)

Further Reading

• COPE Ethical Guidelines for Peer Reviewers — Committee on Publication Ethics
• EQUATOR Network — reporting guidelines for all study types
• Stahel & Moore (2014) "Peer review for biomedical publications: we can improve the system" — BMC Medicine 12:179
• Lovejoy et al. (2011) "An introduction to systematic reviewing" — BMJ Open
• How to peer review — Wiley reviewer guide

Related Skills

• scientific-critical-thinking — evaluating evidence quality and logical reasoning (broader than manuscript review)
• scientific-writing — manuscript authoring; understanding the author's perspective helps write better reviews
• literature-review — systematic evidence gathering; complements Stage 4 (reproducibility) assessment
• scholar-evaluation — quantitative scoring frameworks for automated manuscript assessment