Literature Review

Literature Review

Systematic discovery, extraction, and synthesis of academic research on a defined topic.

When to use this skill vs. others

Workflow

Phase 1: Scope Definition

Before searching, establish the review boundaries:

1. Research question — What specific question does the review answer? Vague topics produce vague reviews. "What techniques exist for X" is weaker than "How do methods for X compare on metric Y across domains Z?"
2. Inclusion criteria — Define what counts:
   • Date range (e.g., 2020–present)
   • Publication type (peer-reviewed, preprints, both)
   • Domains/categories (e.g., cs.CL, cs.AI)
   • Minimum relevance threshold
3. Exclusion criteria — Define what does not count:
   • Tangentially related work
   • Non-primary sources (blog posts, tutorials) unless explicitly included
   • Duplicate or superseded versions
4. Expected output — What form should the review take? Narrative synthesis, tabular comparison, gap analysis, annotated bibliography, or related-work section?

Present the scope to the user for confirmation before proceeding.

Phase 2: Search & Discovery

Execute searches across available sources. Use multiple queries with varying specificity to avoid single-query blind spots.

Primary source: arXiv (via arxiv-search utility)

uv run --with arxiv python scripts/arxiv_search.py "QUERY" --max-results 30 --sort-by relevance

Vary queries systematically:

• Broad topic query: "retrieval augmented generation"
• Field-scoped query: ti:retrieval AND abs:generation AND cat:cs.CL
• Author-anchored query: au:lewis AND abs:retrieval (when key authors are known)
• Recency query: same terms with --sort-by submitted

Secondary sources (via web search/fetch):

• Semantic Scholar API: https://api.semanticscholar.org/graph/v1/paper/search?query=QUERY&limit=20&fields=title,authors,abstract,year,citationCount,externalIds
• Google Scholar (via web search): site:scholar.google.com QUERY
• Connected Papers (for citation graph exploration): https://www.connectedpapers.com/search?q=QUERY

Snowball strategy:

• Forward snowball: find papers that cite a key paper (Semantic Scholar citations endpoint)
• Backward snowball: follow the references of key papers
• Use citation count as a signal for influence, not quality

Phase 3: Screening & Filtering

For each discovered paper, apply the inclusion/exclusion criteria from Phase 1.

Produce a screening table:

Rules:

• Screen on title + abstract first. Read full paper only for borderline cases.
• When uncertain, include. It is cheaper to drop a paper later than to miss it.
• Track exclusion reasons — they inform the review's limitations section.
• Flag papers that appear in multiple search queries as likely high-relevance.

Phase 4: Data Extraction

For each included paper, extract a structured record:

- id: "2301.07041"
  title: "Paper Title"
  authors: ["Author One", "Author Two"]
  year: 2023
  venue: "NeurIPS 2023"
  research_question: "How does X affect Y?"
  methodology: "Controlled experiment with N=1000"
  key_findings:
    - "Finding 1 with quantitative result"
    - "Finding 2 with effect size"
  limitations: "Single-domain evaluation"
  relevance_to_rq: "Directly compares methods A and B on metric Y"
  citation_count: 142

Extraction discipline:

• Record what the paper demonstrates, not what it claims to demonstrate.
• Distinguish empirical findings (data-backed) from interpretive claims (author's framing).
• Note methodology details that enable cross-paper comparison (datasets, metrics, baselines).
• If the paper is available via pdf_url, read it for extraction. Do not extract from abstracts alone for included papers.

Phase 5: Synthesis

Transform extracted records into structured analysis. The synthesis method depends on the output format requested in Phase 1.

Thematic synthesis — Group papers by theme, approach, or finding:

• Identify recurring themes across papers
• Note where papers agree, disagree, or address different aspects
• Highlight methodological trends (what approaches are gaining/losing traction)

Comparative synthesis — Build comparison tables:

Chronological synthesis — Map the evolution of the field:

• What was the state of knowledge at time T?
• What shifted and why?
• Where is the field heading?

Gap analysis — Identify what is missing:

• Questions raised but not answered by existing work
• Methodological gaps (no one has tried approach X on problem Y)
• Domain gaps (studied in domain A but not B)
• Contradictions between studies that remain unresolved