Scientific Literature Search

Scientific Literature Search

Overview

Scientific literature search is the foundation of evidence-based research. A well-executed search maximizes recall (finding all relevant papers) while maintaining precision (avoiding irrelevant results). This guide provides a systematic approach that combines database-specific query strategies, AI-assisted synthesis, and direct content extraction, organized into a three-tiered framework that scales from targeted lookups to comprehensive landscape reviews.

Key Concepts

The PICO Framework

For clinical and biomedical questions, structure queries using the PICO framework:

• P (Population): Who are you studying? (e.g., "Diabetes Mellitus"[MeSH])
• I (Intervention): What treatment or exposure? (e.g., "Metformin"[MeSH])
• C (Comparison): What is the alternative? (e.g., placebo, standard care)
• O (Outcome): What result are you measuring? (e.g., "Cardiovascular Diseases"[MeSH])

PICO queries can be combined with publication type filters to target specific evidence levels:

"Diabetes Mellitus"[MeSH] AND "Metformin"[MeSH] AND "Cardiovascular Diseases"[MeSH] AND ("clinical trial"[Publication Type] OR "meta-analysis"[Publication Type])

Three-Tiered Search Strategy

Literature search is most effective when approached in tiers of increasing breadth:

Tier 1 -- Database-Specific Searches (Most Reliable)

Query established academic databases (PubMed, arXiv, Google Scholar) for peer-reviewed, indexed content. This is the most reliable tier and should always be the starting point.

• PubMed (via Biopython Bio.Entrez): Primary database for biomedical and life science literature. Supports MeSH controlled vocabulary and advanced field tags.
• arXiv (via the arxiv package): Preprint server for physics, mathematics, computer science, and quantitative biology. Results appear faster than peer-reviewed journals.
• Google Scholar (via the scholarly package): Broadest coverage across all academic disciplines. Note: has aggressive rate limits on automated queries.

Best for: finding specific papers, systematic reviews, clinical evidence, preprints.

Tier 2 -- AI-Assisted Web Search (Comprehensive)

Use the Claude API with the web_search_20250305 server-side tool to synthesize broader context, identify research trends, and surface recent developments not yet indexed in databases. Also use general web search (e.g. via the duckduckgo-search package) for protocols, tutorials, and software documentation.

Best for: understanding the research landscape, complex multi-faceted questions, finding recent developments, identifying key researchers.

Avoid for: specific paper lookups (use Tier 1), citation counts (use Google Scholar), systematic reviews requiring reproducibility, searches where exact query terms must be documented.

Tier 3 -- Direct Content Extraction (Deep Dive)

Extract and analyze full-text content, PDFs, and supplementary materials from identified papers using trafilatura (HTML article extraction), pypdf (PDF text), and the Crossref API (DOI → supplementary file URLs).

Best for: detailed methodology extraction, data retrieval, protocol identification, supplementary data access.

PubMed Field Tags

PubMed supports field-specific searching to improve precision:

Boolean Operators

Boolean operators control how search terms combine:

# AND: All terms must be present -- narrows results
results = query_pubmed("CRISPR AND cancer AND therapy")

# OR: Any term can be present -- broadens results (use for synonyms)
results = query_pubmed("(tumor OR tumour OR neoplasm) AND immunotherapy")

# NOT: Exclude terms -- use sparingly to avoid losing relevant papers
results = query_pubmed("cancer immunotherapy NOT review")

Use parentheses to group OR terms together before combining with AND.

arXiv Subject Categories

arXiv organizes preprints by subject category. Biology-related categories include:

Decision Framework

Use this tree to determine which search tier and database to start with:

What type of question are you answering?
├── Clinical / biomedical question
│   ├── Specific drug or treatment → Tier 1: PubMed with PICO query
│   ├── Disease mechanism → Tier 1: PubMed with MeSH terms
│   └── Clinical trial evidence → Tier 1: PubMed filtered by Publication Type
├── Computational / quantitative methods
│   ├── ML model or algorithm → Tier 1: arXiv (cs.LG, cs.AI)
│   ├── Computational biology method → Tier 1: arXiv (q-bio.*) + PubMed
│   └── Software tool or pipeline → Tier 2: AI-assisted web search
├── Broad research landscape
│   ├── Current state of a field → Tier 2: AI-assisted web search
│   ├── Recent developments (last 6 months) → Tier 2: AI-assisted web search
│   └── Cross-disciplinary question → Tier 1: Google Scholar + Tier 2
├── Specific paper or data
│   ├── Known paper details → Tier 1: any database by title/author/DOI
│   ├── Methodology or protocol → Tier 3: full-text extraction
│   └── Supplementary data → Tier 3: DOI-based supplementary fetch
└── Protocols / reagents
    ├── Lab protocol → Tier 2: web search for protocols.io, etc.
    └── Validated reagents → Tier 2: AI-assisted web search

Best Practices

1. Use controlled vocabulary (MeSH) for PubMed searches: Free-text searches miss papers that use different terminology. MeSH terms map synonyms to a single concept, improving recall without sacrificing precision.

   # Free text misses synonyms
   query_pubmed("heart attack treatment")
   # MeSH captures all synonyms
   query_pubmed('"Myocardial Infarction"[MeSH] AND "Drug Therapy"[MeSH]')

2. Include synonyms and alternative terms with OR: Scientific concepts often have multiple names (e.g., tumor/tumour/neoplasm). Group synonyms with OR inside parentheses to avoid missing relevant papers.

   query_pubmed("(myocardial infarction OR heart attack) AND (treatment OR therapy)")

3. Use phrase searching for multi-word concepts: Quoting exact phrases prevents the search engine from splitting terms and matching them independently.

   query_pubmed('"single cell RNA sequencing" AND methods')

4. Filter by publication type when seeking specific evidence: Clinical trials, systematic reviews, and meta-analyses each answer different questions. Use [Publication Type] to target the evidence level you need.

   query_pubmed("COVID-19 vaccine efficacy AND clinical trial[Publication Type]")

5. Start broad, then narrow iteratively: Begin with core concepts (2-3 terms) and review initial results. Add specificity based on what you find -- more terms, date ranges, field tags, or publication types.

   # Step 1: Broad
   results = query_pubmed("CRISPR base editing iPSC", max_papers=20)
   # Step 2: Add MeSH and specificity
   results = query_pubmed(
       '"CRISPR-Cas Systems"[MeSH] AND "base editing" AND "induced pluripotent stem cells" AND efficiency',
       max_papers=20
   )
   # Step 3: Filter by date
   results = query_pubmed(
       '"CRISPR-Cas Systems"[MeSH] AND "base editing" AND "induced pluripotent stem cells" AND efficiency AND ("2022"[Date - Publication]:"2024"[Date - Publication])',
       max_papers=20
   )

6. Cross-reference multiple databases: No single database covers all literature. Use PubMed for biomedical content, arXiv for computational preprints, and Google Scholar for cross-disciplinary coverage.

7. Assess result quality systematically: Evaluate papers for source reliability (peer-reviewed journal), author credentials, recency, study design appropriateness, sample size adequacy, reproducibility, declared conflicts of interest, and citation count.

Common Pitfalls

1. Overly long and specific queries: Packing too many terms into a single query causes missed results because all terms must match simultaneously.

   • How to avoid: Limit queries to core concepts (3-5 terms). Run separate searches for sub-topics and combine results manually.

   # Too specific -- misses relevant papers
   query_pubmed("CRISPR Cas9 gene editing HEK293T cells 2024 efficiency optimization delivery")
   # Better -- core concepts only
   query_pubmed("CRISPR Cas9 gene editing optimization efficiency")

2. Relying on a single database: PubMed has biomedical focus, arXiv covers preprints, Google Scholar spans disciplines. Using only one database guarantees blind spots.

   • How to avoid: Always search at least two databases. For computational biology, combine PubMed and arXiv. For cross-disciplinary topics, include Google Scholar.

3. Ignoring publication dates: Scientific knowledge evolves rapidly. Foundational papers remain relevant, but methods and clinical evidence may be superseded.

   • How to avoid: Check publication dates in all results. For methods papers, prefer the last 3-5 years. For foundational concepts, older papers are acceptable but verify with recent reviews.

4. Skipping title and abstract review before deep-diving: Not all search results that match keywords are actually relevant. Downloading and reading full texts without screening wastes time.

   • How to avoid: Always screen titles and abstracts first. Only extract full text (Tier 3) for papers that pass screening.

5. Using NOT operators too aggressively: The NOT operator can inadvertently exclude relevant papers that mention the excluded term in a different context.

   • How to avoid: Use NOT sparingly. Prefer adding positive terms to narrow results rather than excluding terms. When you must use NOT, verify that excluded results are genuinely irrelevant.

6. Ignoring Google Scholar rate limits: Google Scholar aggressively rate-limits automated queries, which can block further searches.

   • How to avoid: Use Google Scholar sparingly. Add delays between requests. Prefer PubMed or arXiv for bulk searching and reserve Google Scholar for cross-disciplinary checks.

7. Not documenting the search strategy: For systematic reviews and reproducible research, an undocumented search cannot be verified or reproduced.

   • How to avoid: Record your search terms, databases queried, date ranges, and number of results at each stage. This is essential for systematic reviews and good practice for all searches.