Grounded Research Lit

Grounded Research Literature

Use a structured grounded note to perform targeted literature / web research.

This skill is not a generic survey skill. It is a grounding-conditioned research stage.

Its responsibilities are:

1. read a grounded note
2. adapt to the grounded note's schema
3. generate focused search queries
4. save them to a JSON file
5. run research using either:
   • external API backend, or
   • Cursor-native search / browse fallback
6. enforce link opening when required
7. preserve readable source material for opened literature items
8. build structured paper-level notes from opened source material
9. write an initial literature report (lit_initial.md) from opened-source evidence
10. optionally download opened research literature as an auxiliary artifact
11. refine paper notes from successfully downloaded PDFs
12. write the final refined literature report (lit.md) for downstream stages

Pipeline Constants

Load constants from:

• config/research_pipeline.env

Use:

source config/research_pipeline.env

Constants

Backend selection rule

if SEARCH_BACKEND == "external":
    use external API backend only
elif SEARCH_BACKEND == "cursor":
    use Cursor-native research only
else:  # auto
    if BIGMODEL_SEARCH_API_KEY exists:
        use external API backend
    else:
        use Cursor-native research

Important:

• Cursor-native fallback is orchestration logic in this skill
• Do not try to call Cursor-native search through Python
• web_search_reader.py is only for the external backend
• prepare_opened_paper_notes.py is used for both backends
• download_opened_literature.py is a local downloader / backfill helper
• refine_notes_from_downloaded_pdfs.py is the PDF refinement helper used after downloads complete

When to Use

Use this skill when:

• a grounded note already exists
• you want focused literature / web follow-up
• you want search results with evidence quality
• you want a structured literature result rather than raw URLs

Do not use this skill when:

• the input is a raw transcript or raw document
• the upstream grounding stage has not yet been completed
• the goal is a final polished memo (use grounded-summary after this)
• the goal is a broad field survey unrelated to the grounded input

Input

How to get ground_id

Read ground_id.txt from the grounding bundle to get the stable pipeline identifier:

data/grounded_notes/<ground_id>/ground_id.txt

Do NOT generate a new ground_id. All downstream directories reuse the same ground_id.

Grounded note input

Usually:

• data/grounded_notes/<ground_id>/grounded.md

Optional input

• queries_confirmed_path: path to a user-confirmed queries JSON file (e.g., data/lit_inputs/<ground_id>/queries_confirmed.json). When provided, use the queries from this file directly instead of generating new ones.

This skill should adapt to the grounded note's actual schema rather than forcing one universal section layout.

Primary query seed

When queries_confirmed_path is provided, use the queries from that file as the authoritative source — do not re-generate.

When queries_confirmed_path is NOT provided, fall back to:

• When Search Keywords is present in the grounded note, use it as the primary query seed.
• However, do not rely on keywords alone. Always interpret them together with the grounded note's topic, current findings, unresolved issues, risks, and suggested next steps.

Schema-aware reading rules

Map the grounded note into research-relevant semantic slots.

Archive Grounding

• topic ← Archive Overview
• materials ← Included Materials
• processed_items ← Successfully Processed Child Items
• cross_material_signals ← Key Signals Across Materials
• failures ← Skipped / Unsupported / Failed Items
• next_steps ← Suggested Next Steps
• keywords ← Search Keywords

Document Grounding

• topic ← Main Topic / Purpose
• known_points ← Main Points
• conclusions ← Key Findings / Claims
• risks ← Constraints / Risks
• evidence ← Important Non-Textual Elements
• open_questions ← Unresolved Issues
• next_steps ← Suggested Next Steps
• keywords ← Search Keywords

Meeting Grounding

• topic ← Meeting Topic
• known_points ← Main Discussion Points
• conclusions ← Key Conclusions
• risks ← Constraints / Risks
• open_questions ← Disagreements or Unresolved Issues
• next_steps ← Suggested Next Steps
• keywords ← Search Keywords

PPTX Grounding

• topic ← Main Topic / Purpose
• structure ← Deck Structure / Narrative Flow
• known_points ← Main Points
• evidence ← Important Evidence and Assets
• notes_signals ← Speaker Notes Signals
• risks ← Gaps / Risks / Ambiguities
• next_steps ← Suggested Next Steps
• keywords ← Search Keywords

Table Grounding

• topic ← Main Topic / Purpose
• schema_signals ← Main Fields
• known_points ← Key Signals
• anomalies ← Anomalies / Outliers
• cautious_conclusions ← Possible Supported Conclusions
• risks ← Risks / Data Quality Issues
• next_checks ← Suggested Next Checks
• keywords ← Search Keywords

Query-generation emphasis

Generate query groups with grounding-type-aware emphasis.

Problem / background queries

Use for:

• task/background understanding
• domain context
• benchmark/dataset context
• problem framing

Method / solution queries

Use for:

• methods
• baselines
• solution directions
• implementation ideas

Domain / constraint queries

Use for:

• risks
• constraints
• ambiguities
• unresolved issues
• failure modes
• data quality concerns

Required queries.json format

queries.json must use plain string arrays for each query group.

Preferred structure:

{
  "ground_id": "<ground_id>",
  "problem_queries": [
    "query string 1",
    "query string 2"
  ],
  "method_queries": [
    "query string 1",
    "query string 2"
  ],
  "constraint_queries": [
    "query string 1",
    "query string 2"
  ]
}

Rules:

• keep problem_queries, method_queries, and constraint_queries as arrays of strings only
• do not wrap each query as an object
• do not include per-query fields such as query, emphasis, keywords, or rationale
• concise, directly searchable query strings are preferred over verbose natural-language instructions
• top-level metadata such as ground_id is allowed, but query entries themselves must remain plain strings

Output Files

Intermediate files

Under:

• data/lit_inputs/<ground_id>/

Write:

• data/lit_inputs/<ground_id>/queries.json
• data/lit_inputs/<ground_id>/search_results.json

Opened source artifacts

If any literature item is successfully opened and readable, preserve source material under:

• data/lit_inputs/<ground_id>/opened_sources/

Structured paper notes

After search/open, generate structured paper-note artifacts under:

• data/lit_inputs/<ground_id>/opened_paper_notes.jsonl
• data/lit_inputs/<ground_id>/opened_paper_notes/

These artifacts are internal to this skill and do not change downstream contracts. They exist to make the literature report deeper, less snippet-driven, and more evidence-backed.

Initial literature output (internal)

Under:

• data/lit_inputs/<ground_id>/lit_initial.md

This is an internal research-stage intermediate artifact. It may be used to checkpoint the first pass based on opened-page evidence, but it must not be treated as the final output for downstream stages.

Final literature output

Under:

• data/lit_results/<ground_id>/lit.md

This is the final research-stage output that downstream stages should consume. It must be written only after the PDF refinement pass completes or no eligible downloaded PDFs are available.

Optional downloads

If DOWNLOAD_OPENED_LITERATURE=true:

• downloaded files go to:
  • data/lit_downloads/<ground_id>/
• manifest file:
  • data/lit_downloads/<ground_id>/manifest.json

Skill-local scripts

• .cursor/skills/grounded-research-lit/scripts/web_search_reader.py
• .cursor/skills/grounded-research-lit/scripts/prepare_opened_paper_notes.py
• .cursor/skills/grounded-research-lit/scripts/download_opened_literature.py
• .cursor/skills/grounded-research-lit/scripts/refine_notes_from_downloaded_pdfs.py

Required behavior

When invoked, this skill must:

1. infer ground_id from the grounded note path if possible

2. read the grounded note

3. detect or infer the grounding type from the note structure

4. load constants from config/research_pipeline.env

5. load confirmed queries if queries_confirmed_path is provided; otherwise extract and generate:

   • main topic
   • open questions / unresolved issues
   • constraints / risks / ambiguities
   • likely method directions
   • search keywords

6. if queries_confirmed_path is provided: read queries from that file and copy them as queries.json — do NOT regenerate or ask the user again

7. if queries_confirmed_path is NOT provided: use Search Keywords as the primary query seed, generate query groups:

   • problem/background queries
   • method/solution queries
   • domain/constraint queries
   • write queries.json using plain string arrays for each query group; do not use per-query objects with fields such as query, emphasis, keywords, or rationale

8. run research:

   • external backend → use web_search_reader.py
   • Cursor-native backend → use Cursor-native research / browse

9. preserve opened readable sources under opened_sources/

10. generate opened_paper_notes.jsonl and per-paper note markdown files using prepare_opened_paper_notes.py

11. for backend=cursor, explicitly verify the opened-paper count using the exact opened-count verification rule before writing data/lit_inputs/<ground_id>/lit_initial.md 11.5. Paper Opened Confirmation (Agent Inline Check) — this step must be completed before writing data/lit_inputs/<ground_id>/lit_initial.md:

    • Iterate through each paper in opened_paper_notes.jsonl and verify, for each one:
      • whether a corresponding opened source file exists under opened_sources/
      • whether that source file contains sufficient readable content (≥ 500 chars of readable body = sufficient; < 100 chars = severely insufficient)
      • whether the paper note's analysis is substantive (not hollow, no obvious hallucination risk)
    • If a paper is found to be missing a source file or to have severely insufficient source content:
      • must re-search using the paper title
      • attempt to open a new readable page and save it to opened_sources/
      • regenerate the corresponding paper note
    • If a paper still cannot obtain sufficient source content after re-searching, it should be removed from the candidate list (not referenced in lit_initial.md)
    • This confirmation is performed by the agent inline, without relying on external scripts
    • after confirmation is complete, proceed to write lit_initial.md

12. write data/lit_inputs/<ground_id>/lit_initial.md from opened-page evidence and paper notes

13. if DOWNLOAD_OPENED_LITERATURE=true:

• external backend: launch background/auxiliary download when possible
• Cursor-native backend: run download_opened_literature.py after the opened-source artifacts and initial note artifacts exist
• in both cases, treat download as an auxiliary stage between data/lit_inputs/<ground_id>/lit_initial.md and final lit.md

14. wait until the download manifest reaches a terminal state (completed or no_eligible_items) before finalizing the research stage
15. run refine_notes_from_downloaded_pdfs.py to enrich notes from successfully downloaded PDFs
16. write the final lit.md from the refined notes

Backend-specific sufficiency rule

External API backend

• preserve the existing broader external search/open flow
• analyze as many successfully opened relevant literature items as the run actually obtains
• if downloads are enabled, download as many opened relevant literature items as are actually available
• do not impose a fixed minimum successful-download count on the external backend

Cursor-native backend

• before writing data/lit_inputs/<ground_id>/lit_initial.md, you must search and open at least MIN_OPENED_PAPERS unique relevant literature items
• at least $MIN_OPENED_PAPERS opened items is a hard requirement, not a best-effort suggestion
• do not treat snippet-only candidates as opened papers for this count
• the minimum-$MIN_OPENED_PAPERS rule must be enforced using the exact opened-count verification rule, not by rough browsing impressions alone
• if fewer than $MIN_OPENED_PAPERS unique relevant literature items satisfy the exact opened-count verification rule, the Cursor-native research stage is not complete and must continue searching / opening more items
• if DOWNLOAD_OPENED_LITERATURE=true, the Cursor-native backend must also successfully download at least MIN_OPENED_PAPERS unique relevant literature items before the research stage can be treated as complete, unless the run explicitly reports why this target could not be reached after continued search/open/download attempts
• the opened count is an entry sufficiency rule for writing, while the download count is a completion sufficiency rule when download is enabled
• recent-paper enforcement runs after the opened-count check: the agent must ensure at least MIN_RECENT_PAPERS papers published in the last 2 years have been opened before writing lit_initial.md; if insufficient, supplement searches with year-filtered queries must be performed

Critical workflow rule: data/lit_inputs/<ground_id>/lit_initial.md first, lit.md last

The research stage must follow this post-search order:

1. obtain search results
2. open readable literature items
3. preserve opened readable source content
4. build structured paper notes from opened readable content
5. write data/lit_inputs/<ground_id>/lit_initial.md based primarily on those paper notes
6. run or complete literature download as an auxiliary stage
7. refine notes using downloaded PDFs when available
8. write the final lit.md

The goal is:

• fast first-pass analysis from opened readable sources
• deeper final analysis from downloaded PDFs when available
• stable file-flow semantics for downstream stages

data/lit_inputs/<ground_id>/lit_initial.md is internal to this research stage. Downstream stages should consume only the final lit.md.

Paper Opened Confirmation: Inline Check Before Writing lit_initial.md

Before writing data/lit_inputs/<ground_id>/lit_initial.md, the following confirmation process must be completed for all candidate papers. This step is executed by the agent inline, without relying on external scripts.

Subject

Each paper in opened_paper_notes.jsonl.

Verification Rules

For each paper, verify:

Handling Flow

1. Read opened_paper_notes.jsonl to get the paper list
2. Check that each paper's opened_source_path exists under opened_sources/
3. Read each corresponding source file and measure readable body length
4. For any paper with issues:
   • Re-search using the paper title
   • Open a new readable page and save it to opened_sources/
   • Re-run prepare_opened_paper_notes.py
   • If still no sufficient source after re-searching, remove the paper from the candidate list
5. After all confirmations are complete, write lit_initial.md

Judgment Standards

• Sufficient: readable body ≥ 500 chars, analysis ≥ 20 lines → ready to write
• Thin: readable content 100–500 chars, analysis < 20 lines → attempt to expand, or keep with noted limitations
• Severely insufficient: readable content < 100 chars, or no source file at all → must re-search; if irrecoverable, remove
• Suspicious: analysis content clearly misaligned with the source file → must re-search to confirm

The purpose of this step is to eliminate hallucination risk before writing, not to fix it after writing.

Cursor-native artifact production rule

When backend=cursor, do not treat browsing alone as research completion. The Cursor-native branch must produce explicit local artifacts before the writing stage.

The required sequence is:

1. search and identify candidate literature items
2. continue opening and reading literature items until at least $MIN_OPENED_PAPERS unique relevant items satisfy the exact opened-count verification rule
3. for each successfully opened and readable literature item:
   • save one readable source file under data/lit_inputs/<ground_id>/opened_sources/
   • record the opened status in search_results.json
4. after opened-source preservation, run prepare_opened_paper_notes.py
5. confirm that:
   • opened_paper_notes.jsonl exists
   • opened_paper_notes/ contains generated per-paper note files
6. explicitly verify that the opened-paper count is at least $MIN_OPENED_PAPERS using the exact opened-count verification rule

6.5 (Cursor-native only) recent papers enforcement:

Load constant from `config/research_pipeline.env`:
  MIN_RECENT_PAPERS (value depends on `research_mode`)

This check is always active for the Cursor-native backend.
a. From all opened papers in `search_results.json` where:
       - opened == true
       - open_status == "success"
       - is_research_literature == true
     extract the publication year from:
       - the `publish_date` field, or
       - the opened source file content (look for a year in the visible
         metadata or near the title/abstract)
  b. Count papers with year >= (current_year - 1). In 2026, the cutoff is 2025.
  c. If count < MIN_RECENT_PAPERS:
       i.   Extract Search Keywords from the grounded note.
       ii.  Generate supplementary queries by appending year filters for
            the last 2 years to keywords:
              e.g. "[keyword] 2025..2026"
                   "[keyword] 2025 site:arxiv.org"
                   "[keyword] 2026 site:arxiv.org"
       iii. Run WebSearch with these supplementary queries.
       iv.  Use WebFetch to open newly found papers.
       v.   Save each newly opened source under `opened_sources/`.
       vi.  Update `search_results.json` with new opened items
            (append to the existing queries array).
       vii. Re-run `prepare_opened_paper_notes.py`.
       viii.Re-verify: opened count >= `$MIN_OPENED_PAPERS` AND recent count >= MIN_RECENT_PAPERS.
            If still insufficient after supplement, apply the fallback
            policy below only after exhausting the escalation steps in
            section d.

  d. (Escalation — must be attempted before any fallback is taken)
     If supplement search in step c still leaves recent count < MIN_RECENT_PAPERS,
     the agent must escalate before falling back:

       - Broaden the keyword strategy: try shorter core terms, remove
         modifiers, try synonyms (e.g. from "long-context transformer"
         to "transformer", from "RAG" to "retrieval augmented generation")
       - Search arxiv's cs.* categories directly with year filters
       - Search Google Scholar with year filters if available
       - Try searching for "state of the art [keyword] 2025" and
         "latest research on [keyword]" as standalone queries
       - Try the arxiv cs.CV / cs.CL / cs.AI new submission pages
       - Record every escalation attempt in the run report:
           what was tried, how many results were found, how many
           were opened, why each attempt fell short

     After completing all escalation steps, re-count recent papers.
     Only if recent count is still < MIN_RECENT_PAPERS AND the agent can
     document a concrete reason why recent papers do not exist for this topic
     (e.g. the research area literally did not exist until this year),
     then the fallback in section e may be taken.

  e. (Strict fallback — only as last resort, requires explicit justification)
     Fallback means: the minimum is reduced to 1 recent paper if at least
     1 can be found; if even 1 cannot be found after all escalation steps,
     the research stage reports the complete absence of recent literature
     as a finding in the run report and proceeds.

     ⚠️ ANTI-GAMING RULE: The fallback is NOT a permission to skip the
     recent-paper requirement. It exists only for genuinely novel topics
     where research from the last 2 years is physically absent.
     The agent must NOT use fallback as a convenience escape hatch.
     Any fallback taken without documented escalation attempts in section d
     constitutes a skill violation.

     Before taking fallback, the agent must explicitly state in the run
     report:
       - Which escalation steps in section d were attempted
       - How many results each yielded
       - The concrete reason recent papers could not be found
       - Whether the fallback to 1 recent paper or zero-recent finding
         was taken, and why

7. only then write data/lit_inputs/<ground_id>/lit_initial.md
8. if downloads are enabled, run download_opened_literature.py
9. wait for the manifest terminal state
10. run refine_notes_from_downloaded_pdfs.py
11. only then finalize lit.md

⚠️ REQUIRED TOOLS FOR CURSOR-NATIVE BACKEND — READ CAREFULLY

When backend=cursor, you MUST use these tools for search and open:

✅ USE THESE TOOLS:

• WebSearch — Use this tool to search for literature. Provide a targeted search_term and an explanation of what you're looking for.
• WebFetch — Use this tool to fetch and read individual paper/abstract pages from URLs returned by WebSearch.

❌ DO NOT USE:

• MCP browser tools (ListMcpResources, FetchMcpResource, browser_* tools) — these are for different purposes, NOT for literature research
• Python search scripts (web_search_reader.py) — these are for the external API backend only, not Cursor-native
• CallMcpTool for browser/IDE-related operations during research

Required workflow with WebSearch + WebFetch:

1. For each query in queries.json, call WebSearch with the query string
2. From the search results, identify relevant papers (arxiv, conference papers, etc.)
3. For each promising result, call WebFetch to read the paper page
4. Save the fetched content as markdown files under data/lit_inputs/<ground_id>/opened_sources/
5. Create data/lit_inputs/<ground_id>/search_results.json recording each item's opened status
6. After all opens, run prepare_opened_paper_notes.py

⚠️ CRITICAL REMINDER:

The skill file says "Cursor-native fallback is orchestration logic in this skill" and "Do not try to call Cursor-native search through Python". This means you must use the WebSearch and WebFetch tools directly — there is NO Python script that does the searching for you in Cursor-native mode.

Exact opened-count verification rule for Cursor-native backend

For backend=cursor, the minimum opened-paper requirement must be verified explicitly from local artifacts before data/lit_inputs/<ground_id>/lit_initial.md is written.

A literature item counts toward the required opened count only if all of the following are true:

• is_research_literature == true
• opened == true
• open_status == "success"
• opened_source_path is present in search_results.json
• the file referenced by opened_source_path actually exists on disk under data/lit_inputs/<ground_id>/opened_sources/

The following do not count as opened papers for the minimum-$MIN_OPENED_PAPERS rule:

• snippet-only candidates
• items with opened=false
• items whose open_status is not success
• items missing opened_source_path
• items whose opened_source_path is recorded but the file does not actually exist on disk

For backend=cursor, data/lit_inputs/<ground_id>/lit_initial.md must not be written until the explicitly verified opened count is at least $MIN_OPENED_PAPERS.

Required opened-source preservation details for Cursor-native backend

For each successfully opened and readable literature item in the Cursor-native backend, save one markdown file under:

• data/lit_inputs/<ground_id>/opened_sources/

Each saved opened-source file should include, when available:

• title
• source URL
• access date
• venue / year / author metadata visible on the page
• readable abstract / summary text
• readable body text or the most informative page-level content actually available

The corresponding item in search_results.json should record, when available:

• title
• url
• opened
• open_status
• opened_source_path
• is_research_literature

Required local script execution for Cursor-native backend

After search_results.json and opened_sources/ are populated, run:

python .cursor/skills/grounded-research-lit/scripts/prepare_opened_paper_notes.py \
  --search-results data/lit_inputs/<ground_id>/search_results.json \
  --output data/lit_inputs/<ground_id>/opened_paper_notes.jsonl \
  --notes-dir data/lit_inputs/<ground_id>/opened_paper_notes

If DOWNLOAD_OPENED_LITERATURE=true, then run:

python .cursor/skills/grounded-research-lit/scripts/download_opened_literature.py \
  --search-results data/lit_inputs/<ground_id>/search_results.json \
  --output-dir data/lit_downloads/<ground_id> \
  --ground-id <ground_id> \
  --manifest-path data/lit_downloads/<ground_id>/manifest.json \
  --wait \
  --wait-timeout-sec 1800

⚠️ --ground-id is required. --manifest-path is optional (defaults to {output-dir}/manifest.json) but must be provided when also using --wait so that the wait loop checks the correct file. --wait makes the script poll for a terminal-valid manifest state: status=completed OR status=no_eligible_items with downloaded_count>0. A manifest with status=no_eligible_items and downloaded_count=0 means the download failed before producing results (e.g. _iter_items returned 0 items) — the script will keep polling and timeout rather than treat the broken state as terminal.

After the download stage reaches a terminal manifest state, run:

python .cursor/skills/grounded-research-lit/scripts/refine_notes_from_downloaded_pdfs.py \
  --search-results data/lit_inputs/<ground_id>/search_results.json \
  --notes-path data/lit_inputs/<ground_id>/opened_paper_notes.jsonl \
  --notes-dir data/lit_inputs/<ground_id>/opened_paper_notes \
  --manifest-path data/lit_downloads/<ground_id>/manifest.json \
  --wait \
  --wait-timeout-sec 1800

⚠️ Argument names differ from the old --notes-jsonl / --downloads-dir / --output-jsonl names. Use --notes-path, --notes-dir, --manifest-path exactly as shown. --wait makes refine poll the manifest until it reaches a terminal-valid state before reading it, avoiding the case where refine runs before download has finished. The poll loop also correctly ignores no_eligible_items manifests with downloaded_count=0 — it keeps waiting for the next download attempt.

Failure to produce these artifacts means the Cursor-native research stage is incomplete.

Critical writing rule: lit.md must be paper-note-driven

Neither data/lit_inputs/<ground_id>/lit_initial.md nor lit.md may be driven mainly by snippets.

They must use, in order of priority:

1. opened_paper_notes.jsonl
2. files under opened_paper_notes/
3. files under opened_sources/
4. only then search snippets for candidates that could not be opened

After PDF refinement, the final lit.md should prefer:

1. refined note fields derived from downloaded PDFs
2. original opened-page note fields for papers without downloaded PDFs
3. snippet-only candidates as a separate last section