Ref Downloader

Ref Downloader — Agent Runbook

Slim entry for agent mode. The full 8-step manual runbook with code snippets for Mode A debug + PUBLISHER_MAP extension procedure lives in references/agent-runbook.md. Human users see ../../README.md.

<SKILL_DIR> = this folder (skills/ref-downloader in the source repo, or wherever the user copied this skill — e.g. ~/.claude/skills/ref-downloader/). Python scripts live in <SKILL_DIR>/scripts/; config files (config.example.toml, config.local.toml) live at <SKILL_DIR>/.

Mode router

This skill handles two flows. Pick before running.

• Mode A — Reference-list download (original use case). User provides ONE paper (DOI or local PDF) and wants "all of its references". Pipeline: extract_refs.py → validate_refs.py → download_refs.py.

• Mode B — Custom batch download. User provides their own batch of papers — DOIs, paper titles, non-DOI identifiers (arXiv / PMID / Semantic Scholar IDs), OR an abstract query ("Smith 在 Google Scholar 上的文章" / "Nature Energy 2023 papers"). The agent resolves whatever was given to DOIs, then runs validate_refs.py → download_refs.py directly. Skip the wrapper — run_ref_downloader.py assumes a parent DOI and will fail.

Both modes share install, config, per-publisher strategies, failure modes, output layout, and the CloakBrowser opt-in backend.

Trigger family

Key disambiguators:

• "refs OF a paper" (Mode A) vs "these papers themselves" (Mode B).
• Identifies a specific paper (B.2) vs describes a set to discover (B.3). If user names a specific paper with enough metadata to uniquely identify it (title + author + year), it's B.2 (lookup exact); if they describe a class of papers ("all Smith's 2024 Nature papers"), it's B.3 (discover then filter).

Mode A — Reference-list download

When to invoke

Trigger phrases:

• "帮我下载 [paper / DOI] 的参考文献" / "批量下载引用文献" / "把这篇论文的所有引用下载下来"
• "Download all refs of [paper / DOI]" / "Batch-download every reference"
• User provides a DOI (10.x/y form) or local PDF path and asks for "all references" / "全部参考文献"

Don't invoke for:

• Downloading one arbitrary PDF (not a reference list) → user wants one paper itself, route to Mode B as a single-item B.1
• Generic web scraping
• Paper search / Zotero import — different tools

Primary entry

python "<SKILL_DIR>/scripts/run_ref_downloader.py" <DOI_OR_PDF_PATH>

The wrapper handles DOI resolution (Zotero → fitz fallback), output-dir layout, sequential 3-stage pipeline (extract_refs.py → validate_refs.py → download_refs.py), and end-of-run cleanup.

Useful flags:

• --yes — non-interactive (CI/batch), overwrite prompts default-yes
• --auto — forwarded to download_refs.py: skip "press Enter" confirm + shorter challenge wait + async retry queue for manual_pending refs (60s delay, single retry, max 3 concurrent). Use for CI / overnight runs; not for sessions where you want to drive captchas yourself.
• --fail-fast — terminate after first actionable unresolved ref (useful in CI to surface real failures fast)
• --output-dir <path> — override default output location
• --config <path> — alternate TOML config (overrides config.local.toml)

Pre-flight checklist (confirm before running)

1. DOI correct? Echo back to user: 即将下载参考文献：DOI=<doi>
2. Edge fully closed? All msedge.exe processes killed (Task Manager check). The script claims the user's persistent Edge profile and needs exclusive access. (Cloak backend skips this.)
3. Config set? First-run users need <SKILL_DIR>/config.local.toml with [crossref].mailto. Missing config → wrapper prints a WARNING but continues with placeholder defaults.
4. Output location agreed? Default for DOI input: <cwd>/<project_name>_refs/. For PDF input: <pdf_dir>/<pdf_stem>_refs/. Override with --output-dir.

Mode B — Custom batch download

Input variability is the point. Don't refuse — route. The agent handles whatever shape the user gave (paste, file, prose, BibTeX, RIS, abstract query) and resolves it to a clean DOI list before handing off to the pipeline.

Step 0 — Canonicalize input (always runs first)

Before any extraction or routing, normalize the input string:

1. Unwrap delimiters around DOIs: strip leading/trailing {}, <>, (), [], and quote marks.
2. Strip URL prefixes: https://doi.org/, http://doi.org/, https://dx.doi.org/, http://dx.doi.org/ → bare DOI.
3. Full-width / Unicode punctuation to ASCII:
   • ／ (U+FF0F) → /
   • ： (U+FF1A) → :
   • ． (U+FF0E) → .
   • Smart quotes "" '' → straight " '
4. Trim trailing punctuation: .,;)}"' (note }).

Apply Step 0 BEFORE the regex pass in B.1 AND before the canonical dedupe compare in step 5 of the main flow.

B.0 — Normalize non-DOI identifiers

For each non-DOI identifier the user gave:

Network preflight: before firing B.0 lookups, do one cheap sanity probe (e.g. HEAD https://api.crossref.org/). If it fails, tell the user "no network — Mode B can't resolve non-DOI identifiers or do discovery; only direct DOI extraction will work" and let them decide to proceed with B.1 only.

Semantic Scholar rate limit: unauthenticated ≤ 1 req/sec. Pace batches; on 429 back off 30s then retry once; on second 429, drop the entry with skipped (ss_rate_limited).

B.1 — DOI extraction

After Step 0 canonicalization:

1. Regex 10\.\d{4,9}/[^\s,;<>"'{}]+ on the canonicalized input (or file contents). The character class explicitly excludes {} so wrappers like BibTeX doi = {10.x/y} don't leak braces.
2. Strip trailing punctuation .,;)}"'.
3. Dedupe via canonical lowercase compare (see flow step 5).
4. Fallback rule: if regex yields DOIs for less than 50% of the apparent entries (e.g. BibTeX with 20 @article entries but only 5 have ASCII DOIs), send the remaining entries to B.2 title lookup rather than silently dropping them.

B.2 — Title → DOI lookup

For each title:

1. Query Crossref: GET https://api.crossref.org/works?query.title=<urlencode>&rows=5&mailto=<config crossref.mailto>
2. Pick top by score. Note: Crossref score is unbounded relevance, NOT 0–100 — absolute thresholds across queries don't compare. Use relative + content rules:
   • If top1.score / top2.score < 1.5 → ambiguous; show user top 3.
   • Always sanity-check matched_title vs input_title by token overlap (or Levenshtein). If overlap < 50%, mark low-confidence regardless of score ratio.
3. If user gave an author or year ("Smith 2024"), cross-check against the candidate's author[].family and issued.date-parts.
4. Each B.2 result enters the confirm table as one of:
   • confidence=high (top1, ratio ≥ 1.5, overlap ≥ 50%, any author/year match consistent) — included by default.
   • confidence=low (any of: ratio < 1.5, overlap < 50%, no author match) — excluded by default; user must explicitly pick.
   • unresolved (0 candidates or all rejected) — dropped with skipped (no_match).

B.3 — Discovery from abstract description

Triggered by queries like "Smith 在 Google Scholar 上的文章", "topic Y top 20", "Nature Energy 2023". Do NOT scrape Google Scholar (anti-bot + ToS). Interpret "Scholar" semantically and use the ladder below.

Tool ladder (try in order, use what's available):

1. Crossref (api.crossref.org/works?query.author= / query.bibliographic= / query.container-title=) — always available, no auth.
2. OpenAlex (api.openalex.org/works?search= or ?filter=author.id:A...) — free, no auth, broader coverage than Crossref author search, returns DOIs directly.
3. Semantic Scholar (api.semanticscholar.org/graph/v1/paper/search?query=...) — better for topic / abstract search. Rate limit ~1 req/sec unauthenticated; pace requests, on 429 back off 30s then drop the query on a second 429.
4. PubMed (bio-research:pubmed MCP) — if biomedical AND the MCP is loaded in the host framework.
5. WebSearch / Tavily / Exa — last-resort if a web-search-* skill is loaded. Highest hallucination risk; agent MUST round-trip every candidate through Crossref or OpenAlex to verify the DOI exists before accepting.

After discovery: present candidates as a numbered list with title + first-author + year + DOI + source (which API found it). Default top 20; ask if user wants more. User strikes out / picks subset → final list locked.

Open-ended-query clarifier: if the agent's discovery would return more than 50 candidates (e.g. user said "Smith 的所有文章" and the author has 200+ publications), confirm scope with user BEFORE returning — "found 200+; you want all of them, top 20 most cited, or filter by year?".

Mode B flow (after sub-flow resolution)

1. Resolve input via Step 0 → B.0 → B.1 → B.2 (leftovers + B.1 fallback) → B.3 (for queries).
2. Consolidate + dedupe. Canonicalize EVERY DOI (lowercase, strip wrappers, no URL prefix, ASCII slash) BEFORE comparing. 10.X/ABC} and 10.x/abc must collapse to one entry.
3. Confirm with user — FULL TABLE, not just first 5:

   找到 N 个唯一 DOI（去重后）。完整列表：
   
     [ 1] doi=10.xxxx/yyy  source=B.1   confidence=high
          title= ...  author= ...  year= ...
     [ 2] doi=10.zzzz/www  source=B.2   confidence=high
          matched_title= ...  (input: "...")
     [ 3] doi=10.aaaa/bbb  source=B.3   confidence=high
          via=Crossref  (query: "...")
     [ 4] doi=10.cccc/ddd  source=B.2   confidence=LOW
          matched_title= ...  (input: "...")  ← excluded; pick to include
     [ 5] (unresolvable)   source=B.0   from: "PMID:99999"
          ← dropped
     ...
   
   开始下载吗？(y=accept all high-confidence / n=cancel /
   include 4 / exclude 1,3 / show <N> / ...)

   Default: download confidence=high rows only. confidence=low excluded unless user explicitly includes. Unresolvables dropped.
4. Propose project name (context-aware: "组会文献" → groupmtg_<date>; "Smith 综述补充" → smith_review_extras; nothing topical → custom_<date>). Ask user confirm.
5. Append vs new — if <OUTPUT_DIR>/<project_name>/refs_raw.json exists, ask append / new / rename (default: ask again on any other input — DO NOT default-append). Append rules:
   • Canonicalize new DOIs (Step 0) BEFORE compare.
   • Drop new DOIs already present in existing entries (case-insensitive canonical-form compare).
   • New ids start at id = max(existing_ids) + 1.
   • Never renumber existing entries — validate_refs.py keys its incremental skip on id, renumbering re-assigns prior verified metadata to the wrong DOI. Only verified rows are skipped; failed/pending rows revalidate on re-run.
6. Build refs_raw.json (heredoc the agent runs):

   import json
   from datetime import datetime
   dois = [...]  # finalized canonical-lowercase list
   start_id = 1  # or max(existing_ids)+1 in append mode
   data = {
     "parent_doi": "",  # empty string for clean report labels;
                        # validate_refs.py reads as raw JSON, null
                        # would also work but "" is preferred.
     "parent_title": f"Custom batch — {user_label}",
     "extracted_at": datetime.now().isoformat(timespec="seconds"),
     "total": len(dois),
     "with_doi": len(dois),
     "without_doi": 0,
     "references": [
       {"id": i, "doi": d,
        "key": "", "unstructured": "",
        "author": "", "year": "", "journal": "",
        "volume": "", "first_page": ""}
       for i, d in enumerate(dois, start=start_id)
     ],
   }
   with open(f"{project_name}/refs_raw.json", "w", encoding="utf-8") as f:
       json.dump(data, f, ensure_ascii=False, indent=2)

   All metadata fields stay empty — validate_refs.py fills them from Crossref per DOI on success. Rows whose DOI Crossref can't resolve become status=failed with empty metadata (not partially enriched).
7. Run pipeline (skip the wrapper):

   cd <OUTPUT_DIR>
   python <SKILL_DIR>/scripts/validate_refs.py <project_name>
   python <SKILL_DIR>/scripts/download_refs.py <project_name> [--auto] [--fail-fast]