Bib Parse

Bibliography Parser — PDF to .bib

LIBRARY-FIRST RULE: ALWAYS check Paperpile membership by DOI (paperpile lookup-by-doi) for each parsed reference BEFORE generating new BibTeX entries. Topic/title search-library is a lossy discovery aid, NOT a membership test — never tag a reference NEW off a title-search miss. Reuse existing library entries instead of constructing from scratch. Enforced in Phase 2.3 per shared/reference-resolution.md § Membership Check.

Goal: given a PDF file, extract all cited references and produce a clean, validated .bib file.

When to Use

• You have a PDF (paper, report, thesis) and need a .bib file for its references
• Importing references from a paper that doesn't have a companion .bib file
• Reconstructing a bibliography from a document with embedded \begin{thebibliography} or numbered references
• Converting a reference list from any format into BibTeX

When NOT to Use

• You already have a .bib file and just need to validate it — use the configured bibliography validator
• Finding new references on a topic — use an installed literature workflow or scholarly search
• The PDF is already in Paperpile — export via paperpile export-bib instead (faster, more accurate)

Input

A single PDF file path. The PDF should contain a bibliography, reference list, or works cited section.

Output

A .bib file, located by context:

• PDF inside a research project (e.g. articles/, paper-*/, project root) — write to the same directory as the input PDF, named references.bib (per project convention). If references.bib already exists, ask before overwriting — offer to merge or use a different name (e.g., extracted-refs.bib).
• Standalone PDF (ad-hoc extraction with no project home — e.g. a client-managed upload) — write to ~/vault/parsed-bibs/<slug>.bib, where <slug> describes the source (e.g. kasberger-algorithmic-cooperation-geb-2026.bib). This is the standing location for one-off parses (established 2026-06-14). Create the folder if absent.

Any staged Paperpile-import entry (Phase 4.1) is written into a .bib under .paperpile-import/ next to the output .bib.

Phase 1: Read

1.1 Read the PDF

Use split-pdf to read the PDF contents. This handles large PDFs by splitting into chunks. Focus on the bibliography / references section — typically the last few pages. Identify:

• Numbered reference lists (e.g., [1] Smith, J. (2020)...)
• APA / Harvard style reference lists
• \bibitem entries (if the PDF was generated from LaTeX)
• Footnote-based citations

1.2 Skeleton confirmation (mandatory for ≥10 references)

Before running any enrichment queries, render the parsed reference list as a numbered table and ask the user to confirm the split is correct. A misparsed reference list (one citation accidentally split across two rows, or two citations merged into one) makes Phase 2 spend 30+ API calls on the wrong entries — expensive in time and rate-limit budget.

Detected N references. Confirm the split before I enrich each one:

| # | parsed reference (verbatim) |
|---|----------------------------|
| 1 | Bebchuk, L. A., Cohen, A., & Hirst, S. (2017). The agency problems of institutional investors... |
| 2 | Smith, J. (2020). Title... |
| ... | ... |

Anything wrong with the split? (Reply "ok" to proceed, or list the row numbers that need merging/splitting.)

Skip the confirmation step if any of these:

• Fewer than 10 references parsed (cheap to fix after the fact)
• The PDF was generated from LaTeX and \bibitem boundaries are unambiguous
• The user explicitly said "just go" / "no confirmation"

When the user flags a bad split, fix it and re-render the skeleton before proceeding. Don't proceed to Phase 2 with unaddressed split issues — that's exactly the failure case this gate prevents.

Phase 2: Enrich

2.1 Extract reference metadata

For each reference found, extract:

Large reference lists (50+ entries): process in batches of ~20. Verify each batch before moving to the next. Report progress: "Processed 20/87 references...". This caps the rate-limit blast radius if a batch fails.

2.2 Verify and enrich

For each extracted reference, verify and enrich using available tools:

1. Search by title via web search to confirm the reference exists and pull missing metadata (DOI, volume, pages).
2. Use the scholarly CLI for multi-source enrichment:
   • scholarly scholarly-search "<title>" --json — search by title across OpenAlex + S2 + Scopus + WoS
   • scholarly scholarly-paper-detail <paper_id> --json — get full metadata including pre-formatted BibTeX (citationStyles), TLDR summary, and open access PDF link. When BibTeX is available, use it as the primary template instead of manually constructing entries — reduces formatting errors.
   • scholarly openalex-lookup-doi <doi> --json — get full metadata if DOI was found.
3. Cross-check extracted metadata against search results: confirm author names, year, fill in missing DOI/volume/pages/publisher.

Important: do NOT silently replace the extracted metadata with search results. If there's a conflict (e.g., different year, different spelling of author name), flag it as a note field in the .bib entry and keep the original from the PDF.

2.3 Check Paperpile library (LIBRARY-FIRST gate)

For each parsed reference, check if it already exists in Paperpile using the resolution order from shared/reference-resolution.md:

1. Check membership by DOI — paperpile lookup-by-doi --doi <DOI> for each parsed reference (verify the DOI itself first if unsure). This is the membership test. Batch ≥6 references via one Bash sub-agent returning a merged {doi: citekey-or-null} map.
2. Only for DOI-less references, fall back to paperpile search-library with first-author surname + title keywords (limit ≥10) — a discovery aid, not a membership test.
3. Match → ALREADY IN PAPERPILE: reuse the returned Paperpile citekey, skip staging in Phase 4.1. A confirmed DOI miss (or, for DOI-less refs, an author+title miss) → NEW — never tag NEW off a search-library miss alone. Foundational classics (Simon, March, Levinthal, Cyert & March, …) are the most common false NEW; check them by DOI too.

Status summary:

Graceful degradation: if the paperpile CLI is unavailable, skip with a warning and treat all references as NEW.

Phase 3: Construct

3.1 Generate citation keys

Better BibTeX conventions:

• Format: {Author}{Year}-{xx} — e.g., Smith2020-xy
• First author's last name, title-cased
• Four-digit year
• Two-character suffix (Better BibTeX style)
• If merging into an existing .bib, match the key format already in use
• For references already in Paperpile (Phase 2.3), copy the Paperpile citekey into the entry (do not keep the generated key) and backfill the DOI from Paperpile, but keep the richer local metadata (type, journal, volume, pages, full authors — Paperpile's export-bib is @misc/thin). Add note = {key reconciled to Paperpile <key>}. Full rule + first-author-match guard: shared/reference-resolution.md § Key Reconciliation.