Citation Audit

Citation Audit Skill

Purpose

Verify every citation in a manuscript against its actual source. LLMs hallucinate citations, invent arXiv IDs, misattribute findings, and confuse authors. This skill catches all of that by fetching and reading each cited work.

Why This Exists

LLMs are unreliable with citations in three distinct ways:

1. Ghost papers — The paper does not exist. Title, authors, or venue are fabricated.
2. Wrong metadata — The paper exists but the bib entry has the wrong arXiv ID, wrong authors, wrong year, or wrong venue.
3. Inverted claims — The paper exists and the bib is correct, but the manuscript mischaracterizes what the paper says.

All three are invisible to structural audits (cross-reference checks, compilation tests). They require reading the actual cited work.

Inputs

• The manuscript .tex file(s)
• The .bib file
• Web access (to fetch papers from arXiv, conference sites, URLs)

Execution

Phase 1: Extract citation contexts

For each \citep{}, \citet{}, \cite{} in the manuscript:

1. Record the bib key
2. Record the surrounding sentence or paragraph (the claim context)
3. Classify the claim type:
   • FACTUAL: "X et al. found Y" / "X et al. measured Y"
   • METHODOLOGICAL: "We follow X" / "We use the benchmark from X"
   • POSITIONAL: "Unlike X, we..." / "X does not measure..."
   • PARENTHETICAL: "(X, 2024)" — no specific claim, just a reference
4. For FACTUAL and POSITIONAL claims, extract the specific assertion the manuscript makes about the cited work

Phase 2: Verify bib entry metadata

For each bib entry, verify against the actual source:

For arXiv papers (eprint field present):

1. Fetch https://arxiv.org/abs/{eprint_id}
2. Compare: title, authors, year
3. If the fetched paper has a DIFFERENT title/authors than the bib entry, this is a WRONG ID or GHOST PAPER

For conference/journal papers (booktitle or journal field):

1. Search for the paper by title + author on the web
2. Verify: venue, year, author list
3. If the paper cannot be found at the stated venue, flag as UNVERIFIABLE or GHOST PAPER

For web resources (howpublished with URL):

1. Fetch the URL
2. Verify it loads and the content matches the described resource
3. If the URL is dead or redirects to unrelated content, flag as DEAD LINK

For each entry, check:

• Paper exists (reachable via arXiv, DOI, URL, or web search)
• Title matches (exact or near-exact)
• Authors match (at least first author correct)
• Year matches
• Venue matches (if applicable)
• Entry type appropriate (@inproceedings for conferences, @article for journals, @misc for preprints/blogs)

Phase 3: Verify claim accuracy

For each FACTUAL or POSITIONAL claim:

1. Read the cited paper (abstract + relevant sections at minimum)
2. Compare the manuscript's claim against what the paper actually says
3. Classify:
   • ACCURATE — The claim faithfully represents the cited work
   • INACCURATE — The claim mischaracterizes the cited work
   • INVERTED — The claim says the opposite of what the paper found
   • OVERCLAIMED — The claim is stronger than what the paper supports
   • UNDERCLAIMED — The cited work supports a stronger claim than stated
   • UNVERIFIABLE — Cannot access the paper to verify

For INACCURATE and INVERTED findings, provide:

• What the manuscript claims
• What the cited paper actually says
• The specific section/page of the cited paper that contradicts the claim
• A suggested correction

Phase 4: Check for missing citations

Scan the manuscript for:

1. Claims that cite no source but should (empirical claims without attribution)
2. Tools, benchmarks, or datasets mentioned by name without citation
3. Methods described as "standard" or "well-known" that have a canonical citation

Output Format

Per-citation report

### [bib_key] — [VERDICT]

**Bib entry:** [title] by [authors] ([year])
**Actual paper:** [actual title] by [actual authors] ([actual year])
**Metadata match:** title [✓/✗] | authors [✓/✗] | year [✓/✗] | venue [✓/✗]

**Claim in manuscript (line N):** "[exact text]"
**What the paper actually says:** "[summary of actual finding]"
**Claim accuracy:** [ACCURATE / INACCURATE / INVERTED / OVERCLAIMED / UNDERCLAIMED]

**Fix required:** [description of what needs to change, or "None"]

Summary table

| Bib Key | Exists | Metadata | Claim | Verdict |
|---------|--------|----------|-------|---------|
| key1    | ✓      | ✓        | ✓     | PASS    |
| key2    | ✓      | ✗        | ✗     | FAIL    |
| key3    | ✗      | —        | —     | GHOST   |

Verdict categories

• PASS — Paper exists, metadata correct, claims accurate
• METADATA — Paper exists, bib entry has errors (wrong ID, wrong authors, wrong year)
• CLAIM — Paper exists, metadata correct, but manuscript mischaracterizes it
• GHOST — Paper does not exist as described
• DEAD — URL/link is broken
• UNVERIFIABLE — Cannot access the paper to verify

Severity

• CRITICAL: GHOST papers, INVERTED claims
• HIGH: Wrong arXiv IDs, wrong authors, INACCURATE claims
• MEDIUM: Wrong year, wrong venue, OVERCLAIMED
• LOW: Missing citations, incomplete bib entries, UNDERCLAIMED

Important notes

• NEVER trust your own knowledge of papers. ALWAYS fetch and verify. Your training data contains hallucinated citations. The only way to verify is to read the actual source.
• For arXiv papers, always fetch the abstract page to confirm the paper exists and matches.
• For conference papers, search DBLP, ACM DL, or the conference site.
• WebFetch and WebSearch are your primary tools. Do not skip verification because a citation "looks right."
• Blog posts and documentation URLs change. Always check that the URL still works and points to the described content.
• When a bib entry has both an eprint (arXiv ID) and a booktitle (venue), verify both independently.