Audit Reproducibility

Audit Reproducibility

Compare numeric claims in a manuscript (point estimates, standard errors, p-values, counts) against the actual outputs produced by the analysis pipeline. Report PASS / FAIL per claim against the tolerance thresholds defined in .claude/rules/replication-protocol.md.

Core principle: If the paper says ATT = -1.632 (0.584) and the code produces -1.628 (0.591), we verify — numerically — that the difference is within the documented tolerance. No more "looks close enough" eyeballing.

Two directions, not one. Vertically, each claim is checked against the output that produced it. Horizontally, it is checked against every other artifact that displays the same number — the supplement table, the slide deck, the poster. The vertical check passes contentedly while a deck quotes last month's value; only the horizontal one catches that. Declared displays live in the passport's appears_in list — see replication-protocol.md → The horizontal check.

When to use

• Before submission. Catches the "I updated the analysis but forgot to update Table 2" bug.
• Before presenting or teaching from the same numbers. Catches the deck, the poster, or the supplement that was never regenerated after the last rerun.
• Before releasing a replication package. Verifies the code actually reproduces the paper.
• After a major revision. Ensures the paper still matches the latest code.
• Quality-gate in /commit. Pair with a pre-commit invocation on manuscript + analysis changes.

Inputs

• $0 — path to the manuscript (.tex, .qmd, .md, .pdf). Required.
• $1 — path to the outputs directory. Defaults to output/, where every language's pipeline writes (R, Stata via /stata-replication, Python). Recognised alternatives: _targets/objects/ (R targets workflows), any directory the user-specified outputs live in. If output/ does not exist but a pre-v2.6 scripts/<lang>/_outputs/ does, use that and say so in the report.

Workflow

Phase 0: Pre-flight

1. Read replication-protocol.md for the tolerance thresholds currently in effect.
2. Verify the outputs directory exists and is non-empty. If empty or stale (older than the manuscript), prompt the user to re-run their pipeline (e.g., Rscript scripts/R/00_run_all.R) before auditing.
3. Ensure a sessionInfo.txt or equivalent environment capture exists in the outputs dir.

Phase 1: Extract claims from the manuscript

Parse the manuscript for numeric claims. Patterns to match:

• Point-estimate + SE: ATT = -1.632 (0.584), $\beta = 0.342$ (0.091), hat{\tau} = 1.28** with starred significance
• Table cells: & -1.632$^{***}$ & 0.584 & in LaTeX table environments
• Counts: our sample of 2,847 firms, $N = 2{,}847$
• Summary stats: mean = 0.423, SD = 0.087
• P-values: p < 0.01, $p = 0.003$

Record each claim as a tuple:

{
  claim_id: "Table2_col3_ATT",
  location: "Table 2, Column 3, row 'Treatment'",
  kind: "point_estimate" | "standard_error" | "p_value" | "count" | "percentage",
  reported_value: -1.632,
  uncertainty: 0.584,              # only for point estimates
  significance_stars: 3,            # 0-3 or None
  raw_context: "the ATT estimate of -1.632 (0.584) indicates..."
}

Write the extracted claims to quality_reports/reproducibility_claims_[manuscript-name].json so the user can review the extraction before audit.

Phase 2: Extract results from outputs

Scan $1 for corresponding values. Priority order:

1. .rds files — readRDS(path)$coef[["treatment"]] style lookups. Can use Rscript -e "saveRDS(summary(readRDS(...)), '/tmp/audit.rds')" to extract.
2. .tex tables — parse LaTeX table cells directly; match on column headers + row labels.
3. .csv summary files — pandas/readr parse, key-value lookup.
4. .out / .log files (Stata, regress output) — regex extraction.
5. .json — direct key lookup.

Record each extracted result:

{
  source: "output/results.rds",
  lookup_key: "fit_main$coefficients['treated']",
  value: -1.628,
  uncertainty: 0.591,
  p_value: 0.005
}

Phase 2b: Collect every declared display (passport mode)

For each passport claim, read its appears_in list and pull the value as displayed at each entry:

{
  claim_id: "C3",
  displays: [
    { path: "manuscript.tex", locator: "Table 1, Col 2", display_precision: 3, shown: 0.342 },
    { path: "Slides/Lecture04_Results.tex", locator: "frame 'Main result'", display_precision: 2, shown: 0.34 }
  ]
}

Rules for this pass:

• The primary location: is one of the displays, not a separate thing — if appears_in repeats it, that is one display, not two.
• A declared display that cannot be resolved — missing file, or no value at the locator — is recorded as shown: NOT_FOUND. It resolves to FAIL in Phase 4c. Never skip it: a locator that has quietly stopped matching is exactly where a stale number hides.
• A claim with no appears_in list is horizontally unchecked, not horizontally clean. Report it that way (see Phase 5) rather than silently passing it.

Phase 3: Match claims to results

Use fuzzy heuristics when exact labels don't match:

• Name similarity ("treatment effect" ~ "ATT" ~ "treated")
• Magnitude similarity (if two candidates have values within 10% of the reported, prefer the one with closer SE)
• Context hints from the claim's raw_context field (table number, row label, description)

For every claim, produce a match candidate with a confidence score. Claims below 0.7 confidence get flagged as "UNMATCHED — manual review needed" rather than silently passing.

Phase 4: Tolerance check

For each matched claim, apply the thresholds from replication-protocol.md:

Respect any tolerance overrides the user has written into their replication-protocol.md fork (they may loosen for MC noise or tighten for administrative data).

Phase 4b: Disposition — PASS / FAIL / EXPLAINED / UNMATCHED

A tolerance check resolves to one of four dispositions:

• PASS — within tolerance.
• FAIL — outside tolerance, with no defensible alternative recorded. Blocks (exit 1).
• EXPLAINED — outside tolerance, but the author has recorded a concrete, named alternative specification that accounts for the gap (see the downgrade rule). Surfaced in the report and carried into a response-to-referees; does not block.
• UNMATCHED — no computed counterpart found (Phase 3 confidence < 0.7). Never auto-downgradable.

A mismatch is not automatically a failure. In applied work the most common out-of-tolerance result is a defensible alternative spec, not a bug — reghdfe vs feols clustering df, a different bandwidth-selection rule, a different MC seed/reps, or display rounding. The skill's job is to stage the disagreement for a human auditor, not to pronounce the code right and the paper wrong. (The df-adjustment note in "Stata-specific notes" below is the canonical example of a named alternative.)

The manuscript is not the oracle. When the computed value disagrees with the manuscript, do not presume the code is correct and the paper stale — nor the reverse. A refactor may have broken a previously-correct table (the on-disk output is the buggy one), or the paper may carry an old number. The computed value is a challenger, not ground truth. Report a mismatch as "one of {paper, code} must change — isolate which," never "revert the code to match the paper." This prevents the trap of reverting a genuine bug-fix just to make the paper 'reproduce.'

Downgrade rule: FAIL → EXPLAINED

A FAIL may be downgraded to EXPLAINED only when a specific named alternative is recorded for that exact claim — in the passport entry's notes: field (passport mode) or the audit report's author-note column (default mode). Example of a valid note:

"reghdfe vs feols clustering-df adjustment; under the reghdfe small-sample correction the published value is −1.19, within rounding of the script's −1.187. CODE-CORRECTED pending."

The author is the auditor: the skill stages the two-sided comparison (reported value and computed value, both shown); the human writes the one-line named alternative; the skill records it and thereafter respects it. Tag the resolution PAPER-CORRECTED, CODE-CORRECTED, or DEFENSIBLE-ALTERNATIVE.

Hard floor — never downgradable to EXPLAINED:

• A blank note, "unclear", "looks fine", or any note that does not name a concrete alternative spec.
• An UNMATCHED claim (no computed counterpart to compare against).
• A flat numerical contradiction with no alternative offered.

(Citation/existence claims are out of scope here — /verify-claims owns those, and applies the same named-alternative softening on its side.)

Repeated EXPLAINED is a signal (two-strikes)

Reuse the two-strikes rule from review-paper --adversarial and summary-parity.md: if the same claim is downgraded to EXPLAINED in two consecutive audits without ever being corrected to PASS (the author keeps invoking the alternative but never updates paper or code), stop treating it as quietly resolved. Surface it prominently in Phase 5 — "this contested number has been EXPLAINED twice but never corrected" — so a standing disagreement can't hide behind a recorded note indefinitely. In passport mode, detect this by comparing the current status/notes against the prior audit's.

Phase 4c: Horizontal check — the displays must agree with each other

Phase 4 compared the claim to the code. This phase compares the claim's displays to each other, pairwise over the displays collected in Phase 2b:

1. Round both sides to the coarser precision. Two displays agree when they are equal after both are rounded to the smaller of their two display_precision values. 0.34 in a deck against 0.342 in the paper agrees at 2 decimals; 0.29 against 0.342 does not.
2. No tolerance applies. The tolerance: block governs the vertical comparison, where two measurements are compared. Two displays of one number are two copies — after the rounding in step 1 they either match or they do not.
3. Dispositions, in the vocabulary of Phase 4b:
   • PASS — every pair agrees.
   • FAIL — any pair disagrees, or any display is NOT_FOUND. Never downgradable to EXPLAINED: a named alternative spec explains why code and paper differ and has nothing to say about why two copies of one number differ. If a display genuinely shows a different specification, it is a different claim and belongs in its own passport entry.
   • UNCHECKED — the claim declares no appears_in list. Reported, non-blocking; the fix is to declare the displays.
4. Report which side moved, not which side is right. Name the artifact whose value differs from the rest, and when it was last touched; the author decides whether the deck is stale or the paper is. Do not hand-edit one display to match its sibling — re-derive both from the output the claim points at (replication-protocol.md → Anti-patterns).

Phase 5: Report

Write quality_reports/reproducibility_audit_[manuscript-name].md:

# Reproducibility Audit: [Manuscript Title]

**Date:** [YYYY-MM-DD]
**Manuscript:** [path]
**Outputs directory:** [path]
**Tolerance source:** .claude/rules/replication-protocol.md

## Summary

| Status | Count |
|---|---|
| PASS (vertical and horizontal) | N |
| FAIL (diff > tolerance, no named alternative) | M |
| EXPLAINED (out of tolerance, named alternative recorded) | E |
| UNMATCHED (manual review) | K |
| HORIZONTAL DRIFT (declared displays disagree, or a display not found) | H |
| HORIZONTALLY UNCHECKED (claim declares no `appears_in`) | U |
| **Overall verdict** | **PASS / FAIL** (FAIL iff M > 0 or H > 0; EXPLAINED does not fail the audit) |

## PASS (all within tolerance)
| Claim | Reported | Computed | Diff | Tolerance |
|---|---|---|---|---|
| Table2_col3_ATT | -1.632 (0.584) | -1.628 (0.591) | 0.004 / 0.007 | 0.01 / 0.05 |

## FAIL (outside tolerance — BLOCKER)
| Claim | Reported | Computed | Diff | Tolerance | Location in paper | Author note (name a concrete alternative to downgrade → EXPLAINED) |
|---|---|---|---|---|---|---|

## EXPLAINED (out of tolerance; defensible named alternative recorded — non-blocking, carry into response-to-referees)
| Claim | Reported | Computed | Named alternative (why the gap is defensible) | Resolution |
|---|---|---|---|---|
| Table3_col2_ATT | -1.187 | -1.19 | reghdfe vs feols clustering-df adjustment | DEFENSIBLE-ALTERNATIVE |

## UNMATCHED (manual review)
| Claim | Raw context | Candidate sources |
|---|---|---|

## HORIZONTAL DRIFT (one number, two values — BLOCKER)
| Claim | Display A | Display B | Compared at | Values | Which side moved |
|---|---|---|---|---|---|
| C3 | manuscript.tex — Table 1, Col 2 | Slides/Lecture04_Results.tex — frame 'Main result' | 2 decimals | 0.34 vs 0.29 | deck older than the claim's output_file |

## HORIZONTALLY UNCHECKED (no `appears_in` declared)
| Claim | Primary display | Other artifacts to declare |
|---|---|---|

## Environment
[sessionInfo excerpt]

## Next steps
1. Resolve each FAIL row — either correct the manuscript, rerun the analysis, or (if the gap is a defensible alternative spec) record a concrete named alternative to downgrade it to EXPLAINED.
2. Resolve each HORIZONTAL DRIFT row — regenerate the lagging display from the claim's output, never by retyping the other display's value. A `NOT_FOUND` display is fixed by correcting the `locator`, not by dropping the entry.
3. Review UNMATCHED rows — add explicit lookup keys or widen the search scope.
4. Declare `appears_in` for the HORIZONTALLY UNCHECKED rows — every artifact a reader sees the number in.
5. Review EXPLAINED rows before submission — each should map to a sentence in the response-to-referees.
6. After zero FAILs and zero HORIZONTAL DRIFT (EXPLAINED rows allowed), the paper is replication-ready.