Replication Audit

Replication Audit

Examine which findings in a literature have been replicated, failed to replicate, or never been tested. Flag papers that build on non-replicated foundations.

Don't build your dissertation on a foundation made of sand.

Output Path

Per rules/review-artefact-routing.md (auto-loads in research projects (path-scoped to paper-*/ and paper/)):

• Source slug: replication-audit
• Write reports to: reviews/<scope>/replication-audit/<YYYY-MM-DD-HHMM>.md inside the project, where <scope> is the paper slug (e.g., paper-jtp) for paper-level reviews or _project for project-level reviews. Path is relative to the research project root, not the Task-Management repo.
• Never at project root (./CRITIC-REPORT.md-style filenames are forbidden — pre-rule layout).
• Idempotency: if the run timestamp already exists, append a same-run descriptor ({timestamp}-revision.md, {timestamp}-r2.md) — never overwrite the same <YYYY-MM-DD-HHMM> path.
• Index update: if reviews/INDEX.md exists, write a one-line entry under "Latest per source" pointing at the new file. Otherwise review-recap will rebuild the index next time it runs.
• Infrastructure repos (Task-Management, atlas-workspace, etc.): this section does not apply — the path-scoped rule won't load there.

When to Use

• Before committing to a theoretical framework — is the evidence solid?
• Writing a literature review — need to distinguish robust from fragile findings
• Preparing a replication study — need to know what's already been tested
• Evaluating a paper for peer review — are its cited foundations sound?

When NOT to Use

• Your own results — use the referee2-reviewer agent
• Methodological comparison — use method-audit
• Finding papers — use an installed scholarly-search workflow first

Input

A .bib file, PDF directory, topic, or list of key findings to audit. Works best with a focused set of influential papers whose findings underpin a line of research.

Workflow

Phase 1: Findings Inventory

From the corpus (assembled as in other corpus skills), extract the key empirical findings — not paper summaries, but specific claims with effect sizes where available:

Focus on findings that other papers depend on — the claims that, if wrong, would undermine subsequent work.

Phase 2: Replication Search

For each key finding, search for replication attempts:

1. Forward citations — use scholarly scholarly-citations on the original paper's DOI
2. Keyword search — search for "replication" + key terms from the finding via scholarly scholarly-search
3. Replication databases — search for the paper on:
   • ReplicationWiki (via web search)
   • Many Labs projects
   • Replication registries in the relevant field
4. Author self-replication — check if the original authors replicated in a new sample

Dispatch rule. If ≥5 key findings need auditing, batch findings into groups of 4–5 per sub-agent. Each sub-agent runs steps 1–4 for its batch (scholarly-citations, scholarly-search, web searches) and writes replication evidence to /tmp/replication-audit-<n>.json. Main context merges and proceeds to Phase 3 classification. For <5 findings, sequential searches in main context are fine. See _shared/cli-dispatch-policy.md.

For each replication found, record:

• Result: Successful / Failed / Partial / Conceptual (similar but different design)
• Sample: How does it compare to the original (size, population)?
• Method: Exact replication or conceptual?
• Effect size: Same direction? Same magnitude?
• Published where? (Replication failures in top journals carry more weight)