Consistency Audit

REVIEWER_MODEL          = gpt-5.6-sol                  # different family from executor (Claude)
REVIEWER_REASONING      = xhigh                    # always; effort never lowers reviewer quality
REVIEWER_SANDBOX        = read-only                # detect-only; never mutate the paper
REVIEWER_CWD            = <paper-dir>              # so it can read claims.json + sources directly
THREAD_POLICY           = fresh mcp__codex__codex per run; NEVER mcp__codex__codex-reply
TAXONOMY_VERSION        = 0.5                      # references/hack-pattern-taxonomy.md
DETERMINISTIC_FINDINGS  = consistency-audit.deterministic.findings.json   # Step 1 (tool)
SEMANTIC_FINDINGS       = consistency-audit.findings.json                 # Step 4 (validated)
TRACE_POLICY            = forensic (never silently dropped)
TRACE_DIR               = .aris/traces/consistency-audit/<YYYY-MM-DD>_run<NN>/

ROOT=$(git rev-parse --show-toplevel 2>/dev/null || pwd)
# $ARGUMENTS is a paper-dir OR a claims.json path:
LEDGER="$ARGUMENTS"; [ -d "$LEDGER" ] && LEDGER="$LEDGER/claims.json"
# Only the NO-ARGUMENT case defaults to the CWD ledger. An EXPLICIT argument that
# resolves to a missing claims.json must NOT silently fall back to $(pwd) — that
# could audit the wrong paper; let the NO_LEDGER check below fire instead.
[ -z "$ARGUMENTS" ] && LEDGER="$(pwd)/claims.json"
python3 - "$LEDGER" <<'PY'
import json, sys, os
p = sys.argv[1]
if not os.path.isfile(p):
    sys.exit("NO_LEDGER: claims.json not found. Run /evidence-ledger FIRST "
             "(it writes artifact_manifest.json + claims.json).")
d = json.load(open(p, encoding="utf-8"))
print("LEDGER      =", os.path.abspath(p))
print("PAPER_DIR   =", os.path.dirname(os.path.abspath(p)) or ".")
print("PAPER_ID    =", d.get("paper_id", "?"))
print("RUN_LEVEL_L =", d.get("observability_level", 0))
print("CLAIMS      =", len(d.get("claims", [])))
PY

ROOT=$(git rev-parse --show-toplevel 2>/dev/null || pwd)
LEDGER="<abs path to claims.json from Step 0>"
python3 "$ROOT/tools/check_numeric_consistency.py" \
    --ledger "$LEDGER" \
    --out "$(dirname "$LEDGER")/consistency-audit.deterministic.findings.json"
# statistical self-consistency (GRIM / GRIMMER / statcheck) — also pure arithmetic,
# disjoint GRIM###/VAR###/STAT### ids; the orchestrator globs *.findings.json so this
# file is picked up automatically. z-tests are stdlib; t/F/chi2/r use optional scipy.
python3 "$ROOT/tools/check_stat_consistency.py" \
    --ledger "$LEDGER" \
    --out "$(dirname "$LEDGER")/consistency-audit.stat.findings.json"

{
  "finding_id": "NUM001",
  "skill": "consistency-audit",
  "pattern_id": "HP-DELTA-ERROR",
  "title": "Stated improvement contradicts its operands",
  "description": "Text states a 16.7% relative change, but 73.1->78 is 6.7% relative (+4.9 absolute points).",
  "severity": "major",
  "observability_level_required": 0,
  "evidence": [{
    "claim_id": "C003",
    "span": "FooNet reaches 78.0\\% accuracy, improving from a 73.1\\% baseline to 78.0\\% accuracy, a 16.7\\% relative improvement.",
    "location": {"file": ".../delta_inflate.tex", "line": 9, "section": "abstract"},
    "artifact_hash": "a0a6b6…e45804"
  }],
  "verdict_local": "fail",
  "reviewer": {"deterministic": true},
  "false_positive_risk": "low",
  "recommended_reviewer_action": "Ask the authors to reconcile the stated delta with the reported operands; verify relative-vs-absolute convention."
}

LEDGER="<abs path to claims.json from Step 0>"; D="$(dirname "$LEDGER")"
TBASE="$D/.aris/traces/consistency-audit/$(date +%F)"
NN=1; while [ -e "${TBASE}_run$(printf '%02d' "$NN")" ]; do NN=$((NN+1)); done
TRACE_DIR="${TBASE}_run$(printf '%02d' "$NN")"; mkdir -p "$TRACE_DIR"
echo "TRACE_DIR = $TRACE_DIR"   # carry this absolute path into Steps 2 and 5

mcp__codex__codex:
  model: gpt-5.6-sol
  config: {"model_reasoning_effort": "xhigh"}
  sandbox: read-only
  cwd: <absolute PAPER_DIR from Step 0>
  prompt: |
    You are an integrity-forensics reviewer auditing a research paper for INTERNAL
    self-consistency only — the paper against ITSELF. You have NO external ground
    truth and you do NOT judge whether results are "real": you find places where the
    paper contradicts itself, or where the method DESCRIBED differs from the method
    EVALUATED.

    INPUTS (in your working directory, read them directly):
      - claims.json  — the evidence ledger: the authoritative, span-anchored list of
        every checkable claim. This is the ONLY structure you reason over for
        navigation. Each claim has {claim_id, type, text_span (VERBATIM source text),
        location, value?}.
      - the source files the ledger references — you MAY re-open them to confirm a
        span is real, but you may NOT introduce a claim that is not in the ledger.
    RUN OBSERVABILITY LEVEL L = <L from Step 0>.

    HARD RULES (a finding that breaks any of these is worthless):
    1. ANCHOR. Every finding above severity "info" MUST carry >=1 evidence entry
       {claim_id, span}, where claim_id EXISTS in claims.json and span is a VERBATIM
       substring of THAT claim's text_span — copied character-for-character INCLUDING
       LaTeX escapes like \% and \, (do NOT unescape, normalize, or paraphrase). The
       anchor check is a whitespace-normalized substring match (`span in claim`); an
       unescaped or reworded span will fail and be demoted to info. If you cannot
       quote such a span, keep the finding at "info" or drop it.
    2. DISCREPANCY, NOT ACCUSATION. description and recommended_reviewer_action
       describe what a human should CHECK or ASK. Never write "reject", "fabricated",
       or "the authors faked X".
    3. OBSERVABILITY. Set observability_level_required = the LOWEST tier at which the
       discrepancy is DECIDABLE. A purely textual contradiction (text vs text/table
       inside the paper) = 0. Anything that needs the CODE or RESULT FILES to confirm
       = 2 (a PDF-only run reports it as needing L2 — that is correct, not a loss).
    4. HONEST FP RISK. Set false_positive_risk truthfully. Legit "best config"
       labels, deliberately-labeled ablations, standard rounding, deterministic
       metrics, and honestly-labeled pilots are COMMON false positives — say so.
    5. HAND OFF EXTERNAL CLAIMS. For "first / SOTA / state-of-the-art / beats all
       prior work" that internal text cannot settle: do NOT rule. Set
       verdict_local = "needs_external_check" and requires_external_check = true.
    6. pattern_id MUST be one of the HP-* ids listed in the checklist.

    CHECKLIST (run every item; one finding per concrete discrepancy):
     1. NUMBER COHERENCE — the same metric+setting reported with DIFFERENT values
        across abstract / intro / body / table / appendix, or a headline value more
        favorable than its own table row.            [HP-NUM-INFLATE, HP-APPENDIX-CONTRA]
        severity: critical if the contradicted value is the headline claim; else
        major. FP: a genuinely different, NAMED config; standard rounding. level 0.
     2. DELTA ARITHMETIC — "improves by X%": recompute (new-old)/old AND absolute
        points; flag if X disagrees beyond rounding, or relative/absolute are
        conflated to inflate.                                       [HP-DELTA-ERROR]
        severity: major; critical if the corrected delta deflates a "large/
        significant" framing. FP: abs-vs-rel stated explicitly; rounding. level 0.
        (A deterministic pass already caught the single-sentence cases; you catch
        deltas spread ACROSS sentences/tables a regex cannot pair.)
     3. AGGREGATION DRIFT — text says "mean over N seeds" but the number matches the
        BEST seed, or N in text != N in the table; variance hidden.   [HP-AGG-DRIFT]
        severity: major; critical if variance is hidden and the gap is within
        plausible seed spread. FP: best AND mean both reported; labeled pilot. level 0.
     4. DENOMINATOR / POPULATION DRIFT — two tables average over DIFFERENT subsets
        ("all tasks" vs "tasks where the method applies") but the body treats them as
        one number.                                                  [HP-DENOM-DRIFT]
        severity: major. FP: subsets clearly delimited and not conflated. level 0.
     5. METRIC DIRECTION / UNIT — % vs absolute points mixed; a lower-is-better metric
        described as higher-is-better; an improvement stated in the wrong direction.
                                                              [HP-UNIT-DIR-MISMATCH]
        severity: minor, or major if it FLIPS a comparison. FP: an unconventional but
        internally consistent, clearly-defined unit. level 0.
     6. METHOD IDENTITY DRIFT — the body describes method A, but the experimental-
        setup text describes A-lite / A+oracle / extra training data / a different
        backbone than claimed.                                      [HP-METHOD-DRIFT]
        severity: critical (this breaks the central claim). FP: a deliberately-labeled
        ablation varying the method. level 0 for a text-vs-text contradiction; set
        level 2 if confirming what was ACTUALLY run needs the code/results.
     7. ABLATION ATTRIBUTION — a gain credited to component X, but no ablation
        ISOLATES X (X is always bundled with Y).                 [HP-ABLATION-ATTRIB]
        severity: major. FP: an isolating ablation exists elsewhere; the component is
        theoretically inseparable and the paper says so. level 0.
     8. CAPTION vs CONTENT — a figure/table caption describes a method, axis, or N
        that the content does not show.                          [HP-CAPTION-MISMATCH]
        severity: minor, unless the caption is the ONLY place a key result is stated.
        FP: a loose multi-panel summary. level 0 (text) / 1 (source).
     9. SCOPE vs EVIDENCE — "comprehensive / extensive / robust / general / across the
        board" on a thin actual scope (few datasets, N=1-2, one domain).
                                                                  [HP-SCOPE-INFLATE]
        severity: minor->major. FP: scope genuinely broad; qualifiers present.
        level 0. (If the language is specifically "SOTA/first", ALSO set
        needs_external_check and hand to baseline/citation forensics.)
    10. THEOREM SCOPE DRIFT — abstract/title advertise generality; the formal result
        holds only under strong/unstated assumptions.       [HP-THEOREM-SCOPE-DRIFT]
        severity: major. FP: assumptions stated up front AND acknowledged in the
        abstract. level 0. (Flag the internal scope mismatch only; route the headline
        framing to adversarial-case-builder.)
    11. ARGUMENT-CHAIN COHERENCE — the chain the paper claims to walk — motivation
        (intro) -> mechanism (method) -> what the experiments MEASURE — has a
        SUBSTANTIVE missing link: the problem motivated is not the one the method
        addresses, or the method's claimed mechanism does not predict what the
        experiments test. This is about the CONTENT of the chain, not its prose —
        mere stylistic disjointedness / "前言不搭后语" reading / filler wording is the
        surface tell HP-NARRATIVE-ARC-BREAK (presentation-signals, capped minor), NOT this.
                                                            [HP-ARGUMENT-CHAIN-BREAK]
        severity: major; critical if the headline contribution rests on the broken
        link. FP: a dense-but-valid argument the reader must work through; a modular
        paper with explicit cross-references; non-native phrasing. level 0. If the
        missing link is a THEORETICAL relation that must be DERIVED (not narrated), do
        NOT adjudicate the math here — route it to proof-derivation-forensics (family G).
    12. CAUSAL / EVIDENCE LEAP — a relation is CONCLUDED that no experiment in the
        paper actually tests: "A and B correlate, therefore equivalent / causal"; the
        paper studies C but concludes "D affects C" with no experiment that VARIES D;
        equivalence or causation asserted from a correlation or a single setting.
                                                            [HP-CAUSAL-EVIDENCE-LEAP]
        severity: major; critical if it is the central claim. FP: the supporting
        experiment exists elsewhere in the paper (cite it); a deliberately
        observational study that does not claim causation. level 0 (the unsupported
        leap is visible in the text); set level 2 only if confirming NO run varies D
        needs the code/results. If the relation is instead established THEORETICALLY
        (a proof/derivation is given), it is NOT an evidence leap — it is a proof
        obligation: route to proof-derivation-forensics (family G).
    13. NAME / ACRONYM DRIFT — the SAME load-bearing component, method, or module is named
        or expanded INCOMPATIBLY across the paper: one acronym defined with two different
        expansions, or one component referred to by two incompatible names/acronyms
        (abstract ↔ method ↔ experiments), so a reader cannot tell they are the same thing.
                                                            [HP-ACRONYM-DRIFT]
        severity: minor; major if the drift makes a central method/result ambiguous. FP: a
        standard acronym reused for different things; author-declared overloading; a local
        scoped abbreviation; verbose names / bold-spam ALONE (that is the surface tell
        HP-JARGON-STUFF, presentation-signals, NOT this). Require the two spans to name the
        SAME object and be genuinely incompatible. level 0.
    BONUS. SUSPICIOUS REGULARITY — numbers across rows related by a too-clean
        arithmetic pattern (a constant additive/multiplicative offset, implausibly
        smooth monotonicity, identical decimals across unrelated settings).
                                                          [HP-SUSPICIOUS-REGULARITY]
        At L0/L1 this is ALWAYS severity "minor", false_positive_risk "high",
        observability_level_required 2 (so a PDF-only run reports it as needing L2). It is a
        PROMPT TO CHECK, never a "fabricated" grade; it rises to major only at L2
        against the real files. FP: deterministic metrics, small integer scores,
        rounding coincidence, a real linear trend.

    OUTPUT: a single JSON array, and NOTHING ELSE (no prose, no code fence). Each
    element conforms to schemas/finding.schema.json:
      {
        "finding_id": "F001",
        "skill": "consistency-audit",
        "pattern_id": "HP-...",
        "title": "short, neutral",
        "description": "the discrepancy, in reviewer-facing language",
        "severity": "critical|major|minor|info",
        "observability_level_required": 0,
        "evidence": [{"claim_id": "C0xx", "span": "verbatim substring of that claim",
                      "location": {"file": "...", "section": "..."}}],
        "verdict_local": "fail|warn|clean|needs_external_check",
        "requires_external_check": false,
        "false_positive_risk": "low|medium|high",
        "alternative_explanation_checked": "<criticals ONLY — the benign readings you explicitly ruled out: rounding/display precision, unit or metric convention, statistical reporting convention, scope difference. Anchoring proves the text exists, NOT that your interpretation is right; a critical without this field demotes to major>",
        "numeric_basis": [{"claim_id": "C012", "role": "old"}, {"claim_id": "C013", "role": "new"}, {"claim_id": "C014", "role": "stated"}],
        "recommended_reviewer_action": "what to CHECK or ASK — never 'reject'"
      }
    If you find no discrepancy for an item, simply emit nothing for it. An empty
    array [] is a valid, honest result.

    NUMERIC BASIS (criticals of HP-DELTA-ERROR / HP-NUM-INFLATE / HP-APPENDIX-CONTRA
    ONLY — omit the field everywhere else): freeze the exact numeric claims your
    accusation computes over, one role each — old/new/stated for HP-DELTA-ERROR,
    fine/coarse for the other two. The set must be precisely the numeric claims in
    your evidence (no extras, no omissions); the adjudicator re-derives every value
    from the ledger and runs a deterministic counter-check. Missing or invalid basis
    on such a critical demotes it to major.