Git History Bug Audit

Audit: Temporal Walk

This is a long-running CLI tool. It walks every commit affecting the caffeine module from project inception to HEAD, asking Claude per commit to flag/resolve/modify a forward-tracked issue database. Issues that survive to HEAD are verified against current code and emitted as detail.dev-format findings.

The default walk is one of several variants (see "Variant walks" below). The variants are not run automatically by the default walk — so they are easy to forget. The orchestrator (run.py) exists to make that impossible.

When invoked interactively

When the user runs /audit-temporal-walk, do NOT silently start the default walk. Instead, present the full battery so nothing is forgotten, then launch the orchestrator:

1. Run python3 .claude/skills/audit-temporal-walk/run.py --list and show the variants with AskUserQuestion (multi-select; default to all). This menu is the reminder — the user consciously picks the battery each time.
2. Create one tracked task per selected variant so progress is visible.
3. Launch run.py --variants <chosen> (or --all) under nohup/tmux — the battery is multi-hour and must survive the session. Suggest the strongest model and --effort max for a quality-critical run.
4. Point the user at the live logs and the combined findings-ALL.md.

When to run

• Before a major release, as a final-pass audit
• After a long sequence of refactors, to catch half-fixes
• Once per several months as a baseline audit
• Not for routine pre-commit review (use /review-change for that)

How to run

The walker uses the claude CLI's default model (the session's current model) unless --model is passed. For a heavyweight rare-run audit, prefer running it in a session on the strongest model available.

# Walk (long-running; safe to interrupt — resumable):
python3 .claude/skills/audit-temporal-walk/walker.py

# In tmux/nohup for multi-hour reliability (the redirect needs the dir to exist):
mkdir -p .local/audits/$AUDIT_MODEL/audit-temporal-walk-<module>
nohup python3 .claude/skills/audit-temporal-walk/walker.py \
  > .local/audits/$AUDIT_MODEL/audit-temporal-walk-<module>/walk.log 2>&1 &

# Process N commits then stop cleanly (useful for chunked runs):
python3 .claude/skills/audit-temporal-walk/walker.py --max-commits 200

# Disable inner-model tool access (faster, less accurate — see Design notes):
python3 .claude/skills/audit-temporal-walk/walker.py --no-tools

# Inspect state without running:
python3 .claude/skills/audit-temporal-walk/walker.py --summary

# After the walk completes, verify surviving issues against HEAD
# (default --min-confidence=low verifies every survivor):
python3 .claude/skills/audit-temporal-walk/verify.py

# Read the verified findings:
cat .local/audits/$AUDIT_MODEL/audit-temporal-walk-<module>/findings.md

Wall clock depends on model and effort and is dominated by model latency: roughly 8-14 hours on a mid-tier config, and ~24 hours on Opus + max effort (the recommended quality-critical config) for the full caffeine module (~760 commits). Tool-enabled mode (default) adds modest overhead from per-commit Read/Grep round-trips. Resumable from checkpoint after quota exhaustion or interruption.

Variant walks

The default run is a broad bug hunt over caffeine/src/main. The same engine drives several focused variants — each is a separate full (or filtered) walk with its own multi-hour cost, not an addition to the main run. --run-name gives each one a disjoint state-<name>.json / log-<name>/ / worktree-<name>/ so they don't clobber the main walk and can run concurrently. Verify a variant with the matching --run-name (and WALKER_SCOPE for the test walk).

Orchestrated battery (recommended — the don't-forget path)

run.py runs a selected set of variants sequentially (walk + verify each), then aggregates every findings-<name>.md into one findings-ALL.md with a summary table. It is resumable: each variant is checkpointed independently, so re-running picks up where it stopped and skips finished variants cheaply. This is the entry point to prefer — one command runs everything and reports on it.

R=.claude/skills/audit-temporal-walk/run.py
python3 $R --list                       # show the battery
mkdir -p .local/audits/$AUDIT_MODEL/audit-temporal-walk-caffeine
nohup python3 $R --all --effort max \   # whole battery, quality config, in tmux/nohup
  > .local/audits/$AUDIT_MODEL/audit-temporal-walk-caffeine/battery.log 2>&1 &
python3 $R --variants fix-audit,lens-sibling   # a chosen subset
python3 $R --all --report-only          # just rebuild findings-ALL.md

Sequential is deliberate: the variants are independent and --run-name makes them parallel-safe, but running one at a time matches the one-active-script discipline and avoids hammering quota — there is no quality gain from parallelism (sharpness is per-prompt, not per-schedule).

Individual variants (focused one-offs)

To run or resume a single variant directly:

W=.claude/skills/audit-temporal-walk
SCOPE_TEST=caffeine/src/test/java/com/github/benmanes/caffeine/cache/

# (#2) Diff-shape lenses — main scope, one concentrated question each.
#      Run individually; sharpness is the point. Three separate walks.
python3 $W/walker.py --prompt $W/lens-deletion.txt --run-name lens-deletion
python3 $W/walker.py --prompt $W/lens-sibling.txt  --run-name lens-sibling
python3 $W/walker.py --prompt $W/lens-intent.txt   --run-name lens-intent
python3 $W/verify.py --run-name lens-deletion        # etc. per lens

# (#4) Fix-commit walk — only commits whose message looks like a fix
#      (~39% of history). Pass the SAME --grep on every resume.
python3 $W/walker.py --prompt $W/fix-audit.txt --run-name fix-audit \
  --grep 'fix|bug|regression|NPE|race|leak|incorrect|wrong|revert'
python3 $W/verify.py --run-name fix-audit

# (#1) Test-history walk — coverage-regression hunt over the TEST tree.
#      The test scope routes to a disjoint  ...-caffeine-test/  reports dir.
#      Pass WALKER_SCOPE to verify.py too.
WALKER_SCOPE=$SCOPE_TEST python3 $W/walker.py \
  --prompt $W/test-walk.txt --run-name coverage
WALKER_SCOPE=$SCOPE_TEST python3 $W/verify.py --run-name coverage

# (#3) Invariant ledger — carries load-bearing assumptions forward and flags a
#      distant commit that violates one. Violations materialize as issues, so
#      verify/findings work unchanged. Full re-walk (the ledger builds from
#      genesis; it can't be backfilled onto the main run).
python3 $W/walker.py --prompt $W/invariant-ledger.txt --run-name invariants
python3 $W/verify.py --run-name invariants

All variants share the resolved/modified/new finding schema and emit findings-<name>.md. The invariant ledger additionally tracks establish/violate/retire in state-<name>.json (see --summary's "Invariants by status" line); a violation is also written as a normal issue so it flows through verification like any other finding.

What the walker does

For each substantive commit (skipping doc/style/dep-bump only), the walker:

1. Checks out the commit into a managed detached worktree under .local/audits/<model>/audit-temporal-walk-<module>/worktree/
2. Invokes claude -p with cwd=worktree and --tools "Read,Glob,Grep", so the inner model can verify hypotheses against the codebase at that commit's state, not HEAD
3. Shows the commit's diff (scoped to the configured module) and the currently-open tracked issues whose files this commit touches

Claude returns deltas: which open issues this commit resolves, which it modifies (e.g., a contract change makes the issue more dangerous), and any new concerns the commit introduces. Each new concern requires a concrete bug witness — the input or scenario that exposes the failure, expressed strongly enough that a developer could write a failing unit test directly from it.

The pattern catalog and design-priors in per-commit.txt are tuned to caffeine's bug history (operator-order in halving formulas, sibling divergence between sync/async paths, missing lifecycle guards, etc.) and caffeine's documented intentional patterns (lossy buffers, best-effort refresh, async-listener semantics).