Diagnose Failed Run

Diagnose a Failed Retort Run

Overview

A run failing the mechanical gate (test_coverage=0 → all metrics zeroed → status=failed) says only that the tests didn't run/pass under the scorer — not that the model produced bad code. Historically most "failures" in this repo were harness/measurement artifacts, not model defects. This skill is the judgment layer on top of retort diagnose: it establishes the real cause with direct evidence before anyone attributes, reports, or gates on it.

It exists because we repeatedly drew wrong conclusions by inferring from a signature ("flaky", "concurrency", a wrong env var, a wrong tool-permission fix) that end-to-end checks later overturned. The cost of guessing here is hours and bad decisions.

Cardinal rules

1. Ground-truth before concluding. Run the tests yourself, inspect the workspace, read the agent logs. Never attribute a cause from the gate verdict, a duration, or a single log line alone.
2. Label every claim [DIRECT] (observed: command output, file state, a reproduced result) vs [HYPOTHESIS] (inferred, not yet tested). Don't ship a hypothesis as a cause. If a cause is unconfirmed, say so and name the experiment that would confirm it.
3. Read the DB and the agent logs. The recorded scores/session show which tool or step failed; the agent stderr shows why (the permission type, the error). You usually need both — e.g. the session said "read rejected", but only --print-logs stderr revealed the real permission was external_directory.
4. Verify any fix end-to-end (through the harness), not just CLI/unit. A fix that passes a CLI probe can still fail in the runner (a wrong env var passed both).

Inputs in each runs/<cell>/repN/

If _agent_*.log are absent, the run predates log capture; reconstruct from the agent's own session db (e.g. opencode's OPENCODE_DB sibling) and add capture before re-running.

Procedure

1. Mechanical pass first: retort diagnose --experiment-dir <dir> labels each failed run TOOLING (recovers with retort rescore --only-failed) vs GENUINE. Start here; this skill handles the cases it can't auto-resolve.
2. Ground-truth each failure (do NOT skip even for GENUINE):
   • Does code exist, and where? A "no code" verdict is often just a package subdir (app/, src/) the eye/glob missed.
   • Run the tests yourself in a clean copy with deps installed (pytest / cargo test / go test ./... / mvn test / …). Passing tests ⇒ it's a scorer false-fail, not a model miss.
   • Read _agent_stdout.log for the failing tool call + status; read _agent_stderr.log for the reason.
3. Classify the cause (see taxonomy). State it [DIRECT] with the evidence.
4. Isolate intermittent/ambiguous causes with a controlled A/B — change exactly one variable (e.g. shared vs isolated db; baseline vs with-fix batch) and compare. Log what you dropped; don't conclude from n=1.
5. Reproduce-with-full-capture before declaring an intermittent cause — run it enough times to catch one with _agent_*.log captured, then read the events at the abort.
6. Verify the fix end-to-end through the harness, then re-score / re-run.