Agentsop Test Fix Loop

Test-Fix Loop · SOP

One-liner: The test result IS the next prompt. Wiring the verifier is 20% of the work; framing its output as a useful feedback message is 80%.

1. 何时激活 (Activation Rules)

Activate this skill when any of the following triggers fire:

• The user says "have the agent fix until tests pass", "run lint and tests automatically", "iterate until green", or invokes aider --auto-test, cline --yes, or an OpenHands-style headless agent.
• The task has a verifiable success command: a non-zero exit code on failure (pytest, ruff, mypy, eslint, tsc, go test, cargo check, npm run build, make check, …).
• You're wrapping a code-editing LLM in a script/CI step and need to decide: when does the agent return?
• The agent just made an edit and the next message in the loop would be "here's what the verifier said".

Do not activate when:

• Success is subjective (writing prose, designing UX). The loop has no feedback signal worth replaying.
• The verifier is slow + interactive (full E2E suite, multi-min builds). Either async-ify the loop, or run a fast subset (pytest -x -k changed) in the loop and gate the slow suite at PR review.
• The gate is human approval, not a machine check — use the HITL skill.

2. 核心心智模型 (Core Mental Model)

2.1 The test result IS the next prompt

The agent's next turn is conditioned almost entirely on the message you inject between edit-N and edit-N+1. That message — formatted from stdout, stderr, exit_code — is the prompt. The framework labels it "tool result" or "verifier output" but mechanically it is a user-role message the LM consumes verbatim.

⇒ Framing the feedback dominates the model choice. A 4000-line raw pytest dump prompts a worse fix than a 30-line "first failing test, traceback, the diff you just applied" digest, regardless of the model behind it.

2.2 Four primitives

+-----------------+   +-----------------+   +-----------------+   +-----------------+
| 1. Verifier     |   | 2. Capture      |   | 3. Format       |   | 4. Iteration    |
|    command      |   |    (stdout +    |   |    feedback     |   |    bound        |
|                 |   |     stderr +    |   |    message      |   |                 |
| - pytest -x     |   |     exit_code)  |   | - first error   |   | - max N tries   |
| - ruff check    |   | - timeout cap   |   | - last K lines  |   | - escalate /    |
| - mypy --strict |   | - byte cap      |   | - drop noise    |   |   commit / skip |
| - eslint .      |   | - kill on hang  |   | - keep colors=0 |   |                 |
+-----------------+   +-----------------+   +-----------------+   +-----------------+

Drop any one of these and the loop fails:

• No verifier → no signal; the agent guesses "done".
• No capture → the model can't read stderr; tracebacks live in stderr.
• No formatting → 25k-token output distracts the model (see Aider's 25k context-drift threshold).
• No iteration bound → infinite loop; the OpenHands SWE-Bench infinite-loop bug [oh/6357] is the canonical failure case.

2.3 Why a separate skill (vs "just give the agent a bash tool")

Naively: "let the agent run pytest and read the output". This breaks because:

1. The agent doesn't know which command to run (project-specific).
2. The agent dumps the full output into context every iteration, blowing the 25k threshold by iter 3.
3. The agent has no termination contract — it'll keep trying after the test passes "to be safe", or keep trying after 30 failures "to be helpful".
4. The agent makes edits with no audit trail — if iter 4 was the right fix, you can't bisect because nothing is committed.

The loop is a contract: *verifier wiring + output capture + feedback framing

• iteration cap + per-fix git commit*. Treat it as one operation, not five.

2.4 What "green" means

3. SOP 工作流 (Agentic Protocol)

Step 1 · Wire the verifier command

Pick the cheapest verifier that catches the class of bug you care about. Cascade from fastest to slowest:

Rule: bind --lint-cmd and --test-cmd to stages 1–4 combined into one shell command (ruff check . && pytest -x). This way one feedback message covers all signals; you don't loop separately on lint then on tests.

For Aider:

aider --auto-lint --lint-cmd "ruff check ." \
      --auto-test --test-cmd "pytest -x --tb=short"

For Claude Code / generic agent:

result = subprocess.run(
    ["bash", "-c", "ruff check . && pytest -x --tb=short"],
    capture_output=True, text=True, timeout=120
)

Step 2 · Capture stdout + stderr + exit code (all three)

result = subprocess.run(
    cmd, capture_output=True, text=True, timeout=120, env={**os.environ, "NO_COLOR": "1"}
)
captured = {
    "exit_code": result.returncode,
    "stdout": result.stdout,
    "stderr": result.stderr,
    "timed_out": False,
}

Common mistakes:

• Capturing only stdout — tracebacks in pytest go to stdout, but compiler errors in tsc / cargo go to stderr. Always capture both.
• Not setting NO_COLOR=1 — ANSI escapes burn tokens and confuse the model.
• No timeout — a single infinite-loop unit test halts the whole agent.
• No byte cap — a 50MB cargo build log kills your context window.

Step 3 · Format the feedback message (the load-bearing step)

The single biggest lever in this skill. Don't paste raw output. Distill to:

The verifier failed (exit 1, pytest -x --tb=short).

FIRST FAILING TEST:
tests/test_auth.py::test_jwt_expiry — AssertionError: expected 401, got 200

TRACEBACK (last frame):
  File "src/auth.py", line 47, in verify_token
    if exp < now: return None
  TypeError: '<' not supported between instances of 'NoneType' and 'datetime'

YOUR LAST EDIT touched src/auth.py:40-50.

Hypothesis: `exp` is None when the JWT lacks an `exp` claim. Either default
it or guard the comparison.

Formatting recipe:

1. First error only. If there are 12 failing tests, show the first. Subsequent ones often cascade from the first fix.
2. Last frame of the traceback. Earlier frames are usually framework noise.
3. Anchor to the last edit. "You just changed src/auth.py:40-50" makes the model attribute the failure correctly.
4. Drop unchanged-between-iters noise — pytest's collection summary, coverage totals, deprecation warnings.
5. Hard byte cap: target ≤ 2k tokens of feedback. If a single failure doesn't fit, truncate the traceback middle (keep top + bottom).
6. No "please fix" — the framing is enough. Imperative pleas degrade instruction-following in some models.

Step 4 · Bound the iterations

Two limits, both required:

• Hard cap (MAX_ITERS = 5 is a sane default; Aider uses ~3, OpenHands uses 50–100 for SWE-Bench).
• Stall detector: if the same error message appears twice in a row, break early — the model is stuck on the wrong hypothesis.

seen_errors = []
for i in range(MAX_ITERS):
    edit = agent.propose_edit(feedback if i else initial_task)
    apply_edit(edit)
    git_commit(f"agent: iter {i+1}")           # always commit each iter
    verifier = run_verifier()
    if verifier["exit_code"] == 0:
        return Success(iters=i+1)
    feedback = format_feedback(verifier, last_edit=edit)
    if feedback in seen_errors[-1:]:           # exact repeat
        return Stall(reason="same error twice", last=feedback)
    seen_errors.append(feedback)
return Escalate(reason=f"exhausted {MAX_ITERS} iters", last=feedback)

Step 5 · Per-iteration commit (the audit lever)

After every edit, before the verifier runs, commit with a structured message:

git commit -am "agent[iter 3/5]: tighten exp guard in verify_token"

Why mandatory:

• If iter 3 made it worse and iter 5 fixed it the "wrong" way, you can bisect with git log --oneline | head -5.
• The agent never overwrites its own previous attempt — each iter is recoverable.
• git diff HEAD~1 gives the formatter a precise "what you just changed" anchor.

Aider's --auto-commits (on by default) does this. For non-Aider agents, wrap the loop in commit logic yourself.

Step 6 · Detect success precisely

The agent disabling tests to "pass" is the most common pathological success. Add a post-success diff check: git log -p -1 | grep -E '(skip|xfail|@disable)'.

Step 7 · Escalate or commit on exit

When the loop exits without success:

1. Surface the last formatted feedback — that's the message the human needs to read, not the raw pytest log.
2. Leave the WIP commits intact — the user may want to inspect iter 3 even if iter 5 failed.
3. Tag the escalation reason: exhausted, stalled, env_failure, timeout. The user's fix differs per cause.

4. 操作模型 (Operation Models)

Format: Trigger → Action → Output → Evidence.

OP-1 · Wire a one-shot verifier

• Trigger: User wants the agent to verify once after editing, no loop yet.
• Action: Run <lint> && <type> && <test> once, capture three-tuple (stdout, stderr, exit_code).
• Output: Pass/fail signal. If fail, structured digest ready to feed back.
• Evidence: [aider/lint-test] "Aider will try and fix any errors if the command returns a non-zero exit code."

OP-2 · Format raw verifier output into ≤2k-token feedback

• Trigger: Verifier failed; about to construct the next prompt.
• Action: Extract first failure, last traceback frame, anchor to changed file:lines from git diff HEAD~1 --name-only -U0. Strip ANSI, coverage, deprecation warnings. Hard byte cap.
• Output: A digest under 2k tokens with a hypothesis line.
• Evidence: [aider/edit-errors] "Above about 25k tokens of context, most models start to become distracted." Each iteration adds context; keep the per-iter delta tiny.

OP-3 · Bound the loop

• Trigger: About to enter or continue a fix loop.
• Action: Set MAX_ITERS (3–5 interactive, 50–100 SWE-Bench), detect stall (same error twice = break), enforce total wall-clock cap.
• Output: A loop with explicit termination, never while True.
• Evidence: [oh/6357] OpenHands infinite-loop bug + [langgraph/recursion] "Hitting recursion_limit indicates an underlying design flaw" — same lesson.

OP-4 · Commit per iteration

• Trigger: Agent has just applied an edit, before re-running verifier.
• Action: git add -A && git commit -m "agent[iter N]: <one-line>". Never --amend.
• Output: A bisectable audit trail; iter K is always recoverable.
• Evidence: [aider/git] per-edit auto-commit; [cline/auto-approve] Cline mirrors the same "edit→commit→test" rhythm.

OP-5 · Detect success without false positives

• Trigger: Verifier exits 0.
• Action: Confirm (a) tests were actually collected (pytest exit 5 ≠ success), (b) no test was newly skipped/xfailed in the last commit, (c) no || true suppression in the verifier command itself.
• Output: Trusted "green" signal.
• Evidence: pytest exit-code spec; [aider/lint-test] formatter wrapper caveat (auto-formatters that rewrite + return non-zero need double-run).

OP-6 · Handle environment failure (escalate, don't re-prompt)

• Trigger: Verifier output indicates infra issue — ImportError, command not found, OOM, network 503, ConnectionRefused to test DB.
• Action: Do not feed the error back as a code-fix prompt. Surface to user with tag env_failure. The agent cannot fix pytest: command not found by editing source.
• Output: Loop exits; user is told to fix the environment.
• Evidence: SWE-Gym docs note: env failures from "missing system dependencies" must be solved at the harness level, not by the agent.

OP-7 · Partial-success handling

• Trigger: 8 of 10 failing tests now pass; 2 remain.
• Action: Acknowledge progress in the feedback ("8 tests now pass; 2 still fail"), then format only the remaining 2. Reset stall detector — different error class = real progress.
• Output: Loop continues on the smaller error surface; model not whipped for the failures it just fixed.
• Evidence: Empirical: models given "you broke things" framing tend to revert good fixes. Anchor to net delta.

OP-8 · Auto-formatter that rewrites + returns non-zero

• Trigger: ruff format or prettier --write modify files and return non-zero on first pass (means "I changed something").
• Action: Wrap in a two-pass script: pass 1 writes, pass 2 verifies. Treat only pass-2 exit code as the signal.
• Output: Loop doesn't get stuck re-running the same successful format.
• Evidence: [aider/lint-test] explicit guidance on formatter wrappers.

5. 困境决策案例 (Dilemma Cases)

Case 1 · "Pytest output is 4000 lines — the agent fixes the wrong test"

• 困境: A failing pytest run dumps 4k lines (12 failures, collection warnings, deprecation notices, full tracebacks each). The agent reads the last traceback (most recent in the output) and tries to fix that, but the first failure was the root cause; the others cascade from it. Three iterations later the agent has touched 5 files and broken more tests.
• 约束:
  • Cannot truncate to first-error-only naively — some failures are independent (parallel test runners surface them in arbitrary order).
  • The user wants to see all failures in the final report, even if the agent only iterates on one.
• 决策步骤:
  1. Run with pytest -x (--exitfirst) so the test runner itself stops at the first failure. The output is naturally bounded.
  2. If the project genuinely needs all failures listed for the user, run twice: once with -x for the agent loop, once with full output captured into a side-file for the human report. Don't conflate the two streams.
  3. In the formatted feedback, anchor to git diff HEAD~1 --name-only: "your last edit touched X; the first failure is in a test of Y." The anchor breaks the "fix the last thing I read" bias.
• 结果: Bounded feedback, root-cause focused, full report preserved separately.
• 可提取的操作: OP-2. -x for the loop, full run for the human.