Terminal Bench Loop

Terminal-Bench Loop

A repeatable operating skill for driving one Terminal-Bench problem to a passing smoke through Paperclip, with explicit issue topology, bounded runs, board-gated product fixes, and worktree continuity.

This skill is operational + diagnostic, not engineering. It coordinates issues, artifacts, and approvals around a Terminal-Bench loop. It does not authorize code changes — every accepted product fix lands as a separate implementation child issue after a board confirmation.

Canonical execution model: read doc/execution-semantics.md before starting a loop or moving any loop issue. Every loop issue must rest in a state the doc allows: terminal (done/cancelled), explicitly live (active run / queued wake), explicitly waiting (in_review with participant/interaction/approval), or explicit recovery/blocker (blocked with blockedByIssueIds and a named owner).

When to use

Trigger on an assignment whose title or body matches any of:

• "run Terminal-Bench in a loop", "loop <task-name> through Paperclip"
• "drive Terminal-Bench fix-git", "iterate on Terminal-Bench until it passes"
• "Terminal-Bench smoke loop", "bench loop", "smoke loop on <task-name>"
• An attached link to a Terminal-Bench loop parent issue, plus a request to do another iteration

Also use when the user hands you an existing top-level loop issue and asks for the next iteration, diagnosis, or rerun.

When NOT to use

• The assignment is to build or change paperclip-bench itself (Harbor adapter, wrapper, telemetry). Use normal engineering flow on that repo.
• The assignment is to submit a benchmark result for ranking. This skill produces smoke/non-comparable runs by design — escalate full-suite or comparable runs to BenchmarkQualityManager.
• The assignment is a normal Paperclip product bug not surfaced by a Terminal-Bench loop. Use normal investigation.
• You have not been granted permission to install or assign company skills, and the asker actually wants library mutation. Hand that step to an authorized skill-library owner.

Three invariants you must preserve

Every loop iteration and every proposed product fix must hold these three invariants together. They come from /diagnose-why-work-stopped and the user has restated them across the liveness work:

1. Productive work continues. Each loop issue must always have a clear next action owner — agent, board, user, or named blocker. No silent in_review with nothing waiting on it.
2. Only real blockers stop work. Stops happen when something genuinely cannot proceed (board confirmation, QA, missing credentials, exhausted budget). Pseudo-stops must be detected and routed.
3. No infinite loops. Iteration count, wall-clock budget, and a board gate before product fixes are applied keep the loop bounded.

If a proposed iteration violates any of the three, drop it or rework it. State explicitly in the loop issue how each invariant is held this iteration.

Inputs

Collect these on the top-level loop issue before iteration 1. Any input that cannot be supplied is a blocker — name the unblock owner and stop.

• Source issue. The Paperclip issue that asked for the loop. The loop parent links back to it.
• Terminal-Bench task name. Single-task identifier (e.g. terminal-bench/fix-git). Multi-task suites are out of scope for this skill.
• Iteration budget. Maximum number of iterations before the loop must stop without further fixes (typical: 3–5). Also record a per-iteration wall-clock cap.
• Paperclip App worktree issue. The implementation-side issue under the Paperclip App project whose execution workspace owns the isolated worktree. First iteration creates it; later iterations reuse it via inheritExecutionWorkspaceFromIssueId or equivalent.
• Benchmark command. The exact paperclip-bench invocation, including the PAPERCLIPAI_CMD (or equivalent) binding pinned to the Paperclip App worktree under test. Record verbatim on the loop issue.
• Dispatch runner config. The exact Harbor/Paperclip runner dispatch config required for the smoke to actually start a Paperclip heartbeat. For the current Harbor wrapper, record the PAPERCLIP_HARBOR_RUNNER_CONFIG JSON (or equivalent config file) verbatim enough to preserve: assignee, heartbeat_strategy, agent_adapter / agent_adapters, reuse_host_home when local credentials are intentionally needed, and the stop budget. A bare Harbor command that creates BEN-1 as unassigned todo with zero heartbeat-enabled agents is a harness/setup failure, not a valid product diagnosis.
• Latest artifact root. Filesystem or storage path under which paperclip-bench writes run artifacts (manifest, results.jsonl, Harbor raw job folders, redacted telemetry). Each iteration appends; nothing is overwritten.
• Approval policy. Who must accept a proposed product fix before implementation (default: board via request_confirmation; CTO if delegated; never the loop driver alone).

Record each input on the top-level loop issue (description or a dedicated inputs document). If any input changes mid-loop, note the change and the iteration it took effect.

Issue topology

The loop must be representable as a tree, not as prose in comments:

• Top-level loop issue. Long-lived. Holds inputs, iteration counter, current state, links to every iteration child, and the product-rule history. Rests in in_progress while an iteration is running, in_review only when a typed waiter sits directly on the loop parent (execution-policy participant, request_confirmation / ask_user_questions / suggest_tasks interaction, approval, or named human owner), blocked with blockedByIssueIds while a child issue is the gating work (iteration child holding the fix-proposal request_confirmation, or implementation, QA, or CTO review children), done on pass, or cancelled on board-rejection / budget exhaustion.
• Iteration child issues. One per iteration. Each carries: a bounded run issue (smoke), a diagnosis issue (applies /diagnose-why-work-stopped), a fix-proposal document with a request_confirmation interaction, and — only after acceptance — implementation, QA, CTO review, and rerun children. Iteration children are blocked by their predecessors so the executor wakes them in order.
• Paperclip App implementation issue. The first iteration creates a fresh Paperclip App child whose project policy spawns an isolated worktree. Every later iteration's implementation/rerun child references that same execution workspace via inheritExecutionWorkspaceFromIssueId so the same worktree is amended and tested.

Wire dependencies with blockedByIssueIds, never with prose like "blocked by X". When a dependent child is done, the executor auto-wakes the next.

Procedure

0. Read the current execution contract

Before opening or advancing a loop, read doc/execution-semantics.md. Use that document's terms intact when classifying loop-issue state: live path / waiting path / recovery path; post-run disposition; bounded continuation; productivity review; pause-hold; watchdog. Do not invent a new state.

1. Open or reuse the top-level loop issue

• If an existing loop issue is supplied, read it: inputs, iteration counter, last iteration's stop reason, current Paperclip App worktree pointer, latest benchmark command.
• If no loop issue exists, create one under the Paperclip App project (or the project the source issue points at). Title: Terminal-Bench loop: <task-name>. Description captures the inputs above, the iteration budget, and a link to the source issue.
• Verify the worktree pointer still resolves. If the recorded execution workspace was discarded (worktree pruned, project changed), the loop is blocked — name the unblock owner (CodexCoder or the Paperclip App owner) and stop.

2. Open the iteration child

• Increment the iteration counter on the loop issue.
• Create an iteration child titled Iteration N: <task-name>. Its description repeats the inputs and references the loop parent. Block it on the prior iteration's terminal child (if any) so the executor cannot start two iterations in parallel.
• If the iteration counter would exceed the budget, do not create the child. Move the loop issue to cancelled (budget exhausted) or in_review if the user must decide whether to extend the budget.

3. Run the bounded smoke

• The benchmark command must use the Paperclip App worktree under test. Set PAPERCLIPAI_CMD (or the equivalent command binding) to the CLI entrypoint inside that worktree. Never let the smoke run against the operator's current Paperclip checkout.
• The same command block must include the runner dispatch config that makes the benchmark issue actionable. For the current Harbor wrapper, export PAPERCLIP_HARBOR_RUNNER_CONFIG with the intended assignee, heartbeat strategy, agent adapter, credential/home mode, and stop budget. Do not treat a bare uvx harbor run ... as the canonical smoke if it omits the dispatch config; record that as a harness/setup miss and rerun with the recorded config.
• Bound the run by wall-clock and by Paperclip's run-budget controls. If the smoke would exceed the per-iteration cap, kill it and record the truncation reason.
• Capture, in the iteration child or a dedicated run document:
  • Paperclip run id and heartbeat run ids
  • benchmark run id, manifest, results.jsonl row, Harbor raw job folder
  • dispatch config used (PAPERCLIP_HARBOR_RUNNER_CONFIG or equivalent), including assignee and adapter type
  • the exact stop reason reported by the harness (pass, harness fail, verifier fail, timeout, agent gave up, infrastructure error)
  • heartbeat-enabled and heartbeat-observed agent counts when Paperclip telemetry exports them
  • failure taxonomy bucket (task/model, Paperclip product, harness/setup, verifier/infrastructure, security, unclear)
  • artifact paths under the latest artifact root
• Label the iteration as smoke / non-comparable. Comparable runs are out of scope for this skill.

4. Diagnose the exact stop point

Apply the /diagnose-why-work-stopped pattern to the iteration's run, scoped to this loop only — do not pull in unrelated forensic boilerplate. Specifically:

• Walk the Paperclip issue tree the smoke produced under the Paperclip App worktree, node by node, and find the exact (issue, status) combination that stopped progress. Quote evidence: run ids, comment timestamps, status transitions.
• Classify every non-progressing issue in that subtree as truly needs human/board intervention, agent-actionable but not currently routed, or already covered.
• State whether the failure is task/model, Paperclip product, harness/setup, verifier/infrastructure, security, or unclear. Be explicit when evidence is inferred (e.g. cross-company API boundary blocks direct reads).
• If the failure is a Paperclip product gap, frame the fix as a general product rule stated as a contract, and check it against the three invariants above. If the rule would have blocked a recent productive run, narrow it.

Record the diagnosis on the iteration child as a diagnosis document. Do not propose code yet.

5. Decide the next move

Based on the diagnosis, the iteration ends in exactly one of these terminal-for-iteration states:

• Pass. Smoke verifier reports pass. Move the iteration child and the loop parent toward QA/CTO review (Step 8).
• Product fix proposed. A Paperclip product gap was identified. Write the fix proposal as a plan document on the iteration child, then go to Step 6.
• Non-product failure with retry. Failure is harness/setup/infrastructure or model flakiness, the iteration budget is not exhausted, and the loop driver believes a rerun without code changes has signal (e.g. transient infra). Record the rationale on the iteration child and go to Step 7 with no implementation step.
• Real blocker. Named external blocker (credentials, quota, third-party outage, security review). Move the loop issue to blocked, set blockedByIssueIds to the blocker issue (creating one if needed), and name the unblock owner. Stop.
• Budget or board stop. Iteration budget reached, or the board has rejected the next fix proposal. Move the loop issue to cancelled with a comment that summarizes the run history and the reason for stopping.

6. Request board confirmation before any product fix

When the iteration ends in product fix proposed:

• Update the iteration child's plan document with the proposed contract, the three-invariant check, the affected Paperclip surfaces, and the phased subtasks (implementation, QA, CTO review, rerun) — but do not create those subtasks.
• Open the request_confirmation interaction on the iteration child (the same issue that owns the plan document), targeting the latest plan revision. Idempotency key: confirmation:{iterationIssueId}:plan:{revisionId}. Set continuationPolicy to wake_assignee.
• Move the iteration child to in_review. The typed waiter — the request_confirmation interaction — sits directly on it, so its in_review is healthy. Comment links the plan document and names the pending confirmation.
• Move the loop parent to blocked with blockedByIssueIds: [iterationChildId] and a comment naming the board (or whichever approver the approval policy designates) as the unblock owner. Do not move the loop parent to in_review here: the typed waiter lives on the iteration child, not on the parent, so the parent's wait path is the child blocker. This matches the topology rule that the loop parent only sits in in_review when a typed waiter is attached directly to the parent.
• Wait for acceptance. If the board posts a superseding comment that changes the plan, revise the document, then open a fresh confirmation tied to the new revision on the iteration child — the prior one is invalidated. The loop parent's blockedByIssueIds already points at the iteration child, so it does not need to change.
• On rejection, end the loop per the Budget or board stop rule; do not silently retry the same proposal.
• On acceptance, create the implementation, QA, CTO review, and rerun child issues with blockedByIssueIds wired in order, and update the loop parent's blockedByIssueIds to point at the new gating child (typically the implementation child) so the parent stays blocked against real downstream work. The implementation child must inherit the Paperclip App execution workspace (inheritExecutionWorkspaceFromIssueId to the worktree-owning issue) so the fix lands in the same isolated worktree the smoke ran against.

7. Rerun against the same worktree

After implementation and QA complete (or immediately, in the non-product failure with retry case), the rerun child runs the same paperclip-bench invocation with PAPERCLIPAI_CMD still pinned to the Paperclip App worktree under test.

• The rerun must use the same worktree the fix landed in. If the workspace was reset between iterations, the loop is invalid — open a blocker on the loop issue and stop.
• On completion, the rerun child becomes the next iteration's run record. If the smoke now passes, jump to Step 8. Otherwise return to Step 4 with a new iteration child (subject to the iteration budget).