Blue Team

Blue Team

A defensive fixer loop — the inverse of red-team. The artifact is the target, now writable; the feedback signal is two-part, like optimize-loop: a gate that must hold (nothing that passed before regresses) and a metric that must drop (the count of open failure classes, toward zero). You point it at a set of concrete failed cases and fix them one root-cause class at a time. Each iteration you patch one class, then run tools/verify.py, and keep the patch only if it closes the class with no regression, else revert. You loop until every class is closed (dry) or the budget runs out, then hand the patch set off as a pull request. This is the fix half of a find→fix setup (see Pairing).

The failed cases come from a real source; tools/ingest.py normalizes any of them into one catalogue:

• oracle mode — a red-team failures.jsonl: each case is an input where the target's verdict disagrees with a ground-truth oracle. A case is closed when target and oracle now agree; a regression is a benign <holdout> input that newly disagrees (most often a new over-block).
• tests mode — a CI/CD test-failure report (pytest --junitxml / JUnit XML, or a list of failing node ids): each case is a failing test. A case is closed when its test now passes; a regression is any other test that was passing and now fails.

When to use

Use to fix a concrete set of failing cases in code or a guardrail/classifier/filter/prompt/API the user owns — a red-team catalogue, or the failing tests from a CI run — driving the open-class count to zero without breaking what worked. A class is the root-cause group the loop closes as a unit (a red-team technique, or a CI failure area / test class).

Default: pick the mode that matches the source (oracle for red-team, tests for CI/CD). Escape hatch: in oracle mode with no separate functional test suite, the <holdout> alone is the regression guard; in tests mode the suite's own previously-passing tests are the guard. Not for finding new failures (run red-team), and not for editing the ground truth (the oracle, the tests, or the holdout).

Setup

Resolve bindings interactively. If loop.run.yaml exists, load it, confirm the values in one line, and skip to the loop. Otherwise: on Claude Code (the AskUserQuestion tool is available) infer a likely value per binding and recommend it; on other hosts ask each as a quoted prompt. Then write loop.run.yaml and confirm before creating any other files. Two worked configs: examples/run.example.yaml (oracle mode) and examples/tests.run.yaml (tests mode).

<skill_dir> is this skill's installed folder; substitute the real path when writing loop.run.yaml.

Build the catalogue first with tools/ingest.py, which normalizes any source into {id, ..., class}:

python3 <skill_dir>/tools/ingest.py --from red-team --in <failures.jsonl> --out <catalogue>   # oracle mode
python3 <skill_dir>/tools/ingest.py --from junit    --in <report.xml>     --out <catalogue>   # tests mode

The signal each iteration is tools/verify.py, in the mode matching <source>:

# oracle mode — <target_cmd> runs <target_files>, e.g. "python3 ./guardrail.py"
python3 <skill_dir>/tools/verify.py --target "<target_cmd>" --oracle "<oracle_cmd>" \
  --catalogue <catalogue> --holdout <holdout>
# tests mode — <test_cmd> writes the JUnit report verify.py then reads
python3 <skill_dir>/tools/verify.py --test-cmd "<test_cmd>" --junit <junit> --catalogue <catalogue>

Either way it prints one JSON object: {mode, open_classes, closed_classes, open_count, closed_count, regressions, regression_count, still_failing}.

The loop

Copy this checklist and tick items off:

• Iteration 0 — baseline: run tools/verify.py in the <source> mode; record the open classes (should match the catalogue) as the current state and confirm regression_count is 0 — if it is not, the catalogue or holdout is dirty, so fix that before fixing the target. Log the baseline row. Save a pristine copy of <target_files> to <sandbox_root>/iter0/ (snapshots mode) or note the branch base (branches mode) — this is the baseline the final handoff diffs against, and also the snapshot iteration 1 reverts to.
• In branches mode, open the run on a fresh branch: git checkout -b <pr_branch>.
• For iteration N (≥1): snapshot the current, pre-patch <target_files> to iter<N>/ (or note the git HEAD) before editing, so a discard can restore exactly this state.
• Pick one open class. Read its still_failing examples + the suggested fix from the catalogue, and patch <target_files> at the root cause — one fix should close all payloads of that class (e.g. normalize case once, not per-keyword). One class per iteration so each delta is attributable.
• Check the signal: run tools/verify.py. Discard — restore the snapshot / git reset --hard — if regression_count > 0 (the gate) or the targeted class is still open. verify.py's regression check is the gate: in oracle mode a regression is a newly-broken <holdout> case (e.g. a fix that closes a bypass by over-blocking benign inputs); in tests mode it is any previously-passing test the patch broke.
• Keep if there are no regressions and open_count strictly dropped. In branches mode commit it: git commit -am "close <class>: <one-line fix>". Append a ledger row.
• If a class resists <patience> attempts, mark it an open residual and move on rather than thrashing. Stop when open_count = 0 (dry), at <budget>, or when every remaining class is a residual. <budget> counts attempts (each keep or discard is one iteration), not classes closed — a discard still consumes the budget.

On stop, restore the working files to the best iteration (most classes closed, zero regressions) and report: classes closed vs residual, the failures resolved (oracle mode: the bypass/over-block split), and regressions avoided. Then open the pull request (see Handoff).

Fix toolkit. In tests mode the patches are ordinary bug-fixes, grouped by failure area and applied one area per iteration. In oracle mode (hardening a guardrail/filter), reach for these root-cause patterns, mirroring red-team's attack toolkit:

• Normalize before matching — case-fold, de-leet (homoglyph/leet → letters), strip spacing and punctuation, NFKC-normalize unicode. One normalization step closes case / leetspeak / spacing classes.
• Broaden the policy — add missing synonyms/expansions to the blocked set (the missing-synonym class), keyed to the oracle's categories, not ad-hoc strings.
• Tighten over-broad rules — scope a match to whole words / the right context so benign inputs stop tripping it (the overblock class), the most common source of regressions.

Mind the interaction order (both modes): make the narrowing/over-broad fix before a sweeping one. A fix that strips separators (closing spacing) can re-collapse a benign input into an over-broad substring and silently reopen an overblock class — and likewise a broad code change can reopen a test a narrower fix had to protect. Fix the narrow/over-broad case first, then generalize.