Autoforge

AutoForge — Autonomous Optimization Framework

Stop reflecting. Start converging. Every iteration is measured, logged, and validated — not vibed.

AutoForge replaces ad-hoc "improve this" prompts with a rigorous optimization loop: define evals, run iterations, track pass rates in TSV, report live to your channel, and stop only when math says you're done. Multi-model cross-validation prevents the "same model grades its own homework" blind spot.

Four modes. One convergence standard.

AutoForge — Top-Agent Architecture

Overview

Agent (you)
├── State: results.tsv, current target file state, iteration counter
├── Iteration 1: evaluate → improve → write TSV → report
├── Iteration 2: evaluate → improve → write TSV → report
├── ...
└── Finish: report.sh --final → configured channel

Sub-Agent = You

"Sub-Agent" is a conceptual role, not a separate process. You (the top-agent) execute each iteration yourself: simulate/execute → evaluate → write TSV → call report.sh. The templates below describe what you do PER ITERATION — not what you send to another agent.

For code mode, run tests using the exec tool.

Multi-Model Setup (recommended for Deep Audits)

For complex audits, you can split two roles across different models:

Flow: Optimizer and Validator alternate. Optimizer iterations have status improved/retained/discard. Validator iterations confirm or refute the pass rate. Spawn validators as sub-agents with sessions_spawn and explicit model.

When to use Multi-Model: Deep Audits (>5 iterations expected), complex ground truth, or when a single model is blind to its own errors.

When Single-Model suffices: Simple CLI audits, prompt optimization, code with clear tests.

Configuration

AutoForge uses environment variables for reporting. All are optional — without them, output goes to stdout.

Hard Invariants

These rules apply always, regardless of mode:

1. TSV is mandatory. Every iteration writes exactly one row to results/[target]-results.tsv.
2. Reporting is mandatory. Call report.sh immediately after every TSV row.
3. --dry-run never overwrites the target. Only TSV, *-proposed.md, and reports are written.
4. Mode isolation is strict. Only execute steps for the assigned mode.
5. Iteration 1 = Baseline. Evaluate the original version unchanged, status baseline.

Modes — Read ONLY Your Mode!

You are assigned ONE mode. Ignore all sections for other modes.

Your mode is in the task prompt. Everything else is irrelevant to you.

TSV Format (same for ALL modes)

Header (once at loop start):

printf '%s\t%s\t%s\t%s\t%s\n' "iteration" "prompt_version_summary" "pass_rate" "change_description" "status" > results/[target]-results.tsv

Row per iteration:

printf '%s\t%s\t%s\t%s\t%s\n' "1" "Baseline" "58%" "Original version" "baseline" >> results/[target]-results.tsv

Use printf not echo -e! echo -e interprets backslashes in field values. printf '%s' outputs strings literally.

5 columns, TAB-separated, EXACTLY this order:

Escaping rules:

• Tabs in text fields → replace with spaces
• Newlines in text fields → replace with |
• Empty fields → use hyphen - (never leave empty)
• $ and backticks → use printf '%s' or escape with \$ (prevents unintended variable interpolation)
• Unicode/Emoji allowed, count as 1 character (not bytes)

Status rules (based on pass-rate comparison):

• baseline — Mandatory for Iteration 1. Evaluate original version only.
• improved — Pass rate higher than previous best → new version becomes current state
• retained — Pass rate equal or marginally better → predecessor remains
• discard — Pass rate lower → change discarded, revert to best state

Reporting (same for ALL modes)

After EVERY TSV row (including baseline):

bash scripts/report.sh results/[target]-results.tsv "[Skill Name]"

After loop ends, additionally with --final:

bash scripts/report.sh results/[target]-results.tsv "[Skill Name]" --final

The report script reads AF_CHANNEL, AF_CHAT_ID, and AF_TOPIC_ID from environment. Without them, it prints to stdout with ANSI colors.

Stop Conditions (for ALL modes)

Priority — first matching condition wins, top to bottom:

1. 🛑 Minimum iterations — If specified in task (e.g. "min 5"), this count MUST be reached. No other condition can stop before.
2. 🛑 Max 30 iterations — Hard safety net, stop immediately.
3. ❌ 3× discard in a row → structural problem, stop + analyze.
4. ✅ 3× 100% pass rate (after minimum) → confirmed perfect, done.
5. ➡️ 5× retained in a row → converged, done.

Counting rules:

• 3× 100% = three iterations with pass_rate == 100%, not necessarily consecutive.
• 5× retained and 3× discard = consecutive (in a row).
• baseline counts toward no series.
• improved interrupts retained and discard series.

At 100% in early iterations: Keep going! Test harder edge cases. Only 3× 100% after the minimum confirms true perfection.

Recognizing Validator Noise

In multi-model setups, the Validator can produce false positives — fails that aren't real issues:

• Config path vs tool name confusion (e.g. agents.list[] ≠ agents_list tool)
• Inverted checks ("no X" → Validator looks for X as required)
• Normal English as forbidden reference (e.g. "runtime outcome" ≠ runtime: "acp")
• Overcounting (thread commands counted as subagent commands)

Rule: If after all real fixes >3 discards come in a row and the fail justifications don't hold up under scrutiny → declare convergence, don't validate endlessly.

Execution Modes

mode: prompt

Only read if your task contains mode: prompt!

Per Iteration: What you do

1. Read current prompt/skill
2. Mentally simulate 5 different realistic scenarios
3. Evaluate each scenario against all evals (Yes=1, No=0)
4. Pass rate = (Sum Yes) / (Eval count × 5 scenarios) × 100
5. Compare with best previous pass rate → determine status
6. On improved: propose minimal, surgical improvement
7. Write TSV row + call report.sh
8. Check stop conditions

At the End

Best version → results/[target]-proposed.md + report.sh --final

mode: code

Only read if your task contains mode: code!

Per Iteration: What you do

1. Create sandbox: SCRATCH=$(mktemp -d) && cd $SCRATCH
2. Write current code to sandbox
3. Execute test command (with timeout 60s)
4. Measure: exit_code, stdout, stderr, runtime
5. Evaluate against evals → calculate pass rate
6. On improved: minimal code improvement + verify again
7. Write TSV row + call report.sh
8. Check stop conditions

Code Eval Types

At the End

Best code → results/[target]-proposed.[ext] + report.sh --final

mode: audit

Only read if your task contains mode: audit!

⚠️ DO NOT write your own code. Only test CLI commands of the target tool (--help + read-only).

Two Variants

Simple Audit (CLI skill, clear commands):

• 2 iterations: Baseline → Proposed Fix
• For tools with clear --help output and simple command structure

Deep Audit (complex docs, many checks):

• Iterative loop like prompt/code, same stop conditions
• For extensive documentation with many checkpoints (e.g. config keys, tool policy, parameter lists)
• Recommended: Multi-Model setup (Opus Optimizer + external Validator)

Simple Audit Flow

1. Write TSV header
2. Iteration 1 (Baseline): Test every documented command → pass rate → TSV + report
3. Iteration 2 (Proposed Fix): Write improved SKILL.md → expected pass rate → TSV + report
4. Improved SKILL.md → results/[target]-proposed.md
5. Detail results → results/[target]-audit-details.md (NOT in TSV!)
6. report.sh --final