Spec Optimize

Iterative Optimization Loop

Run metric-driven iterative optimization. Define a goal, build measurement scaffolding, then run parallel experiments that converge toward the best solution.

Workflow Contract Summary

When To Use

Use when a measurable outcome can improve through iterative experiments, hard gates, and/or LLM-as-judge scoring.

When Not To Use

Do not use for ordinary implementation, vague improvement requests without a metric, debugging without a feedback loop, or unbounded spend/concurrency. Name the destination when routing out: a bug with no stable repro and no measurement loop is spec-debug's work (or spec-work for settled implementation) — diagnosing the bug inside this workflow to turn it into an optimization goal is adopting the wrong workflow, not adapting it; route out explicitly instead of drafting a spec around the diagnosis.

Inputs

An optimization spec or goal, mutable/immutable scope, measurement command or scaffold plan, budget limits, experiment settings, repository instructions, and baseline evidence.

Outputs

A measurement scaffold and experiment log, scored experiment results, kept/rejected variants, final integrated changes when appropriate, and post-run recommendations.

Artifacts

Run state under .spec-first/workflows/spec-optimize/<spec-name>/, experiment worktrees/results, strategy digests, and no hidden workflow state outside the documented log.

Failure Modes

Missing metric, missing measurement command, unsafe scope, excessive or uncapped budget, failed baseline, write verification failure, or unavailable dispatch/worktree backend.

Workflow

Validate the spec and budget, establish the baseline, run bounded experiments, measure and write results immediately, select winners, integrate only verified improvements, and summarize evidence.

Downstream Consumers

Code review、benchmark maintainer、在性能/相关性变更时参与的 release reviewer，以及检查 experiment logs 的人工审查者。

Scenario Capability

Follows docs/contracts/workflows/scenario-capability-matrix.md (default). Overrides: none

Interaction Method

Use the platform's blocking question tool: AskUserQuestion in Claude Code (call ToolSearch with select:AskUserQuestion first if its schema isn't loaded) or request_user_input in Codex. Fall back to numbered options in chat only when no blocking tool exists in the harness or the call errors (e.g., Codex edit modes) — not because a schema load is required. Never silently skip the question.

Input

<optimization_input> #<invocation arguments supplied by the current host> </optimization_input>

If the input above is empty, ask: "What would you like to optimize? Describe the goal, or provide a path to an optimization spec YAML file."

Optimization Spec Schema

Reference the spec schema for validation:

references/optimize-spec-schema.yaml

Experiment Log Schema

Reference the experiment log schema for state management:

references/experiment-log-schema.yaml

Quick Start

For a first run, optimize for signal and safety, not maximum throughput:

• Start from references/example-hard-spec.yaml when the metric is objective and cheap to measure
• Use references/example-judge-spec.yaml only when actual quality requires semantic judgment
• Prefer execution.mode: serial and execution.max_concurrent: 1
• Cap the first run with stopping.max_iterations: 4 and stopping.max_hours: 1
• Avoid new dependencies until the baseline and measurement harness are trusted
• For judge mode, start with sample_size: 10, batch_size: 5, and max_total_cost_usd: 5

For a friendly overview of what this skill is for, when to use hard metrics vs LLM-as-judge, and example kickoff prompts, see:

references/usage-guide.md

Admission And Budget Gate

Do not run spec-optimize as an expensive substitute for ordinary work. Before Phase 1, confirm the run has all of these:

• A repeatable measurement target: metric.primary.type, metric.primary.name, and metric.primary.direction
• At least one cheap degenerate gate that rejects broken variants before judging or ranking
• A measurement command, or a concrete plan to build the harness before any experiment is dispatched
• Explicit scope.mutable and scope.immutable boundaries
• Explicit experiment budget: stopping.max_iterations, stopping.max_hours, and stopping.plateau_iterations
• Execution budget: execution.mode and execution.max_concurrent
• For judge mode, a finite metric.judge.max_total_cost_usd, unless the user explicitly approves uncapped spend

If any item is missing, stop and help the user create a safe spec or route the work to the current host's plan/work/debug entrypoint instead. Do not continue with an open-ended optimization loop.

Measurement-Only Calibration Mode

When the invocation or approved spec selects mode:measurement-only, load references/measurement-only-calibration.md. This mode compares an explicitly identified baseline and candidate against the same repeatable task/corpus. It must run an A/A noise floor before A/B, use a pre-registered acceptance threshold, classify broken runs separately from regressions, and produce only a measurement artifact plus stop/defer guidance.

Before the first measurement, materialize the approved frozen inputs as a run-local measurement-admission-input.json, then run node scripts/measurement-admission.cjs admit --input <path>. Persist the returned normalized admission and admission_sha256 beside the measurement artifact and bind every A/A and A/B attempt to that digest. A rejected admission stops before invoking the measurement command. After A/A, run the helper's allow-ab command with the normalized admission, digest, attempts, and observed noise floor; A/B is forbidden unless it returns ab_allowed: true.

Measurement-only mode does not mutate either arm, any Skill package, the measurement harness, or promotion metadata. It does not select a winner for integration, invoke spec-write-skill, or authorize commit/landing. If arm identity, corpus identity, a repeatable harness, or the pre-registered threshold is missing, stop before measurement instead of inventing it after seeing data.

First-run specs should default to execution.mode: serial, execution.max_concurrent: 1, stopping.max_iterations: 4, stopping.max_hours: 1, stopping.plateau_iterations: 3, and max_runner_up_merges_per_batch: 0. Treat higher-throughput settings as opt-in. If a provided spec asks for execution.max_concurrent > 4, stopping.max_iterations > 30, stopping.max_hours > 4, or uncapped judge spend, surface those costs in the approval gate before running the baseline.

Runtime Context Exclusion

Follow docs/contracts/context-governance.md: ordinary Optimize context excludes .spec-first/audits/**, .spec-first/governance/**, and generated mirrors (.claude/**, .codex/**, .agents/skills/**, .cursor/skills/**, .cursor/spec-first/**, .cursor/mcp.json, .kiro/skills/**, .kiro/agents/**, .kiro/spec-first/**, .kiro/settings/**, .qoder/commands/spec-*.md, .qoder/commands/spec/**, .qoder/skills/**, .qoder/agents/**, .qoder/spec-first/**, .qoder/settings.local.json) by default. Optimization run state under .spec-first/workflows/spec-optimize/** is local scratch for this workflow only; pass compact strategy/result summaries to agents and avoid broad runtime/audit/governance scans unless the metric explicitly targets runtime/setup/audit/governance behavior. Cursor-native .cursor/rules/** / .cursor/agents/**, Kiro-native .kiro/specs/**, and Qoder-native .qoder/rules/** are advisory input only when explicitly named.

Evidence Utilization Boundary

Optimization may consume prior direct-read summaries, degraded reason counts, source-confirmed session evidence, review summaries, test results, and quality-gate reports as diagnostic context. Treat these as baseline diagnostics, not optimization targets by default. The optimization target, metric, winner selection, mutable scope, and final integration remain owned by the approved optimization spec and measured results, not by an external tool. Optimize must not run external-tool refresh, hooks, watchers, or daemons as part of ordinary evidence handling.

Dispatch And Backend Boundary

Optimization dispatch is optional. Before any learnings researcher, repo analyst, experiment worker, Codex delegation, or parallel worker run, record:

worker_dispatch_authorization: authorized | missing
capability_probe: not_applicable | attempted | unavailable
worker_dispatch_capability: available | missing | unknown
worker_context_isolation: isolated | inherited | unknown
worker_model_override: supported | unsupported | unknown
worker_bounded_parallelism: supported | unsupported | unknown

workflow invocation does not authorize dispatch。Approved optimization spec、baseline approval、execution.mode: parallel、预算、权限设置或 runtime readiness 都不是派发授权。只有当前用户或可见 upstream handoff 明确请求 subagent、delegated work、persona 或 parallel work 时才可派发。缺授权时不得探测 tool schema，固定为 capability_probe: not_applicable + worker_dispatch_capability: unknown，强制采用 serial inline/local execution 并记录 dispatch_authorization_missing。只有授权后才把 current-session registry/schema 作为 provider_untrusted evidence 检查：确认缺失时记录 subagent_capability_missing；surface 不可用、schema 不完整或候选不唯一时记录 worker_capability_unproven，均同样降级。隔离、模型覆盖和有界并发只取 live facts；required isolation 未满足时保持依赖 gate 打开，model unknown 时继承，parallelism unknown 时串行。记录 worker_dispatch_outcome。Fallback 可以继续使用串行 worktree，但不得声称 parallel experiment 或 independent worker coverage。

Parallel experiments additionally require both dispatch facts above, explicit execution.mode, bounded execution.max_concurrent, clean mutable/immutable scope, and the worktree readiness probes below. Worktree-backed mutation happens in experiment worktrees; Codex delegation must fall back after repeated failures when the serial/local path can continue. The orchestrator owns final integration: selecting kept experiments, merging or cherry-picking winners, reverting non-winners, cleaning worktrees, updating experiment logs, and presenting post-completion actions. Workers never stage, commit, merge, push, or mutate the authoritative experiment log.

Persistence Discipline

CRITICAL: The experiment log on disk is the single source of truth. The conversation context is NOT durable storage. Results that exist only in the conversation WILL be lost.

The files under .spec-first/workflows/spec-optimize/<spec-name>/ are local scratch state. They are ignored by git, so they survive local resumes on the same machine but are not preserved by commits, branches, or pushes unless the user exports them separately.

This skill runs for hours. Context windows compact, sessions crash, and agents restart. Every piece of state that matters MUST live on disk, not in the agent's memory.

If you produce a results table in the conversation without writing those results to disk first, you have a bug. The conversation is for the user's benefit. The experiment log file is for durability.

Core Rules

1. Write each experiment result to disk IMMEDIATELY after measurement — not after the batch, not after evaluation, IMMEDIATELY. Append the experiment entry to the experiment log file the moment its metrics are known, before evaluating the next experiment. This is the #1 crash-safety rule.

2. VERIFY every critical write — after writing the experiment log, read the file back and confirm the entry is present. This catches silent write failures. Do not proceed to the next experiment until verification passes.

3. Re-read from disk at every phase boundary and before every decision — never trust in-memory state across phase transitions, batch boundaries, or after any operation that might have taken significant time. Re-read the experiment log and strategy digest from disk.

4. The experiment log is append-only during Phase 3 — never rewrite the full file. Append new experiment entries. Update the best section in place only when a new best is found. This prevents data loss if a write is interrupted.

5. Per-experiment result markers for crash recovery — each experiment writes a result.yaml marker in its worktree immediately after measurement. On resume, scan for these markers to recover experiments that were measured but not yet logged.

6. Strategy digest is written after every batch, before generating new hypotheses — the agent reads the digest (not its memory) when deciding what to try next. The strategy digest is derived, reconstructable state; the experiment log remains the canonical resume and audit source. Persist every hypothesis decision, kept/reverted result, metric, and audit field needed to reconstruct the digest in the experiment log before replacing or deleting the digest.

7. Never present results to the user without writing them to disk first — the pattern is: measure -> write to disk -> verify -> THEN show the user. Not the reverse.

Mandatory Disk Checkpoints

These are non-negotiable write-then-verify steps. At each checkpoint, the agent MUST write the specified file and then read it back to confirm the write succeeded.

Format of a verification step:

1. Write the file using the native file-write tool
2. Read the file back using the native file-read tool
3. Confirm the expected content is present
4. If verification fails, retry the write. If it fails twice, alert the user.

File Locations (all under .spec-first/workflows/spec-optimize/<spec-name>/)

On Resume

When Phase 0.4 detects an existing run:

1. Read the experiment log from disk — this is the ground truth
2. Scan worktree directories for result.yaml markers not yet in the log
3. Recover any measured-but-unlogged experiments
4. Continue from where the log left off

Phase 0: Setup

0.1 Determine Input Type

Check whether the input is:

• A spec file path (ends in .yaml or .yml): read and validate it
• A description of the optimization goal: help the user create a spec interactively
• Not an optimization input at all — a bug report, an unstable repro, or a "fix it" request with no measurable outcome: do not absorb it by fixing the bug or converting the diagnosis into a spec here. Name spec-debug (unstable/unresolved failures) or spec-work (settled implementation) in the reply, route out, and stop this workflow.

0.2 Load or Create Spec

If spec file provided:

1. Read the YAML spec file. The orchestrating agent parses YAML natively -- no shell script parsing.
2. Validate the spec against every rule in the validation_rules section of references/optimize-spec-schema.yaml. That section is the single source of truth for what a valid spec requires; do not rely on a remembered subset. Conditional rules such as exclusive-resource serial execution, singleton-rubric requirements, uncapped judge spend approval, high-throughput approval, and stopping criteria live there.
3. If validation fails, report errors and ask the user to fix them

If description provided:

1. Analyze the project to understand what can be measured

2. Detect whether the optimization target is qualitative or quantitative — this determines type: hard vs type: judge and is the single most important spec decision:

   Use type: hard when:

   • The metric is a scalar number with a clear "better" direction
   • The metric is objectively measurable (build time, test pass rate, latency, memory usage)
   • No human judgment is needed to evaluate "is this result actually good?"
   • Examples: reduce build time, increase test coverage, reduce API latency, decrease bundle size

   Use type: judge when:

   • The quality of the output requires semantic understanding to evaluate
   • A human reviewer would need to look at the results to say "this is better"
   • Proxy metrics exist but can mislead (e.g., "more clusters" does not mean "better clusters")
   • The optimization could produce degenerate solutions that look good on paper
   • Examples: clustering quality, search relevance, summarization quality, code readability, UX copy, recommendation relevance

   IMPORTANT: If the target is qualitative, strongly recommend type: judge. Explain that hard metrics alone will optimize proxy numbers without checking actual quality. Show the user the three-tier approach:

   • Degenerate gates (hard, cheap, fast): catch obviously broken solutions — e.g., "all items in 1 cluster" or "0% coverage". Run first. If gates fail, skip the expensive judge step.
   • LLM-as-judge (the actual optimization target): sample outputs, score them against a rubric, aggregate. This is what the loop optimizes.
   • Diagnostics (logged, not gated): distribution stats, counts, timing — useful for understanding WHY a judge score changed.

   If the user insists on type: hard for a qualitative target, proceed but warn that the results may optimize a misleading proxy.

3. Design the sampling strategy (for type: judge):

   Guide the user through defining stratified sampling. The key question is: "What parts of the output space do you need to check quality on?"

   Walk through these questions:

   • What does one "item" look like? (a cluster, a search result page, a summary, etc.)
   • What are the natural size/quality strata? (e.g., large clusters vs small clusters vs singletons)
   • Where are quality failures most likely? (e.g., very large clusters may be degenerate merges; singletons may be missed groupings)
   • What total sample size balances cost vs signal? (default: 30 items, adjust based on output volume)

   Example stratified sampling for clustering:

   stratification:
     - bucket: "top_by_size"     # largest clusters — check for degenerate mega-clusters
       count: 10
     - bucket: "mid_range"       # middle of non-solo cluster size range — representative quality
       count: 10
     - bucket: "small_clusters"  # clusters with 2-3 items — check if connections are real
       count: 10
   singleton_sample: 15          # singletons — check for false negatives (items that should cluster)

   The sampling strategy is domain-specific. For search relevance, strata might be "top-3 results", "results 4-10", "tail results". For summarization, strata might be "short documents", "long documents", "multi-topic documents".

   Singleton evaluation is critical when the goal involves coverage — sampling singletons with the singleton rubric checks whether the system is missing obvious groupings.

4. Design the rubric (for type: judge):

   Help the user define the scoring rubric. A good rubric:

   • Has a 1-5 scale (or similar) with concrete descriptions for each level
   • Includes supplementary fields that help diagnose issues (e.g., distinct_topics, outlier_count)
   • Is specific enough that two judges would give similar scores
   • Does NOT assume bigger/more is better — "3 items per cluster average" is not inherently good or bad

   Example for clustering:

   rubric: |
     Rate this cluster 1-5:
     - 5: All items clearly about the same issue/feature
     - 4: Strong theme, minor outliers
     - 3: Related but covers 2-3 sub-topics that could reasonably be split
     - 2: Weak connection — items share superficial similarity only
     - 1: Unrelated items grouped together
     Also report: distinct_topics (integer), outlier_count (integer)

5. Guide the user through the remaining spec fields:

   • What degenerate cases should be rejected? (gates — e.g., "solo_pct <= 0.95" catches all-singletons, "max_cluster_size <= 500" catches mega-clusters)
   • What command runs the measurement?
   • What files can be modified? What is immutable?
   • Any constraints or dependencies?
   • If this is the first run: recommend execution.mode: serial, execution.max_concurrent: 1, stopping.max_iterations: 4, and stopping.max_hours: 1
   • If type: judge: recommend sample_size: 10, batch_size: 5, and max_total_cost_usd: 5 until the rubric and harness are trusted

6. Write the spec to .spec-first/workflows/spec-optimize/<spec-name>/spec.yaml

7. Present the spec to the user for approval before proceeding

0.3 Search Prior Learnings

Read references/agents/learnings-researcher.md. Dispatch a generic subagent seeded with that local prompt only when the Dispatch And Backend Boundary permits it; otherwise search inline with the same bounded scope and record the matching fallback reason. Do not dispatch a standalone agent by type/name. If relevant learnings exist, incorporate them into the approach.

0.4 Run Identity Detection

Check if optimize/<spec-name> branch already exists:

git rev-parse --verify "optimize/<spec-name>" 2>/dev/null

If branch exists, check for an existing experiment log at .spec-first/workflows/spec-optimize/<spec-name>/experiment-log.yaml.

Present the user with a choice via the platform question tool:

• Resume: read ALL state from the experiment log on disk (do not rely on any in-memory context from a prior session). Recover any measured-but-unlogged experiments by scanning worktree directories for result.yaml markers. Continue from the last iteration number in the log.
• Fresh start: archive the old branch to optimize-archive/<spec-name>/archived-<timestamp>, clear the experiment log, start from scratch

0.5 Create Optimization Branch and Scratch Space

git checkout -b "optimize/<spec-name>"  # or switch to existing if resuming

Create scratch directory:

mkdir -p .spec-first/workflows/spec-optimize/<spec-name>/

Phase 1: Measurement Scaffolding

This phase is a HARD GATE. The user must approve baseline and parallel readiness before Phase 2.

Bundled scripts. Phases 1 and 3 call helper scripts that ship in this skill's scripts/ directory (measure.sh, parallel-probe.sh, experiment-worktree.sh). The Bash tool's working directory is the user's project, not the skill directory, so a bare scripts/<name> path will not resolve — invoke each by the skill's own absolute path. Every runnable block below already sets SKILL_DIR inline (shell state does not persist between Bash tool calls, so each block must carry it); replace the <absolute path ...> placeholder with the directory you loaded this spec-optimize SKILL.md from before running. The shape:

SKILL_DIR="<absolute path of the directory containing this SKILL.md>"
bash "$SKILL_DIR/scripts/<name>"

1.1 Clean-Tree Gate

Verify no uncommitted changes to files within scope.mutable or scope.immutable:

git status --porcelain

Filter the output against the scope paths. If any in-scope files have uncommitted changes:

• Report which files are dirty
• Ask the user to commit or stash before proceeding
• Do NOT continue until the working tree is clean for in-scope files

1.2 Build or Validate Measurement Harness

Resolve scripts/measure.sh, scripts/parallel-probe.sh, and scripts/experiment-worktree.sh relative to this skill's loaded directory. The measurement working directory remains the project directory named by the optimization spec.

Before the first measurement command, freeze and display this run-local execution envelope:

measurement_execution_authorization: authorized | missing
measurement_command: <exact command string from the approved spec>
measurement_working_directory: <resolved absolute cwd>
measurement_environment_names: [<inherited or overlaid variable names; never secret values>]
measurement_expected_effects: [read-only | writes-project-files | writes-local-state | network | other]

An approved optimization spec, clean tree, executable harness, baseline approval, or shell permission does not set this fact. Require the current user or visible upstream handoff to authorize the displayed command, resolved cwd, environment names, and expected effects before any invocation of measure.sh, direct measurement command, or parallel probe that executes it. When missing, return measurement_execution_authorization_missing with zero measurement command executions. If any frozen field changes later, invalidate the authorization and present the new envelope before continuing. This is a workflow-level effect gate; measure.sh remains a bounded executor and does not infer semantic authorization from the command text.

If user provides a measurement harness (the measurement.command already exists):

1. Run it once via the measurement script:

   SKILL_DIR="<absolute path of the directory containing this SKILL.md>"
   bash "$SKILL_DIR/scripts/measure.sh" "<measurement.command>" <timeout_seconds> "<measurement.working_directory or .>"

2. Validate the JSON output:
   • Contains keys for all degenerate gate metric names
   • Contains keys for all diagnostic metric names
   • Values are numeric or boolean as expected
3. If validation fails, report what is missing and ask the user to fix the harness

If agent must build the harness:

1. Analyze the codebase to understand the current approach and what should be measured
2. Build an evaluation script (e.g., evaluate.py, evaluate.sh, or equivalent)
3. Add the evaluation script path to scope.immutable -- the experiment agent must not modify it
4. Run it once and validate the output
5. Present the harness and its output to the user for review

1.3 Establish Baseline

Run the measurement harness on the current code.

If stability mode is repeat:

1. Run the harness repeat_count times
2. Aggregate results using the configured aggregation method (median, mean, min, max)
3. Calculate variance across runs
4. If variance exceeds noise_threshold, warn the user and suggest increasing repeat_count

Record the baseline in the experiment log:

baseline:
  timestamp: "<current ISO 8601 timestamp>"
  gates:
    <gate_name>: <value>
    ...
  diagnostics:
    <diagnostic_name>: <value>
    ...

If primary type is judge, also run the judge evaluation on baseline output to establish the starting judge score.

1.4 Parallelism Readiness Probe

Run the parallelism probe script:

SKILL_DIR="<absolute path of the directory containing this SKILL.md>"
bash "$SKILL_DIR/scripts/parallel-probe.sh" "<project_directory>" "<measurement.command>" "<measurement.working_directory>" <shared_files...>

Read the JSON output. Present any blockers to the user with suggested mitigations. Treat the probe as intentionally narrow: it should inspect the measurement command, the measurement working directory, and explicitly declared shared files, not the entire repository.

1.5 Worktree Budget Check

Count existing worktrees:

SKILL_DIR="<absolute path of the directory containing this SKILL.md>"
bash "$SKILL_DIR/scripts/experiment-worktree.sh" count

If count + execution.max_concurrent would exceed 12:

• Warn the user
• Suggest cleaning up existing worktrees or reducing max_concurrent
• Do NOT block -- the user may proceed at their own risk

1.6 Write Baseline to Disk (CP-1)

MANDATORY CHECKPOINT. Before presenting results to the user, write the initial experiment log with baseline metrics to disk:

1. Create the experiment log file at .spec-first/workflows/spec-optimize/<spec-name>/experiment-log.yaml
2. Include all required top-level sections from references/experiment-log-schema.yaml: spec, run_id, started_at, baseline, experiments, and best
3. Seed experiments as an empty array and seed best from the baseline snapshot (use iteration: 0, baseline metrics, and baseline judge scores if present) so later phases have a valid current-best state to compare against
4. Optionally seed hypothesis_backlog: [] here as well so the log shape is stable before Phase 2 populates it
5. Verify: read the file back and confirm the required sections are present and the baseline values match
6. Only THEN present results to the user

1.7 User Approval Gate

Present to the user via the platform question tool:

• Baseline metrics: all gate values, diagnostic values, and judge scores (if applicable)
• Experiment log location: show the file path so the user knows where results are saved
• Parallel readiness: probe results, any blockers, mitigations applied
• Clean-tree status: confirmed clean
• Worktree budget: current count and projected usage
• Judge budget: estimated per-experiment judge cost and configured max_total_cost_usd cap (or an explicit note that spend is uncapped)

Options:

1. Proceed -- approve baseline and parallel config, move to Phase 2
2. Adjust spec -- modify spec settings before proceeding
3. Fix issues -- user needs to resolve blockers first

Do NOT proceed to Phase 2 until the user explicitly approves.

If primary type is judge and max_total_cost_usd is null, call that out as uncapped spend and require explicit approval before proceeding.

State re-read: After gate approval, re-read the spec and baseline from disk. Do not carry stale in-memory values forward.

Phase 2: Hypothesis Generation