Agentsop Prompt Compilation

prompt-compilation — The Compile-Readiness Gate

"It's unproductive to launch optimization runs using a poorly designed program or a bad metric." — DSPy core team [dspy.ai/learn/optimization/overview/]

"Compile when you can measure. The optimizer maximizes your metric — garbage metric in, garbage prompt out." — this skill's operating principle (synthesized from the line above + DSPy Case C)

This is an enhancement-overlay decision skill. It answers exactly one question the broad [[dspy]] library skill buries under API surface: have you earned the right to run an optimizer yet, and which one? It produces a go / no-go gate plus an optimizer pick. It defers every implementation detail — Signature syntax, module choice, compile() calls, save/deploy — to [[dspy]] and the full workflow in [[agentsop-dspy]]. It defers metric construction to [[agentsop-metric-design]]; this skill only checks the metric exists and is validated, then uses it as the gate.

The trap it removes: people reach for MIPROv2(auto="heavy") because the API is right there, before they have a metric worth maximizing or enough data to avoid memorization. Compilation is a hyperparameter search costing hundreds-to-thousands of LM calls ($2–$40+, minutes-to-hours) [dspy.ai/faqs/]. Spending that on an un-validated metric or 8 examples is pure waste.

1. 何时激活 (When to Activate)

Activate when all three of these are plausibly true (the gate then confirms them):

• A hand-tuned prompt has plateaued. The team has manually iterated few-shot examples / wording past the point of obvious returns. Symptom from [[agentsop-dspy]] §1: "the team manually tunes few-shot examples; a metric exists but isn't being used to drive prompt design."
• A metric exists (or can be built). There is a metric(example, pred) -> bool|float — or one can be written and human-validated. Without this, do not activate; the optimizer has nothing to maximize.
• Labeled examples exist. There is a dev set. The count determines which optimizer is even legal (§3, §4.2).

Concrete triggers in intent or codebase:

Do NOT activate when:

• The task is one-shot or the Signature/I-O contract is still churning daily — compile only after it stabilizes [dspy.ai/learn/optimization/overview/]; otherwise you pay compile cost for prompts you'll throw away.
• No metric is possible and none will be built — then this is verbose prompting, not compilation. Route to [[agentsop-metric-design]] first; if the user refuses any success criterion, the gate stays closed.
• Compliance requires verbatim human-authored prompts — optimized prompts are machine-generated artifacts.
• You only need parse safety (a typed Signature), not quality optimization — that is [[agentsop-dspy]] Stage 1 and the signature-design overlay. Promoting prose to a typed Signature and compiling it are two different gates.

2. 核心心智模型 (Core Mental Model)

"Compile when you can measure. The optimizer maximizes your metric — garbage metric in, garbage prompt out."

An optimizer (MIPROv2, GEPA, BootstrapFewShot) is a black-box search over prompt instructions + few-shot demos that maximizes metric(pred, example). It has no taste. It will faithfully chase whatever the metric rewards, biases and all. From [[agentsop-dspy]] Case C: "DSPy will optimize toward whatever the metric rewards. A bad metric becomes a bad program at scale." Two corollaries make this a gate, not a step:

1. The metric is the precondition, not a tunable. Before any compute is spent, the metric must (a) exist and (b) agree with human judgment on ≥20 spot-checks. An un-validated metric means the expensive search optimizes the metric's blind spot. This is non-negotiable [dspy.ai/learn/evaluation/metrics/; Case C]. Construction is [[agentsop-metric-design]]'s job; this skill only checks the receipt.

2. Data scale is a hard floor, not a preference. Below the optimizer's example floor you are not training, you are memorizing. The DSPy 20/80 train/val split exists because "prompt-based optimizers often overfit to small training sets" [dspy.ai/learn/optimization/overview/]. The floor differs per optimizer (§4.2).

The two-gate picture

            ┌──────────────────────────────────────────────┐
 Gate 1     │  METRIC: exists?  AND  human-validated ≥20?   │  ── No ─► STOP. Build/validate metric ([[agentsop-metric-design]]).
 (measure)  └──────────────────────────────────────────────┘
                              │ Yes
                              ▼
            ┌──────────────────────────────────────────────┐
 Gate 2     │  EXAMPLES: ≥ floor for the optimizer I want?  │  ── No ─► Pick a lower-floor optimizer, collect data,
 (data)     └──────────────────────────────────────────────┘           or STOP (use LabeledFewShot as a floor).
                              │ Yes
                              ▼
                   COMPILE (cheap probe first: auto="light")

The cost reality (why the gate is worth having)

Compilation runs hundreds-to-thousands of LM calls. Reference run: ~3.2k API calls, $2–$3, 6–20 min for auto="light"; auto="heavy" on 1000+ examples can hit tens of dollars and hours [dspy.ai/faqs/]. Cost scales with num_trials × |trainset| × |program LM calls|. The gate's entire job is to stop you spending that on a metric or dataset that cannot pay it back.

3. SOP (Standard Operating Procedure)

0. Confirm activation (§1): plateaued hand-tuning + metric + examples.
1. GATE 1 — metric exists AND validated ≥20 human spot-checks?   [hard]
2. GATE 2 — example count ≥ floor for the chosen optimizer?      [hard]
3. PICK    — choose optimizer by data scale + feedback signal.   (§4.2)
4. BUDGET  — estimate cost; cap it; choose a cheap optimizer LM.
5. PROBE   — run auto="light" (or smallest config) as a signal.
6. DECIDE  — gain ≥ threshold → escalate; else go back, don't escalate.

Gate 1 — Metric (the "can you measure" gate)

• Is there a metric(example, pred, trace=None) -> bool|float? If not → STOP, route to [[agentsop-metric-design]].
• Has it been validated against a human on ≥20 spot-checks (≥30 if open-ended), with ≥80% agreement? If not → STOP and validate first. "Never compile against a metric you haven't human-validated on ≥20 spot-checks" [[[agentsop-dspy]] Case C; dspy.ai/learn/evaluation/metrics/].
• Is the metric a single holistic LLM-judge? Then it inherits length / self-preference / position bias and the optimizer will chase those biases — decompose it ([[agentsop-metric-design]]) before compiling [arxiv.org/pdf/2506.02592].

Exit: a validated metric callable + a calibration receipt. Otherwise the gate is closed; do not proceed.

Gate 2 — Examples (the "enough data" gate)

Count labeled examples. The DSPy-documented thresholds: ≥30 = minimum useful, ~300 = recommended, 200+ required for MIPROv2 to avoid overfitting [dspy.ai/learn/optimization/overview/]. The floor is per optimizer:

• < 10 → only LabeledFewShot(k=8) (no search, weakest) — usually means STOP, collect data.
• ~10+ with textual feedback → GEPA is legal and sample-efficient (this inverts the usual "more data" rule).
• ~30–50 → BootstrapFewShot / BootstrapFewShotWithRandomSearch.
• 200+ → MIPROv2.

Exit: the example count clears the floor of the optimizer you intend to run. If not, either drop to a lower-floor optimizer or stop.

Step 3 — Pick optimizer by data scale + signal

Use the table in §4.2. The single most important branch: do you have textual error feedback (test diffs, schema violations, judge rationales like "answer was verbose")? If yes, GEPA needs only ~10 examples and converges faster because it reflects on the text of the feedback, not just a scalar [dspy.ai/tutorials/gepa_ai_program/; arxiv.org/abs/2507.19457]. If no, fall back to the data-scale ladder.

Step 4 — Budget

Estimate before launching (§4.4). Cap spend. Use a cheap optimizer LM (e.g. gpt-4o-mini) even when the task LM is expensive — community-reported parity at a fraction of cost [github.com/stanfordnlp/dspy/issues/1596]. Set dspy.configure(track_usage=True) to log actual spend.

Step 5 — Probe cheap first

Run auto="light" (MIPROv2) or the smallest config first. Docs: "start with moderate values, observe behavior, and scale up only if you see clear gains" [github.com/stanfordnlp/dspy/issues/1596]. Never start at auto="heavy".

Step 6 — Decide on the probe result

• light gives <2% lift → do not escalate. The bottleneck is the program/metric, not the optimizer. Loop back to [[agentsop-dspy]] Stage 1 (signature ambiguous? wrong decomposition?) [dspy.ai/learn/optimization/overview/].
• light gives 2–10% lift → escalate to medium; go to heavy only if data ≥300 and you have a held-out test set distinct from val.
• Always confirm the final gain on a held-out test set, not the val set used during optimization.

4. 操作模型 (Operations)

4.1 — Operation registry (Trigger / Action / Output / Evidence)

OP-1 — ReadinessGate (entry, hard gate)

• Trigger: Anyone about to call an optimizer's compile().
• Action: Run Gate 1 (metric exists + validated ≥20) then Gate 2 (examples ≥ optimizer floor). Any failure → STOP.
• Output: Decision token compile-ready | not-ready + the failing gate.
• Evidence: [dspy.ai/learn/optimization/overview/]; [[agentsop-dspy]] Case C, §3 three-stage gate.

OP-2 — MetricExistsCheck

• Trigger: Gate 1.
• Action: Confirm a metric(example, pred, trace=None) -> bool|float exists. If absent → route to [[agentsop-metric-design]], gate stays closed.
• Output: Metric callable or a STOP.
• Evidence: Anti-pattern "compiling without a metric" [dspy.ai/learn/optimization/overview/].

OP-3 — MetricValidatedCheck

• Trigger: Metric exists (OP-2).
• Action: Require ≥20 (open-ended ≥30) human spot-checks at ≥80% agreement, plus a calibration receipt. If a single holistic LLM-judge, require decomposition first.
• Output: validated flag + receipt reference.
• Evidence: [[agentsop-dspy]] Case C step 4; [dspy.ai/learn/evaluation/metrics/]; [arxiv.org/pdf/2506.02592].

OP-4 — ExampleFloorCheck

• Trigger: Gate 2.
• Action: Count labeled examples; compare to the floor of the intended optimizer (LabeledFewShot any; GEPA ~10+feedback; Bootstrap ~30–50; MIPROv2 200+).
• Output: cleared | below-floor + the legal optimizer set.
• Evidence: [dspy.ai/learn/optimization/overview/] (30 min / 300 rec / 200+ MIPROv2).

OP-5 — OptimizerByDataScale

• Trigger: Both gates passed.
• Action: Pick from the §4.2 table by example count and feedback signal. Prefer GEPA when textual feedback exists.
• Output: Optimizer choice + config.
• Evidence: [dspy.ai/learn/optimization/optimizers/]; [dspy.ai/api/optimizers/GEPA/overview/].

OP-6 — CostBudgetGuard

• Trigger: Before launching compile.
• Action: Estimate num_trials × |trainset| × calls; cap spend; set a cheap optimizer LM; enable track_usage=True.
• Output: A budget ceiling + chosen optimizer LM.
• Evidence: [dspy.ai/faqs/]; [github.com/stanfordnlp/dspy/issues/1596].

OP-7 — CheapProbeFirst

• Trigger: Compile-ready, budget set.
• Action: Run auto="light" (or smallest config) first. Never start at heavy.
• Output: A cheap lift signal.
• Evidence: [github.com/stanfordnlp/dspy/issues/1596]; [[agentsop-dspy]] Case A.

OP-8 — EscalateOrReturn

• Trigger: Probe finished.
• Action: <2% lift → return to program/metric design (do NOT escalate). 2–10% → medium. heavy only if data ≥300 + held-out test set. Confirm gain on held-out test.
• Output: Escalate / iterate-back decision.
• Evidence: [[agentsop-dspy]] Case A; [dspy.ai/learn/optimization/overview/].

4.2 — Optimizer-by-data-scale table

4.3 — Readiness checklist (copy/paste)

GATE 1 — MEASURE
[ ] metric(example, pred, trace=None) -> bool|float exists
[ ] validated vs human: >=20 spot-checks (>=30 open-ended), >=80% agreement
[ ] receipt saved {n_spot_checks, agreement, judge_model, task_model, date}
[ ] NOT a lone holistic LLM-judge (else decompose via [[agentsop-metric-design]] first)

GATE 2 — DATA
[ ] labeled examples counted: N = ____
[ ] N clears the floor of the optimizer chosen below
[ ] (prompt-based optimizers) 20/80 train/val split planned; GEPA uses standard split

PICK + BUDGET
[ ] optimizer chosen by §4.2 (GEPA if textual feedback)
[ ] cost ceiling set; cheap optimizer LM chosen; track_usage=True
[ ] plan: probe auto="light" first, escalate only on >=2% lift, confirm on held-out test

4.4 — Cost budget heuristics

Source: [dspy.ai/faqs/]; [github.com/stanfordnlp/dspy/issues/1596]; [arxiv.org/abs/2507.19457].

5. 困境决策案例 (Dilemma Cases)