Agentsop Dspy

DSPy SOP — Programming, Not Prompting

"DSPy isn't a prompt-optimization agent framework. It's the LLM compiler for the shortest, cleanest code." — Eito Miyamura [eito.substack.com/p/dspy-the-most-misunderstood-agent]

"Prompts are effectively the weights of an LLM application." — Core philosophy [arxiv.org/abs/2310.03714]

1. 何时激活 (When to activate)

Activate this skill when any of the following triggers are present in the user's intent or codebase:

Do NOT activate when:

• The task is one-shot ("just answer this question once") — use raw client.messages.create.
• No evaluation metric is possible and none is willing to be built — DSPy without a metric is just verbose prompting.
• Prompts must remain human-authored verbatim for compliance, audit, or stylistic reasons.
• The team is in rapid exploration mode where the task signature itself is changing daily — compile only after the signature stabilizes [dspy.ai/learn/optimization/overview/].

2. 核心心智模型 (Core mental model)

DSPy's full name is Declarative Self-improving Python. The three primitives form a PyTorch-like compile chain [arxiv.org/abs/2310.03714]:

┌─────────────┐    ┌──────────┐    ┌──────────────┐    ┌─────────┐
│  Signature  │ →  │  Module  │ →  │ Teleprompter │ →  │ Compile │
│ (what)      │    │ (how)    │    │ (optimizer)  │    │ (tune)  │
└─────────────┘    └──────────┘    └──────────────┘    └─────────┘
   I/O spec       Predict/CoT/      MIPROv2/GEPA/      Bake demos
   field names     ReAct/PoT        BootstrapFewShot   + instructions
   = semantic     = strategy        = search algorithm  into JSON

Three mental shifts the agent must internalize:

1. Prompts are weights. The prompt string is not the artifact you ship — the compiled program (a JSON of demonstrations + instructions + structural choices) is. You ship program.json, not a .txt prompt [dspy.ai/tutorials/saving/].

2. Signatures carry semantic load. question -> answer is not the same as query -> response. DSPy uses the field names as the only natural-language hint the optimizer has about intent before it sees data. Name them like you'd name function parameters in well-written code [dspy.ai/learn/programming/signatures/].

3. Compile is a hyperparameter search, not a one-shot call. Compilation runs hundreds-to-thousands of LM calls (typically $2–$3 USD, 6–20 minutes, 3.2k API calls in the reference run). Costs scale with num_trials × |trainset| × |program LM calls| [dspy.ai/faqs/].

The PyTorch analogy is load-bearing. Signatures ≈ nn.Module.forward() shape contract. Modules ≈ nn.Linear / nn.Transformer. Teleprompters ≈ torch.optim.Adam. compile() ≈ training loop. save()/load() ≈ checkpoint.

3. SOP 工作流 (SOP workflow)

The DSPy team is explicit about a three-stage gate [dspy.ai/learn/]:

"It's unproductive to launch optimization runs using a poorly designed program or a bad metric."

Do not skip stages. Each stage has an exit criterion.

Stage 1 — Programming (no optimizer yet)

1. Pin the task as a Signature. Start inline ("question -> answer"); upgrade to a class-based dspy.Signature with InputField(desc=...) / OutputField(desc=...) when types matter or fields need disambiguation.
2. Pick the lowest-power Module that works. Default to dspy.ChainOfThought. Use dspy.Predict for trivial classification, dspy.ReAct only when tools are needed, dspy.ProgramOfThought for arithmetic-heavy tasks [dspy.ai/learn/programming/modules/].
3. Compose with plain Python control flow. Subclass dspy.Module, instantiate sub-modules in __init__, call them in forward(). No special DSL.
4. Run zero-shot on 5–10 hand-picked examples. Look at outputs with dspy.inspect_history(n=3).

Exit criterion: the un-optimized program produces plausible outputs on 5+ examples. Not great — plausible.

Stage 2 — Evaluation (no optimizer yet)

5. Build a dev set. Documented sweet spot: 30 examples = minimum useful, 300 = recommended, 200+ required for MIPROv2 to avoid overfitting [dspy.ai/learn/optimization/overview/].
6. Write a metric: def metric(example, pred, trace=None) -> float|bool. Start with exact-match; only escalate to LLM-as-judge when the task demands it (open-ended generation, multi-criteria).
7. Run dspy.Evaluate(devset=dev, metric=metric, num_threads=16) and record a baseline score.

Exit criterion: baseline score is stable across two runs (cache-free) AND the metric agrees with human judgment on 10 spot-checks.

Stage 3 — Optimization (compile)

8. Pick optimizer by data + signal regime (Section 4 table). Decide which model optimizes vs. which model is the task model — they can differ.
9. Use the unusual 20/80 split (20% train, 80% val) for prompt-based optimizers. GEPA uses standard ML splits (maximize train) [dspy.ai/learn/optimization/overview/].
10. Start auto="light". Only escalate to "medium"/"heavy" if dev-set gains flatten and budget allows.
11. Save the compiled program: compiled.save("v1.json") for state, or compiled.save("./v1/", save_program=True) for whole-program (preferred for production with metadata) [dspy.ai/tutorials/saving/].
12. Deploy via FastAPI (dspy.asyncify) or MLflow (mlflow.dspy.log_model) [dspy.ai/tutorials/deployment/].

Exit criterion: compiled program beats baseline on a held-out test set (not the val set used in optimization) by ≥ task-relevant delta.

When to iterate back

Loop to Stage 1 if optimization plateaus. Per the docs: "Is your task well-defined? Do you need more data? Should your evaluation metric change?" — these are the questions to re-ask, not "should I try a different optimizer?" [dspy.ai/learn/optimization/overview/].

4. 操作模型 (Trigger / Action / Output / Evidence)

4.1 Choose the optimizer

4.2 Module selection

4.3 Metric design

4.4 Cost guardrails

5. 困境决策案例 (Dilemma cases — ≥3)

Case A — "Optimizer cost vs gain: when is it worth compiling?"

困境 (Dilemma): User has a 3-stage RAG pipeline. Hand-tuned prompts already hit 72% on dev. MIPROv2 auto="heavy" would cost ~$40 and 4 hours. Worth it?

约束 (Constraints):

• 250 labeled examples (above MIPROv2 200-example floor) [dspy.ai/learn/optimization/optimizers/].
• Prompts already manually iterated — diminishing returns suspected.
• Pipeline LM = GPT-4o ($-per-call adds up at trial scale).

决策步骤 (Decision steps):

1. Check whether the prompts were ever validated against the metric, or just eye-balled. If eye-balled, even auto="light" (~$2) typically yields 10–30%+ on hand-tuned baselines per the paper's GPT-3.5/Llama2 results (25%/65% lift over standard few-shot) [arxiv.org/abs/2310.03714].
2. Run auto="light" first as a cheap signal. The docs explicitly recommend "start with moderate values, observe behavior, and scale up only if you see clear gains" [github.com/stanfordnlp/dspy issue #1596].
3. If light gives <2% lift, do not escalate to heavy. Instead, revisit Stage 1: is the signature ambiguous? Is the program structure (3 stages) actually right?
4. If light gives 5–10% lift, run medium. Only escalate to heavy if data ≥ 300 and you have a held-out test set distinct from val.
5. Use gpt-4o-mini as the optimizer LM even when the task LM is gpt-4o. Community evidence: parity at fraction of cost [github.com/stanfordnlp/dspy issue #1596].

结果 (Outcome): Typical: light exposes whether more compute helps. Often the answer is "no — fix the program/metric first."

可提取的操作 (Extractable operation): Never start compilation at auto="heavy". Always probe with light and use a cheap optimizer LM.

Case B — "Swap the underlying LM: recompile, transfer, or both?"

困境: Compiled program for GPT-4o works at 85%. Need to switch to Llama-3-8B for cost. Re-use the GPT-4o-compiled program.json or recompile?

约束:

• Compiled program contains demos + instructions that may exceed the smaller model's context coherence.
• Recompile cost ≈ another $2–5.
• The DSPy doctrine: "If you optimize a complex pipeline for GPT-4, it usually breaks on a smaller model like Llama-3-8b" [acldigital.com — Death to Prompting].

决策步骤:

1. Always recompile when changing the task LM family. This is the headline value prop of DSPy: "swap the LLM definition and re-compile your program" [ganeshkedari.substack.com/p/stop-writing-prompts-a-guide-to-dspy].
2. The optimizer will discover the smaller model needs more examples / simpler reasoning steps and adjusts automatically — you do not edit prompts.
3. Use BootstrapFinetune as a follow-on: optimize prompts on the big model, then distill into a 1B–7B student. Typical setup: student=Llama-3.2-1B-Instruct, teacher=gpt-4o-mini [dspy.ai/api/optimizers/BootstrapFinetune/].
4. If demos in the saved program reference GPT-4o-style verbose CoT, the small model may parrot length without reasoning. Recompile is mandatory, not optional.
5. Keep both program.gpt4o.json and program.llama8b.json checked in; A/B in production.

结果: Recompiled programs typically recover 70–90% of the larger-model performance at 1/10–1/50 the per-call cost. The "transfer without recompile" path is reliably worse.

可提取的操作: Treat the compiled program as a (program × LM) pair. Changing the LM invalidates the artifact — recompile.

Case C — "Metric design: proxy correctness vs ground truth"

困境: Open-ended customer-support response task. No exact-match metric possible. LLM-as-judge "feels right" but the team worries the judge will be biased toward verbose, hedged outputs.

约束:

• 400 labeled examples with a reference response (not the unique correct response — one acceptable response).
• Production users penalize verbosity.
• DSPy will optimize toward whatever the metric rewards. A bad metric becomes a bad program at scale.

决策步骤:

1. Refuse to ship a single-LLM-judge as the optimization metric. Document evidence: LLM judges exhibit self-preference, recency, rubric-order, and provenance biases [arxiv.org/pdf/2506.02592, arxiv.org/pdf/2509.26072].
2. Decompose the judge into orthogonal sub-judges, each a dspy.Predict(Assess) call with a single yes/no question (factual? on-topic? concise? non-hedging?). Documented pattern [dspy.ai/learn/evaluation/metrics/].
3. Use trace is not None to return bool during compile, float during eval — same metric function, two modes. Avoids the optimizer overfitting to score noise.
4. Spot-check the metric on 20 examples with a human judge first. If sub-judges disagree with human on >20% of cases, fix the metric before compiling. Garbage metric → garbage compiled program.
5. If sub-judge feedback is rich (e.g. "answer was verbose"), pipe textual feedback into dspy.GEPA instead of MIPROv2 — GEPA leverages text feedback for faster, more sample-efficient convergence [dspy.ai/api/optimizers/GEPA/overview/, arxiv.org/abs/2507.19457].
6. Add a length penalty as a separate scalar in the metric — don't rely on the judge to penalize verbosity (judges over-prefer length).

结果: Multi-dimension metric with explicit length penalty + GEPA's textual feedback typically beats single-judge + MIPROv2 by 10–13% on AIME-style benchmarks [arxiv.org/abs/2507.19457] and is the empirically robust path.

可提取的操作: Never compile against a metric you haven't human-validated on ≥ 20 spot-checks. Decompose multi-criteria metrics. Prefer GEPA when you can express textual feedback.