Agentsop LLM Artifact Versioning

Case B — "Prompt change shipped without an eval — and now we can't reproduce or roll back cleanly"

困境: Under deadline pressure, an engineer edits the synthesizer system prompt directly in the running config to fix one bad answer, redeploys, and moves on. No new bundle, no eval run, no tag. A week later a different regression appears in prod, and a model-snapshot rollover also landed in the same window. Now: which prompt is live? Was it ever evaluated? Can we roll back to before the prompt edit without also reverting unrelated changes?

约束:

• The live prompt no longer matches any committed bundle manifest — the manifest is stale.
• The model alias rolled forward in the same window, so even "revert the prompt" leaves an untested (old-prompt × new-model) combination.
• No eval was run on the edited prompt, so there is no score to compare against — the regression gate was bypassed entirely.
• Customers are affected; time pressure is real again.

决策步骤:

1. Stop the bleeding with a bundle rollback, not a prompt rollback. Deploy the last tagged bundle deploy/v<n> (OP-6, R1 §8). Because the bundle pins prompt and model and config together, this restores a known-good combination — it does not produce the untested (old-prompt × new-model) Frankenstein that a prompt-only revert would.
2. Reconstruct what was live. The ad-hoc prompt edit is the root cause of the reproducibility loss: behavior was changed outside the bundle. Capture the edited prompt's content hash now so it is at least recoverable, then delete the out-of-band path.
3. Re-introduce the fix the right way. Put the prompt fix into VCS (config-as-code, OP-3), bump the model pin to the dated snapshot it will actually run on (OP-2), and build a new bundle deploy/v<n+1>.
4. Run the gate. [[agentsop-regression-gate]] runs the pinned eval set on v(n+1) vs the parent. The original one-bad-answer case should be in the eval set now (turn the incident into a test). Merge only on pass.
5. Add a pre-deploy guard. Refuse to deploy if the live prompt hashes do not match the tagged bundle's manifest — converts "someone edited prod directly" from a silent reproducibility hole into a deploy-time error.

结果: Rollback restores service immediately because the bundle is atomic. The fix re-lands behind the gate. The lesson the team internalizes: a prompt change shipped without an eval-linked bundle is invisible to rollback and reproduction — it is exactly the failure mode this overlay exists to prevent.

可提取的操作: No behavior-affecting change ships outside a versioned, eval-linked bundle. Roll back by bundle tag (atomic) — never by reverting one component, which yields untested combinations.

6. 反模式与边界 (Anti-patterns & boundaries)

Anti-patterns

1. Versioning code but not prompts/configs. Git tracks the Python; the prompt text lives in a dashboard and the model id in an env var. The "deploy" is then unreproducible — the committed code does not determine behavior. Prompts are weights (R1 §1); version them.
2. "latest" (or any alias) model pin. gpt-4o, claude-3-5-sonnet, latest are moving targets that change behavior with no version bump (R1 §3). Pin dated snapshots at every call site (OP-2).
3. No eval-set linkage. A bundle that records a score but not the eval set that produced it cannot be compared to its successor — "score went up" across two eval sets is meaningless (R1 §5, OP-4). Pin eval_set_sha.
4. Per-component tags instead of one bundle tag. Tagging prompts, models, and configs separately makes rollback non-atomic — you revert one part and ship a never-tested combination (R1 §8, OP-6). One bundle, one tag.
5. Editing live prod components out of band. A "quick" dashboard prompt edit (Case B) breaks reproducibility and rollback silently. All behavior changes flow through a new bundle.
6. Treating the bundle like a single ML model. Logging just the compiled program and calling it "the artifact" drops the model pins, retrieval config, and eval set. The artifact is a composite (R1 §7) — save_program captures one component, not the bundle (R1 §2).
7. Retrieval config left implicit. Index pointer, embed model, and top_k unbundled means a re-embed or a knob change silently alters answers (OP-8, R1 §6). Pin them in the manifest.
8. Deleting old bundles to "save space." Disk is cheap; rollback targets are precious. Keep prior bundle tags reachable.

Boundaries (when this overlay is overkill)

• Single-component PoC. One prompt, one model, no retrieval, never reproduced — a flat v1.json is enough. Adopt this overlay at the second component or the first real deploy.
• Vendor owns the lifecycle. OpenAI Prompt Library / Anthropic Workbench / managed RAG — let the vendor version it.
• Pre-stabilization exploration. If the app shape changes daily, bundle versioning has no shelf life yet — stabilize first (same boundary as dspy-sop).
• Versioning ONE compiled prompt. Use [[agentsop-per-model-artifacts]] directly; this overlay only adds value once there are multiple components to bundle.
• The CI comparison mechanism itself. This overlay requires a regression gate but does not implement it — see [[agentsop-regression-gate]].

7. 跨框架对照 (Cross-framework comparison)

How four mechanisms handle (or fail to handle) the whole-bundle versioning problem. None of them natively version the composite; each covers a slice.

DSPy save_program — one component, not the bundle

• Stores: per-predictor instructions, bootstrapped demos, signature shapes (R1 §2) [dspy.ai/tutorials/saving/].
• Does NOT store: model identity, dataset, metric, retrieval config, eval-set version.
• Role in the bundle: program.json (or a save_program=True dir) is one component the manifest references. The overlay pins the rest. Per-prompt lifecycle of this component → [[agentsop-per-model-artifacts]].

LangChain Hub — versioned text templates, model as a hint

• Stores: prompt template text, partial variables, model name as a non-enforced hint.
• Does NOT store: eval scores, dataset binding, retrieval config, recompile lineage.
• Role in the bundle: a viewer/diff tool for the prompt component. It does not pin a model, bind an eval set, or run a gate. Use Hub as the prompt viewer; the bundle manifest is the source of truth. hub.pull("user/rag-qa:v3") resolves a prompt version, not a deploy.

MLflow Model Registry — the closest analogy, built for one model

• Stores: pickled model/program, param + metric history per run, signature, versions with explicit Stage labels (None/Staging/Production/Archived) [~/.claude/skills/mlflow/SKILL.md, mlflow.org/docs/latest/model-registry.html].
• Does NOT store natively: the heterogeneity — many model snapshots, multiple prompts, retrieval config, eval-set hash (add via params/tags).
• Role in the bundle: borrow the primitives — versioning, stage promotion, reproduce, compare (R1 §7). Press it into service by logging the whole manifest as one "model" with components as params/artifacts. The gap: it was built for the single-weights case; the LLM bundle is a composite. The analogy holds for versioning + promotion + rollback, not for the artifact's shape.

Plain git + manifest — the lightweight default

• Stores: whatever you put in deploy/manifest.<version>.json — prompt hashes, compiled program shas, model snapshots per call site, retrieval config, eval_set_sha, framework versions.
• Strengths: zero infra, fully grep-able and git-blame-able; the bundle tag (deploy/v<n>) gives atomic deploy + rollback (R1 §8); the regression gate ([[agentsop-regression-gate]]) wires to the pinned eval set in CI.
• Limitations: no UI, no RBAC, no built-in compare. Scales to small/medium teams.
• Decision rule: start here. Migrate the registry layer to MLflow when you exceed team scale; keep the manifest discipline regardless.

Summary

Companion skills

• [[agentsop-per-model-artifacts]] — versions ONE compiled prompt as a (program × LM × dataset) triple. This overlay wraps that as one component of the whole bundle. Use it for per-prompt swap-tests, snapshot churn, and dataset-hash discipline.
• [[agentsop-regression-gate]] — the CI mechanism that re-runs the pinned eval set on new bundle vs parent and fails the PR on regression. This overlay requires it (Step 4, OP-4) but does not implement it.
• dspy-sop — the compile loop that produces the compiled-program component.

Quick-reference appendix

Manifest skeleton

{
  "bundle_version": "v7", "built_at": "...", "commit": "<git-sha>", "parent_bundle": "v6",
  "prompts": [{"name": "synth_system", "path": "prompts/synth_system.txt", "sha256": "..."}],
  "compiled_programs": [{"name": "rag_synth", "path": "artifacts/rag_synth/.../v3.json", "sha256": "..."}],
  "models": [
    {"call_site": "router",      "provider": "openai",    "snapshot": "gpt-4o-mini-2024-07-18"},
    {"call_site": "synthesizer", "provider": "openai",    "snapshot": "gpt-4o-2024-08-06"},
    {"call_site": "judge",       "provider": "anthropic", "snapshot": "claude-3-7-sonnet-20250219"}
  ],
  "retrieval": {"index_path": "indices/kb/.../<corpus-sha8>/", "embed_model": "text-embedding-3-small",
                "dim": 1536, "top_k": 8, "reranker": "bge-reranker-v2", "corpus_sha8": "a8f1c2e9"},
  "eval_set_sha": "9c3f...", "scores": {"dev": 0.81, "test": 0.79},
  "frameworks": {"dspy": "2.6.1", "python": "3.11.8"}
}

Decision tree

Preparing to deploy / "what's in prod?" ─► Step 1-3 (enumerate, pin, bundle)
   │
Score regressed, nothing committed changed? ─► Case A: confirm eval_set unchanged,
   │                                            check sample-hashes, re-bundle, gate
Prompt edited live without eval? ──────────► Case B: roll back by BUNDLE tag (atomic),
   │                                            re-land fix behind the gate
Need rollback? ─────────────────────────────► OP-6: deploy previous bundle tag
   │
Only ONE compiled prompt to version? ───────► use [[agentsop-per-model-artifacts]] instead
Need the CI comparison itself? ─────────────► use [[agentsop-regression-gate]]

Pre-deploy guards (recommended)

• no-alias-pin: fail if any manifest model id lacks a date suffix or is latest (OP-2).
• manifest-matches-live: fail deploy if live component hashes ≠ tagged bundle (Case B).
• eval-set-pinned: fail bundle if eval_set_sha missing or scores recorded without it (OP-4).
• regression-gate: re-run pinned eval set on new bundle vs parent — see [[agentsop-regression-gate]].

Key references

• DSPy save_program / (program × LM × data) triple: dspy-sop-skill/SKILL.md (R1 §1, §2).
• Per-prompt artifact lifecycle, snapshot pinning, silent drift, config-as-artifact: d-per-model-artifacts-skill/SKILL.md (R1 §3–§6).
• Registry analogy (versioning, stage transitions, reproduce): ~/.claude/skills/mlflow/SKILL.md (R1 §7).
• OpenAI deprecations: platform.openai.com/docs/deprecations
• Anthropic deprecations: docs.anthropic.com/en/docs/about-claude/model-deprecations
• This skill's evidence file: references/R1-source-evidence.md.