Agentsop Per Model Artifacts

Case C — "LlamaIndex ServiceContext deprecation: which artifacts to migrate"

困境: Team upgrades from LlamaIndex 0.9.x to 0.11.x. ServiceContext(llm=..., embed_model=...) is deprecated in favor of Settings.llm = ... / Settings.embed_model = ... (LlamaIndex SKILL §4.4). There are 14 indices in production, each baked with various embed models including text-embedding-ada-002 (deprecated) and text-embedding-3-small (current). What needs to be touched?

约束:

• Code-level migration: ServiceContext → Settings is mechanical, ~1 hour.
• Index-level migration: re-embedding the largest corpus (4M docs, text-embedding-3-small) is ~$200 and 6 hours.
• Indices on text-embedding-ada-002 MUST be re-embedded — that model is going away (R1 §3).
• Mixing embedding spaces silently is the worst-case failure (LlamaIndex SKILL anti-pattern A2).

决策步骤:

1. List every index and its manifest (per R2 Recipe 7). If any index lacks a manifest.json with embed_model and dim, treat as unknown — must re-embed defensively.
2. Bucket indices by embed model:
   • On deprecated embed (text-embedding-ada-002): must re-embed. New artifact path indices/<name>/openai__text-embedding-3-small__1536/<new-corpus-sha8>/.
   • On current embed: code-only migration (ServiceContext → Settings). Index artifact unchanged. Update manifest with new llama_index_version.
3. Refuse to start the new app if Settings.embed_model at runtime doesn't match a loaded index's manifest.embed_model (R2 Recipe 7 loader guard). This converts a silent data-corruption bug into a startup error.
4. For each re-embedded index, write a NEW manifest with a NEW corpus_sha8 even if the source corpus is unchanged — different model = different artifact, period.
5. Keep the old index files for 30 days under indices/<name>/openai__text-embedding-ada-002__1536/_archived/. Roll back if production answers degrade. (Anti-pattern: deleting the old index "to save space" before validating the new one.)
6. Tag the new app version only after at least one rep query per re-embedded index passes a regression test (top-3 results overlap with golden set ≥80%).

结果: 9 of 14 indices needed code-only migration; 5 needed re-embed. Total cost ~$300 and 1 engineer-day. The discipline of per-index manifests caught two indices that had been silently broken (ada-002 query vector against 3-small index from a botched earlier migration).

可提取的操作: Framework deprecations cascade into per-artifact decisions. The "easy" code migration is rarely the full migration — interrogate every artifact's framework binding (embed model, edit format, tokenizer version) before declaring a deprecation handled.

6. 反模式与边界 (Anti-patterns & boundaries)

Anti-patterns

1. Assuming portability across models. A prompt.txt or compiled.json with no model in its path or metadata is implicitly "for any model." Empirically, prompts transfer down (big → small) badly and sideways unpredictably (R1 §5). The artifact must encode its LM.

2. No model in the artifact name/path. artifacts/v3.json is unauditable. artifacts/rag_synth/openai/gpt-4o-2024-08-06/devset-v3-a8f1c2e9/v3.json is. Long paths are good; they replace metadata files no one reads.

3. Pinning to an alias instead of a snapshot. gpt-4o, claude-3-5-sonnet, llama-3-8b-instruct — all moving targets. Pin gpt-4o-2024-08-06, claude-3-5-sonnet-20241022, and an HF revision SHA respectively.

4. No regression gate on artifact PRs. Every artifact replacement is implicitly a behavior change. CI must re-run held-out test and compare to parent artifact. Without this, regressions land silently and surface in production.

5. Hand-editing compiled JSON. "Just one little instruction tweak" invalidates the metric-optimality the artifact carries. Either re-compile or don't touch. Pre-commit hook should enforce.

6. Skipping the parent_artifact pointer. Without lineage, you cannot answer "what was the last known-good artifact" during an incident. Every REGISTRY entry must reference its predecessor (or null for the very first).

7. Single artifact, multiple LMs. Production hits two LMs (e.g. failover routing GPT-4o ↔ Sonnet 3.5). Each LM needs its OWN artifact, with its own swap-tested score. A "shared" artifact biased toward whichever LM compiled it.

8. No dataset hash. Two "v3 devset" files that differ by 50 examples produce different artifacts that look identical in the registry. dataset_sha256_8 in the path makes drift visible.

9. Forgetting framework version. A DSPy 2.4 artifact loaded by DSPy 2.6 may silently behave differently due to module-level changes. dspy_version in REGISTRY enables targeted rollback.

10. Deleting old artifacts on new compile. Disk is cheap, history is precious. Keep old artifacts checked in. Production deploys read git tags, not HEAD — the old artifact stays reachable for rollback.

11. Treating LangChain Hub as the lifecycle. Hub is a viewer + diff tool. It doesn't bind to model, dataset, or run a regression gate. Pair with REGISTRY.jsonl or MLflow (R1 §8).

12. Promoting an artifact without swap-test on the actual production LM. "It scored well in compile" — but compile was against the same LM that production uses, right? Not always: the optimizer LM can differ from the task LM (dspy-sop §4.4). Swap-test against the actual production LM before tagging.

Boundaries (when this SOP is overkill)

• One artifact, one LM, never changes. A research demo or proof-of-concept can use compiled.save("v1.json") flat. Adopt this SOP when adding the second artifact or moving to a managed environment.
• Pure inference-time prompts with no compile loop. A system prompt manually authored and unchanging is just a config file — version it normally, but the regression gate and swap-test machinery are not load-bearing until the prompt is metric-optimized.
• Provider with built-in artifact registry. OpenAI's Prompt Library, Anthropic's Workbench saved prompts, vendor-managed RAG — let the vendor own the lifecycle. This SOP is for org-controlled artifacts.
• Framework changes more often than the LM. If you re-write the program every week, the compile artifact has no shelf life and the registry overhead exceeds the audit value. Stabilize the program first.

7. 跨框架对照 (Cross-framework comparison)

How four ecosystems handle the per-model artifact problem.

DSPy — save_program is the artifact

• What it stores: instructions per predictor, bootstrapped demos, signature shapes (for whole-program save).
• What it does NOT store: the LM identity, the dataset, the metric, the run cost. These must be added by the SOP (REGISTRY.jsonl).
• Two save modes:
  • compiled.save("v1.json") — state JSON, needs source Python at load. Use for code-review-friendly diffs.
  • compiled.save("v1/", save_program=True) — pickled program + state. Use for portability across source-tree refactors.
• Reload: program_cls().load("v1.json") requires the Python class import; dspy.load("v1/") for whole-program.
• Versioning practice: DSPy docs leave it to the user. The dspy-sop SKILL Case B says "Treat the compiled program as a (program × LM) pair" but doesn't prescribe directory layout — this SOP fills that gap.

LangChain Hub — versioned text templates

• What it stores: prompt template text, partial variables, model name as a hint (not enforced).
• What it does NOT store: evaluation scores, dataset binding, recompile lineage.
• API:

  from langchain import hub
  hub.push("user/rag-qa", prompt, parent_commit_hash=last)
  hub.pull("user/rag-qa:v3")

• Versioning practice: Hub gives you a UI diff between versions but not a regression gate or model-pinning enforcement. Use Hub as the viewer; keep your local REGISTRY.jsonl as the gate.
• Limitation: Hub is org-shared; per-model variants pollute the namespace unless you name them rag-qa-gpt4o / rag-qa-sonnet35.

Manual jsonl + git — lightweight, fully local

• What it stores: whatever you put in REGISTRY.jsonl. Recipe 1 schema (R2) covers model, dataset, metric, cost, lineage.
• Strengths: zero infra, fully grep-able, git-blame-able. Works for solo developers and small teams.
• Limitations: no UI, no multi-tenant access control, no eval-vs-baseline comparison out of the box (must script). Scales to ~50 artifacts before the jsonl gets unwieldy.
• Decision rule: start here. Migrate to MLflow when you exceed 50 artifacts OR have >5 contributing developers OR need a stage-promotion UI.

MLflow Model Registry — org-scale

• What it stores: model artifact (pickled DSPy program via mlflow.dspy.log_model), full param/metric history per run, signature, version with explicit Stage labels (None / Staging / Production / Archived).
• What it does NOT store natively: dataset hash binding (must add as a param), provider-snapshot-deprecation calendar (add via tags).
• API:

  with mlflow.start_run():
      mlflow.log_params({"model": MODEL, "dataset_sha8": SHA8})
      mlflow.log_metrics({"dev_score": d, "test_score": t})
      mlflow.dspy.log_model(compiled, "program",
                            registered_model_name=f"rag_synth-{MODEL}")
  client.transition_model_version_stage(
      name=f"rag_synth-{MODEL}", version=v, stage="Production")

• Strengths: UI, RBAC, multi-team. Promotion via stage labels replaces git tags.
• Limitations: infra overhead (Postgres + artifact store). Overkill for solo.

Aider model defaults — config artifacts, not data artifacts

• What it stores: .aider.conf.yml per-model edit format, params (R2 Recipe 6).
• Versioning practice: This SOP applied to config: pre-commit hook + bench delta gate on changes.
• Lesson: "artifact" generalizes beyond compiled JSON. Aider's edit-format choice IS a per-model artifact even though it's a string in YAML. Treat it like one.

LlamaIndex index files — embedding-bound artifacts

• What it stores: vector store, doc store, index store — all bound to one specific embedding model + dimension.
• Versioning practice: R2 Recipe 7. Per-index manifest.json with embed_model, dim, corpus_sha8. Loader refuses to query if Settings.embed_model at runtime mismatches manifest.
• Lesson: embedding model is part of index identity, identical in structure to LM being part of compiled-prompt identity. Same SOP applies.

Summary table

Quick-reference appendix

Filesystem layout (canonical)

artifacts/
├── REGISTRY.jsonl
└── <program>/
    └── <provider>/
        └── <model-snapshot>/        # e.g. gpt-4o-2024-08-06
            └── <dataset>-<sha8>/    # e.g. devset-v3-a8f1c2e9
                ├── v1.json
                ├── v2.json
                └── v3.json

REGISTRY.jsonl one-liner

{"path":"...","sha256":"...","program":"...","provider":"...","model":"...","dataset":"...","dataset_sha256_8":"...","optimizer":"...","optimizer_config":{...},"metric":"...","dev_score":0.0,"test_score":0.0,"cost_usd":0.0,"wall_seconds":0,"dspy_version":"...","compiled_at":"...","commit":"...","parent_artifact":null}

Pre-commit hooks (required)

• no-edit-artifact: rejects any change under artifacts/** that doesn't also append to REGISTRY.jsonl.
• no-alias-pin: scans Python source for model="gpt-4o", model="claude-3-5-sonnet" etc. with no date suffix; fails.
• no-deprecated-pin: scans for snapshots within 90 days of deprecation per scripts/deprecation_table.py; fails without override label.

CI gates (required)

• artifact-regression: re-run held-out test on new artifact + parent; fail if Δ < −2 points.
• swap-test-on-snapshot-change: when a new provider snapshot is announced (via webhook or scheduled scan), open a PR running OP-4 against affected artifacts.

Decision tree

New compile? ──► Stage 1-3 (compile, version, gate)
   │
New snapshot from provider? ──► Stage 4 swap-test
   │   ├── |Δ| ≤ 2 pts ──► tag transferable; ship
   │   └── |Δ| > 2 pts ──► Stage 5 recompile
   │
Framework deprecation? ──► R2 Recipe 7 pattern: list artifacts, bucket by binding, migrate per-artifact
   │
Canary score drift on stable snapshot? ──► Case B path: recompile against same snapshot
   │
Family swap (GPT-4o ↔ Sonnet)? ──► Stage 5 recompile, always; swap-test gives the magnitude, not the decision

Key references

• DSPy compiled-prompt model dependency: /Users/5imp1ex/Desktop/Skill-Workplace/output/dspy-sop-skill/SKILL.md Dilemma Case B.
• Aider per-model edit format defaults: /Users/5imp1ex/Desktop/Skill-Workplace/output/aider-sop-skill/SKILL.md §"edit format" table.
• LlamaIndex ServiceContext → Settings deprecation: /Users/5imp1ex/Desktop/Skill-Workplace/output/llamaindex-sop-skill/SKILL.md §4.4 / anti-pattern A4.
• DSPy save_program API: dspy.ai/tutorials/saving/.
• MLflow Model Registry: mlflow.org/docs/latest/model-registry.html.
• OpenAI deprecation policy: platform.openai.com/docs/deprecations.
• Anthropic model deprecations: docs.anthropic.com/en/docs/about-claude/model-deprecations.
• This SOP's reference files: references/R1-source-evidence.md, references/R2-versioning-recipes.md.