Agentsop Observability Setup

Observability Setup — Which Backend + One-Line Autolog

"Instrument before you need it. The cheapest debugging session is a trace you already have."

This is an ENHANCE overlay. The local skills [[langsmith]], [[phoenix]] and [[mlflow]] each teach one backend deeply. This skill sits one level up: it answers the question those skills cannot — which one, and how do I turn it on in a single line right now — then hands off to the chosen backend's own skill for depth.

1. 何时激活 (When to activate)

Activate at one of three moments — earlier is always cheaper:

Do NOT activate when:

• You have already chosen a backend and just need its API — defer to [[langsmith]] / [[phoenix]] / [[mlflow]] directly.
• The task is classic ML experiment tracking with no LLM calls — that is plain MLflow, no decision needed.
• You only need to read one rendered prompt right now — that is [[agentsop-prompt-history-inspect]] (the cheaper first move).

The relationship to [[agentsop-prompt-history-inspect]]: that skill is the reactive first move (dump one prompt, no setup). This skill is the proactive layer — wire persistent tracing so the next "why" is answered by a trace that already exists, not a frantic re-run.

2. 核心心智模型 (Core mental model)

   Instrument BEFORE you need it
   ───────────────────────────────────────────────
   pick backend      one-line          verify         add eval
   by constraints →  autolog turns  →  a trace    →   hooks (judge,
   (stack/scale/      it on            actually        datasets,
    budget)           (~1 line)        landed          alerts)
   ───────────────────────────────────────────────
   cost of skipping each stage compounds: a missing trace at deploy
   becomes a multi-hour blind-debug later (AP-15, §6).

Three load-bearing ideas:

1. Choosing is the hard part, not wiring. Every modern backend offers ~one-line autolog (langsmith env vars, px.launch_app(), mlflow.<framework>.autolog()). The real cost is picking the one that won't lock you in or under-serve you at scale (§4 decision table, §5 cases).
2. The cheapest debugging is a trace you already have. A trace recorded proactively costs near-zero; reconstructing one after a production bug costs hours and may be impossible (no repro).
3. One backend, owned end-to-end. Don't wire two trace backends "to be safe" — pick one, standardize on it, and let OpenTelemetry GenAI semantics keep you portable if you outgrow it (§7).

3. SOP (Standard operating procedure)

A five-step path. Each step gates the next.

Step 1 — Pick the backend by constraints

Run the decision table in §4 (OP-1). Inputs: stack (is it LangChain/LangGraph? OSS-only? already on MLflow?), scale (dev-only vs high-volume production), budget/hosting (managed-OK vs must-self-host). Output: exactly one backend.

Step 2 — One-line autolog

Wire the single call/env-var for the chosen backend (OP-2). Resist building a custom tracing layer first — autolog gets you a trace today; you can refine later.

Step 3 — Verify traces actually land

Run one real LM call, then open the trace UI and confirm the call appears with inputs, outputs, latency, and token counts (OP-3). A backend that is "configured" but shows no traces is the #1 silent failure (wrong project name, env var not exported, sampling at 0).

Step 4 — Add eval hooks

Once raw traces flow, attach what makes them actionable: an LLM-as-judge or rule evaluator, a dataset built from real traces for regression testing, and basic cost/latency alerts (OP-4). This is what turns "we have logs" into "we catch regressions before users do."

Step 5 — Hand off to the backend skill

Hand off to the chosen backend's own skill ([[langsmith]] / [[phoenix]] / [[mlflow]]) for the deep API once tracing and evals are flowing (OP-6).

Step gating, explicitly. Do not start Step n+1 until Step n is observably true. The most common failure is jumping from Step 2 (wired) straight to Step 4 (evals) without Step 3 (verify) — you build evaluators on top of a trace stream that was silently empty the whole time. Each step has a one-line proof: Step 1 → a backend name written down; Step 2 → an autolog call in the code; Step 3 → one trace visible in the UI with token counts; Step 4 → one evaluator firing on that trace. If any proof is missing, you are not at that step yet.

4. 操作模型 (Operational model — Trigger / Action / Output / Evidence)

OP-1 · Backend decision table (the core of this skill)

• Trigger: Stage 1 — you have an LM project and no backend chosen yet.
• Action: Match your dominant constraint to a row. First matching row wins; ties broken by the "tie-breaker" column.

• Output: Exactly one backend name + its skill to hand off to.
• Evidence: backend frontmatter in [[langsmith]]/[[phoenix]]/[[mlflow]]; langgraph-sop AP-15; llamaindex-sop "LangSmith / Phoenix for observability"; dspy-sop MLflow + LangFuse lines. See references/R1-source-evidence.md.

OP-2 · One-line autolog per backend

• Trigger: Stage 2 — backend chosen, turn it on.
• Action: Use the minimal turn-on for the chosen backend:

  # LangSmith — env vars only, no code change (auto-traces LangChain/LangGraph):
  #   export LANGSMITH_TRACING=true
  #   export LANGSMITH_API_KEY=...   export LANGSMITH_PROJECT=my-app
  # (raw SDK / non-LangChain: wrap calls with langsmith.wrappers.wrap_openai or @traceable)
  
  # Phoenix — launch local app + auto-instrument:
  import phoenix as px; px.launch_app()
  from phoenix.otel import register
  register(auto_instrument=True)        # picks up installed framework instrumentors
  
  # MLflow — one autolog call per framework:
  import mlflow
  mlflow.set_tracking_uri("http://localhost:5000")
  mlflow.dspy.autolog()                 # or mlflow.langchain / mlflow.openai / mlflow.crewai

• Output: Traces start flowing on the next LM call — no per-call instrumentation written by hand.
• Evidence: [[langsmith]]/[[phoenix]]/[[mlflow]] quick-start sections; dspy-sop mlflow.dspy.log_model deploy line; prompt-history-inspect op-001 (mlflow.dspy.autolog).

OP-3 · Verify a trace actually landed

• Trigger: Stage 3 — autolog wired, before trusting it.
• Action: Run one real call, open the UI (LangSmith project / px localhost / MLflow :5000), confirm the run shows inputs + outputs + latency + token count. If empty: check project name, that the env var is exported in this process, and that sampling ≠ 0.
• Output: One verified trace = the backend is real, not just configured.
• Evidence: AP "configured-but-empty" is the most common silent failure; verify-before-trust is standard SOP (references/R2).

OP-4 · Add eval hooks on top of raw traces

• Trigger: Stage 4 — traces flow; make them actionable.
• Action: Attach (a) an evaluator — LLM-as-judge (Phoenix) or dataset evaluators (LangSmith) or logged metrics (MLflow); (b) a dataset built from real traces for regression testing; (c) cost/latency alerts. Defer the deep API to the chosen backend's skill.
• Output: Traces become a regression-catching system, not just a log.
• Evidence: [[langsmith]]/[[phoenix]] evaluation sections; llamaindex-sop "eval loop is what surfaces failures"; dspy-sop metric pattern.