Langchain Incident Runbook

LangChain Incident Runbook

Overview

3:07am. PagerDuty: "LangChain p95 latency > 10s for 5 minutes." You open LangSmith, filter by service="triage-agent" over the last 15 minutes, and the first trace is 43 seconds long — an agent is on step 24 of 25 iterations, bouncing between the same two tools on a vague user prompt ("help me with my account"). The cost dashboard shows $400 spent in the last 10 minutes, up from a $6/hour baseline. This is P10: create_react_agent defaults to recursion_limit=25 with no cost cap; vague prompts never converge; the spend hits before GraphRecursionError surfaces. First move is not to push a code fix — it is to flip recursion_limit=5 via config reload and add a middleware token-budget cap per session, then deal with the stuck sessions.

Or: same alert, different signature. p95 is healthy at 1.8s, but p99 is 12s and spiky. The spikes correlate with instance starts in Cloud Run. P36: Python + LangChain + embedding preloads = 5–15s cold start; Cloud Run scales to zero by default, so first-request p99 is 10x p95. First move is --min-instances=1 (or a keepalive pinger), not more CPU.

The shape of the page decides the first move. This runbook gives you:

1. The LLM-specific SLO set most teams do not have: p95 TTFT <1s, p99 total latency <10s, error-rate <0.5%, cost-per-req <$0.05 — with Prometheus burn-rate recording rules that page on user-visible regression.
2. A triage decision tree with three root paths (latency / cost / error-rate), each with a 3-step diagnostic and first-response action.
3. Provider outage runbook wired to .with_fallbacks(backup) so failover is a config flip, not a code change.
4. Agent-loop containment via recursion_limit tuning and middleware token-budget caps so runaway agents stop burning cost before the GraphRecursionError.
5. Post-incident debug bundle (cross-ref langchain-debug-bundle) and write-up template.

Pinned: langchain-core 1.0.x, langgraph 1.0.x, langsmith 0.3+. Primary pain anchors: P10 (agent runaway), P36 (cold start). Adjacent: P29 (per-process rate limiter), P30 (max_retries=6 means 7 attempts), P31 (Anthropic cache RPM).

Prerequisites

• LangSmith workspace with tracing enabled (free tier is fine for runbook work)
• Prometheus + Alertmanager or equivalent (Datadog, Grafana Cloud) — burn-rate rules assume PromQL
• langchain-observability skill applied — metrics are grounded in what LangSmith callbacks emit
• A backup model factory from langchain-rate-limits — the failover playbook assumes .with_fallbacks() is already wired
• On-call rotation + PagerDuty (or equivalent) integrated with the alert pipeline

Instructions

Step 1 — Define the LLM-specific SLO set

HTTP-style SLOs miss what users actually feel. Define four, publish them, wire burn-rate alerts to the symptom.

Prometheus recording rule pattern for p99 latency burn-rate (replicate for TTFT, error-rate, and cost):

groups:
  - name: langchain_slo
    interval: 30s
    rules:
      - record: langchain:p99_latency_5m
        expr: histogram_quantile(0.99, sum(rate(langchain_request_duration_seconds_bucket[5m])) by (le, service))
      - alert: LangChainP99LatencyBurn
        expr: langchain:p99_latency_5m > 10
        for: 5m
        labels: { severity: page, team: llm }
        annotations:
          summary: "LangChain p99 > 10s for {{ $labels.service }}"
          runbook: "https://runbooks/langchain-incident-runbook#latency"

See LLM SLOs for the canonical set (free / paid / enterprise tiers), burn-rate recipes (fast + slow), and a TTFT-specific rule that requires streaming to be instrumented.

Step 2 — Triage decision tree: which root path?

The alert name tells you the root path. Do not mix diagnostics across paths — the first-response action differs.

Alert fired
  ├── Latency (p95/p99 breach, TTFT breach)
  │     ├── 1. Provider status page (Anthropic, OpenAI) green? → if red, Step 3
  │     ├── 2. Cold start pattern? (p99 >> p95, correlates with instance starts) → P36
  │     └── 3. Streaming configured? (TTFT only makes sense with .stream/.astream)
  │
  ├── Cost (spend/req or absolute spend/hour breach)
  │     ├── 1. Agent recursion depth? (LangSmith: max steps per trace) → P10
  │     ├── 2. Retry rate elevated? (callback log: attempt count / logical call) → P30
  │     └── 3. Token-use per req regression? (input + output tokens from callbacks)
  │
  └── Error rate (5xx + uncaught exceptions)
        ├── 1. Provider 429/500 spike? (distinguish client 4xx from provider 5xx)
        ├── 2. Auth? (API key rotation, expired token, org quota exhausted)
        └── 3. Schema drift on structured output? (Pydantic ValidationError in traces)

For each leaf, Latency Triage and Cost Overrun Response give the LangSmith filter query, the exact metric to inspect, and the remediation.

Step 3 — Provider outage: detect, circuit-break, fail over

Detection precedes failover. Do not flip fallbacks on an application bug.

1. Detect via three signals — all three should agree before declaring a provider outage:
   • Vendor status page watcher (status.anthropic.com, status.openai.com) — poll every 30s, surface into Slack
   • In-app canary probe — a 1-req/min call to each configured provider with a trivial prompt, tracked as a separate SLO
   • Error-rate spike on the primary provider in your own metrics (distinguishes a real outage from your app's bug)
2. Circuit-break the primary: a CircuitBreaker middleware (see langchain-middleware-patterns if available, or a simple aiobreaker-backed runnable) opens after N consecutive APIError / APITimeoutError within a window. Once open, calls skip the primary and go straight to the backup. This bounds the latency cost of a down provider.
3. Fail over via .with_fallbacks(backup) — the fallback chain is already wired (see langchain-rate-limits). During an outage, either flip a feature flag that swaps the default factory, or temporarily set the primary's max_retries=0 so the chain reaches the fallback immediately.
4. Comms: post a user-facing status page entry ("Degraded performance on feature X — monitoring upstream provider") and an internal Slack update with the canary graph attached. Provider Outage Playbook has the full comms template and the circuit-breaker middleware snippet.

Step 4 — Agent loop containment: stop the bleed before GraphRecursionError

P10 is the most common cost-spike cause. create_react_agent defaults to recursion_limit=25, meaning 25 model calls per user turn — with Claude Sonnet at ~$3/MTok input, a 10k-token tool-call loop burns real money per minute.

Three containment layers, applied in order:

1. Set recursion_limit per agent depth — interactive chat agents rarely need more than 5–8 steps; background research agents can justify 15; never leave the default 25 in production.

   from langgraph.prebuilt import create_react_agent
   
   agent = create_react_agent(
       llm, tools,
       recursion_limit=8,  # P10 — was default 25
   )

2. Middleware token-budget cap per session — a callback that tracks cumulative input + output tokens for a session id and raises a custom BudgetExceeded exception once the cap is hit. The agent terminates cleanly; the user sees a polite "I could not finish this task in budget, try rephrasing" instead of a spinning UI until GraphRecursionError.

3. Circuit on repeated tool calls — a LangGraph edge that routes to END when the same tool has been called with the same args twice in a row. This is a cheap heuristic for "the agent is stuck in a loop."

Cost Overrun Response has the middleware implementation, the repeat-tool edge pattern, and a per-tenant budget enforcement example.