Langchain Cost Tuning

LangChain Cost Tuning (Python)

Overview

An engineer shipped a new research agent Tuesday. By Friday the Anthropic bill had grown 6x while traffic grew 1.4x. The cost dashboard — wired to on_llm_end — showed spend up maybe 2x. Reconciling against the provider console on Monday surfaced two compounding bugs: (1) the agent's ChatOpenAI fallback kept the default max_retries=6, so each logical call billed as up to 7 requests (P30); (2) retry middleware was registered below token accounting, so every retry fired on_llm_end twice — the aggregator summed both emissions while LangSmith deduped them by generation ID, undercounting the dashboard by ~50% against actual billed rate (P25).

The fix took an afternoon: cap retries at 2, tag retries with a stable request_id, and migrate token accounting to AIMessage.usage_metadata read from astream_events(version="v2"). Finding the bug took a week. This skill is that week compressed into a runbook.

Cost tuning for a LangChain 1.0 production app has five levers, each with a sharp failure mode:

• Token accounting — on_llm_end lags streams by 5-30s (P01); retries double-count (P25); Anthropic cache savings aggregate per-call, never per-session (P04).
• Retry discipline — max_retries=6 default on ChatOpenAI (P30); Anthropic 50 RPM tier throttles cached and uncached calls against the same budget (P31).
• Agent loop caps — create_react_agent defaults to recursion_limit=25; vague prompts burn a session's budget before GraphRecursionError surfaces (P10).
• Caching — InMemoryCache ignores bound tools in the cache key and returns wrong answers (P61); RedisSemanticCache ships with a 0.95 threshold that hits <5% of the time (P62).
• Model tiering — Running claude-opus-4-5 on intent classification is 30-60x more expensive than claude-haiku-4-5 for a task the cheaper model solves at equal quality.

Pin: langchain-core 1.0.x, langchain-anthropic 1.0.x, langchain-openai 1.0.x. Pain-catalog anchors: P01, P04, P10, P23, P25, P30, P31, P61, P62.

Prerequisites

• Python 3.10+
• langchain-core >= 1.0, < 2.0
• At least one provider package: pip install langchain-anthropic langchain-openai
• redis-py >= 5.0 for budget middleware (optional; in-process dict works for dev)
• Provider console access (Anthropic, OpenAI) to reconcile usage_metadata against billed spend — you will need this to verify any instrumentation fix

Instructions

Step 1 — Read usage_metadata, never response_metadata["token_usage"]

LangChain 1.0 standardizes all provider usage into AIMessage.usage_metadata. response_metadata["token_usage"] still exists as a compatibility shim but its shape is provider-specific (Anthropic nests under usage, OpenAI flat, Gemini uses different keys). Code that reads it directly will break when you switch providers or when a provider SDK upgrades.

from langchain_core.messages import AIMessage

def read_usage(msg: AIMessage) -> dict:
    """Canonical shape: input_tokens, output_tokens, input_token_details,
    output_token_details. Safe across Anthropic, OpenAI, Gemini."""
    meta = msg.usage_metadata or {}
    details_in = meta.get("input_token_details", {}) or {}
    details_out = meta.get("output_token_details", {}) or {}
    return {
        "input": meta.get("input_tokens", 0),
        "output": meta.get("output_tokens", 0),
        "cache_read": details_in.get("cache_read", 0),       # Anthropic
        "cache_creation": details_in.get("cache_creation", 0),
        "reasoning": details_out.get("reasoning", 0),        # OpenAI o1/o3
    }

Include reasoning in your output-billable total for o1/o3. A call with output_tokens=500 and reasoning=2000 actually bills 2500 output tokens.

Step 2 — Stream-accurate aggregation via astream_events(version="v2")

on_llm_end fires once after the stream closes, so dashboards lag by stream duration (P01). Anthropic populates usage_metadata on the message_start and message_delta events; OpenAI populates only the final chunk. Both show up as on_chat_model_stream events in astream_events.

async def metered_invoke(chain, inputs, meter):
    async for event in chain.astream_events(inputs, version="v2"):
        if event["event"] == "on_chat_model_stream":
            chunk = event["data"]["chunk"]
            if getattr(chunk, "usage_metadata", None):
                meter.record(
                    run_id=event["run_id"],
                    usage=chunk.usage_metadata,
                )

See Token Accounting Pitfalls for the full streaming-delta behavior across providers and reconciliation against provider dashboards.

Step 3 — Dedup retries on run_id, not prompt hash

Retry middleware runs the model twice on transient errors. Both emit usage events. If the aggregator keys on prompt hash, it looks like one call cost twice as much. If it keys on run_id (LangChain assigns one per generation attempt), you can attach a stable request_id at the chain level and dedupe on that (P25).

from uuid import uuid4

class RetryAwareMeter:
    def __init__(self):
        self._seen: set[str] = set()
        self.totals = {"input": 0, "output": 0, "cache_read": 0}

    def record(self, run_id: str, usage: dict, request_id: str | None = None):
        # Keep only the last emission per logical request.
        # On retry: same request_id, different run_id -> overwrite.
        key = request_id or run_id
        if key in self._seen:
            # Retry emission — subtract prior, add new (last wins).
            prior = self._prior_by_key.get(key, {})
            for k in self.totals:
                self.totals[k] -= prior.get(k, 0)
        self._seen.add(key)
        self._prior_by_key[key] = usage
        self.totals["input"]  += usage.get("input_tokens", 0)
        self.totals["output"] += usage.get("output_tokens", 0)
        details = usage.get("input_token_details", {}) or {}
        self.totals["cache_read"] += details.get("cache_read", 0)

Inject request_id via config={"metadata": {"request_id": str(uuid4())}} on each invoke. The meter reads event["metadata"]["request_id"] alongside run_id.

Alternative: place token accounting above retry middleware in the chain — retries happen inside, so only the successful attempt emits. This is simpler but makes retries invisible to observability, which you usually want to see.

Step 4 — Model tiering: draft cheap, finalize expensive

Most chains have a structural split: a cheap "understand the request" call and an expensive "produce the final artifact" call. Running the expensive model on both roughly triples cost for no quality gain.

Per-1M pricing snapshot, 2026-04 (verify current prices before shipping at https://www.anthropic.com/pricing and https://openai.com/api/pricing/):

Anthropic cache reads cost 10% of input. Cache creation costs 125% of input. Break-even is ~4 uses of a cached prefix. See Cache Economics.

Decision tree:

input
  └── intent classification / routing
      └── gpt-4o-mini OR claude-haiku-4-5        (~$0.15-$1 per 1M in)
  └── generation / reasoning
      ├── single-pass, low-stakes
      │   └── gpt-4o-mini                         (draft)
      ├── single-pass, high-stakes (extraction, contracts)
      │   └── claude-sonnet-4-6                   (finalize)
      ├── multi-step reasoning
      │   └── gpt-o3-mini OR claude-sonnet-4-6    (plan)
      └── mission-critical long-horizon
          └── claude-opus-4-5                     (expensive, used sparingly)

Tiering is wrong when quality degrades silently — high-stakes extraction on Haiku misses entities the Sonnet would catch. Always evaluate both tiers on a gold set before committing. See Model Tiering for the evaluation harness and a worked draft-then-finalize chain.

Step 5 — Aggregate Anthropic cache usage per session and tenant (P04)

usage_metadata["input_token_details"]["cache_read"] reports per-call. To see whether caching is paying for itself you need to aggregate per-session or per-tenant and compare against cache-creation cost.

class CacheLedger:
    def __init__(self, tenant_id: str):
        self.tenant_id = tenant_id
        self.read = 0           # billed at 0.1x input rate
        self.creation = 0       # billed at 1.25x input rate
        self.uncached_input = 0 # billed at 1.0x input rate

    def ingest(self, usage: dict):
        details = usage.get("input_token_details", {}) or {}
        self.read     += details.get("cache_read", 0)
        self.creation += details.get("cache_creation", 0)
        total_input = usage.get("input_tokens", 0)
        self.uncached_input += total_input - self.read - self.creation

    def savings_vs_no_cache(self, price_per_1m_input: float) -> float:
        # What we paid with cache vs. paying full price on all input.
        actual = (self.uncached_input * 1.00
                  + self.creation      * 1.25
                  + self.read          * 0.10) * price_per_1m_input / 1_000_000
        naive  = (self.uncached_input + self.creation + self.read) * price_per_1m_input / 1_000_000
        return naive - actual

Persist CacheLedger to Redis or Postgres keyed by (tenant_id, day). If savings is negative over a 24h window, caching is costing more than it saves — your cached prefix is either too short or hit too rarely. See Cache Economics.

Step 6 — Cache keys must include bound tools (P61)

set_llm_cache(InMemoryCache()) hashes the prompt string only. A chain that binds different tool sets will return wrong answers from the cache. This is the most dangerous cache failure mode — it silently returns semantically incorrect responses rather than missing.

Do not use InMemoryCache on any chain that calls bind_tools(). Use SQLiteCache or RedisSemanticCache with a composite key:

import hashlib, json

def tool_aware_key(prompt: str, tools: list) -> str:
    tools_fingerprint = hashlib.sha256(
        json.dumps([t.args_schema.model_json_schema() for t in tools],
                   sort_keys=True).encode()
    ).hexdigest()[:16]
    return f"{tools_fingerprint}:{hashlib.sha256(prompt.encode()).hexdigest()}"

Cross-reference: langchain-middleware-patterns covers cache-key layering in middleware order (redact → cache → model) to avoid cross-tenant PII leaks.

Step 7 — Tune semantic-cache threshold; ship a calibrated value, not the default (P62)

RedisSemanticCache defaults to score_threshold=0.95. On real workloads this hits under 5% of the time. Production-tuned values land between 0.85 and 0.90. Ship with a calibrated threshold, not the default:

1. Collect 200 real query pairs labeled "should return same answer" (positive) and "should return different answer" (negative).
2. Embed both sides of each pair with your embedding model.
3. For thresholds in [0.80, 0.82, 0.84, …, 0.95], compute:
   • hit rate (% positive pairs above threshold)
   • false positive rate (% negative pairs above threshold)
4. Pick the lowest threshold where FPR < 2%.
5. Ship with a daily audit: sample 1% of cache hits, log to review queue.

If the curve flattens above 0.92 on positives, your embeddings are too weak for semantic caching — consider exact-match SQLiteCache instead. See Cache Economics for the calibration worksheet.