LLM Pipeline

llm-pipeline

Wire multiple LLM calls into a reliable, controllable production pipeline. You chain steps where one call's validated output feeds the next, put a router in front of providers so an outage fails over instead of taking you down, and engineer the cross-cutting concerns: timeouts, bounded retries, fallbacks, caching, and cost caps.

Treat the LLM as an unreliable network dependency, not a local function call. Every rule below follows from that: providers have outages, rate limits, and latency tails, so no single provider is a single point of failure and no call is allowed to run unbounded.

Do you even need a pipeline?

This skill is the orchestration around calls. If you only have one call, you are in the wrong place.

A pipeline is a DAG you designed. The moment the model picks its own next step, it is an agent — go build that instead.

Design the chain as a typed DAG

Each step is a pure-ish function: (typed input) -> (typed output via structured output). Chaining small single-purpose steps beats one mega-prompt — reported ~20% output-quality gain — because each step is debuggable, cacheable, and retryable in isolation.

Rules:

• The structured output of step N is the input contract of step N+1. Validate it (Pydantic / JSON Schema) at the seam. A schema-valid object that fails validation here never poisons the next call.
• Keep steps small and single-purpose. "Extract entities" and "classify sentiment" are two steps, not one prompt doing both. Smaller steps route to cheaper models and cache better.
• Mark independent steps for parallel fan-out. If step B and step C both only need step A's output, run them concurrently — see ../parallel/SKILL.md. Sequential only where there is a real data dependency.
• Tag each step idempotent or side-effecting. Retries and replays must be safe; a step that writes to a DB or sends an email is not safe to blindly retry.

# Bad: free text flows between steps; step 2 silently mis-parses step 1
entities = client.responses.create(model="gpt-4o", input=f"Extract entities: {doc}").output_text
summary  = client.responses.create(model="gpt-4o", input=f"Summarize for {entities}").output_text

# Good: each step emits a validated object; the seam is a contract you assert on
from pydantic import BaseModel

class Entities(BaseModel):
    people: list[str]
    orgs: list[str]

step1 = client.responses.parse(model="gpt-4o", input=f"Extract: {doc}",
                               text_format=Entities, timeout=30)
ents: Entities = step1.output_parsed          # parse fails HERE, not downstream
summary = summarize(ents)                      # typed input, not a string blob

Put a gateway/router in front

Use a router so a failure on one deployment fails over to another, and so you can swap or load-balance models from config without touching call sites.

LiteLLM is the de-facto open-source LLM gateway (current stable line v1.83.3-stable). It exposes a unified OpenAI-format completion() across 100+ providers, with built-in retry/fallback, cost tracking, and budget management — usable as a Python SDK or as a Proxy Server.

Fallback semantics worth knowing: a request to an order=1 deployment that fails (connection error, 404, 429, ...) auto-tries order=2, then order=3; each order level gets its own num_retries before escalating; exhausting orders falls through to configured fallbacks. There are specialized buckets — content_policy_fallbacks (ContentPolicyViolationError), context_window_fallbacks (ContextWindowExceededError), and default_fallbacks.

from litellm import Router

router = Router(
    model_list=[
        {"model_name": "smart",
         "litellm_params": {"model": "anthropic/claude-sonnet-4-6", "timeout": 30}},
        {"model_name": "smart-backup",
         "litellm_params": {"model": "openai/gpt-4o", "timeout": 30}},
    ],
    fallbacks=[{"smart": ["smart-backup"]}],
    context_window_fallbacks=[{"smart": ["smart-backup"]}],
    num_retries=2,            # per order level, bounded
    timeout=30,               # hard cap, never unbounded
)
resp = router.completion(model="smart",
                         messages=[{"role": "user", "content": prompt}])

Build vs buy: raw SDK + a thin retry/timeout wrapper (fewest deps, fine for one provider) → LiteLLM SDK/proxy (widest provider coverage, fallbacks for free) → hosted gateway (Bifrost/Portkey, when you want it operated for you). Full config — all fallback buckets, redis cache params, budget/rate-limit settings, cost callbacks — is in references/litellm-router.md.

Reliability mechanics

• Always set a timeout. A 30–60s hard cap per call. Without it, one slow upstream hangs the request indefinitely.
• Bounded retries with exponential backoff + jitter. LiteLLM defaults: backoff from INITIAL_RETRY_DELAY 0.2s up to MAX_RETRY_DELAY 10s, with jitter to avoid thundering herds. Never while True.
• Only retry idempotent steps. Retrying a step that sent an email or charged a card double-fires. Gate retries behind the idempotency tag from the DAG design.
• Circuit-break and degrade gracefully. When a provider is down and fallbacks are exhausted, return a partial result, a cached result, or a cheaper-model result — never hang and never 500 the user if a degraded answer exists.

# Bad: naked call in a for-loop; no timeout, unbounded effect, no fallback
for _ in range(10000):
    try:
        return client.chat.completions.create(model="gpt-4o", messages=msgs)
    except Exception:
        continue          # hammers a down provider, blows the budget, may never exit

# Good: router does bounded retries + backoff + fallback; you degrade on exhaustion
try:
    return router.completion(model="smart", messages=msgs)   # timeout + num_retries set
except Exception:
    return cached_or_cheaper_answer(msgs)                     # graceful degradation