Litellm

LiteLLM AI Gateway Operations

Use this skill to operate LiteLLM as an organization's AI gateway: run the proxy (litellm --config config.yaml), route requests to 100+ LLM providers through one OpenAI-compatible API, manage model lists, routing and reliability, virtual keys, teams, budgets and rate limits, caching, guardrails, observability, and spend — and diagnose failures with evidence. LiteLLM ships two surfaces: a Python SDK (litellm.completion(), in-process) and the proxy (a FastAPI service on port 4000 with keys, budgets, and an admin UI). This is a tool skill for the named tool. Engine selection and serving methodology belong to ml-engineering; operating a single engine belongs to vllm or llama-cpp.

Operating contract

1. Record the deployment before tuning it. Capture the pinned image or pip version, config.yaml, model list, routing, budgets, env-var references, and data stores in the proxy config record. That record is the rollback unit.
2. Confirm the target, scope, and rollback path before mutating. Read-only discovery (health probes, /v1/models, logs, spend queries) may proceed without confirmation. Mutations — config changes, key mint/revocation, restarts, image upgrades, DB migrations — require an explicit human directive naming the deployment.
3. A proxy that responds is not a proxy that serves. /health/liveliness returning 200 proves liveness only. Verify at the delivery boundary: a representative /v1/chat/completions request returns tokens and x-litellm-model-id names the deployment you expected.
4. Keep evidence bounded. Summarize logs and configs; never dump full logs, .env contents, master keys, or provider credentials into chat. Spend logs and debug output can contain prompt content — redact before sharing.
5. Pin versions. LiteLLM releases weekly and changes defaults; every claim here was checked against 1.97.0 (2026-08-22). Re-verify version-sensitive behavior against your installed release before relying on it.

The litellm-health script

scripts/litellm-health is a read-only probe for a running proxy. It issues GET requests only, never writes files, and emits bounded output.

scripts/litellm-health --help                                   # no proxy needed
scripts/litellm-health --url http://127.0.0.1:4000 --json
scripts/litellm-health --check health --check readiness --json
scripts/litellm-health --check models --check model_info \
  --key "$LITELLM_MASTER_KEY" --json

Exit codes: 0 all checks passed, 1 issues found or a fatal error, 2 usage error, 124 timeout. Checks: health (GET /health/liveliness, unauthenticated), readiness (GET /health/readiness, unauthenticated; 503 when the configured DB is unreachable), models (GET /v1/models, requires key), and model_info (GET /model/info, requires key). Keys are sent as Authorization: Bearer <key>. The script never sends data anywhere except the proxy you name.

Operating loop

1. Identify the deployment: pinned version/image digest, how it runs (bare, Docker, Compose, Helm), config source (file, store_model_in_db, or both), and data stores (Postgres? Redis?).
2. Collect evidence: litellm-health --json; GET /v1/models and /model/info with a key; response headers (x-litellm-call-id, x-litellm-model-id, x-litellm-model-api-base, x-litellm-version); --detailed_debug logs or LITELLM_LOG=DEBUG for the outbound request.
3. Triage against the symptom: classify provider vs gateway errors (see troubleshooting); check cooldown state, budgets, DB connectivity.
4. Act with confirmation: bounded, scoped changes after a human directive, with the rollback path named first.
5. Verify: re-run the probe and a representative chat request at the delivery boundary.

Quickstart: one config, many providers

model_list:
  - model_name: gpt-4o                     # name clients request
    litellm_params:
      model: openai/gpt-4o                 # routed string (provider prefix required)
      api_key: os.environ/OPENAI_API_KEY   # resolved inside the proxy process
  - model_name: claude-sonnet
    litellm_params:
      model: anthropic/claude-sonnet-4-5
      api_key: os.environ/ANTHROPIC_API_KEY

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY   # require auth on every call

Start with litellm --config config.yaml --port 4000. Success logs Proxy initialized with Config, Set models:. Clients call the OpenAI surface: /v1/chat/completions, /chat/completions, /v1/embeddings, /v1/images/generations, /v1/audio/transcriptions, plus /responses, Anthropic-compatible /messages, /model/info, /health/liveliness, /health/readiness. Any OpenAI SDK works unchanged: openai.OpenAI(base_url="http://localhost:4000", api_key=<virtual key>). Details and the SDK surface: quickstart reference.

Config and routing

• Entries sharing a model_name form one load-balanced group; each entry is a deployment with its own hashed model_id used for health and cooldown tracking.
• router_settings.routing_strategy — simple-shuffle (default, recommended; weighted by rpm/tpm or weight under litellm_params), least-busy, latency-based-routing, usage-based-routing (docs warn against it in prod), cost-based-routing.
• Reliability: litellm_settings.num_retries (per-deployment and request-level overrides exist; num_retries is not the provider SDK's max_retries), fallbacks / context_window_fallbacks / content_policy_fallbacks, cooldowns (allowed_fails, cooldown_time), deployment order for priority, enable_pre_call_checks: true to enforce context windows and region filters pre-call (opt-in).
• With store_model_in_db: true, UI/API writes deep-merge over YAML in Postgres and win on key conflicts — editing those YAML keys later has no effect while the DB row exists. Details: config and routing reference.

Keys, teams, budgets, spend

• general_settings.master_key (must start sk-) is the admin credential and UI password. Virtual keys (POST /key/generate) scope models, budgets, and rpm/tpm per workload; keys are stored hashed and never contain provider credentials.
• Budgets require Postgres. Without a connected DB, budgets fail open (a startup warning is the only signal) and key endpoints return No connected db. — never run a budget-sensitive deployment DB-less.
• Team keys enforce team (+ team-member) budgets only; the owner's personal budget does not apply. Rate limits do not apply to proxy admins. Spend lands in /spend/logs and /global/spend; store_prompts_in_spend_logs defaults to false. Details: keys and budgets reference.

Caching and guardrails

• Response cache: litellm_settings.cache: true + cache_params.type: redis for multi-instance production (in-memory is per-process; disk/S3/GCS exist). Per-request controls: cache: {ttl, no-cache, namespace} in the body.
• Semantic caches (qdrant-semantic, redis-semantic, valkey-semantic) embed the whole messages array and can replay stale answers across similar multi-turn turns — docs recommend excluding agentic traffic from semantic caching.
• Guardrails run pre_call, post_call, during_call, or logging_only (there is no all mode); Presidio PII masking is OSS. Violations fail with HTTP 400 and an embedded verdict; x-litellm-applied-guardrails names what ran. Details: caching and guardrails reference.

Observability and logging

• Callbacks: litellm_settings.success_callback / failure_callback / callbacks (Langfuse, OTel, Prometheus, Datadog, Sentry, ...). Prometheus /metrics requires auth since 1.85.0 — give the scraper a bearer key or set require_auth_for_metrics_endpoint: false.
• Forensic response headers: x-litellm-call-id, x-litellm-model-id, x-litellm-model-api-base, x-litellm-version, x-litellm-response-cost.
• Privacy: turn_off_message_logging: true keeps metadata but drops content from callbacks; redact_user_api_key_info: true redacts key/user/team identifiers. Debug with --detailed_debug, LITELLM_LOG=DEBUG, or per-request "litellm_request_debug": true. Details: observability reference.

Deployment

• Postgres is mandatory for keys, teams, spend, budgets, and UI state; Redis >=7 is required for more than one instance (shared rate-limit counters, cooldowns, cache).
• Pin image tags (ghcr.io/berriai/litellm:vX.Y.Z — semver tags since 1.84.0; -stable suffixes are gone, main-latest is deprecated). Images are cosign-signed.
• Prisma migrations run at startup by default; on Kubernetes use the migration job pattern with DISABLE_SCHEMA_UPDATE=true on serving pods. One Uvicorn worker per pod; size the DB pool as MAX_DB_CONNECTIONS / (instances x workers). Details: deployment reference.