Vllm

vLLM Inference Serving

Use this skill to operate vLLM as a production inference server: deploy it with Docker or Kubernetes, configure the model and engine (quantization, tensor parallelism, KV cache, context length), serve the OpenAI-compatible API surface, benchmark throughput and latency with comparable evidence, tune continuous batching, operate the GPUs underneath, and upgrade or roll back safely. This is a tool skill for one named engine. Serving methodology — engine selection, quantization trade-offs, deployment plans, regression triage — belongs to ml-engineering; local single-node GGUF serving with the llama.cpp stack belongs to llama-cpp. This skill owns the day-to-day operation of vLLM itself.

Operating contract

1. Record the deployment before tuning it. Capture the vLLM version or image digest, model and revision, quantization, parallelism, max-model-len, KV cache settings, batching limits, GPU inventory, and workload. The serving config template exists for exactly this.
2. Confirm the target, scope, and rollback path before acting. Read-only discovery (health probes, /metrics, nvidia-smi) may proceed without confirmation. Mutations — restarting a server, changing serving args, scaling replicas, upgrading the image — require an explicit human directive naming the deployment.
3. A server that responds is not a server that serves. /health returning 200 proves liveness, not that the model loaded or that inference works. Verify at the delivery boundary: /v1/models reports the served model and a representative request returns generated tokens.
4. Benchmark before and after every change. vLLM flags, defaults, and behavior change between releases; an unmeasured tuning change is a guess. Compare only matched conditions (version, model, GPU, context, batch, workload) and record the evidence in the benchmark run record.
5. Keep evidence bounded. Summarize logs, configs, and metrics; never dump full server logs, .env files, or HF tokens into chat. --enable-log-requests with debug logging can leak prompt content; keep request logging off or redacted in shared sessions.

The vllm-health script

scripts/vllm-health is an agent-first, read-only probe for a running vLLM server. It issues GET requests only, never mutates, and emits bounded JSON.

scripts/vllm-health --help                    # no server needed
scripts/vllm-health --url http://127.0.0.1:8000 --json
scripts/vllm-health --check health --check models --json
scripts/vllm-health --check metrics --timeout 10 --json

Exit codes: 0 all checks passed, 1 issues found or a fatal error, 2 usage error, 124 timeout. Checks: health (/health), version (/version), models (/v1/models), load (/load), and metrics (a bounded prefix of /metrics). The script never sends data anywhere and never writes files.

Operating loop

1. Identify the deployment: vLLM version or image digest, model and revision, served model name, parallelism, and how it is deployed (bare vllm serve, Docker, Kubernetes).
2. Collect evidence: run vllm-health --json for health, version, models, and load; check /metrics counters (vllm:num_requests_running, vllm:num_requests_waiting, vllm:gpu_cache_usage_perc); inspect GPU state with nvidia-smi.
3. Triage against the symptom: map the problem to the evidence (OOM → KV cache or gpu_memory_utilization; high latency → batching, TTFT vs TPOT; model not found → served name or chat template; slow start → model download or compile cache).
4. Act with confirmation: bounded, scoped changes after a human directive, with a rollback path named first.
5. Verify: re-run the probe and the representative request at the delivery boundary, and re-benchmark if the change affects performance.

Deployment: Docker and Kubernetes

• Docker: the official image is vllm/vllm-openai (Docker Hub). Run with GPU access, the Hugging Face cache mounted, the HF token for gated models, port 8000 published, and --ipc=host (or a --shm-size) for the shared memory tensor parallelism relies on. See references/01-deployment.md.
• Kubernetes: a Deployment with nvidia.com/gpu (or amd.com/gpu) resources, a PVC for the model cache, an emptyDir backed by Memory at /dev/shm, liveness/readiness probes on /health port 8000, and a Service. Raise probe failureThreshold for large models that take minutes to load — a premature kill shows up as KeyboardInterrupt: terminated in the container log.
• Pin image tags to a release (for example vllm/vllm-openai:v0.26.0) instead of latest, and persist the compile cache (default ~/.cache/vllm) across restarts so torch.compile artifacts are reused.

Model configuration

• Model identity: --model is the HF repo or local path; --revision pins the exact weights. --served-model-name sets the name clients must use in /v1 requests and in the model field of responses. --trust-remote-code is required for some model repos and should be reviewed before use.
• Context length: --max-model-len bounds prompt plus output per request. Unset, it derives from the model config; -1/auto picks the largest length that fits GPU memory. It is the single biggest driver of KV cache size.
• Quantization-aware serving: pass --quantization (or -q) only when the model weights require it (GPTQ/AWQ/GGUF checkpoints load their scheme from config). Weight types and activation dtypes must match what the kernels support; a quantized model served at the wrong dtype fails to load or silently degrades. Hardware support varies by method (see references/02-model-configuration.md).
• Tensor parallelism: --tensor-parallel-size N shards one model across N GPUs in the same node; --pipeline-parallel-size splits layers across nodes. TP requires NVLink/fast interconnect and equal per-GPU memory; startup logs the memory profiling result, which is the evidence that the model fits.
• KV cache: --gpu-memory-utilization (default 0.92) caps the fraction of GPU memory the model plus KV cache may use. --kv-cache-dtype fp8 shrinks the cache for long contexts on supported GPUs. The engine logs GPU KV cache size: N tokens and the implied max concurrency — record both; they tell you how many concurrent requests of a given length the box can hold.

OpenAI-compatible API surface

• Basic endpoints: /health (liveness), /version, /v1/models (served models), /load (load metrics), /metrics (Prometheus). Inference: /v1/completions and /v1/chat/completions (chat requires the model to ship a chat template, or pass --chat-template); /v1/embeddings for pooling models; /v1/responses for the Responses API.
• Streaming, tool calling (--enable-auto-tool-choice --tool-call-parser openai), structured outputs, and parallel sampling are server-side options that change request/response behavior — verify each against the installed release rather than assuming parity.
• Exposing the server beyond loopback requires an explicit decision about bind address, API keys, TLS or a trusted reverse proxy, and firewall rules. Development-only endpoints (/reset_prefix_cache, weight transfer, profiling) must not be exposed in production.

Benchmarking: throughput and latency

• Online serving benchmark: run vllm bench serve against a live server with a representative dataset (ShareGPT, a local custom JSONL, or your own prompts) and fixed --num-prompts, --request-rate, and --max-concurrency. It reports request throughput (req/s), output token throughput (tok/s), total token throughput, and TTFT/TPOT/ITL percentiles.
• Offline throughput: vllm bench throughput measures raw engine throughput without the HTTP path; use it for engine-only comparisons, not end-to-end user latency.
• Comparable evidence: the benchmark run record template freezes version, model, quantization, parallelism, context, batching, GPU, dataset, and load pattern. Never compare numbers across different conditions as if one variable changed. TTFT is a latency metric; token throughput is a throughput metric — an optimization that helps one can hurt the other.
• For production capacity testing, vLLM's docs recommend the separate GuideLLM framework; this skill's scope is the bundled vllm bench tools.