Vllm

vLLM — high-throughput serving of open-weight models

vLLM is the inference engine you put in front of an open-weight model when many requests hit it at once. Its job — and this skill's — is throughput under concurrency: keep the GPU busy across dozens of simultaneous requests, not squeeze one prompt out fast. You own the vllm serve flags; the box those flags run on is a runpod/modal concern.

Why not just loop a transformers generate()? Naive serving runs one request at a time and pads every batch to the longest sequence, so the GPU idles. vLLM fixes both:

• PagedAttention stores the KV cache in non-contiguous fixed-size blocks (like OS virtual-memory paging), so there is almost no padding/reservation waste and long contexts pack tightly.
• Continuous batching admits and retires requests token-by-token instead of per-batch, so a new request joins the running batch immediately rather than waiting for the slowest one to finish.

Net effect: an order-of-magnitude more concurrent throughput than single-request serving. If you only ever have one user on a laptop, that machinery is wasted — that is ollama, not this.

Version & setup reality (verify at author time — vLLM ships ~weekly)

• Latest stable is ~0.25.x (July 2026) — releases land roughly weekly. Check PyPI / releases; do not pin to a number you read here.
• The V1 engine is the default since v0.8.0 and recent releases have removed the legacy V0 path, so treat V1 as the only engine. It runs the scheduler + core loop in a separate process and turns on chunked prefill and prefix caching out of the box, so you rarely tune the scheduler by hand (V1 guide, accessed 2026-07). VLLM_USE_V1 historically toggled it — if you see it referenced, it is legacy.
• Install: pip install vllm (default build is CUDA/NVIDIA). ROCm, CPU, TPU and other backends have separate install paths — see the docs' installation matrix. Needs Python + a supported GPU.
• Auth is OFF by default. A bare vllm serve is an open endpoint on 0.0.0.0:8000. To require a bearer token, pass --api-key <KEY> (or set VLLM_API_KEY); clients then send Authorization: Bearer <KEY>. Do not expose an unauthenticated server publicly.

vllm serve → an OpenAI-compatible server

vllm serve Qwen/Qwen3-8B                 # download from HF (or a local path) and serve on :8000
vllm serve /models/qwen3-8b --api-key sk-local-xyz --port 8000 --host 0.0.0.0

The server speaks the OpenAI wire protocol, so any OpenAI client works unchanged — just repoint base_url and use a dummy (or your --api-key) key. Endpoints (online serving docs, accessed 2026-07):

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="sk-local-xyz")  # key = your --api-key
r = client.chat.completions.create(
    model="Qwen/Qwen3-8B",               # the id `/v1/models` reports (or a LoRA adapter name)
    messages=[{"role": "user", "content": "Name three primes."}],
)
print(r.choices[0].message.content)

The model field must match what /v1/models returns — the served model id or a LoRA adapter name (below), not an arbitrary string.

Parallelism: fit the model, then scale out

Two orthogonal knobs. Reach for parallelism only when the model does not fit one GPU — a model that fits should stay on a single GPU (no split), because every split adds communication overhead.

• --tensor-parallel-size N — shard each layer's weights across N GPUs on one node. Use this first: it needs fast intra-node links (NVLink / PCIe) because GPUs sync every layer. N must divide the model's attention-head count. This is how you serve a model too big for one GPU but fitting the node.
• --pipeline-parallel-size M — split the model by layer stages across M nodes. Tolerates slower inter-node network. Use it when the model does not fit even a full node.

Rule of thumb from the docs (parallelism & scaling, accessed 2026-07): set tensor-parallel = GPUs per node, pipeline-parallel = number of nodes.

vllm serve meta-llama/Llama-3.3-70B-Instruct --tensor-parallel-size 4          # 1 node, 4 GPUs
vllm serve <huge-model> --tensor-parallel-size 8 --pipeline-parallel-size 2    # 2 nodes × 8 GPUs

Multi-node needs a Ray cluster wired up first — that orchestration is a runpod/modal concern, not a vLLM flag.

Quantization: smaller weights ≠ automatic speedup

Quantization shrinks the weights so a model fits fewer/smaller GPUs and frees VRAM for KV cache. Select a method with --quantization (often auto-detected from the checkpoint's config). vLLM supports AWQ, GPTQ/GPTQModel, FP8 (W8A8), compressed-tensors (LLM Compressor), INT4/INT8, bitsandbytes and more (quantization docs, accessed 2026-07).

vllm serve TheModel/Qwen3-8B-AWQ --quantization awq
vllm serve neuralmagic/Model-FP8   --quantization compressed-tensors   # FP8 needs Ada/Hopper+

The honest tradeoffs:

• Support is GPU-arch-specific — FP8 W8A8 wants Ada/Hopper-class (or AMD) hardware; a method that is fast on one GPU may be unsupported or emulated on another. Check the compatibility table.
• Quant is a memory win, not a guaranteed throughput win. At low batch / low concurrency the workload is memory-bandwidth-bound and a quant can help; but dequant overhead can cost latency, and at high batch you may be compute-bound where a weight-only quant does little. Benchmark your quant on your hardware at your real concurrency before assuming it is faster.
• Quality drops — usually small for 8-bit/well-calibrated 4-bit, larger for aggressive 4-bit. Serve a quant to save VRAM, not as a free lunch. Picking which quantized checkpoint to pull is an open-weights decision.

LoRA adapters: serve the fine-tuning output

vLLM serves LoRA adapters on top of one loaded base model, so your finetuning/unsloth output goes live without merging or a second server (LoRA docs, accessed 2026-07).

vllm serve meta-llama/Llama-3.2-3B-Instruct \
  --enable-lora \
  --lora-modules sql=/adapters/sql-lora legal=/adapters/legal-lora \
  --max-loras 2 \        # how many adapters resident at once
  --max-lora-rank 16     # must be >= the rank the adapter was trained at

Route to an adapter by naming it in the request model field: "model": "sql" hits the SQL adapter, "model": "meta-llama/Llama-3.2-3B-Instruct" hits the untuned base — same server, no reload.

Load/unload at runtime with VLLM_ALLOW_RUNTIME_LORA_UPDATING=True, then POST /v1/load_lora_adapter / POST /v1/unload_lora_adapter. Base + adapter must match (same family and dims), and --max-lora-rank must be ≥ the trained rank or load fails.

Memory & throughput: where OOM comes from

The two knobs that cause (and cure) most OOM (engine args, accessed 2026-07):

• --gpu-memory-utilization (default ~0.92) — fraction of each GPU vLLM may claim. Weights are loaded, then the rest of this budget becomes the KV-cache pool. Raising it toward 1.0 buys more concurrent sequences but risks OOM from activation/CUDA-graph spikes; lower it if you get OOM at load or under burst.
• --max-model-len — max context (prompt + output) per request; auto-derived from the model config if unset. This is the single biggest OOM lever: KV cache scales with max-model-len × concurrent sequences. A model whose weights fit will still OOM if you leave the full 128K context on and let many long requests batch. Cap --max-model-len to what you actually need.

The mental model: KV_pool = gpu_memory_utilization × VRAM − weights, and concurrent_sequences ≈ KV_pool ÷ (bytes_per_token × context_len). Too-long context or too-high utilization eats the pool. --max-num-seqs and --max-num-batched-tokens cap the batch to trade latency vs throughput. Full KV math + an OOM playbook: references/memory-and-throughput.md.