vLLM Model Serving

What it does

When your agent uses it

• Standing up a production LLM API behind an OpenAI-style endpoint
• Tuning inference latency and throughput for a served model
• Running a model on a GPU with limited memory using quantization
• Processing a large file of prompts offline without a server

Example prompts

• “Serve Llama 3 8B Instruct with vLLM and show how to call it from the OpenAI Python SDK.”
• “Write a Docker command to run vLLM with the OpenAI-compatible server on one GPU.”
• “Set up Prometheus monitoring for our vLLM deployment and name the metrics to watch.”
• “Run prompts.txt through vLLM in offline batch mode and save the outputs.”

Requirements

• Python with the `vllm` package
• A GPU for serving models
• Docker for the container deployment step

What it can do on your machine

Read from SKILL.md and the folder at commit 773a529. It shows what the files ask for, not the result of running them.

• Tool permissions

  Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.

  From allowed-tools in the SKILL.md frontmatter.

• Runs code

  Shell commands in SKILL.md call:

  • pip
  • python
  • curl
  • docker

  From the folder's file list and the shell code blocks in SKILL.md.

• Network

  Links to these hosts (documentation or services it may open):

  • docs.vllm.ai
  • github.com
  • discuss.vllm.ai

  From URLs in SKILL.md, links to its own repository left out.

• Credentials

  Names no API keys, tokens, secrets or passwords.

  From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.

Context cost

vLLM Model Serving loads about 2.3k tokens when it runs, and up to ~9k if it reads all its reference files. Until then it costs about 80 tokens; SKILL.md has 491 words of instructions outside code blocks.

Always · name and description, kept in context so the agent knows when to use it

~80

When it runs · the whole SKILL.md, loaded when a task matches

~2.3k

With references · SKILL.md plus every file in references/, read only if the agent opens them

~9k

Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.

Safety

Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.

SKILL.md

The full file from Orchestra-Research/AI-Research-SKILLs at commit 773a529, republished under its MIT licence (© Orchestra-Research). 491 words, ~2,257 tokens.

name

serving-llms-vllm

description

Serves LLMs with high throughput using vLLM's PagedAttention and continuous batching. Use when deploying production LLM APIs, optimizing inference latency/throughput, or serving models with limited GPU memory. Supports OpenAI-compatible endpoints, quantization (GPTQ/AWQ/FP8), and tensor parallelism.

version

1.0.0

author

Orchestra Research

license

MIT

tags

vLLM, Inference Serving, PagedAttention, Continuous Batching, High Throughput, Production, OpenAI API, Quantization, Tensor Parallelism

dependencies

vllm, torch, transformers

pip install vllm

from vllm import LLM, SamplingParams

llm = LLM(model="meta-llama/Llama-3-8B-Instruct")
sampling = SamplingParams(temperature=0.7, max_tokens=256)

outputs = llm.generate(["Explain quantum computing"], sampling)
print(outputs[0].outputs[0].text)

vllm serve meta-llama/Llama-3-8B-Instruct

# Query with OpenAI SDK
python -c "
from openai import OpenAI
client = OpenAI(base_url='http://localhost:8000/v1', api_key='EMPTY')
print(client.chat.completions.create(
    model='meta-llama/Llama-3-8B-Instruct',
    messages=[{'role': 'user', 'content': 'Hello!'}]
).choices[0].message.content)
"

Deployment Progress:
- [ ] Step 1: Configure server settings
- [ ] Step 2: Test with limited traffic
- [ ] Step 3: Enable monitoring
- [ ] Step 4: Deploy to production
- [ ] Step 5: Verify performance metrics

# For 7B-13B models on single GPU
vllm serve meta-llama/Llama-3-8B-Instruct \
  --gpu-memory-utilization 0.9 \
  --max-model-len 8192 \
  --port 8000

# For 30B-70B models with tensor parallelism
vllm serve meta-llama/Llama-2-70b-hf \
  --tensor-parallel-size 4 \
  --gpu-memory-utilization 0.9 \
  --quantization awq \
  --port 8000

# For production with caching and metrics
vllm serve meta-llama/Llama-3-8B-Instruct \
  --gpu-memory-utilization 0.9 \
  --enable-prefix-caching \
  --enable-metrics \
  --metrics-port 9090 \
  --port 8000 \
  --host 0.0.0.0

# Install load testing tool
pip install locust

# Create test_load.py with sample requests
# Run: locust -f test_load.py --host http://localhost:8000

curl http://localhost:9090/metrics | grep vllm

# Run vLLM in Docker
docker run --gpus all -p 8000:8000 \
  vllm/vllm-openai:latest \
  --model meta-llama/Llama-3-8B-Instruct \
  --gpu-memory-utilization 0.9 \
  --enable-prefix-caching

Batch Processing:
- [ ] Step 1: Prepare input data
- [ ] Step 2: Configure LLM engine
- [ ] Step 3: Run batch inference
- [ ] Step 4: Process results

# Load prompts from file
prompts = []
with open("prompts.txt") as f:
    prompts = [line.strip() for line in f]

print(f"Loaded {len(prompts)} prompts")

from vllm import LLM, SamplingParams

llm = LLM(
    model="meta-llama/Llama-3-8B-Instruct",
    tensor_parallel_size=2,  # Use 2 GPUs
    gpu_memory_utilization=0.9,
    max_model_len=4096
)

sampling = SamplingParams(
    temperature=0.7,
    top_p=0.95,
    max_tokens=512,
    stop=["</s>", "\n\n"]
)

# Process all prompts in one call
outputs = llm.generate(prompts, sampling)

# vLLM handles batching internally
# No need to manually chunk prompts

# Extract generated text
results = []
for output in outputs:
    prompt = output.prompt
    generated = output.outputs[0].text
    results.append({
        "prompt": prompt,
        "generated": generated,
        "tokens": len(output.outputs[0].token_ids)
    })

# Save to file
import json
with open("results.jsonl", "w") as f:
    for result in results:
        f.write(json.dumps(result) + "\n")

print(f"Processed {len(results)} prompts")

Quantization Setup:
- [ ] Step 1: Choose quantization method
- [ ] Step 2: Find or create quantized model
- [ ] Step 3: Launch with quantization flag
- [ ] Step 4: Verify accuracy

# Search for AWQ models
# Example: TheBloke/Llama-2-70B-AWQ

# Using pre-quantized model
vllm serve TheBloke/Llama-2-70B-AWQ \
  --quantization awq \
  --tensor-parallel-size 1 \
  --gpu-memory-utilization 0.95

# Results: 70B model in ~40GB VRAM

# Compare quantized vs non-quantized responses
# Verify task-specific performance unchanged

vllm serve MODEL \
  --gpu-memory-utilization 0.7 \
  --max-model-len 4096

vllm serve MODEL --quantization awq

vllm serve MODEL --enable-prefix-caching

vllm serve MODEL --enable-chunked-prefill

vllm serve MODEL --trust-remote-code

vllm serve MODEL --max-num-seqs 512

vllm serve MODEL --tensor-parallel-size 4  # Not 3

vllm serve MODEL --speculative-model DRAFT_MODEL

© Orchestra-Research, MIT. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file

Files

SKILL.md and 4 other files (references) in 12-inference-serving/vllm of Orchestra-Research/AI-Research-SKILLs.

• SKILL.md
• references/optimization.md
• references/quantization.md
• references/server-deployment.md
• references/troubleshooting.md

Open the folder on GitHubat commit 773a529

Used in 5 other repositories

We found 6 copies of this SKILL.md (exact, near-identical or edited) in other folders, from 5 other GitHub owners. This page covers the copy in Orchestra-Research/AI-Research-SKILLs, which our catalogue first saw on October 7, 2026.

• RedWoodOG/Hermes-Desktopskills/mlops/inference/vllm
• Tommy-yw/RunbookHermesskills/mlops/inference/vllm
• davila7/claude-code-templatescli-tool/components/skills/ai-research/inference-serving-vllm
• OpenLAIR/dr-clawskills/inference-serving/vllm
• ArgentAIOS/argentos-coreskills/hermes/mlops/inference/vllmskills/community/mlops/inference/vllm