Vllm

vLLM

Purpose

• Use this skill to deploy local or remote LLM inference with vllm.
• Prefer it when you need OpenAI-compatible serving, high throughput, and modern GPU utilization.
• vLLM is strongest for decoder-only chat and completion models.
• It is a good default for production inference when latency and token throughput matter.

Install

• Install from PyPI:

pip install vllm

• Verify the install:

python -c "import vllm; print(vllm.__version__)"

• Match your CUDA, NVIDIA driver, and PyTorch stack before deploying on GPUs.
• If deployment is containerized, prefer pinning a known-good image or package version.

What vLLM Gives You

• OpenAI-compatible HTTP API for chat and completion workflows
• PagedAttention for efficient KV cache memory usage
• Continuous batching for higher aggregate throughput
• Tensor parallelism for multi-GPU serving
• Quantization support for lower memory footprints
• Streaming token responses
• Good support for major open-weight model families

Common Models

• meta-llama/Llama-3.1-8B-Instruct
• meta-llama/Llama-3.1-70B-Instruct
• Qwen/Qwen2.5-7B-Instruct
• Qwen/Qwen2.5-14B-Instruct
• mistralai/Mistral-7B-Instruct-v0.3
• microsoft/Phi-3-medium-4k-instruct
• google/gemma-2-9b-it

Basic Serve

• Minimal OpenAI-compatible server:

python -m vllm.entrypoints.openai.api_server --model meta-llama/Llama-3.1-8B-Instruct

• By default this serves on port 8000.
• The API base path is http://localhost:8000/v1.
• Health check endpoint:

curl http://localhost:8000/health

Key Flags

• --port: bind a non-default server port
• --tensor-parallel-size: split one model across multiple GPUs
• --gpu-memory-utilization: cap how much GPU RAM vLLM should attempt to use
• --max-model-len: set maximum sequence length for inference
• --quantization: load quantized weights when supported

Serve Examples

• Serve on port 8080:

python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --port 8080

• Raise the context window and tune memory usage:

python -m vllm.entrypoints.openai.api_server \
  --model Qwen/Qwen2.5-7B-Instruct \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.92

• Use four GPUs with tensor parallelism:

python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-70B-Instruct \
  --tensor-parallel-size 4

Quantization

• vLLM commonly works with these quantization paths:

• awq

• gptq

• bitsandbytes

• fp8

• Example with AWQ:

python -m vllm.entrypoints.openai.api_server \
  --model Qwen/Qwen2.5-7B-Instruct-AWQ \
  --quantization awq

• Example with GPTQ:

python -m vllm.entrypoints.openai.api_server \
  --model TheBloke/Mistral-7B-Instruct-v0.2-GPTQ \
  --quantization gptq

• Example with FP8:

python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --quantization fp8

• AWQ and GPTQ reduce VRAM pressure at some accuracy or compatibility cost.
• bitsandbytes is useful when using 4-bit or 8-bit HF-compatible flows.
• FP8 is hardware-sensitive and best validated on your actual deployment target.

Call Through the OpenAI Client

• vLLM exposes an OpenAI-style API, so the standard OpenAI Python client is a common fit.
• Point the client at the local base URL:

from openai import OpenAI

client = OpenAI(
    api_key="dummy",
    base_url="http://localhost:8000/v1",
)

response = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[
        {"role": "system", "content": "You are a concise assistant."},
        {"role": "user", "content": "Summarize why continuous batching matters."},
    ],
    temperature=0.2,
    max_tokens=256,
)

print(response.choices[0].message.content)

• base_url='http://localhost:8000/v1' is the key integration setting.
• Many SDK-based applications can switch from OpenAI-hosted inference to vLLM with only the base URL and model name changed.

Curl Chat Example

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages": [
      {"role": "system", "content": "You are concise."},
      {"role": "user", "content": "Explain PagedAttention in two sentences."}
    ],
    "temperature": 0.2,
    "max_tokens": 128
  }'

Async Engine Usage in Python

• Use the async engine when embedding vLLM inside a Python application instead of only exposing the HTTP server.
• This pattern is useful for custom services, pipelines, or batched internal inference.

import asyncio

from vllm import AsyncEngineArgs, AsyncLLMEngine, SamplingParams


async def main() -> None:
    engine_args = AsyncEngineArgs(
        model="meta-llama/Llama-3.1-8B-Instruct",
        gpu_memory_utilization=0.9,
        max_model_len=8192,
    )
    engine = AsyncLLMEngine.from_engine_args(engine_args)
    sampling_params = SamplingParams(temperature=0.2, max_tokens=128)

    request_id = "req-1"
    prompt = "List three use cases for OpenAI-compatible local inference."

    async for output in engine.generate(prompt, sampling_params, request_id):
        if output.finished:
            print(output.outputs[0].text)


asyncio.run(main())

• Use unique request IDs for concurrent work.
• Reuse one engine per process rather than creating a new engine for every request.
• Keep sampling params explicit for reproducibility in evaluation workflows.

Multi-GPU Notes

• Scale large models with --tensor-parallel-size 4 or another GPU count that matches the host.
• Ensure all GPUs are visible and comparable in capability.
• Cross-GPU communication performance matters, so NVLink or high-bandwidth PCIe topology helps.
• Validate memory headroom with the exact context size and batch profile you plan to serve.