Vllm Deploy Docker

vLLM Docker Deployment

A Claude skill describing how to deploy vLLM with Docker using the official pre-built images or building the image from source supporting NVIDIA GPUs with CUDA. Instructions include NVIDIA CUDA support, example docker run and a minimal docker-compose snippet, recommended flags, and troubleshooting notes. For AMD, Intel, or other accelerators, please refer to the vLLM documentation for alternative deployment methods.

What this skill does

• Deploy vLLM with docker using pre-built images (recommended for most users) or build from source for custom configurations
• Provide example commands for running the OpenAI-compatible server with GPU access and mounted Hugging Face cache
• Point to build-from-source instructions when a custom image or optional dependencies are needed
• Explain common flags: --ipc=host, shared cache mounts, and HF_TOKEN handling

Prerequisites

• Docker Engine installed (Docker 20.10+ recommended)
• NVIDIA GPU(s) with appropriate drivers and CUDA toolkit installed
• Optional: curl for API tests
• A Hugging Face token if pulling private models or to avoid rate-limits: HF_TOKEN

Quickstart using Pre-built Image (recommended)

Run a vLLM OpenAI-compatible server with GPU access, mounting the HF cache and forwarding port 8000:

docker run --rm --gpus all \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  --env "HF_TOKEN=$HF_TOKEN" \
  -p 8000:8000 \
  --ipc=host \
  vllm/vllm-openai:latest \
  --model Qwen/Qwen2.5-1.5B-Instruct

• --gpus all exposes all GPUs to the container. Adjust if you need specific GPUs.
• --ipc=host or an appropriately large --shm-size is recommended so PyTorch and vLLM can share host shared memory.
• Mounting ~/.cache/huggingface avoids re-downloading models inside the container.

Note: vLLM and this skill recommend using the latest Docker image (vllm/vllm-openai:latest). For legacy version images, you may refer to the Docker Hub image tags.

Build Docker image from source

You can build and run vLLM from source by using the provided docker/Dockerfile. First, check the hardware of the host machine and ensure you have the necessary dependencies installed (e.g., NVIDIA drivers, CUDA toolkit, Docker with BuildKit support). For ARM64/aarch64 builds, refer to the "Building for ARM64/aarch64" section.

Basic build command

DOCKER_BUILDKIT=1 docker build . \
  --target vllm-openai \
  --tag vllm/vllm-openai \
  --file docker/Dockerfile

The --target vllm-openai specifies that you are building the OpenAI-compatible server image. The DOCKER_BUILDKIT=1 environment variable enables BuildKit, which provides better caching and faster builds.

Build arguments and options

• --build-arg max_jobs=<N> — sets the number of parallel compilation jobs for building CUDA kernels. Useful for speeding up builds on multi-core systems.
• --build-arg nvcc_threads=<N> — controls CUDA compiler threads. Recommended to use a smaller value than max_jobs to avoid excessive memory usage.
• --build-arg torch_cuda_arch_list="" — if set to empty string, vLLM will detect and build only for the current GPU's compute capability. By default, vLLM builds for all GPU types for wider distribution.

Using precompiled wheels to speed up builds

If you have not changed any C++ or CUDA kernel code, you can use precompiled wheels to significantly reduce Docker build time:

• Enable precompiled wheels: Add --build-arg VLLM_USE_PRECOMPILED="1" to your build command.
• How it works: By default, vLLM automatically finds the correct precompiled wheels from the Nightly Builds by using the merge-base commit with the upstream main branch.
• Specify a commit: To use wheels from a specific commit, add --build-arg VLLM_PRECOMPILED_WHEEL_COMMIT=<commit_hash>.

Example with precompiled wheels and options for fast compilation:

DOCKER_BUILDKIT=1 docker build . \
  --target vllm-openai \
  --tag vllm/vllm-openai \
  --file docker/Dockerfile \
  --build-arg max_jobs=8 \
  --build-arg nvcc_threads=2 \
  --build-arg VLLM_USE_PRECOMPILED="1"

Building with optional dependencies (optional)

vLLM does not include optional dependencies (e.g., audio processing) in the pre-built image to avoid licensing issues. If you need optional dependencies, create a custom Dockerfile that extends the base image:

Example: adding audio optional dependencies

# NOTE: MAKE SURE the version of vLLM matches the base image!
FROM vllm/vllm-openai:0.11.0

# Install audio optional dependencies
RUN uv pip install --system vllm[audio]==0.11.0

Example: using development version of transformers:

FROM vllm/vllm-openai:latest

# Install development version of Transformers from source
RUN uv pip install --system git+https://github.com/huggingface/transformers.git

Build this custom Dockerfile with:

docker build -t my-vllm-custom:latest -f Dockerfile .

Then use it like any other vLLM image:

docker run --rm --gpus all \
  -p 8000:8000 \
  --ipc=host \
  my-vllm-custom:latest \
  --model Qwen/Qwen2.5-1.5B-Instruct