Serving LLMs On Instinct

Serving LLMs on AMD Instinct

Get a vLLM endpoint running on AMD Instinct GPU hardware.

Prerequisites

• ROCm driver and amd-smi installed on the GPU host
• Docker running and accessible (check with docker ps)
• /dev/kfd and /dev/dri present on the GPU host
• HuggingFace token in HF_TOKEN env var (required for gated models; not required for Qwen3 or Gemma). For gated models (Llama 3.2, Gemma, etc.), the HF token must belong to an account that has accepted the model's license at huggingface.co/<model_id>. A valid token without license acceptance will fail with an opaque "Engine core initialization failed" error.
• For remote GPU: SSH key access configured (ssh <user>@<host> must work without a password prompt). If only password access is available, set up keys first: ssh-copy-id <user>@<host>

Data files

Read these files directly to get model and GPU configuration:

• data/recipes_cache.json -- model configs synced from vllm-project/recipes. Each entry under models.<HF_ID>.recipe contains the full recipe with model.base_args, model.base_env, features.tool_calling.args, features.reasoning.args, hardware_overrides.amd.extra_args, hardware_overrides.amd.extra_env. The top-level docker_image field has the latest resolved vLLM ROCm image.

• data/gpu_overrides.json -- GPU-specific configuration. Contains docker_flags (mandatory for all AMD Instinct), gpu_configs keyed by gfx_version with env_defaults and workarounds, and legacy_models for models not yet in vLLM recipes.

• data/blacklist.json -- models in vLLM recipes that cannot be served as LLM endpoints. Includes diffusion/image/audio generation models, embedding models, rerankers, ASR models needing audio pipelines, and models requiring unreleased vLLM nightly builds. Check this before attempting to serve a model. If the user requests a blacklisted model, explain why it won't work and suggest an alternative.

If the user doesn't specify a model, default to Qwen/Qwen3.5-9B: dense multimodal with MTP, Apache 2.0 license (no HF token needed), fits on a single GPU, strong reasoning and tool-calling.

Step 1: Detect the GPU

python3 scripts/detect.py
# Remote:
python3 scripts/detect.py --host user@hostname

Returns JSON with gfx_version, vram_gb, gpu_count, rocm_version.

If gfx_version is unknown: amd-smi ran but found no GPU. Check lsmod | grep amdgpu.

Step 2: Validate the environment

python3 scripts/validate.py --auto-fix
# Remote:
python3 scripts/validate.py --auto-fix --host user@hostname

Returns JSON with ready (bool), errors, warnings, fixes_applied. Do not proceed if ready is false.

Step 3: Refresh recipes (if stale)

Check fetched_at in data/recipes_cache.json. If older than 24 hours or the file is missing, refresh:

python3 scripts/sync_recipes.py

This shallow-clones vllm-project/recipes from GitHub and fetches the latest Docker tag from Docker Hub. Takes ~10 seconds. If it fails, the existing cache still works.

Step 4: Construct the Docker command

Read data/recipes_cache.json and data/gpu_overrides.json directly. Build the Docker command by combining:

1. Docker flags from gpu_overrides.json > docker_flags (mandatory for all AMD GPUs)
2. HF cache mount: -v ~/.cache/huggingface:/root/.cache/huggingface (if a shared model cache directory exists on the host, check whether models--* directories are at the cache root or inside a hub/ subdirectory -- mount accordingly to /root/.cache/huggingface or /root/.cache/huggingface/hub)
3. Port: -p <port>:<port> (default 8000)
4. Environment variables: merge gpu_configs.<gfx_version>.env_defaults with the recipe's model.base_env and hardware_overrides.amd.extra_env. Always add --env HF_TOKEN=${HF_TOKEN}.
5. Docker image: use docker_image from recipes_cache.json top level (unless the model needs a pinned image, e.g. GLM-4.5 needs v0.15.1). If the user specifies a Docker image version, check it against the recipe's model.min_vllm_version. Warn if the image is older -- the model may crash on startup with an opaque "Engine core initialization failed" error.
6. Model ID: --model <HF_ID>
7. vLLM args: combine the recipe's model.base_args + hardware_overrides.amd.extra_args + features.tool_calling.args + features.reasoning.args. Add --enable-auto-tool-choice if not present. For multi-GPU, add --tensor-parallel-size N (see VRAM estimation below). For MoE models on multi-GPU, also add --distributed-executor-backend mp.
8. Port arg: --port <port>

If the exact model ID is not in recipes_cache.json, check for a base model match by stripping date/version suffixes (e.g., Kimi-K2-Instruct matches Kimi-K2-Instruct-0905). Use the base model's recipe if found.

If no recipe match, check legacy_models in gpu_overrides.json. If not there either, use a generic config with --enable-auto-tool-choice --trust-remote-code --tool-call-parser hermes.

Precision variant selection: Recipes may offer variants (default, fp8, nvfp4). Check gpu_configs.<gfx_version>.precision.native in gpu_overrides.json before selecting a variant. On gfx942 (MI300X), only bf16, fp16, fp8_fnuz, and int8 are hardware-native. MXFP4 and NVFP4 compute is emulated (dequant to BF16 during matmul), but weights stay compressed in VRAM so quantized models still fit in less memory. On gfx950 (MI350X), MXFP4 is hardware-native.

VRAM estimation and fit check: Before constructing the Docker command, estimate whether the model fits the available hardware:

python3 scripts/estimate_vram.py --model-id <HF_ID> --vram-gb <per_gpu_vram> --tp <N>

This queries the HuggingFace Hub API (no model download) and returns JSON with:

• weight_memory_gb -- total weight size
• kv_cache_bytes_per_token -- KV cache cost per token at BF16
• fit.weights_fit -- whether weights fit at the given TP
• fit.recommended_max_model_len -- max context the GPU can serve
• fit.context_limited -- true if KV cache limits context below the model's native max
• fit.min_tp_required -- minimum TP needed (only if weights don't fit)

Understanding the overhead: The script reserves ~4 GB for vLLM's runtime overhead (activation profiling, HIP graph capture, internal buffers). During startup, vLLM runs a profiling forward pass to measure peak activations, then captures HIP graphs for optimized decode. This startup peak is higher than steady-state. The remaining_for_kv_gb field reflects what's left after weights and this overhead.

Use remaining_for_kv_gb to decide:

1. remaining_for_kv_gb >= 6: safe to run. If context_limited: true, add --max-model-len <recommended_max_model_len> to the vLLM args. Mention the FP8 KV cache option (--kv-cache-dtype fp8) if the user needs longer context (fit.max_seq_len_fp8_kv shows the gain).
2. remaining_for_kv_gb between 2 and 6: tight but worth trying. Launch normally. If vLLM OOMs during HIP graph capture (check container logs for "out of memory" after "capturing CUDA/HIP graphs"), retry with --enforce-eager added to the vLLM args. This skips graph capture and frees 1-2 GB. The only cost is slightly higher decode latency.
3. remaining_for_kv_gb < 2: too tight. Will likely OOM during the activation profiling step. Do not attempt.
4. weights_fit: false with multiple GPUs: re-run with --tp <min_tp_required> and check again.
5. weights_fit: false, not enough GPUs: look for quantized alternatives in this order: a. Recipe variants: the recipe may have fp8 or mxfp4 variants with a different model_id that points to a quantized checkpoint. b. Same provider: many providers release quantized versions alongside the base model (e.g. Qwen/Qwen3.5-122B-FP8 from Qwen). Search HuggingFace for <provider>/<model-name> with FP8/GPTQ/AWQ suffixes. c. AMD quantized: AMD's Quark team publishes quantized models under the amd/ org on HuggingFace (e.g. amd/Kimi-K2-Instruct-w-mxfp4-a-fp8). Search for amd/<model-name> variants. Run estimate_vram.py on the quantized model ID to verify it fits, then use that model ID instead.
6. Still doesn't fit: tell the user the model requires more VRAM than available and suggest either a smaller model or multi-GPU hardware. Do not attempt to launch.

Docker command template:

docker run -d --name vllm-<model-slug> \
  <docker_flags> \
  -v <hf_cache_mount> \
  -p <port>:<port> \
  --env <key>=<value> (for each env var) \
  --env HF_TOKEN=${HF_TOKEN} \
  <docker_image> \
  --model <model_id> \
  <vllm_args> \
  --port <port>