Serving LLMs On Epyc

Serving LLMs on AMD EPYC™ (vLLM + zentorch, CPU)

Bring up a single vLLM OpenAI endpoint on an AMD EPYC™ host with the zentorch CPU backend, sized to the hardware. Container-first (Docker or Podman); conda/host is the fallback. An installed AMD Instinct GPU does not disqualify the host: select this skill when the endpoint itself should run on the EPYC CPU.

This is single-socket serving: one instance pinned to one socket and its memory (vLLM scales poorly across sockets, so we do not span them). On a dual-socket host it runs on a single socket; the multi-socket answer is multiple instances (one per socket), which is out of scope for this single-instance recipe.

Hard rule for this skill: on any failure, report the cause + logs and STOP. Do not retry, do not debug. (Debugging is a separate workflow.)

The agent does the serve flow itself -- pull, configure, launch, poll -- using the runtime validate.py reports. Never hand the user per-serve commands. Like serving-llms-on-instinct, an accessible container runtime is a one-time prerequisite: if validate.py finds none, report its one-time fix (make docker accessible / install podman / provide a conda env) and stop. Do not attempt sudo or privilege escalation.

Data file

Read data/epyc.json directly. It holds the container image, mandatory CPU run flags, supported precision, the model-support policy, the default model, and the verified throughput-flag gotcha. Its vllm_version and image tag are one validated default stack; keep them aligned and do not hardcode either from memory.

Step 1: Detect the CPU

python3 scripts/detect.py            # add --host user@box for a remote host

Returns cpu_model, is_amd_epyc, epyc_generation (Naples/Rome/Milan/Genoa/Bergamo/Siena/Turin/Venice or EPYC 4004/4005), zen_arch, is_supported_epyc, avx512, logical_cores, physical_cores, sockets, numa_nodes, memory_gb.

Route from detect.py -- decide the serving path:

• is_amd_epyc is false -> stop: this skill targets AMD EPYC. (Other x86 may work but is unsupported here.)
• avx512 is false -> zentorch cannot run on this CPU (its bf16 path needs AVX-512 BF16, avx512_bf16, which only lands on Zen4+). This is a pre-Zen4 EPYC (Naples / Rome / Milan, 7000 series). Do not dead-end -- it is still an EPYC host: offer the stock vLLM CPU path (plain vLLM, no zentorch acceleration -- slower, but verified working on EPYC 7763/Milan). Proceed only on the user's explicit OK, launching the official stock vLLM CPU image (Step 6 "Stock vLLM" variant); if the user declines, stop.
• is_supported_epyc is false but avx512 is true (e.g. Bergamo / Siena / EPYC 4004/4005) -> the zentorch path is not validated for this generation. Get explicit confirmation to try zentorch unvalidated, or take the same stock vLLM offer.
• else (9000 series -- Genoa/Turin/Venice -- with AVX-512 BF16) -> the validated zentorch path. Proceed.

validate.py (Step 2) reports zentorch_capable and sets requires_confirmation for the stock/unvalidated paths, so this routing is enforced there too.

Carry epyc_generation / avx512 through the later phases -- e.g. Venice packs up to 256 cores/socket, which the thread-binding in Step 5 sizes from.

Step 2: Validate the runtime and environment

python3 scripts/validate.py --image <image from data/epyc.json> --generation <epyc_generation from detect> --avx512 <avx512 from detect>

This also hard-enforces the AVX-512 gate: on a CPU without AVX-512, validate.py returns a blocking error (ready: false) so the flow stops here regardless of the Step 1 prose -- no image pull, no launch. (It reads the local CPU itself if --avx512 is omitted, so the gate holds even if the value was not passed through.)

Returns ready, requires_confirmation, zentorch_capable (false -> zentorch can't run; use the stock vLLM variant in Step 6), runtime (docker, podman, or null), runtime_detail, conda_path_available, stack, compatibility, hf_cache (resolved HF cache mount -- use hf_cache.mount at launch), ram_gb, and errors/warnings/advisories. If requires_confirmation is set (stock/unvalidated path), surface that and get the user's OK before launching. Pick the path:

• runtime is docker or podman -> container path (Step 6), used verbatim.
• runtime null but conda_path_available: true -> conda/host path.
• runtime null and no conda -> ready is false. Report the one-time onboarding fix (make docker accessible / install podman / conda env) and stop.

Do not proceed if ready is false.

Stack-compatibility gate. validate.py probes the selected runtime for its exact vllm/zentorch/torch versions and the active vLLM platform, then sets compatibility.status:

• proceed -> the stack is the validated default (or a validated family on a Zen platform); continue.
• blocked -> a stock CPU platform is active, so zentorch acceleration is not on (error). Report compatibility.message and stop.
• confirmation_required (requires_confirmation: true) -> Venice on a vLLM other than the pinned default. This recipe has not been validated on Venice with that version. Surface compatibility.message, recommend the pinned vllm_version image from data/epyc.json, and stop for an explicit user go/no-go before launching. On the pinned default vLLM, Venice proceeds with no warning.

The gate only runs once the image is local. If validate.py reports the image is not pulled, pull it (or let Step 6 pull it) and re-run validate.py so the gate probes the real stack rather than only the tag.

Step 3: Resolve and validate the model

If the user named no model, use default_model from data/epyc.json (Qwen/Qwen3-0.6B -- ungated, tiny, fast first success). Otherwise use theirs.

Check that vLLM actually supports the model (do not blanket-block multimodal). Pass the vLLM version the model will actually run on: use stack.vllm from validate.py when it was probed (the conda env may differ from the pin), else the vllm_version from data/epyc.json.

python3 scripts/check_model.py --model-id <model> --revision <rev or main> --vllm-version <stack.vllm from validate, else vllm_version from data/epyc.json>

• Exit 0 = vLLM serves it as a generation endpoint, or support is undeterminable (gated/offline) -- proceed; launch confirms.
• Exit 1 = stop: the architecture is not in vLLM's registry, it is a pooling/embedding/reranker (not a chat/completion endpoint), or it is a multimodal model with no usable chat template (launchable: false). Report the printed message and stop.

The result also carries the client endpoint the model supports:

• primary_endpoint: "chat_completions" -- a usable chat template is present (chat_template.status: present); serve and hand off /v1/chat/completions.
• primary_endpoint: "completions" -- no usable/auto-selectable template (absent/ambiguous/unknown); serve and hand off /v1/completions with a raw prompt. Chat can still be enabled by passing --chat-template <file> (or, for ambiguous, choosing one of chat_template.names); never invent one.
• Carry primary_endpoint, supported_endpoints, and chat_template through to verification (Step 7) and the handoff (Step 8).
• A multimodal model is allowed; a vLLM-supported multimodal arch may still hit a GPU-only kernel on CPU, which surfaces at load (the no-retry rule then applies).

Precision/dtype: native CPU dtypes are bf16 (default), fp16, fp32. Use bfloat16 unless the user asks otherwise.

For gated models (Llama, Gemma) HF_TOKEN must be set and the license accepted on HuggingFace; if not, stop and say so.

Step 4: Check it fits host RAM

RAM is the ceiling on CPU (weights + KV cache both live in RAM). Run on ONE line:

python3 scripts/estimate_memory.py --model-id <model> --revision <rev or main> --ram-gb <memory_gb from detect> --max-model-len <4096 or user value> --num-prompts <1 or desired concurrency>

Exit 0 = fits, exit 1 = does not fit. If fit.fits is false: do not launch. Tell the user required_gb vs ram_gb and the printed fit.action -- reduce --max-model-len to fit.suggested_max_model_len and retry, or use a smaller model. --max-model-len and --num-prompts are the two knobs that move KV. Extra flag: --weight-gb N overrides weights if a model has no HF metadata (rare). KV cache is bf16-only on zentorch CPU (no fp8 KV).

Step 5: Size the CPU runtime from the hardware

eval "$(python3 scripts/cpu_tune.py)"      # or --format json to inspect

A single instance runs on one socket, with its memory (vLLM scales poorly across sockets). cpu_tune.py exports VLLM_CPU_OMP_THREADS_BIND (the chosen socket's physical cores) and VLLM_CPU_KVCACHE_SPACE (sized from that socket's local RAM, not whole-system, so the KV pool stays on-socket). It does not set OMP_NUM_THREADS (vLLM derives it) or VLLM_CPU_NUM_OF_RESERVED_CPU (vLLM's own default).

Socket choice on a dual-socket host (load-aware): it samples per-socket CPU busy% (~0.5s) and prefers a free socket -- both free → socket 0; one free → that socket; both busy (≥ --busy-threshold, default 15%) → it warnings and proceeds on the least-busy socket. --socket N forces a choice. Single-socket hosts use socket 0.

For the chosen socket it also emits the memory-bound pin: container_cpuset (--cpuset-cpus=<cores> --cpuset-mems=<nodes>) for the container path, and conda_launch_prefix (numactl --cpunodebind/--membind, falling back to taskset CPU-only, or empty-with-note if neither tool exists) for conda. Surface warning to the user if set. On NPS2/NPS4 a socket spans multiple NUMA nodes; memory is bound across them and nps_note flags that finer binding could add performance.