Find Serving Recipe

Find a Serving Recipe

<!--
SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
SPDX-License-Identifier: Apache-2.0
-->

Known-good serving configurations exist for far more model and hardware combinations than this repository's recipes/ tree carries, and deriving one from scratch on GPU time is the most expensive way to obtain one. This skill defines "the catalog": an ordered set of sources, each with a trust level, so that a recipe scan finds what exists and never presents an unverifiable config as deployable.

Two invocation contexts, with different outputs:

• Baseline selection (the interviewer's recipe-catalog scan): the dossier feeds the proposal-and-confirmation flow. This skill only finds and ranks; proposing and capturing confirmation stay with the interviewer.
• Mid-engagement candidate hunting: before deriving a candidate configuration from first principles, check whether a published recipe already answers the performance question. A found recipe becomes a HYPOTHESIS for the normal candidate pipeline (consult, materialize, adversarial review); it never bypasses that pipeline and it NEVER re-selects the engagement baseline.

Ground rules

1. Never answer from memory. Every claim in the dossier comes from a source fetched or read during this invocation, cited with its path or URL and, where available, a commit or date. If a source cannot be reached and no usable local copy exists, say so explicitly and move to the next tier; do not reconstruct what the catalog "probably" contains.
2. Refuse to guess a hardware match. If no source names the target GPU or a documented equivalent, report no match for that source. Do not silently map one SKU onto another.
3. The tag-resolvability gate, applied to every candidate at every tier. Resolve the recipe's container image before ranking it:
   • nvcr.io/... images: query the NGC registry API for the tag.
   • Docker Hub images: https://hub.docker.com/v2/repositories/<org>/<repo>/tags/<tag>. A candidate whose image tag does not resolve, or resolves only to a mutable tag (latest, dev, nightly without a digest), is ceiling-only: it may inform expectations but must never be handed to a deploy step. A digest-pinned image (@sha256:...) passes regardless of tag mutability.
4. Carry the model card forward. Before any tier, read the Hugging Face model card for the target model. It frequently points at the authoritative recipe for that model, states a minimum engine version, and is the only source of the model author's sampling and context correctness settings. Record those settings in the dossier; no catalog carries them.
5. Stop at the first tier that yields a candidate meeting the confidence bar for the invocation context (deployable for baseline selection; hypothesis-grade for candidate hunting). Later tiers may still be consulted for expected performance.
6. Freshness is part of the verdict, never a reason to discard. Serving recipes rot unevenly. Split every candidate into its DURABLE content (topology, parallelism dimensions, precision and quantization, KV-cache dtype, memory fractions, sizing, workload fit, and the reasoning behind those choices) and its VERSION-BOUND content (exact flag names and defaults, the image tag, kernel and backend selections, measured performance). Durable content is first-class evidence regardless of the recipe's age; carry it into the dossier and into any candidate's rationale. Version-bound content ages: record the engine version the recipe pins (image tag, min_*_version, or commit) and its verification or publication date, and compare against the engine's CURRENT stable release (registry tag list, engine release page, or the catalog's latest entry). When the pinned engine is behind current, verify each flag against the current CLI and port renamed ones, treat the recipe's measured performance as historical (a shape hint, not a target), pin the current image, and re-verify before anything is graded deployable. A recipe verified on the current or immediately previous minor release with unchanged flags may be graded deployable; anything older is hypothesis until re-verified. A version label alone is evidence of relabeling, not revalidation; only a recorded re-verification resets the clock. Prefer the fresher of otherwise comparable candidates, but never let a fresher, thinner recipe displace the durable lessons of an older, richer one.

Tier 0: this repository

• docs/fern/pages/recipes/_catalog/: the schema-validated machine-readable index (index.yaml, recipes/*.yaml, validated by validate.py against schema.json). Match on model.hf_id, targets[].hardware, targets[].runtime.framework, and targets[].topology. Two fields live at different levels: status (validated or experimental) is an ENTRY-level property and gates every target under it, while recommended is a TARGET-level property. Prefer targets with recommended: true under entries with status: validated; an experimental entry caps all of its targets at hypothesis. Surface the entry's gaps list in the dossier rather than hiding it.
• The matched entry's deploy.asset points at the recipes/<model>/.../deploy.yaml DynamoGraphDeployment and its benchmark manifests; expected_performance.summary carries the measured numbers when available: true.

A Tier 0 match with a resolvable image is the best possible outcome: zero translation, attached perf, and a validated flag. Use it and stop. A Tier 0 entry that structurally matches but fails the verdict bar (mutable or unresolved image, experimental status, unsatisfiable prerequisites, stale pin) is recorded with its lower grade and the walk CONTINUES: a structural match is not a usable match.

Tier 1: authoritative external catalogs

Consult in this order when Tier 0 yields no candidate meeting the required verdict.

1. vllm-project/recipes (vLLM engine). Consume the JSON API, not the YAML files: https://recipes.vllm.ai/models.json, then /<hf_org>/<hf_repo>.json, then /<hf_org>/<hf_repo>/hw/<hardware>.json. Read the MODEL-LEVEL JSON first and take three things from it, because they do not appear on the per-hardware response: the version floor at $.model.min_vllm_version (variants may carry their own under $.variants.<name>.min_vllm_version), the verification signal at $.meta.hardware, a hardware-to-status map (a hardware key absent from that map is unverified for this model even if /hw/<hardware>.json renders a config), and the freshness date at $.meta.date_updated. Then fetch the per-hardware response and branch on its deploy_type: for single_node it returns argv, docker_image, env, and a hardware_profile whose gpu_count is the total; for multi_node it returns head_argv, worker_argvs, node_count, and a hardware_profile whose gpu_count is PER NODE, so the total requirement is node_count times gpu_count (the strategy_spec names the interconnect assumption, for example InfiniBand for multi-node TP). Never read the flat single-node fields from a multi-node response; a missing argv is a schema branch, not an absent recipe. Two caveats: most recipes fall back to a mutable latest image (the tag gate then classifies them ceiling-only unless the engagement pins its own image), and the exported JSON drops the prefill/decode strategy_overrides; for disaggregated topology, read the recipe's YAML from the git repo and compose against its strategies.json.
2. NVIDIA/srt-slurm-recipes for frontier models on Blackwell-class hardware, especially disaggregated and multi-node. Recipes are SLURM-shaped but carry the full engine configuration, worker split, and image. Exclude **/agentic/ and *-sa/ paths from deployable candidates: those port externally tuned benchmark configs and are quarantined to ceiling-only (see Tier 4). Expect a meaningful fraction of container tags to fail the gate.
3. llm-d/llm-d guides/ ("well-lit paths"). Tested, benchmarked, Kubernetes-native recipes with sized prefill/decode Deployments; when the target model is covered, the best public source of sized disaggregation topology. Two checks before any llm-d candidate is graded above hypothesis: (a) IMAGE CAPABILITY: the guide's image component may select a stock upstream image or an llm-d-hosted variant (ghcr.io/llm-d/llm-d-*) that carries patches upstream lacks, such as the NVSHMEM fix for RoCE; record which, and treat a patched-variant dependency as a prerequisite the target must satisfy, not a stock image; (b) COORDINATION TRANSLATION: guides that use LeaderWorkerSet (LWS_WORKER_INDEX, LWS_GROUP_SIZE, LWS_LEADER_ADDRESS) compute ranks and addresses in shell at start-up, which has no literal DGD equivalent; those manifests are not a mechanical translation and stay hypothesis until the multi-node coordination is re-expressed in Dynamo's own terms and verified. Only single-pod-per-worker guides with literal args: arrays on stock images translate near-mechanically.