Deploy Dynamo Recipe

Deploy Dynamo Recipe

<!--
SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
SPDX-License-Identifier: Apache-2.0
-->

Purpose

Deploy exactly one assigned Dynamo Kubernetes DGD and return a small smoke-test artifact. This skill does not search the recipe catalog, choose or substitute a DGD, tune knobs, benchmark performance, or create new recipes.

Input ownership:

• First Optimization Iteration: user-interviewer provides the canonical user-provided DGD path and SHA256.
• Subsequent Optimization Iterations: hypothesis-challenger provides the candidate deploy.yaml or DGD.
• The synthesized user_workload.yaml supplies the Kubernetes context, namespace, optional storage class, and baseline DGD path-and-hash record.

Inputs

Required:

• assigned DGD manifest path and SHA256
• handoff provenance: user-interviewer for iteration 0 or hypothesis-challenger for iteration > 0
• exact <EXP_ROOT>/user_workload.yaml path and SHA256
• target namespace and kubectl context from user_workload.yaml
• experiment root created by user-interviewer
• zero-based optimization iteration
• previous deployment root for iteration > 0

Optional:

• storage class, only when model-cache PVCs need one
• smoke prompt; default to Simply output the phrase: NVIDIA Dynamo

Secrets:

• Never ask the user to paste token values into the agent conversation.
• Treat Kubernetes secrets referenced by the selected manifests as pre-existing cluster prerequisites.
• Check referenced secrets only by name. If one is missing, record a blocker; do not ask for its value or create it.
• Current recipes commonly expect hf-token-secret with key HF_TOKEN for gated Hugging Face model access.

Workflow

Recompute the supplied user_workload.yaml SHA256 before using its Kubernetes and workload context. Recompute the assigned DGD SHA256 and require it to match the handoff before creating run-scoped copies. At iteration 0, also require the assigned path and SHA256 to equal deployment.dgd_path and deployment.dgd_sha256 in user_workload.yaml.

1. Create The Deployment Directory

Create exactly one directory for the assigned candidate:

<EXP_ROOT>/artifacts/deploy-iter-<NNN>/

Create applied_manifests/ beneath it. Copy the assigned DGD and every explicitly handed-off support manifest used by the deployment into that directory with stable names such as deploy.yaml, model-cache.yaml, model-download.yaml, and model-validate.yaml — normalizing the filename at copy time. A recipe may ship variant-specific manifests (recipes/deepseek-v4/* ship model-download-fp8.yaml and model-download-nvfp4.yaml): select the one matching the assigned DGD's precision and copy it as model-download.yaml. Few recipes ship a validation job at all. Copy what the handoff actually contains. Never modify the handed-off source files.

Update these run-scoped copies in place when a compatibility fix is required, then reapply them. Record every change and reason in deployment_ledger.json; do not retain numbered intermediate copies. After a successful smoke test, applied_manifests/ must contain exactly one final file per manifest type used, and those files must be the exact set that produced the successful deployment. If the deployment is blocked, retain only the latest attempted copies and mark the ledger blocked. Create logs/ only when a targeted failure log must be retained.

2. Validate The Assigned DGD

Run read-only checks first:

set -euo pipefail
kubectl --context "${KUBE_CONTEXT}" get namespace "${NAMESPACE}"
# CRD presence gate: a Forbidden here is tolerated because the server dry-run below
# re-checks it authoritatively; a confirmed absence stops before any mutation.
crds="$(kubectl --context "${KUBE_CONTEXT}" get crd 2>&1 || true)"
case "${crds}" in
  *Forbidden*) echo "WARN: cluster-scope CRD list forbidden for this identity; deferring to server dry-run" ;;
  *dynamographdeployment*) : ;;
  *) echo "Dynamo CRDs missing"; exit 1 ;;
esac
# Advisory reads: storage classes and node inventory inform sizing but a namespace-scoped
# identity may lack cluster-scope list rights. Record a Forbidden as a run limitation; do not fail.
kubectl --context "${KUBE_CONTEXT}" get storageclass || echo "WARN: storageclass list forbidden; record as limitation"
kubectl --context "${KUBE_CONTEXT}" get nodes -o wide || echo "WARN: node list forbidden; record as limitation"

Every kubectl call in this skill pins --context "${KUBE_CONTEXT}" (the contract's kube_context); never rely on the ambient current-context.

Validate the selected path without mutating the cluster:

kubectl --context "${KUBE_CONTEXT}" apply --dry-run=server -n "${NAMESPACE}" \
  -f <assigned-dgd-yaml>

Review the assigned DGD and any support manifests explicitly included in the handoff. Check:

• DGD name and frontend service name
• model-cache PVCs and storage class needs
• model download or validation jobs
• secrets referenced by secretKeyRef, envFromSecret, or imagePullSecrets
• GPU requests, node selectors, tolerations, and GPU SKU expectations

Stop before mutation if required namespace, CRDs, PVC prerequisites, secret names, storage class, images, or GPU capacity are missing. Also verify before mutation that the assigned manifest changes no knob listed in the contract's resources.pinned and that total concurrent GPU holdings stay within resources.gpu_ceiling.

When checking GPU capacity, count every pod that is bound to a node (spec.nodeName set) and not in a terminal phase (Succeeded/Failed) as holding its full GPU request. Do not filter on phase == Running: pods in ImagePullBackOff, ContainerCreating, or init hold their reservations. Exclude nodes whose taints the assigned manifest does not already tolerate, and never add new tolerations for other tenants' reservation taints. Evaluate fit by expanding the DGD into its full multiset of pod demands (every component, every replica) and placing them against per-node free blocks while decrementing remaining capacity — two pods cannot count the same free GPUs. Honor each pod's node selectors, required affinity/anti-affinity, and tolerations during placement. If the DGD cannot be faithfully expanded into pod demands, report capacity as unknown, not sufficient. When resources.gpu_ceiling is set in the workload contract, also verify the run's total concurrent GPU holdings stay within it.

If a manifest must change only to work with the target cluster, such as resolving a storage class placeholder or adding a required node-taint toleration, update only the copy under applied_manifests/. Preserve the handed-off source and record the exact change and reason in deployment_ledger.json. Do not change performance knobs.

3. Retire The Previous Iteration

For iteration > 0, read the previous deployment ledger and delete only its DGD by exact name, namespace, and context. Wait for the DGD and its operator-owned workloads to terminate before applying the new candidate. Record the deletion in the new deployment ledger, and write torn_down_at into the RETIRED iteration's deployment_ledger.json (the sole permitted modification of a previous iteration directory).

set -euo pipefail
kubectl --context "${PREVIOUS_KUBE_CONTEXT}" delete dynamographdeployment "${PREVIOUS_DGD}" \
  -n "${PREVIOUS_NAMESPACE}" --wait=true --timeout=10m
kubectl --context "${PREVIOUS_KUBE_CONTEXT}" wait --for=delete pod \
  -l nvidia.com/dynamo-graph-deployment-name="${PREVIOUS_DGD}" \
  -n "${PREVIOUS_NAMESPACE}" --timeout=10m

Do not delete or modify the previous deployment directory or its successful YAML, except for writing torn_down_at into its deployment_ledger.json at teardown time. Create new run-scoped copies in the new iteration directory. Preserve shared PVCs, model-cache jobs, namespaces, and secrets.

4. Apply Support Manifests

Follow user-provided deployment instructions when they give a specific sequence. Otherwise:

If the effective cluster context differs from what <EXP_ROOT>/manifest.yaml records, update the manifest's cluster-context entry before mutating anything.

Read each support manifest's kind and metadata.name; never infer a Kubernetes resource name from its filename. Set DOWNLOAD_JOB and VALIDATE_JOB from the corresponding Job manifests. The run-scoped copies are already normalized to the stable filenames above, so the applies below reference those names directly; skip a block when the recipe ships no such manifest.

set -euo pipefail
kubectl --context "${KUBE_CONTEXT}" apply -f "${DEPLOY_ROOT}/applied_manifests/model-cache.yaml" -n "${NAMESPACE}"
kubectl --context "${KUBE_CONTEXT}" apply -f "${DEPLOY_ROOT}/applied_manifests/model-download.yaml" -n "${NAMESPACE}"
job_state=""
for _ in $(seq 1 200); do  # 200 x 30s = 100 min bound
  # NOTE: match by substring - a successful Job on Kubernetes 1.31+ carries BOTH
  # SuccessCriteriaMet and Complete conditions, so the jsonpath returns them space-separated.
  job_state="$(kubectl --context "${KUBE_CONTEXT}" get "job/${DOWNLOAD_JOB}" -n "${NAMESPACE}" \
    -o jsonpath='{.status.conditions[?(@.status=="True")].type}')"
  case "${job_state}" in *Failed*) echo "download job failed"; exit 1;; *Complete*) break;; esac
  sleep 30
done
case "${job_state}" in *Complete*) : ;; *) echo "download job timed out"; exit 1;; esac

If a validation job exists, run it after download and before the DGD:

set -euo pipefail
kubectl --context "${KUBE_CONTEXT}" apply -f "${DEPLOY_ROOT}/applied_manifests/model-validate.yaml" -n "${NAMESPACE}"
job_state=""
for _ in $(seq 1 120); do  # 120 x 30s = 60 min bound
  job_state="$(kubectl --context "${KUBE_CONTEXT}" get "job/${VALIDATE_JOB}" -n "${NAMESPACE}" \
    -o jsonpath='{.status.conditions[?(@.status=="True")].type}')"
  case "${job_state}" in *Failed*) echo "validate job failed"; exit 1;; *Complete*) break;; esac
  sleep 30
done
case "${job_state}" in *Complete*) : ;; *) echo "validate job timed out"; exit 1;; esac

5. Apply The Assigned DGD

Apply only the run-scoped copy of the assigned manifest:

set -euo pipefail
kubectl --context "${KUBE_CONTEXT}" apply -f "${DEPLOY_ROOT}/applied_manifests/deploy.yaml" -n "${NAMESPACE}"
kubectl --context "${KUBE_CONTEXT}" get dynamographdeployment -n "${NAMESPACE}"
kubectl --context "${KUBE_CONTEXT}" get pods -n "${NAMESPACE}" -o wide
kubectl --context "${KUBE_CONTEXT}" get svc -n "${NAMESPACE}"

Do not apply sibling variants. Do not change engine arguments, GPU counts, replica topology, routing, or other performance settings unless the parent supplied them as part of the assigned candidate. Kubernetes-only compatibility patches are allowed when required and recorded.