LLM D Deploy Stack

Deploy llm-d Stack Skill

Follow these instructions to deploy the llm-d benchmarking stack on GKE.

Terminology & Variables

This skill utilizes the following core variables:

• SPEC (also referred to as strategy or guide): This represents the name of the llm-d well-lit path guide being targeted. It maps directly to GKE overlay folder structures (llmd-<spec>)

  • Allowed Values:
    • optimized-baseline (Standard optimized baseline configuration).
    • precise-prefix-cache-routing (Includes precise cache routing for multi-turn workloads).
    • predicted-latency-routing (Includes dynamic latency-based routing overlays).
    • pd-disaggregation (Separates prefill and decode onto dedicated model servers. TPU v6e only, and only with the qwen/qwen3-32b model).

• CRITICAL WARNING:

  1. NEVER run any teardown-*.sh scripts (e.g., teardown-llmd-optimized-baseline.sh) when attempting to "fix and rerun" a deployment or remove a workload. These scripts default to a full core platform teardown (ACP_TEARDOWN_CORE_PLATFORM=true) and will completely destroy the GKE cluster and all its resources. If a deployment fails, debug in place or re-run the deploy scripts. ONLY run teardown script when the user explicitly asks to tear down the stack and confirm with you first.
  2. NEVER Store a secret value or retrieve a value of a secret in memory or write it to any file. Instead, always use gcloud secrets describe [SECRET_NAME] OR kubectl get kubectl get secret [SECRET_NAME] -o yaml. If a secret does not exist ask the user to create it.

1. Prerequisites & Cluster Setup

1. Ask the user: "Is there an existing GKE cluster? (yes/no)"
2. If NO:
   • Ask the user: "What platform name would you like to use for the new cluster, and what is your Google Cloud project ID?"
   • Use sed to inject these values into ${ACP_REPO_DIR}/platforms/gke/base/_shared_config/platform.auto.tfvars:

     sed -i 's/^platform_name.*/platform_name = "<platform_name>"/g' "${ACP_REPO_DIR}/platforms/gke/base/_shared_config/platform.auto.tfvars"
     # If platform_default_project_id doesn't exist, append it:
     grep -q "^platform_default_project_id" "${ACP_REPO_DIR}/platforms/gke/base/_shared_config/platform.auto.tfvars" || echo "platform_default_project_id = \"\"" >> "${ACP_REPO_DIR}/platforms/gke/base/_shared_config/platform.auto.tfvars"
     sed -i 's/^platform_default_project_id.*/platform_default_project_id = "<project_id>"/g' "${ACP_REPO_DIR}/platforms/gke/base/_shared_config/platform.auto.tfvars"

   • Proceed to Section 2: Configure Model & Accelerator (to configure variables and create cluster).
3. If YES:
   • Verify connectivity to the target cluster:

     kubectl cluster-info

   • Ask the user: "Is the llm-d stack already deployed on this cluster? (yes/no)"
   • If YES:
     • Skip to Section 4: Hugging Face Token Setup (we only need to deploy the model server, which is done in Section 5).
   • If NO:
     • Proceed to Section 2: Configure Model & Accelerator (to configure the deployment variables).

2. Configure Model & Accelerator

1. Ask the user: "Which model would you like to run?"
   • Allowed Models:
     • google/gemma-4-31b-it
     • qwen/qwen3-32b (default)
     • qwen/qwen3-32b-fp8
     • redhatai/gemma-4-31b-it-fp8-block
   • Validation: If the user inputs anything else, flag it as invalid, display the allowed models, and ask again.
2. Ask the user: "Which accelerator would you like to use?"
   • Allowed Accelerators:
     • rtx-pro-6000 (default)
     • h100 (translates to nvidia-h100)
     • h200 (translates to nvidia-h200)
     • v6e (TPU, translates to google-tpu-v6e)
   • Validation: If the user inputs anything else, flag it as invalid, display the allowed accelerators, and ask again.
3. Update configuration:
   • Use sed to inject the chosen model and accelerator into ${ACP_REPO_DIR}/platforms/gke/base/use-cases/inference-ref-arch/examples/llmd/_shared_config/llmd.auto.tfvars:

     echo "llmd_model_id = \"<chosen_model>\"" >> "${ACP_REPO_DIR}/platforms/gke/base/use-cases/inference-ref-arch/examples/llmd/_shared_config/llmd-shared.auto.tfvars"
     echo "llmd_accelerator_type = \"<chosen_accelerator>\"" >> "${ACP_REPO_DIR}/platforms/gke/base/use-cases/inference-ref-arch/examples/llmd/_shared_config/llmd-shared.auto.tfvars"

3. Select & Deploy Well-Lit Path Guide

1. Ask the user: "Which llm-d well-lit path guide would you like to deploy?"

   • Allowed Options:
     • optimized-baseline (corresponds to llmd-optimized-baseline-vllm-with-hf-model.md)
     • precise-prefix-cache-routing (corresponds to llmd-precise-prefix-cache-routing-vllm-with-hf-model.md)
     • predicted-latency-routing (corresponds to llmd-predicted-latency-routing-vllm-with-hf-model.md)
     • pd-disaggregation (corresponds to llmd-pd-disaggregation-vllm-with-hf-model.md)
   • Validation: If the user inputs anything else, flag it as invalid, display the list of allowed options, and ask again.
   • Validation: If the user chose pd-disaggregation, the accelerator MUST be v6e and the model MUST be qwen/qwen3-32b. If either differs, explain that this guide is currently TPU-only in this repository and return to Section 2 to reconfigure.

2. Deploy the baseline stack: Run the deployment script corresponding to the chosen guide to create the cluster (if new) and deploy the baseline infra/services:

   • For optimized-baseline:

     "${ACP_REPO_DIR}/platforms/gke/base/use-cases/inference-ref-arch/examples/llmd/deploy-llmd-optimized-baseline.sh"

   • For precise-prefix-cache-routing:

     "${ACP_REPO_DIR}/platforms/gke/base/use-cases/inference-ref-arch/examples/llmd/deploy-llmd-precise-prefix-cache-routing.sh"

   • For predicted-latency-routing:

     "${ACP_REPO_DIR}/platforms/gke/base/use-cases/inference-ref-arch/examples/llmd/deploy-llmd-predicted-latency-routing.sh"

   • For pd-disaggregation:

     "${ACP_REPO_DIR}/platforms/gke/base/use-cases/inference-ref-arch/examples/llmd/deploy-llmd-pd-disaggregation.sh"

3. Run validation:

   • Run the following command to verify the chosen custom compute class exists on the cluster:

     kubectl get computeclasses

   • Ensure that the accelerator type you chose appears in the list. If it does not, warn the user that the accelerator is unsupported or missing its custom compute class.
   • For pd-disaggregation, the required compute class is tpu-v6e-2x4 (8 chips), not the tpu-v6e-2x2 used by the other guides.