Model Garden Deployment

Agent Platform Model Garden Deploy Skill

This skill provides instructions for deploying Open Models from Agent Platform Model Garden to endpoints, and subsequently undeploying them to clean up resources.

1P Tuned Model Copy & Deployment

If you need to copy a 1P (First-Party) Tuned Model from a source project to a destination region or project and deploy it to a newly created endpoint, refer to the 1P Tuned Model Copy & Deployment Guide.

Safety & Confirmation Tiers (CRITICAL)

Before executing any commands on behalf of the user, you MUST adhere to the following safety tiers based on the action requested:

1. Tier R: Read-only (list, describe, list-deployment-config)
   • Rule: No confirmation needed. You may execute these commands immediately to gather information for the user.
2. Tier M: Mutating & Reversible (deploy, undeploy-model)

   • Rule: This requires explicit user confirmation. You MUST present a clear dry-run confirmation card containing:

     1. Exact proposed request: the :deploy request body for a deployment (§3), or the gcloud command code block for undeploy-model.
     2. Model identifier, destination project ID, and target region.
     3. Machine type and accelerator configuration.
     4. Estimated hourly cost ($/hr).
     5. Endpoint display name.
     6. Explicit confirmation prompt asking the user to approve before execution. You MUST wait for their explicit confirmation before executing. For undeploy-model, you MUST first verify that the endpoint and deployed model exist; if describe or list returns a 404 or empty result, you MUST halt and inform the user rather than attempting undeployment.

   • Same-turn restriction: Do not run the command in the same turn as presenting the confirmation prompt. End your turn after asking and wait for the user's reply; only execute after explicit approval. Printing a preview and then calling the tool before the user can answer does not count as obtaining confirmation.

3. Tier D: Destructive & Irreversible (delete)
   • Rule: This requires explicit typed confirmation. You MUST output a text message explaining the irreversible nature of endpoint or model deletion and asking the user to type "I confirm" or "Yes, delete it" before executing the deletion command.

[!IMPORTANT]

Always Output Complete Text Response (NEVER Emit Empty Text): After executing any tool call (such as the :deploy API call, gcloud ai endpoints delete, gcloud ai endpoints list, or status checks), you MUST formulate and return a complete, informative textual response to the user. Explicitly report the operation ID, endpoint name/ID, error message, or list of resources. NEVER finish a turn with empty text or silence.

1. Prerequisites

Before deploying, ensure you have the correct project and region set. The commands below use placeholder variables PROJECT_ID and LOCATION_ID.

Ensure you are authenticated:

gcloud auth login
gcloud auth application-default login
gcloud config set project $PROJECT_ID

2. Discovering Deployable Models

You can list models available in Model Garden and check if they can be self-deployed.

gcloud ai model-garden models list

To see what machine types and accelerators are supported for a specific model, pass a MODEL_ID you obtained from the models list output above. Substitute <PUBLISHER>/<FAMILY>@<VERSION-ID> below with the exact string from the catalog output — the placeholder is deliberately not a real model ID:

gcloud ai model-garden models list-deployment-config \
    --model="<PUBLISHER>/<FAMILY>@<VERSION-ID>"

[!NOTE] Some models, especially Hugging Face models, might require a Hugging Face Access Token for deployment.

[!TIP] Model Recommendation Instructions: Whenever you are about to name a specific model version in a response, do NOT recommend from memory. This applies in all of the following situations — not just direct deploy requests:

• The user asks to deploy a model without naming one.
• You are volunteering a next-step suggestion after a list, describe, or undeploy operation (e.g. "Would you like me to deploy <model> to this endpoint?").
• The user asks a general "what should I use?" / "what's a good model for X?" question.
• You are filling in a MODEL_ID value in an example command you are showing the user (as opposed to a placeholder like <PUBLISHER>/<FAMILY>@<VERSION-ID>).

New model versions ship frequently and older ones may be deprecated, so training-corpus knowledge of which models exist is unreliable. Follow this procedure:

1. Clarify the use case if it isn't already clear from context (task type, quality vs. latency vs. cost priorities, hardware/quota constraints, license constraints). Skip if the user has already given enough signal.
2. Query the live catalog with gcloud ai model-garden models list. Narrow with --filter when appropriate (e.g. --filter="name~gemma", --filter="name~llama", --filter="name~qwen", --filter="name~deepseek"). Never name a specific model version to the user until you have seen it in the catalog output for this project.
3. Pick the latest generally-available version in the family that fits the use case. When multiple size variants exist, pick the one that matches the user's hardware/cost tolerance. Prefer a newer major version over an older one unless it is marked preview/experimental and the user explicitly asked for a stable option.
4. Verify the exact model ID is deployable with gcloud ai model-garden models list-deployment-config --model="<publisher>/<family>@<version>" before naming it in your response.
5. Cite the model ID verbatim in your recommendation, exactly as it appears in the catalog. Do not paraphrase to a family label ("Gemma", "Llama").

The MODEL_ID values in the §3 examples below are intentionally non-substantive placeholders (<PUBLISHER>/<FAMILY>@<VERSION-ID>). Do NOT replace them with a remembered model name for a user-facing recommendation — always re-run steps 2-4 first, then cite the exact string from the catalog.

2.1 Region Availability Check (Gemini + LoRA only)

For first-party Gemini or LoRA deploys, you must verify region availability before proceeding. Load the full instructions with load_skill_resource(skill_name='agent-platform-deploy', file_path='references/region_availability.md').

Skip this for open-weights models (Gemma, Llama, DeepSeek, Qwen) and for Gemini-tuned models — they have no per-region publisher endpoint restriction. Go straight to §3.

3. Deploying a Model

[!WARNING] Deploying models, especially large ones, consumes significant compute resources and incurs costs.

1. You MUST compute an hourly $ estimate for the requested --machine-type before proposing a deploy. Try each source below in order, falling through to the next on any failure:

   • Run scripts/calculate_cost.py. The accelerator type and count are fixed per machine type in Model Garden and derived automatically. Example:

     python3 scripts/calculate_cost.py \
         --machine-type=g2-standard-48

     If the script exits non-zero (unknown --machine-type — a routine state for machines in the Model Garden catalog but not yet in the price snapshot, e.g. A4/B200 today), fall through to the next source. Do NOT invent a number.

   • Fall back to Agent Platform prediction pricing

if no source above produced a number. Read the accelerator + hourly rate directly off that page and cite the URL in the estimate you present to the user.

2. You MUST present this cost estimation to the user and warn them that this is the list price, which may differ from their actual bill due to potential discounts, reservations, or non-us-central1 regions.
3. You MUST ALWAYS request explicit confirmation from the user agreeing to the estimated cost before executing any deploy command.

To deploy an open-weights Model Garden model, call the :deploy API directly with curl.

If a deployment is rejected for quota, report the API's error verbatim.

[!IMPORTANT]

• Cost Pushback & Hardware Renegotiation: If the user pushes back on cost (e.g., "That is too expensive, can you try a smaller configuration?"), or requests an invalid or unsupported hardware combination (e.g. g2-standard-48g with 4x H100 GPUs), explain the constraint or invalidity clearly, check list-deployment-config to identify the supported alternative (e.g., g2-standard-24 with 2x L4 or g2-standard-12 with 1x L4), compute its cost estimate with a single query, and immediately render a complete Tier M dry-run confirmation card for that recommended configuration in the same response.
• Region Failover & Quota Exhaustion: When a deployment fails due to quota or capacity in the requested region (e.g. QUOTA_EXCEEDED or RESOURCE_EXHAUSTED), identify an alternative supported region (e.g. us-east4 or us-east1), compute its cost estimate with a single query, and immediately render a complete Tier M dry-run confirmation card with the new --region and exact command in the same response. State the alternative region directly without making unverified capacity claims.
• Efficient Tool Execution (No Redundant Calls): Do NOT execute redundant models list, list-deployment-config, or --help commands if the model ID, region, or hardware configuration are already known or resolved. Run each discovery command strictly once.
• Single Status Check & Response Formatting (CRITICAL):
  • When initiating a deployment (the :deploy call in §3), the response immediately returns a long-running operation. Formulate and return your textual confirmation response with the operation ID and endpoint display name immediately. Do NOT call operations describe in the same turn as deployment initiation.
  • When the user explicitly asks to check deployment status (e.g., "Please check to see the status of the deployment" or "Can you check if the deployment has finished?"):
    • NEVER run sleep commands, while loops, or repeated polling calls.
    • Execute gcloud ai operations describe <OP_ID> --region=<REGION> strictly ONCE.
    • ALWAYS output a full textual response reporting the operation status (e.g. "The deployment operation projects/.../operations/... is currently in progress / running (created at ...). Asynchronous model deployment typically takes 10–15 minutes to complete").
    • Only proceed with sending a test prediction if the status check confirms the endpoint is already serving and ready.
• Alphanumeric Project ID: Always specify the alphanumeric Project ID (e.g. my-gcp-project) for --project, NOT the numeric project number (e.g. 123456789012). If given a numeric project number and its Project ID is not available, pass the number inside a fully-qualified resource name, e.g. gcloud ai endpoints list --region=projects/123456789012/locations/us-central1, or as a positional resource, gcloud ai endpoints describe projects/123456789012/locations/us-central1/endpoints/<ENDPOINT_ID> --region=us-central1. If a command is still refused because core/project is set to a project number, the sandbox's own project was seeded as a number: every gcloud ai call then needs an explicit --project=<PROJECT_ID>, which a resource name cannot substitute for. Report that instead of retrying.
• Valid User-Specified Hardware Priority: When the user specifies an explicit, valid hardware configuration (e.g. g2-standard-96 with 8 NVIDIA_L4 GPUs, or g2-standard-12 with 1 NVIDIA_L4 GPU), honor that requested configuration for the dry-run preview and cost estimation rather than overriding it with default recommendations. However, if the requested configuration is invalid or unsupported (e.g. mismatched GPU count such as g2-standard-12 with 2 L4 GPUs, or non-existent machine shapes), follow the Cost Pushback & Hardware Renegotiation rule above: explain the invalidity clearly, identify the supported alternative (e.g. g2-standard-24 with 2 L4 GPUs), calculate its cost, and immediately present the confirmation card for the valid alternative.
• Endpoint Display Name: If the user specifies or requests an endpoint name or display name (e.g. 'usersim-gemma-eval-...'), you MUST always include --endpoint-display-name="<NAME>" in the deploy command.