SageMaker Production Defaults

SageMaker Production Defaults

The difference between a demo endpoint and one you can leave running is: it scales with traffic, it tells you when it breaks, and you can debug it later. This skill makes those three the default rather than optional extras.

By the time this skill runs, the planner has chosen a real-time endpoint, IAM has a usable role, and image-selection has resolved a container URI + AMI version. This skill turns those into an actual deployment.

What gets created

For every endpoint, the skill creates these as a unit:

1. SageMaker Model — image + env vars + execution role + S3 artifacts
2. Endpoint config — instance type, initial count, optional data capture
3. Endpoint — the real-time endpoint serving inference
4. Autoscaling target + policy — target tracking on invocations per instance
5. CloudWatch alarms — latency, errors, platform overhead

An inference-component deployment (deploy_ic.py) creates the same set with two changes: the endpoint config carries the execution role and ManagedInstanceScaling, and an inference component carries the model. Its autoscaling target is the component, not the variant.

Data capture (logging requests/responses to S3) is off by default — useful for debugging but creates ongoing S3 costs the user didn't necessarily ask for. Enable with --enable-data-capture.

All resources get a consistent tag set including CreatedBy=agentic-deploy-skills for later cleanup.

Defaults and reasoning in references/deployment-template.md.

Running the deployment

For a text-generation LLM (vLLM), first package the pinned model snapshot as a SageMaker model.tar.gz artifact and upload it to an account-controlled S3 bucket. SageMaker extracts it under /opt/ml/model, so the deployment does not fetch or execute repository code at runtime:

python scripts/deploy.py \
    --model-name qwen3-medical \
    --image-uri "$IMAGE_URI" \
    --inference-ami-version "$AMI" \
    --role-arn "$ROLE_ARN" \
    --model-s3-uri "s3://<your-model-bucket>/qwen3-medical/model.tar.gz" \
    --instance-type ml.g5.xlarge \
    --region "$REGION" \
    --env SM_VLLM_MODEL=/opt/ml/model \
    --env SM_VLLM_HOST=0.0.0.0 \
    --env SM_VLLM_TRUST_REMOTE_CODE=false \
    --env SM_VLLM_MAX_MODEL_LEN=4096

For an embedding model (TEI, often on CPU):

python scripts/deploy.py \
    --model-name bge-large-embeddings \
    --image-uri "$IMAGE_URI" \
    --role-arn "$ROLE_ARN" \
    --instance-type ml.c6i.2xlarge \
    --region "$REGION" \
    --env HF_MODEL_ID=BAAI/bge-large-en-v1.5

Note: TEI deployments do not need --inference-ami-version. That flag is vLLM-specific. TEI env vars are also simpler (HF_MODEL_ID instead of SM_VLLM_*, no host or trust-remote-code to configure).

Where each value comes from:

The script creates resources in order with error handling, waits for InService (up to 30 min), surfaces failure reasons, registers autoscaling and alarms, and prints a summary including the teardown command. Outputs a JSON blob on stdout with endpoint/config/model names for downstream scripting.

The scripts ship with this skill. If the installed copy is missing the scripts/ directory (some harnesses copy only SKILL.md on install), fetch them from the source repo rather than re-implementing them from this description.

Model-loading default: pre-stage pinned weights in S3 and pass --model-s3-uri; the container loads them from /opt/ml/model without a runtime Hub dependency. Loading from the Hub is an explicit exception: expect a 5–15+ minute download after endpoint startup, provide a token for gated models, and keep SM_VLLM_TRUST_REMOTE_CODE=false unless the specific architecture requires reviewed custom code. deploy.py waits 30 minutes.

InService is not success — smoke-test before declaring victory

InService only means the container answered /ping. In MMS-based containers (HF Inference Toolkit) the Java front-end answers pings even while the Python worker crash-loops — an endpoint can be InService and serve nothing. Two checks, always:

1. One real invocation.

   • Real-time: invoke_endpoint.py (below) with a minimal payload; require an HTTP 200 with a sane body.
   • Async: upload one input to S3, call invoke-endpoint-async, poll the output URI for a few minutes (see "Invoking async endpoints"). A result object = success; an object at the failure URI, or nothing appearing, = broken.

2. Scan the endpoint logs for worker-crash markers — catches the crash-loop case even when the smoke request merely times out:

   aws logs filter-log-events \
       --log-group-name /aws/sagemaker/Endpoints/<endpoint-name> \
       --filter-pattern '?"Worker died" ?"Load model failed" ?"ImportError"' \
       --region <region> --max-items 5

   Inference-component deployments log to /aws/sagemaker/InferenceComponents/<component-name> instead. deploy_ic.py scans that group automatically while it waits.

General rule for denied diagnostics: when a read-only call the workflow uses for diagnosis is denied (a restricted role without logs:FilterLogEvents, servicequotas:ListServiceQuotas, and so on), say so in one line and carry on with the checks that do work. Never block a deployment on a permission needed only for diagnosis, and never read a denied call as evidence that nothing is wrong.

Only report the deployment complete after both pass. If the log scan hits, surface the actual traceback from CloudWatch — not the InService status.

Testing a real-time endpoint

Once the endpoint is InService, test it with the bundled helper. It is cross-platform and BOM-safe — use it instead of hand-writing a payload file and calling invoke-endpoint directly:

# macOS / Linux
python3 scripts/invoke_endpoint.py \
    --endpoint-name <endpoint-name> \
    --payload '{"inputs": "Hello"}' \
    --region "$REGION"

# Windows (PowerShell)
python scripts\invoke_endpoint.py `
    --endpoint-name <endpoint-name> `
    --payload-file payload.json `
    --region $REGION

It accepts either --payload '<json>' (inline) or --payload-file <path>, validates JSON, writes the request body as plain UTF-8, invokes the endpoint, and prints the response body to stdout.

The UTF-8 BOM gotcha (Windows)

If you write the request payload yourself on Windows, do not use Set-Content -Encoding UTF8 — depending on the PowerShell version it prepends a UTF-8 byte-order mark (BOM). SageMaker's JSON parser rejects a BOM with a 400 ModelError:

Unexpected UTF-8 BOM (decode using utf-8-sig): line 1 column 1 (char 0)

This is not a model, endpoint-health, or image problem — only the file encoding of the request body. invoke_endpoint.py avoids it entirely (it even strips a BOM from a --payload-file that already has one). If you must call the CLI directly, write the body as BOM-free UTF-8:

# BOM-free UTF-8 — use this
[System.IO.File]::WriteAllText((Resolve-Path "payload.json"), $json, [System.Text.UTF8Encoding]::new($false))

aws sagemaker-runtime invoke-endpoint `
    --endpoint-name <endpoint-name> `
    --content-type application/json `
    --body fileb://payload.json `
    --region $REGION `
    response.json

Fallback: if any invocation fails with Unexpected UTF-8 BOM, rewrite the payload as BOM-free UTF-8 (or re-run via invoke_endpoint.py) and retry once before treating the endpoint or model as broken.

Invoking a generative reranker (vLLM)

Generative rerankers (Qwen3-Reranker etc. — routed to the HuggingFace vLLM DLC by hf-cloud-serving-image-selection) are causal LMs scored by their first generated token, not chat models. Use the completions API with a raw prompt, not the messages/chat API: chat templating does not reliably honor chat_template_kwargs such as {"enable_thinking": false}, and a wrong template silently returns near-identical scores for every query–document pair instead of erroring.

Payload shape (Qwen3-Reranker's expected format — substitute {query} / {document}):

{
  "prompt": "<|im_start|>system\nJudge whether the Document meets the requirements based on the Query and the Instruct provided. Note that the answer can only be \"yes\" or \"no\".<|im_end|>\n<|im_start|>user\n<Instruct>: Given a web search query, retrieve relevant passages that answer the query\n<Query>: {query}\n<Document>: {document}<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n",
  "max_tokens": 1,
  "temperature": 0,
  "logprobs": 20
}

The trailing <|im_start|>assistant\n<think>\n\n</think>\n\n suffix is load-bearing: it pre-fills an empty thinking block so the first generated token is the yes/no judgment. Score from the returned logprobs: P("yes") / (P("yes") + P("no")). Sanity check the endpoint with one relevant pair (expect >0.9) and one irrelevant pair (expect <0.05) — near-identical scores across pairs mean the prompt template is wrong, not that the model is broken.

The same rule generalizes: for any thinking-mode model where the prompt must be byte-exact, prefer the raw completions API over chat.

Picking the image URI

The agent reads the image URI from AWS's Deep Learning Containers catalog — pick the row that matches the model family (HuggingFace vLLM for LLMs, TEI for embeddings, etc.), substitute <region> with the deployment region, and pass to deploy.py --image-uri.

For vLLM images specifically (both huggingface-vllm and the AWS vllm fallback), also check the tag's CUDA version:

# Example: HuggingFace vLLM 0.21.0 from the catalog
IMAGE_URI="763104351884.dkr.ecr.eu-west-1.amazonaws.com/huggingface-vllm:0.21.0-transformers5.8.1-gpu-py312-cu130-ubuntu22.04"

# cu130 tag → must pass --inference-ami-version
python deploy.py --image-uri "$IMAGE_URI" \
    --inference-ami-version al2-ami-sagemaker-inference-gpu-3-1 \
    ...

For tags with cu129 or lower, omit --inference-ami-version. See hf-cloud-serving-image-selection for the full vLLM AMI lookup table and the env-var requirements for each image family.

Scale to zero for real-time endpoints

A real-time endpoint reaches zero instances only when it hosts inference components. The variant-scoped target that deploy.py registers cannot go below one instance. deploy_ic.py builds the component-based shape instead.

Use it when traffic is sparse or scheduled, and the client tolerates a multi-minute first request. Do not use it for interactive traffic with an SLA: the wake takes minutes, and every request during the wake fails.

python scripts/deploy_ic.py \
    --model-name qwen3-scale-to-zero \
    --image-uri "$IMAGE_URI" \
    --inference-ami-version "$AMI" \
    --role-arn "$ROLE_ARN" \
    --model-s3-uri "s3://<your-model-bucket>/qwen3-scale-to-zero/model.tar.gz" \
    --instance-type ml.g5.xlarge \
    --region "$REGION" \
    --env SM_VLLM_MODEL=/opt/ml/model \
    --env SM_VLLM_HOST=0.0.0.0 \
    --env SM_VLLM_TRUST_REMOTE_CODE=false \
    --env SM_VLLM_MAX_MODEL_LEN=4096

How it differs from deploy.py

InferenceAmiVersion still belongs on the variant, and it coexists with ManagedInstanceScaling (verified on cu130 + ml.g5.xlarge).

Four pieces make zero work, and all four are required. Target tracking cannot leave zero, because it cannot divide by zero copies. Drop the step policy or its alarm and the endpoint scales to zero once, then never answers again. deploy_ic.py wires all four.

Measured behaviour

Qwen/Qwen3-0.6B from the Hub, ml.g5.xlarge, us-east-1, July 2026:

Scale-in is not tunable through this skill: Application Auto Scaling creates the AlarmLow itself with a 10 s period and 90 evaluation periods, so 15 minutes of idle datapoints are required before it fires.

Pre-stage production weights and pass --model-s3-uri. Besides avoiding runtime Hub access, this shortens every wake: model artifacts are loaded again on every wake, so their retrieval sits on the critical path of the first request after each idle period.

Sizing the component

ComputeResourceRequirements is a scheduling reservation, not a cap. A component that requests 1024 MB runs vLLM with several GB of host memory without trouble.

The schedulable pool is far smaller than the instance memory. On ml.g5.xlarge (16 GiB) the scheduler accepts 1024 MB and rejects 4096 MB. Over-asking gives an instant, confusing failure:

There is not enough hardware resources on the instances for this endpoint to
create a copy of the inference component.

That message appears even when the endpoint has a healthy instance. Treat it as "the request is too large", not "add instances". Start at the 1024 MB default and raise it only when several components share one instance.

--accelerator-devices must match SM_VLLM_TENSOR_PARALLEL_SIZE for multi-GPU models.

Invoking and testing

Pass the component name, and allow for the wake:

python3 scripts/invoke_endpoint.py \
    --endpoint-name <endpoint-name> \
    --inference-component-name <component-name> \
    --payload '{"prompt": "hello", "max_tokens": 16}' \
    --wait-for-capacity 900 --region "$REGION"

The 400 has no capacity error is the wake signal, not a fault: it publishes the metric that triggers the step policy. With --wait-for-capacity the helper retries every 30 s until a copy serves the request. Without it, a cold endpoint always looks broken.

Teardown order

teardown.py handles both shapes, but the order is load-bearing:

1. alarms (<endpoint>-* and <component>-*)
2. scaling policies and the scalable target, on the component resource id
3. inference components
4. endpoint, endpoint config, model

Two behaviours make this necessary:

• delete-endpoint does not delete the components. They survive, keep reporting InService, and block a new component with the same name. Always delete components first.
• Component deletion is refused during transient states — CREATE_IN_PROGRESS while the container boots, and UPDATE_RC_IN_PROGRESS while a scaling action changes the copy count. The script retries every 15 s for 15 min; a teardown right after a scaling event legitimately takes several minutes.

The TargetTracking-inference-component/<ic>-AlarmHigh|Low alarms belong to Application Auto Scaling. Deleting the policy removes them, so the script does not touch them (verified: no alarms remain after teardown).