Replicate

Replicate platform operations

This skill is about how a model runs in production on Replicate — clients, async, deployments, Cog packaging, webhooks, scaling, and spend. It is the platform-engineering counterpart to image prompt craft. If the question is what prompt, aspect ratio, or model family produces a good image, that is replicate-images, not this skill. Here the mental model is: a prediction is a job. You either wait for it, poll it, or get pinged about it — and where it runs (shared cold pool vs a private warm deployment) is a cost-and-latency dial you set deliberately.

Pinned facts (verified 2026-06-02): Python client replicate 1.x (latest 1.0.7), Python 3.8+; JS client replicate on npm; auth via REPLICATE_API_TOKEN. A 2.0.0aN alpha exists on PyPI but is NOT the default — pin replicate>=1,<2 so a fresh install never silently pulls it.

Decision: how should this model run?

Pick the row by latency tolerance and whether your process can block. Do not default to run() for everything — a 10-minute job inside a web request will time out and burn a worker.

Rule: if a human or HTTP request is waiting longer than a few seconds, do not block on run() — switch to predictions + webhook. Why: synchronous timeouts kill the request but the GPU job keeps running and billing.

Auth & install

export REPLICATE_API_TOKEN=r8_...           # clients read this env var automatically
pip install 'replicate>=1,<2'               # pin: 2.0.0aN is alpha; unpinned can pull it
npm install replicate                        # Node client

Python 3.8+ is required. Never pip install replicate unpinned in a Dockerfile or requirements file — a rebuild months later can resolve to the 2.0 alpha and break your imports. Why: the alpha is a full Stainless/httpx rewrite with a different surface.

Run a model

import replicate

output = replicate.run(
    "black-forest-labs/flux-schnell",
    input={"prompt": "a red bicycle", "num_outputs": 1},
)

Since client 1.0.0, file outputs come back as FileOutput objects, not URL strings. Treating one as a string is the single most common bug.

# Bad — output[0] is a FileOutput, not a str; this writes the repr, not the bytes
open("out.png", "w").write(output[0])

# Good — read the bytes, or take the URL explicitly
with open("out.png", "wb") as f:
    f.write(output[0].read())          # bytes
print(output[0].url)                    # hosted URL if you'd rather link

Rule: call .read() for bytes or .url for the link. Why: silently coercing a FileOutput to a string corrupts the file and the error surfaces far from the cause.

Long-running & async

For jobs over a few seconds, create a prediction instead of blocking:

client = replicate.Client()
prediction = client.predictions.create(
    model="owner/model",
    input={"prompt": "..."},
)
prediction.reload()                      # refresh status from the API
while prediction.status not in ("succeeded", "failed", "canceled"):
    time.sleep(2)
    prediction.reload()

Set a deadline so a stuck job auto-cancels instead of billing forever, and cancel() on cleanup paths. Why: a hung prediction with no deadline is silent, open-ended GPU spend. Poll with a small backoff, not a tight loop — you are charged for the prediction, not the polling, but a tight loop wastes your own process and rate budget. Full polling loop with backoff and 5xx handling is in references/webhooks-and-async.md.

Webhooks

For fire-and-forget, hand Replicate a URL and filter to the events you care about:

client.predictions.create(
    model="owner/model",
    input={"prompt": "..."},
    webhook="https://your.app/hooks/replicate",
    webhook_events_filter=["completed"],   # not every intermediate "logs" event
)

Rule: ALWAYS verify the signature before trusting a webhook body. Why: the URL is public — anyone can POST forged completions to it. Replicate signs each delivery; the secret has a whsec_ prefix. Reconstruct signed content as {webhook-id}.{webhook-timestamp}.{body}, HMAC-SHA256 with the base64-decoded secret, base64-encode, and constant-time compare against the webhook-signature header (a space-separated v1,<sig> list). The clients expose a verification helper; the full Python and Node recipe (including idempotency via webhook-id) is in references/webhooks-and-async.md.

Deployments

A deployment is a private, dedicated API endpoint for one model version that autoscales from zero to hundreds of instances. Reach for it when you need warm instances, a private endpoint, or a hard spend cap — not for one-off runs.

• min_instances — the warm floor. Set >0 to kill cold starts for latency-sensitive traffic; every warm instance bills whether or not it serves a request.
• max_instances — the spend cap. The ceiling on concurrent instances; protects you from a traffic spike turning into a surprise bill.
• Hardware (NVIDIA T4, A100, H100, ...) is switchable without touching code — change the deployment config, not predict.py.
• Rolling updates ship a new version with no downtime; built-in monitoring covers latency, throughput, error rate, and GPU memory.

You still call a deployment with run() / predictions.create() — it just routes to your private instances. Create/update via HTTP API, the clients, or CLI; fields and the rolling-update flow are in references/deployments-api.md.