Runpod

RunPod: GPU compute, two products, one bill

RunPod sells GPU time two ways and they bill on opposite philosophies. Get the choice wrong and you either pay a steep premium for idle work or you pay 24/7 for a box that sits warm doing nothing. Everything below is the RunPod-specific operational playbook: which product a workload belongs on, how to write a worker that does not waste cold-start seconds, and which knobs actually move the number on the invoice.

The two products:

• Pods — rent a GPU container by the hour. It runs continuously while it is up, billed every hour whether busy or idle. Your dev box, your training job, your Jupyter.
• Serverless — per-second autoscaling workers. Billed only while a worker is actually running a job, from worker start to full stop, rounded up to the second. Costs roughly 2-3x the equivalent hourly pod rate, but idle gaps cost nothing on flex workers.

RunPod charges zero egress/ingress fees — bandwidth in and out is free, unlike the hyperscalers. That removes one variable from cost math: you only reason about GPU-seconds and storage.

When to use

• Deploying inference as a Serverless endpoint (vLLM LLM serving, image gen, custom model).
• Running a training / fine-tuning job or a Jupyter / dev box on a GPU Pod.
• Writing or debugging a serverless worker handler (handler(job), async, streaming).
• Building a custom worker Docker template / image, or wiring a network volume.
• Cost control on RunPod: cold starts, idle bleed, runaway max workers, idle volume charges.
• Picking a GPU SKU and the pod-vs-serverless tradeoff for a given workload.

When NOT to use

• Python-native serverless GPU with snapshot autoscaling and no Dockerfile → that is modal.
• Calling hosted, prebuilt model endpoints you do not host → that is replicate.
• Pulling/pushing weights, datasets, model cards on the Hub → that is huggingface.
• Running models locally on your own machine → that is ollama.
• Provider-agnostic cross-cloud spend dashboards → that is cost-tracking.
• Generic container packaging → that is docker (here we cover only the RunPod image shape).

Decision: Pod vs Serverless

Rule: if the GPU would sit busy more than ~60% of the time, a Pod is cheaper than serverless even with active-worker discount — model the two before committing.

GPU SKU quick-pick (2026 pod $/hr, lower on Community Cloud)

Rule: pick the smallest GPU the model fits in VRAM. Defaulting to H100 is up to ~7x the cost for zero speedup when the workload is memory-bound and fits on an L40S or 4090.

Serverless worker handler

A worker is a Python file using the runpod SDK. The minimum: a function that reads job["input"], returns a dict, and is registered with runpod.serverless.start.

import runpod

def handler(job):
    job_input = job["input"]
    prompt = job_input["prompt"]
    # ... run the model ...
    return {"output": f"echo: {prompt}"}

runpod.serverless.start({"handler": handler})

Async handler (for awaiting model calls) and a streaming generator both work:

import runpod

async def handler(job):
    job_input = job["input"]
    return {"output": await run_model(job_input)}

# Streaming: yield chunks from an async generator instead of returning once.
async def stream_handler(job):
    async for token in generate(job["input"]["prompt"]):
        yield {"token": token}

runpod.serverless.start({"handler": stream_handler, "return_aggregate_stream": True})

Test locally before you push an image. A broken handler still burns build minutes and cold-start seconds when discovered on the platform.

# One-shot: reads ./test_input.json, runs the handler once, prints output.
python worker.py

# HTTP server emulating the real endpoint at http://localhost:8000.
python worker.py --rp_serve_api

Concurrency (concurrency_modifier), job cancel, refresh-worker, and the run / runsync / stream / status / cancel / health HTTP endpoints live in references/serverless-workers.md.

Templates & images

A custom template is a Docker image plus environment variables. Pin the base; an unpinned or :latest base re-pulls on cold start and lengthens it.

# Pin the CUDA base — never bare :latest.
FROM runpod/base:0.6.2-cuda12.4.1

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY handler.py .

# The worker entrypoint runs the handler.
CMD ["python", "-u", "handler.py"]

vLLM shortcut. For OpenAI-compatible LLM serving, use the prebuilt worker-vllm image instead of writing a handler. Its AsyncEngineArgs are set via UPPERCASE env vars:

# Template env vars on the endpoint — these map to vLLM AsyncEngineArgs.
MODEL_NAME=mistralai/Mistral-7B-Instruct-v0.3
MAX_MODEL_LEN=8192

Network volumes

Attach a network volume when data outgrows what you want in the image:

• Datasets larger than container disk.
• Weights shared across many workers (download once, mount everywhere).
• Checkpoints that must survive a worker / pod restart.

Two traps to state up front:

1. A volume locks the endpoint/pod to one data center and adds network latency. That shrinks the pool of available GPUs in that DC — you can get stuck waiting for capacity.
2. Idle volume cost is double. Pod volume disk bills ~$0.10/GB/mo while running but ~$0.20/GB/mo while the pod is stopped. A forgotten volume on a stopped pod bleeds money.

Rule: bake small, static weights into the image; reserve volumes for large mutable data (datasets, checkpoints). Full storage tables are in references/cost-and-scaling.md.