Ollama

Ollama — run open-weight LLMs on one box

Ollama serves GGUF models from a local daemon at http://localhost:11434, exposing both a native HTTP API and an OpenAI-compatible layer. Your job: reach for the right command, the right endpoint, and the right quant for the hardware in front of you — and recognize when the model does not fit and the work belongs on a remote GPU instead.

This skill owns: install/serve, pull/tag, the local API (native + OpenAI-compat), Modelfiles, quantization choice, and VRAM/RAM sizing on a single machine.

When to use / when not

Use when the model runs on this machine: pulling/running a model, fixing an OOM, choosing Q4 vs Q8, authoring a Modelfile, or wiring an app to localhost:11434.

Go elsewhere when:

• Hosting behind a managed/remote GPU, autoscaling, or serverless inference → runpod, modal, replicate, together-fireworks, fal. Ollama is local, single-box, no autoscale.
• Downloading raw weights, datasets, hf/transformers, repo management → huggingface.
• Designing chunking / retrieval / reranking around a model → rag or embeddings-search.
• Orchestrating multi-step calls, routing, pipeline evals → llm-pipeline / agent-eval.
• Writing the prompt/system-message content itself → prompt-engineering.

(Those siblings live in the catalog by id; link them only once their SKILL.md exists on disk.)

Quickstart

ollama serve                 # start the daemon (a desktop install already runs it)
ollama pull qwen3:8b         # download a model + tag; :8b is explicit — avoid bare :latest
ollama run qwen3:8b          # interactive REPL, or: ollama run qwen3:8b "summarize this"
ollama ps                    # what is LOADED in VRAM right now + when it unloads (keep_alive)
ollama list                  # what is on disk (pulled), not what is loaded
ollama show qwen3:8b         # template, params, context length, quant of a model
ollama rm qwen3:8b           # free disk; ollama stop qwen3:8b unloads from memory

ps vs list is the OOM-debug split: list is disk, ps is memory. A model only eats VRAM once a request loads it; it unloads after keep_alive (default 5m).

Pick a model + quant

Quantization trades VRAM for quality. The everyday default is Q4_K_M: roughly half the memory of fp16 for ~3–5% quality loss. Q8_0 is near-lossless at ~1 byte/param. fp16 is the unquantized ceiling at 2 bytes/param.

Sizing formula (weights only) — a rule of thumb, not a per-model spec sheet:

weights_GB ≈ params(B) × bytes_per_param × 1.2   # ×1.2 = runtime overhead
bytes_per_param:  Q4_K_M ≈ 0.5   Q8_0 ≈ 1.0   fp16 = 2.0
# then ADD the KV cache (see below) — it is NOT in this number.

These bytes/param are conservative round-downs of the measured k-quant rates: llama.cpp's quantize benchmark reports Q4_K_M ≈ 4.89 bits/weight (~0.6 byte/param) and Q8_0 ≈ 8.5 bits/weight (~1.06 byte/param) on Llama-3.1-8B (llama.cpp quantize README, accessed 2026-06-02). Rounding to 0.5 / 1.0 keeps the estimate on the safe side; the per-row GB figures in the table below are derived from this formula, not vendor-published numbers — verify with ollama show.

KV cache is the trap. It grows ~linearly with num_ctx and lives in VRAM on top of the weights. At long context (e.g. 128K) a 70B can add tens of GB of cache — often more than people budget for. If you are tight: cap num_ctx, or shrink the cache with OLLAMA_KV_CACHE_TYPE=q8_0 (or q4_0). See references/hardware-sizing.md for the KV math and a per-context table.

Ollama runs a llama.cpp-backed engine (GGUF) by default, with a scheduler that reduces OOM crashes and improves multi-GPU placement. On Apple Silicon it can use an MLX backend (shipped in Ollama 0.19, per ollama.com/blog/mlx, 2026-03-30), but only on Macs with >32 GB of unified memory — below that gate it stays on the llama.cpp engine. None of this invents memory you don't have: when the box can't hold the model, that's a runpod/modal job, not a quant downgrade.

The API

Two surfaces, same daemon. Use native /api/chat when you want Ollama-specific fields (keep_alive, format as a JSON schema, think); use the OpenAI-compat /v1 layer to reuse an existing OpenAI SDK unchanged.

Native chat (/api/chat), non-streaming:

curl http://localhost:11434/api/chat -d '{
  "model": "qwen3:8b",
  "messages": [{"role": "user", "content": "Name three primes."}],
  "stream": false,
  "options": {"temperature": 0.2, "num_ctx": 8192},
  "keep_alive": "10m"
}'

stream defaults to true (NDJSON, one object per line, final object has done: true + timing stats). options.num_ctx sets the context window for this request — it does not persist; bake it into a Modelfile if you want it permanent.

OpenAI-compatible — point any OpenAI SDK at localhost:11434/v1 with a dummy key:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")  # key is ignored
resp = client.chat.completions.create(
    model="qwen3:8b",
    messages=[{"role": "user", "content": "Name three primes."}],
    temperature=0.2,
)
print(resp.choices[0].message.content)

Structured output — pass a JSON schema as format (native) so the model is constrained to valid JSON:

curl http://localhost:11434/api/chat -d '{
  "model": "qwen3:8b",
  "messages": [{"role": "user", "content": "Extract name and age from: Ana is 30."}],
  "stream": false,
  "format": {
    "type": "object",
    "properties": {"name": {"type": "string"}, "age": {"type": "integer"}},
    "required": ["name", "age"]
  }
}'

Tool calling (tools), multimodal (images as base64), embeddings (/api/embed), and the full field tables live in references/api.md. Endpoint map at a glance: /api/generate, /api/chat, /api/embed, /api/create, /api/pull, /api/show, /api/ps, /api/tags.