Start Temps Cluster

Start a local multi-node Temps cluster (Docker-in-Docker)

Wraps tools/dev-cluster/ — a docker-compose harness that runs a real Temps control plane plus 3 worker nodes, each a privileged DinD container, with the actual overlay network (VXLAN, compute_cidr allocation, cluster DNS) wired up exactly as it would be on real hardware. Use this whenever a fix or feature needs more than one node to prove anything — single-node start-temps cannot exercise cross-node scheduling, the overlay, or cluster DNS at all.

Topology

┌──── temps-underlay (10.42.0.0/24, docker bridge, the "VPC") ────┐
│                                                                  │
│  postgres            10.42.0.5   (TimescaleDB pg18, internal)   │
│  control-plane        10.42.0.10  DinD + `temps serve`           │
│    host ports 80 → 80, 443 → 443 (NOT 8080 — see Gotcha below)  │
│                                                                  │
│  worker-1  10.42.0.21   worker-2  10.42.0.22   worker-3 10.42.0.23│
│    each: privileged DinD + `temps agent`, own dockerd            │
└──────────────────────────────────────────────────────────────────┘

Allocator carves /24s from 172.20.0.0/16, one per worker:
  worker-1 → 172.20.0.0/24
  worker-2 → 172.20.1.0/24
  worker-3 → 172.20.2.0/24

Cross-node container traffic flows over a VXLAN tunnel pinned to the
underlay IPs above. Within-node traffic uses the unchanged
`temps-app-network` Docker bridge (same as single-node temps).

The temps source tree is bind-mounted read-write at /workspace inside every DinD container, and cargo build --bin temps runs inside Linux — no cross-compiling from macOS. The cluster builds from whichever checkout you run ./dev-cluster in. cd into the right worktree first if you're testing a fix that isn't on the checkout's current branch.

Prerequisites

• Docker Desktop (or any Linux Docker host) running — docker version must succeed.
• ~3 GB free RAM, ~5 GB free disk (compiled binary cache + 4 worker volumes + TimescaleDB volume).
• GeoLite2-City.mmdb at the repo root of the checkout you're running from (<checkout>/GeoLite2-City.mmdb, gitignored) — the proxy plugin refuses to start without it. Download it from MaxMind's GeoLite2 program (free registration required) and place it at the repo root. If you keep multiple worktrees of this repo, copying the file between them is faster than downloading it again.

Only one cluster at a time, across the ENTIRE machine

Unlike start-temps (which allocates a port "slot" per checkout so many worktrees run side by side), this harness uses fixed container names (temps-dev-control-plane, temps-dev-worker-1/2/3, temps-dev-postgres) and fixed host ports (80/443). There is no per-checkout isolation — starting a second dev-cluster from a different worktree while one is already running will collide.

Before running up, always check first:

docker ps -a --filter "name=temps-dev-" --format "{{.Names}}\t{{.Status}}"

If containers are already there and belong to someone else's in-progress work, don't down/reset them without asking — treat it like any other shared, hard-to-reverse state.

There's also a separate mTLS-hardening variant (docker-compose.harden.yml, driven by e2e-harden.sh, container prefix temps-harden-*, 2 workers only) used for testing require_mtls / node-identity hardening specifically. It's a different compose project name/port set from the base cluster, but still only one can run at a time on its own.

Quick start

cd <checkout>/tools/dev-cluster
./dev-cluster up          # first run: ~5-10 min (compiles the workspace inside Linux)
                           # subsequent runs: ~10-30s (cargo build no-ops)

up blocks until: control plane's TCP listener is up, all 3 workers have registered as nodes rows, and every worker has an allocated compute_cidr. It prints admin credentials and the join-token path on success.

./dev-cluster status              # node + overlay state across the cluster
./dev-cluster logs control-plane   # or worker-1/worker-2/worker-3/postgres
./dev-cluster shell worker-1       # shell into a worker's DinD host
./dev-cluster restart control-plane  # bounce after a binary rebuild
./dev-cluster down                # stop, KEEP volumes (fast restart)
./dev-cluster reset               # stop + DELETE all volumes (fresh slate, asks to confirm)

Gotcha: the script says port 8080, the cluster actually serves 80/443

docker-compose.yml publishes 80:80 and 443:443 (production-shaped — TEMPS_ADDRESS=0.0.0.0:80), matching what e2e-harden.sh and every API recipe below assume (http://localhost/...). But dev-cluster's own up command waits on /dev/tcp/127.0.0.1/8080 and prints web UI: http://localhost:8080 — that's stale/drifted from the compose file and will never resolve. Use port 80, not 8080, for every API/browser interaction. If ./dev-cluster up looks stuck on "waiting for control plane", check whether it's actually already up on the right port before assuming a real hang:

curl -s -o /dev/null -w '%{http_code}\n' http://localhost/api/health

If that returns a real HTTP status, the cluster is up — the wait loop itself is just polling the wrong port and will eventually time out at 20 minutes regardless. Don't wait for it; move on once curl confirms readiness.

Login: admin@local.dev / password on line 2 of .state/admin.txt.

J=/tmp/tc_cookies.txt
PW=$(sed -n '2p' tools/dev-cluster/.state/admin.txt)
curl -s -c $J -X POST http://localhost/api/auth/login -H 'Content-Type: application/json' \
  -d "{\"email\":\"admin@local.dev\",\"password\":\"$PW\"}" -o /dev/null -w "login -> %{http_code}\n"

Verifying the overlay actually works (bare check, no API)

The README documents this network as temps-overlay — that name is wrong. Verified live: the actual Docker network on every worker is named temps0 (docker network ls inside a worker shows it). Use temps0, not temps-overlay, or docker run fails with "network temps-overlay not found".

./dev-cluster shell worker-1
docker run -d --rm --name target --network temps0 --ip 172.20.1.50 nginx:alpine
exit

./dev-cluster shell worker-2
docker run --rm --network temps0 --ip 172.20.0.50 alpine sh -c \
  'apk add -q curl && curl -sf --max-time 5 http://172.20.1.50/ | head -c150'
# "<!DOCTYPE html>..." => bridge + VXLAN + FDB + routes + dual-attach all confirmed working

(IPs above match worker-1 → 172.20.1.0/24, worker-2 → 172.20.0.0/24 — the allocator assigns CIDRs in registration order, not worker-number order, so confirm actual assignments with ./dev-cluster status rather than assuming worker-N always gets the Nth-numbered /24.)

Recipe: deploying across specific nodes (multi-replica / anti-affinity)

NIDS=$(curl -s -b $J http://localhost/api/internal/nodes | \
  python3 -c "import sys,json;print(','.join(str(n['id']) for n in json.load(sys.stdin)['nodes']))")

PID=$(curl -s -b $J -X POST http://localhost/api/projects -H 'Content-Type: application/json' \
  -d '{"name":"cluster-test","directory":".","main_branch":"main","preset":"dockerfile","storage_service_ids":[],"source_type":"docker_image"}' \
  | python3 -c "import sys,json;print(json.load(sys.stdin).get('id',''))")
EID=$(curl -s -b $J http://localhost/api/projects/$PID/environments | \
  python3 -c "import sys,json;print(json.load(sys.stdin)[0]['id'])")

curl -s -b $J -X PUT http://localhost/api/projects/$PID/environments/$EID/settings -H 'Content-Type: application/json' \
  -d "{\"replicas\":2,\"target_nodes\":[$NIDS],\"anti_affinity\":true,\"exposed_port\":8080}"

curl -s -b $J -X POST http://localhost/api/projects/$PID/environments/$EID/deploy/image -H 'Content-Type: application/json' \
  -d '{"image_ref":"nginxinc/nginx-unprivileged:alpine","health_check_path":"/"}'

Use nginxinc/nginx-unprivileged:alpine, not plain nginx:alpine. Every deployed container runs cap_drop: ALL + no-new-privileges; stock nginx needs to chown/bind a privileged port and crashes under that hardening. preset must be "dockerfile" (not "docker" — invalid) even for source_type: "docker_image". Pre-pull the image on every node first (docker exec temps-dev-worker-N docker pull <image>) if ensure_image_on_remote isn't wired for your image, or the deploy will fail on nodes without it cached.

Recipe: pinning a database service to one node, an app to another

SID=$(curl -s -b $J -X POST http://localhost/api/external-services -H 'Content-Type: application/json' \
  -d '{"name":"test-db","service_type":"postgres","node_id":<worker-1-node-id>,"topology":"standalone","parameters":{"database":"app","username":"app_user","password":"<pick-one>"},"members":[]}' \
  | python3 -c "import sys,json;print(json.load(sys.stdin).get('id',''))")

Then create the app project with "storage_service_ids":[$SID] (this is the link mechanism — set at project creation, there is no later update endpoint for it) and set target_nodes to a different node than the one above, to exercise cross-node service linking. Note a project can only link one service per service_type (e.g. one postgres, one redis) — linking a second postgres service to the same project fails.

Recipe: toggling cluster DNS

Cluster DNS defaults off. The setting is a top-level cluster_dns object with an enabled field — NOT multi_node.cluster_dns_enabled (that key doesn't exist in the real schema; writing it via raw SQL succeeds silently but does nothing, and disappears on the next control-plane restart since settings get re-normalized from the typed Rust struct on load). Verified live:

docker exec temps-dev-postgres psql -U temps -d temps -c \
  "update settings set data = jsonb_set(COALESCE(data::jsonb,'{}'),'{cluster_dns,enabled}','true'::jsonb)::json where id=1;"
docker exec temps-dev-postgres psql -U temps -d temps -tAc \
  "select data::jsonb->'cluster_dns' from settings where id=1;"   # => {"enabled": true}

Two restarts are required after this, not zero:

1. docker restart temps-dev-control-plane — a raw SQL write bypasses ConfigService's in-memory settings cache; the control plane won't see the change until it re-reads settings on startup (there's no live invalidation path for a write that didn't go through the settings API).
2. docker restart temps-dev-worker-1 temps-dev-worker-2 temps-dev-worker-3 (or at least whichever workers are relevant to your test) — the per-node DNS resolver is only spawned once, during initial agent startup. An agent that was already running when you flip cluster_dns.enabled will keep operating in "disabled" mode indefinitely; it does not poll for this setting changing live. This is a known gap (tracked as part of ADR-024, proposed not yet implemented) — don't lose time assuming a stuck deploy means the fix under test is broken; check docker logs <worker> | grep -i resolver for "DNS resolver started" before assuming anything else is wrong.

After both restarts, worker containers hit a transient DinD containerd: timeout waiting for containerd to start stall fairly often on a busy host — this is the same known containerd-stall gotcha as during initial up. Don't machine-gun docker restart on it; one extra restart is fine, but if it's still stalling after that just wait — it clears on its own within roughly a minute. Confirm real recovery via a fresh heartbeat, not just "container status is Up":

docker exec temps-dev-postgres psql -U temps -d temps -tAc \
  "select extract(epoch from now() - last_heartbeat) from nodes where name='worker-2';"
# a small number (single-digit seconds) means it's actually back

If a deployment fails with "Worker route/DNS propagation did not complete within 10s ... N active node(s) have never ACKed a DNS generation at all", that message IS the diagnosis — it's telling you exactly this: enable cluster DNS, then restart the agents, in that order, before deploying.