Scaling

Scaling: survive many requests at once

Performance makes one request faster. Scaling makes many requests survive at the same time. Different problem, different toolbox — don't reach for this one when a single endpoint is slow for a single user (that is ../performance/SKILL.md).

The whole job in one line: diagnose the bottleneck, apply the cheapest lever that moves it, re-measure under load. Repeat until the next bottleneck appears or you hit your target.

Prime directive: never add infrastructure without a measurement first. A replica, a queue, or a third app instance you bought on a hunch costs money every month and usually moves the wrong tier. Measure, then add.

Start here — order of operations

Levers are ordered by payoff per dollar. Caching is nearly free and wins biggest; replicas and autoscaling cost forever. Climb the ladder in order; stop the moment the symptom clears.

Step 0 — find the bottleneck before you add anything

Use the USE method (Utilization, Saturation, Errors) on every resource — CPU, memory, disk, network, and the DB connection pool. For each one ask: how busy (U), how much is queued/waiting (S), and any errors (E).

• Saturation predicts collapse earlier than utilization. A CPU at 70% util with a growing run-queue is closer to falling over than one at 90% with no queue. Watch queue depth, run-queue length, and connection-pool wait time — those spike before throughput craters.
• Read p95/p99, never the mean. The mean hides the tail your users actually feel; if p50 is 80 ms and p95 is 4 s, a meaningful slice of traffic is having a bad time and the average lies about it.
• The dashboards, alerts, and SLOs that produce these numbers are ../monitoring/SKILL.md / observability's job. Scaling consumes USE signals; it doesn't build the collectors.

Output of Step 0 is one sentence: "the bottleneck is the DB connection pool / app CPU / origin cache-miss rate." Don't proceed without it.

Lever 1 — Caching (cheapest, do first)

Cache layers, outermost to innermost — each one removes work the layer behind it would have done:

• Cache the expensive read, not the cheap one. Caching a 2 ms lookup adds a network hop and an invalidation bug for nothing; cache the 400 ms aggregate everyone hits.
• Target >80% hit ratio for general traffic (static-heavy/CDN routinely hits 95%+). A ratio consistently <60% means a broken strategy — wrong cache keys, TTL too low, or churny data that shouldn't be cached.
• Beware the cold-cache miss storm. On deploy or cache flush, every request misses at once and stampedes the origin — the cache that was protecting you now amplifies the load. Mitigating that (stampede protection, single-flight, request coalescing) is cache correctness → ../redis/SKILL.md.
• If the underlying query is just slow, caching only hides it. Fix the query — N+1, missing index — via ../performance/SKILL.md before papering over it with TTL.

Lever 2 — Connection pooling + queues

Pool DB connections. Each Postgres connection is a backend process with real memory cost; apps that open a connection per request exhaust max_connections fast.

Bad:  app → opens a fresh DB connection per request → "too many clients already"
Good: app → PgBouncer (transaction mode) → small pool of reused server connections

• Use transaction pooling for stateless web apps: a server connection is held only for the duration of a transaction and released on COMMIT/ROLLBACK — reported real-world effect is roughly 100× effective connection capacity.
• Sizing rule: keep (number_of_pools × default_pool_size) < max_connections − ~15 (leave headroom for superuser/admin slots). Set default_pool_size ≈ 1.5–2× vCores for CPU-bound OLTP — more connections than cores just adds context-switch contention, not throughput.
• Gotcha — transaction pooling breaks session-scoped features: prepared statements (pre-PG14 protocol), SET / session GUCs, advisory session locks, and LISTEN/NOTIFY. Route those to a session-pooling pool or refactor them out. Don't discover this in production.

Shed spiky writes into a queue. Queue-based load leveling puts a queue between a bursty producer and a constrained consumer so the consumer drains at its own steady rate; the queue absorbs the spike instead of the synchronous tier melting.

• Move anything that doesn't need a synchronous answer — emails, webhooks, thumbnails, exports — off the request path.
• Queue semantics (race-free rate limits, stalled-job recovery, fencing tokens, durable SKIP LOCKED queues) belong to ../redis/SKILL.md and ../postgresdb/SKILL.md. Scaling decides that you defer work; those decide it's done correctly.