Redis

Redis — cache, lock, rate limiter, queue done correctly

Redis is four primitives wearing one server: a cache, a distributed lock, a rate limiter, and a job queue. Each has a correctness contract that has nothing to do with whether SET returns OK. You almost certainly already "have Redis working" — a value goes in, a value comes out. What you do not yet have is a cache that won't stampede your database, a lock that won't release someone else's hold, a rate limiter that survives concurrent requests, and a queue whose jobs aren't silently lost when a worker dies. This skill is engine- and pattern-level, client-SDK-agnostic: examples are redis-cli and Lua, so they hold across node-redis, ioredis, redis-py, go-redis, and Lettuce. Every pattern applies unchanged to Valkey and any Redis-protocol store, since Valkey is a BSD fork of Redis 7.2 and stays protocol-compatible.

When to use / When NOT

Use when you are:

• Adding or reviewing a cache: TTL strategy, cache-aside vs write-through, key naming, stampede/thundering-herd prevention, negative caching, invalidation.
• Building a distributed lock: "only one worker does X", cron de-duplication, leader-ish mutual exclusion — the SET NX PX + Lua-release + fencing-token contract, and the Redlock call.
• Building a rate limiter: per-user/IP/API-key throttling (fixed window, sliding-window log, sliding-window counter, token bucket) and why naive INCR+EXPIRE is a race.
• Running a queue / background jobs on Redis: Lists vs Streams vs a library (BullMQ, Sidekiq, RQ, Celery), at-least-once delivery, acks, stalled-job recovery, idempotency.
• Deciding eviction & persistence: maxmemory-policy, RDB vs AOF, cache mode vs database mode.
• Reviewing Redis code for foot-guns: KEYS *, unbounded keys, non-atomic read-modify-write, missing TTL, a lock released by the wrong owner.

NOT for (route instead):

• Durable SQL queue (SELECT … FOR UPDATE SKIP LOCKED), engine-level SQL caching → ../postgresdb/SKILL.md. Boundary: if the durable store is the queue, that's postgresdb; if Redis is the queue, you're here.
• Vector similarity / semantic search as the primary job → ../vector-db/SKILL.md. Redis 8 vector sets are only noted here.
• Analytical / clickstream event store → ../clickhouse-analytics/SKILL.md.
• System-level capacity / load strategy, and app/CDN/HTTP-layer caching (Cache-Control, ISR, edge) → ../scaling/SKILL.md. This skill owns the Redis-specific data contract underneath.
• Provisioning a managed Redis (cluster sizing, dashboards) → ../fly-io/SKILL.md / ../aws-essentials/SKILL.md. This skill gives the client-side contract, not console clicks.
• Language-runtime async/job concepts unrelated to Redis → ../nodejs/SKILL.md or your language skill.

Non-negotiables

1. Every cache key has a TTL. No TTL in a cache = a memory leak that eventually evicts or 500s.
2. Never KEYS * in production — use SCAN. KEYS is O(N) and blocks the single thread; one bad pattern freezes every client.
3. Read-modify-write must be atomic. App-side GET then SET/INCR races under concurrency. Use one Lua EVAL, a single atomic command, or WATCH/MULTI.
4. A lock = random token + PX + Lua compare-and-delete. Plain DEL releases whoever holds it now (maybe not you); a lock with no PX is held forever if the owner crashes.
5. Pick maxmemory-policy deliberately. The default noeviction makes a full cache fail every write — correct for a database, fatal for a cache.
6. A queue must ack and recover. RPOP straight into a worker loses the job if the worker dies mid-task; use a processing list or Streams consumer groups.
7. Cap unbounded structures. Lists grow forever without LTRIM; streams without MAXLEN ~.
8. One Lua = one short atomic step. Redis is single-threaded; a long script or huge MULTI/EXEC blocks all clients, not just yours.
9. Namespace keys as svc:entity:id (e.g. cart:user:42) so scans, eviction, and humans can reason.
10. Know cache vs database durability before choosing persistence. Cache = lose-it-and-rebuild; database = RDB/AOF and noeviction. Decide which Redis is before you tune it.

Decision table — which primitive, which structure

Caching

Namespace + TTL with jitter. A fixed TTL on a batch of keys written together synchronizes their expiry — they all die in the same second and stampede the origin at once. Add jitter.

# Bad: 1000 keys warmed in a loop with the same TTL all expire together
SET product:42 "$json" EX 300
# Good: spread expiry with per-key jitter (300s ± up to 60s)
SET product:42 "$json" EX $(( 300 + RANDOM % 60 ))

Cache-aside (read path). Why: it is the default for read-heavy data; the trap is forgetting the TTL and forgetting to cache the miss.

# Bad: cache only hits, never the miss → every request for an absent id hits the DB
val = GET product:42
if val == nil: val = db.fetch(42); SET product:42 val EX 300   # absent rows still hammer the DB

# Good: cache-aside with negative caching
val = GET product:42
if val == "__MISS__": return null                              # cached miss, no DB call
if val == nil:
    val = db.fetch(42)
    if val == null: SET product:42 "__MISS__" EX 30            # short negative TTL
    else:           SET product:42 val EX 300

Stampede prevention. When a hot key expires, N concurrent requests all miss and hammer the origin. Two mitigations:

• Recompute lock — one request rebuilds, the rest briefly serve stale or wait. The lock TTL must exceed worst-case origin latency, or two requests rebuild anyway.

  # On miss, try to claim the rebuild; only the winner recomputes
  SET product:42:lock 1 NX PX 5000   # 5s > slowest DB rebuild
  # winner: rebuild, SET product:42, then DEL product:42:lock
  # losers: serve last-known value or retry after a short sleep

• Probabilistic early expiration (XFetch) — refresh before TTL with rising probability so one request renews early while others still hit a warm key. Math + write-through/read-through/ write-behind in references/caching.md.

Invalidation: on write, either delete the key (next read rebuilds) or write-through. Never rely on manual invalidation instead of a TTL — you will miss a path and serve stale forever.

Distributed locks

# Bad: GET-then-DEL races — between your GET and DEL the lock can expire and a NEW owner takes it,
#      and your DEL frees THEIR lock.
GET resource:lock        # == my token?
DEL resource:lock        # ← may delete someone else's lock

# Good: claim with a random token + PX, release only if the token is still mine (atomic Lua).
SET resource:lock "$TOKEN" NX PX 30000     # NX = only if absent; PX = auto-expire so a crash frees it

Release must be a compare-and-delete in one Lua step:

-- KEYS[1] = lock key, ARGV[1] = my token. Returns 1 if I held and released it, else 0.
if redis.call('GET', KEYS[1]) == ARGV[1] then
  return redis.call('DEL', KEYS[1])
else
  return 0
end

TTL vs work duration. If the protected work can outlast PX, the lock expires mid-task and a second worker starts. Either size PX above the worst case, or run a watchdog that renews the TTL (again via a token-checked Lua PEXPIRE) while work continues — never blindly PEXPIRE.

Fencing tokens. A GC pause or a long syscall can freeze the lock holder past expiry; another worker acquires, then the paused one wakes and writes — both think they hold the lock. The only fix is a fencing token: a monotonically increasing number issued with the lock and checked by the protected resource, which rejects any write carrying a stale token. No Redis lock alone provides this.

Redlock decision. Single-instance SET NX PX is fine when occasional double-execution is tolerable (idempotent work, best-effort cron de-dup). For hard mutual exclusion, a single Redis lock is not enough and multi-instance Redlock is contested (it assumes bounded clock drift and no long pauses) — fence the resource or use a lease/consensus system. Full Redlock walk-through and critique in references/locks-and-rate-limiting.md.