Redis Host

Redis host

Production Redis is one host, built by config/redis/ and reached by the app containers, the proxy and the cert scripts at a floating private IP (BLOT_REDIS_HOST). config/redis/README.md describes the setup and holds the steps for building a host, cutting over, resizing and disaster recovery. This skill does not repeat them: it says which README section to use, which read-only checks to propose for each alert, how to read them, and what must never be done without asking.

SSH aliases: redis is the Redis host (ec2-user, port 22, key-only), blot is the app host.

Related: investigate-production-container-restarts (a Redis stall shows up there as [LOCK COMPROMISED] restarts; its section on the Redis host has the TCP memory background) and investigate-slow-upstream-responses.

Rules

1. Ask the operator before EVERY production command: ssh redis, ssh blot, docker exec, docker logs, and any AWS call that changes something (launch, terminate, assign an address, modify an attribute, write to S3). Read-only commands too. State the exact command. An approval covers that command only, not the next one, not "the same kind of thing", and not what other skills auto-approve. Read-only AWS describe-* / ls calls are fine without asking, but say what you ran.
2. Use launch.sh --drill for any throwaway host (a rehearsal, a restore test, a cutover practice). A drill host is tagged BlotDrill=true and can never upload backups, so it cannot overwrite or prune the production ones.
3. Tear down by explicit instance ID only, never with a filter-based terminate. Immediately before, re-check that instance's BlotDrill tag and that it is not a production instance (not the one holding the floating IP or serving BLOT_REDIS_HOST). Production instances have termination protection; if a terminate is refused for that reason, stop and ask rather than lifting it.
4. Never roll back to a host whose interface held the floating IP recently. The VPC keeps delivering to the other interface for seconds (README, "Timings and rollback"), which costs [LOCK COMPROMISED] restarts. Go forward to a fresh host instead.
5. No deploys or proxy restarts around a cutover. Only cut over in the windows cutover.sh allows, and always --dry-run first.
6. Private details stay out of public issues and PRs. Instance IDs, public IPs, account IDs, key names: they go in the gitignored data/ directory (the repo is public). Private IPs already in the README are fine.
7. The shell is zsh, which does not word-split variables. Run snippets that loop over several IDs under bash.
8. Do not run anything that writes to the live Redis (CONFIG SET, BGSAVE, readonly.sh, SLOWLOG RESET) or changes the host (sysctl -w, bootstrap.sh) on your own initiative. Propose it, with the exact command and what it will do, after the read-only checks say it is needed.

What triggers this

app/scheduler/check-redis-host.js runs every 5 minutes on the master and emails REDIS_HOST_ALERT with the subject Redis host: <summary> (template app/helper/email/admin/REDIS_HOST_ALERT.txt). Each condition is emailed once when it starts and once when it clears. Conditions are states; events come from app/scheduler/redis-host-events.js, which compares Redis's counters with the ones seen at the previous check (a gap over 15 minutes starts a new baseline, so a long outage does not produce a flood).

The daily email has one line, from app/scheduler/daily/redis-server.js: **Redis:** memory 27% (resize in ~47 days), disk 21% (19 GB free), saved 3m ago, backed up 30m ago. These need action:

To print the current report without sending an email, the README has NODE_PATH=app node app/scheduler/check-redis-host.js; on production that runs inside an app container, so it is a production command (ask first).

Method

1. Read the alert (or the daily line) and find its row above. Note the time, and whether a cutover, deploy or resize happened near it. Check the incident log below for the same pattern.
2. Propose the read-only checks for that alert (next section), one command at a time, and get approval for each (rule 1).
3. Read the result against the notes. Say what you think happened and how sure you are before proposing a fix.
4. A fix is either a README procedure (resize, disaster recovery, bootstrap to re-apply config) or a one-off change on the host. Propose it with the exact commands and wait for a yes. After it, repeat the check that showed the problem to confirm it cleared.
5. Report to the operator in chat, then append an entry to the incident log.

Triage per alert

Host checks run as ssh redis '<command>' (use ssh -n when the command does not read stdin). redis6-cli needs no auth.

Background save failing / last save failed

ssh redis 'redis6-cli INFO persistence'
ssh redis 'sudo tail -n 100 /var/log/redis6/redis6.log'
ssh redis 'df -h / /backups; free -m; sysctl vm.overcommit_memory'

• rdb_last_bgsave_status:err, with rdb_last_bgsave_time_sec and latest_fork_usec (in INFO stats) for how long the last good one took.
• The log names the reason. Can't save in background: fork: Cannot allocate memory is overcommit (vm.overcommit_memory must be 1; bootstrap.sh sets it) or too little free RAM for the fork's copy-on-write: memory is too high for the host, so see the resize section. No space left on device or Write error saving DB on disk is the root disk (/var/lib/redis6).
• Writes are refused (MISCONF) until a save succeeds, so it is urgent. Redis retries on its own save schedule; a manual BGSAVE is a write to the host, so propose it separately once the cause is fixed.
• On a disk problem, find what filled it (sudo du -xh --max-depth=2 / | sort -h | tail) before deleting anything, and ask before deleting.

Redis refused writes (OOM, MISCONF, NOREPLICAS)

ssh redis 'redis6-cli INFO errorstats'
ssh redis 'redis6-cli INFO memory | grep -E "^(used_memory_human|maxmemory_human|mem_fragmentation_ratio):"'
ssh redis 'redis6-cli CONFIG GET min-replicas-to-write'

• OOM: memory reached maxmemory. Treat as the memory alert below, but urgent: the app is serving 503s.
• MISCONF: the failed-save section above.
• NOREPLICAS: min-replicas-to-write is not 0. It should be 0; readonly.sh sets 99 and may have been left on after a freeze or a rehearsal (readonly.sh <host> off undoes it, which is a write: propose it).
• The app side: Error saving file in database lines, and [LOCK COMPROMISED] restarts, in the containers' logs (see below).

Other Redis errors

INFO errorstats for the type and count. READONLY is expected for a moment during a cutover; outside one, clients are talking to a replica: check redis6-cli ROLE on the host BLOT_REDIS_HOST points at, and which host has /etc/blot-redis/floating-ip. WRONGTYPE or ERR in bulk is an app bug: find the error text and the calling code in the containers' logs and fix it in the app.

Rejected connections

ssh redis 'redis6-cli INFO clients'
ssh redis 'redis6-cli CLIENT LIST | grep -o " addr=[0-9.]*" | sort | uniq -c | sort -rn | head'

connected_clients against maxclients. The per-address counts show which host (app container host, proxy, a script) is leaking connections. Fixing that means restarting or fixing the client, which is a production change to propose, not to do.

Restarted or switched host

Expected after a cutover, a resize or a planned reboot. If none, check:

ssh redis 'uptime; redis6-cli INFO server | grep -E "^(run_id|uptime_in_seconds|redis_version):"'
ssh redis 'sudo journalctl -u redis6 --since "2 hours ago" | tail -n 50'
ssh redis 'sudo dmesg | grep -i -E "oom|killed process" | tail'
ssh redis 'sudo tail -n 100 /var/log/redis6/redis6.log'

An OOM kill (dmesg, journald) means memory: the resize section. A reboot means look at why. Data written since rdb_last_save_time is lost after a crash (save policy and the hourly backups are all there is: no AOF), so check redis6-cli INFO persistence and, in the app, whether sync validation needs a run. Expect [LOCK COMPROMISED] restarts in the app containers from the outage itself.

Slow Redis commands

ssh redis 'redis6-cli SLOWLOG GET 20'
ssh redis 'redis6-cli INFO stats | grep latest_fork_usec'
ssh redis 'ls ~/perf/*.log 2>/dev/null && tail -n 20 ~/perf/redis-sample.log'

Each entry has the command, its first arguments, the duration in microseconds and the client address. One slow command blocks everything behind it on Redis's single thread, so the usual suspects are a big reply (SMEMBERS/LRANGE/MGET over huge values), a Lua script, KEYS or an unbounded SCAN, or a fork stall at the :00 backup (compare the time with latest_fork_usec and the ~/perf/redis-sample.log BGSAVE columns, if the collectors from config/redis/perf/ are installed). Find the calling code by grepping the repo for the command and the key pattern, then fix it in the app. Do not SLOWLOG RESET before the entries are recorded in the report.

TCP memory (nearing tcp_mem[1], pressure, sample)

ssh redis 'tail -n 30 ~/tcpmem.log'
ssh redis 'grep TCP: /proc/net/sockstat; cat /proc/sys/net/ipv4/tcp_mem'
ssh redis 'nstat -az TcpExtTCPMemoryPressures TcpExtTCPMemoryPressuresChrono TcpExtPruneCalled TcpExtTCPRcvQDrop'

mem and tcp_mem are pages. Healthy use is dozens of sockets with empty queues and a count far below tcp_mem[1]. A count that keeps rising with few sockets is leaked kernel accounting (the old host's problem in September and October 2026, behind the incident log's first entry): the fix is a new host (resize section), not tuning. A count that rises with many real sockets (ss -tm) is a client leak or burst: find the client. The stopgap for a leak, sudo sysctl -w net.ipv4.tcp_mem=... above the count, is lost on reboot and is a host change to propose. Background and the burst test are in investigate-production-container-restarts.

Memory over 80% of maxmemory / resize now

ssh redis 'tail -n 20 ~/redis-mem.log'
ssh redis 'redis6-cli INFO memory | grep -E "^(used_memory_human|used_memory_peak_human|maxmemory_human|mem_fragmentation_ratio):"'
ssh redis 'redis6-cli INFO keyspace'

Compare with the daily email's projection: steady growth is the normal path to a resize, a sudden jump is something writing a lot (a sync, a runaway job) and is worth finding first. maxmemory is about 70% of RAM, so it is the headroom for the save's copy-on-write that is being used up. Fix: README, "Increasing the Redis server size", then the guardrails below. The daily email warns at 70% so there are days to plan; the 80% alert means do it now. Avoid --bigkeys, --scan or DEBUG commands on the live master (one CPU); never KEYS. MEMORY USAGE <key> is cheap for a suspect key.