Investigate Production Container Restarts

Investigate production container restarts

Blot runs three named app containers plus an airlock sidecar (see scripts/deploy/constants.js):

• blot-container-blue (port 8088, siteConfig) — failover, sites+blogs
• blot-container-green (port 8089, siteConfig) — dashboard/brochure/sync
• blot-container-yellow (port 8090, blogsConfig) — preview+published blogs
• blot-airlock — separate egress sidecar, not part of blue/green/yellow

Every container runs docker create --restart unless-stopped, so Docker itself will silently restart a crashed container — a restart is not inherently a deploy problem, but it's also not nothing. This skill works out which of three things happened:

1. the deploy script's own create/replace cycle (expected, once per container per deploy);
2. the deploy's automated rollback after a failed health check;
3. an unplanned crash that Docker silently recovered from and which would otherwise go unnoticed: an in-process V8 OOM, a Linux-level OOM kill, or the deliberate crash when a sync loses its folder lock.

Confirm with the user before running anything against production that isn't on the auto-approved list below, and stick to read-only commands (log tailing, docker inspect, docker logs, read-only redis-cli) unless a state-changing action has been explicitly authorized.

Auto-approved (no confirmation needed) — these read-only commands over ssh blot may be run without asking:

• docker ps -a and docker inspect on blot-container-{blue,green,yellow} (release ID, CreatedAt, RestartCount, OOMKilled, exit code)
• cat ~/docker-health-check.log
• dmesg | grep -i kill (or the kills helper)
• docker logs <container> with --since/--until, including grepping for FATAL ERROR / JavaScript heap out of memory
• grep <request-id> /var/instance-ssd/logs/access.log (or the req helper) to find the triggering request

Anything else (state-changing commands, restarts, deploys) still needs explicit user approval.

1. Identify the most recent deployment

SSH host is blot. Each container is created with -e BLOT_RELEASE_ID=<commitHash> (scripts/deploy/util/generateDockerCommand.js) and named ${REGISTRY_URL}:${commitHash} as its image tag — this is the ground truth for "what deploy is currently running," independent of docker ps's uptime column:

ssh blot "docker ps -a --format 'table {{.Names}}\t{{.Status}}\t{{.CreatedAt}}\t{{.Image}}'"
ssh blot "for c in blot-container-blue blot-container-green blot-container-yellow; do echo \$c:; docker inspect \$c --format '{{range .Config.Env}}{{println .}}{{end}}' | grep BLOT_RELEASE_ID; done"

Cross-reference the commit hash against GitHub to see what actually shipped and when:

git log -1 <commit-hash>
gh run list --workflow=deploy.yml --limit 5

If all three containers share the same BLOT_RELEASE_ID and a similar CreatedAt, that confirms they were replaced together by one deploy run — the baseline for "expected" restarts. A container with a different (older) BLOT_RELEASE_ID than its siblings, or a much older CreatedAt, means it didn't pick up the latest deploy — that's itself worth explaining (failed health check → rollback left it on the old image; see step 4).

2. docker ps -a — is this actually a restart worth investigating?

ssh blot "docker ps -a --format 'table {{.Names}}\t{{.Status}}\t{{.CreatedAt}}'"

• Up <a few minutes> with a CreatedAt matching the deploy time = the container was recreated by the deploy itself (docker create + docker start, per container, once). Normal, expected, not a crash.
• Up <a few minutes> with a CreatedAt from before the deploy = the same container object restarted (not recreated) after the deploy finished — this is Docker's --restart unless-stopped kicking in after a crash, not part of the deploy process. This is the case worth digging into.
• RestartCount > 0 confirms Docker has restarted this container object at least once since it was created:

ssh blot "docker inspect <container> --format 'OOMKilled={{.State.OOMKilled}} ExitCode={{.State.ExitCode}} StartedAt={{.State.StartedAt}} RestartCount={{.RestartCount}}'"

OOMKilled only reflects a Docker/cgroup-level OOM kill of the whole container — it's usually false even when the Node process inside hit its own --max-old-space-size limit and crashed on its own. Don't treat OOMKilled=false as ruling out memory as the cause; check both failure modes in step 3 regardless.

Also check the auto-restart health-check script's own log, which is a separate mechanism from Docker's --restart policy — it appends a line every time it force-restarts a container it decided was unhealthy:

ssh blot "cat ~/docker-health-check.log"

3. Determine the failure mode

Several distinct causes look identical in docker ps but require different evidence and point to different fixes. Check each — don't stop at the first one that seems plausible. Grep the end of the previous run's log for FATAL ERROR, LOCK COMPROMISED and Unhandled 'error' event before anything else; between them they cover most crashes.

V8/Node heap OOM (in-process crash, not a Linux OOM kill)

ssh blot "docker logs <container> --since <before-crash> --until <after-crash> 2>&1 | grep -B5 'FATAL ERROR\|JavaScript heap out of memory'"

Each fatal error also leaves a Node report (JS stack at the crash, heap spaces, resource usage; env vars excluded) in /var/www/blot/data/node-reports/<container>/report.*.json on the host. These outlive log rotation and container recreation, so check them first (ls/cat there is read-only):

ssh blot "ls -lt /var/www/blot/data/node-reports/<container>/ | head"

If the container's logs have rotated past the crash, openresty still has it: the error log shows a burst of connect() failed / prematurely closed errors to the container's port in the crash second, and the access log shows the in-flight requests that died with it (502, long request time, up=127.0.0.1:<port>):

ssh blot "zcat -f /var/instance-ssd/logs/error.log* | grep ':8090' | grep -c 'connect() failed'"

The crash timestamp is the Starting server on ... line that follows the restart in the container's log (search forward from there to find where the previous run's log ends). Almost always an application-code problem — something holding too much data in memory for a single request — not a memory-limit tuning issue on its own. Read the log lines immediately before the crash to find the triggering request (step 5).

Linux cgroup OOM kill

ssh blot "dmesg | grep -i kill"
# or the bashrc helper, which converts dmesg's boot-relative timestamps to human-readable and filters for OOM events:
ssh blot "kills"

Confirm the killed process was actually node (not esbuild, chromium, or something unrelated sharing the container), and compare its anon-rss at kill time against the container's configured memory limit:

ssh blot "docker inspect <container> --format '{{.HostConfig.Memory}}'"

Lost sync folder lock ([LOCK COMPROMISED], ECOMPROMISED)

A sync holds a per-blog folder lock in Redis (app/sync/lock.js): 10s TTL, renewed by a heartbeat EXTEND every 3s over the shared node-redis connection. If a renewal doesn't reach Redis before the TTL lapses, the lock is lost, and app/sync/index.js deliberately rethrows so the process stops writing. It mostly hits green (which runs the syncs).

Signature:

• the log ends with one or more [LOCK COMPROMISED] {...} diagnostics blocks, then Error: Lock was lost: blog:<id>:folder-lock thrown from app/sync/index.js with code: 'ECOMPROMISED';
• OOMKilled=false, and no Node report (it isn't a fatal V8 error);
• several blogs' locks are usually lost in the same second, because they share the stalled connection.

ssh blot "docker logs <container> --since <before> --until <after> 2>&1 | grep -n 'LOCK COMPROMISED\|Lock was lost\|\[LOCK\]'"

Read the heartbeat instrumentation:

• [LOCK] slow heartbeat <key> tickDelay=… roundTrip=… is logged for any renewal over 500ms.
  • tickDelay near 0 means the timer fired on time, so the event loop isn't to blame.
  • A large roundTrip means the EXTEND sat on the connection.
  • Several heartbeats finishing in the same second, with round trips 3s apart (e.g. 7.5s / 4.5s / 1.5s), means the connection was blocked for the whole span.
• [LOCK] stall probe … probePing=… idle=… qbuf=… omem=… comes from a separate Redis connection, taken while a heartbeat is outstanding.
  • A fast probePing means Redis itself is responsive.
  • Empty qbuf/omem on the shared connection means nothing is backed up inside Redis either.
  • Both together mean the bytes are stuck in a TCP send queue or in flight.
• elu= on request lines is event loop utilisation over that request's lifetime.
  • A /health request that took several seconds with elu=0.02 proves the process sat idle, waiting on I/O.
  • elu=1.00 on a 1ms request means nothing.
• Baseline: count slow heartbeat lines per container since the deploy. A healthy connection logs few or none.

Then check the Redis host. That's where the root cause has been, and it is a separate production host (ssh redis), so confirm with the user first:

ssh -n redis 'grep TCP: /proc/net/sockstat; cat /proc/sys/net/ipv4/tcp_mem; nstat -az TcpExtTCPMemoryPressures TcpExtTCPMemoryPressuresChrono TcpExtPruneCalled TcpExtTCPRcvQDrop TcpExtTCPOFODrop'
ssh -n redis 'tail -20 ~/tcpmem.log'   # 5-minute history, from /etc/cron.d/blot-redis (config/redis/bin/tcpmem-log.sh)

• sockstat's mem and tcp_mem are both in pages. Once mem is above tcp_mem[1], the kernel is in memory-pressure mode. Above tcp_mem[2], each socket may buffer only a minimal amount of incoming data.
• In that state small commands get through, but any pipelined burst sent to Redis is dropped and re-sent with growing back-off, which stalls everything queued behind it for seconds.
• TCPMemoryPressures counts entries into pressure since boot.
• TCPMemoryPressuresChrono is the total time spent in it, in ms. It is only added to when the kernel leaves pressure, so a value that jumps dates the episode.
• Prune and drop counters that rise with each burst confirm the receiving side is discarding data.

From 25 Sep to 6 Oct 2026 the Redis host's kernel TCP memory accounting had leaked past tcp_mem[2], with near-empty real socket buffers.

• That caused the green crashes of 26 Sep and 6 Oct, and green's routine 1–2s slow heartbeats.
• Raising net.ipv4.tcp_mem above the leaked count (runtime sysctl -w) cleared it at once.
• The real fix is a kernel update and reboot. See blotcms/blot#2041 for the full write-up, the checklist, and whether that has happened yet.

To confirm a stall is on the network path, and not in the app:

• Burst test. Pipe about 250KB of inline PING\r\n into one fresh TCP connection to Redis, and time the replies. The Redis address is BLOT_REDIS_HOST in a container's env; send the script over stdin as ssh blot 'script=$(cat); node -e "$script"' < burst.js.
  • Run it from the app host, then from a throwaway container on blotnet, e.g. docker run --rm -i --network blotnet --entrypoint node <a live container's image> -.
  • A healthy path answers in tens of milliseconds with 0 retransmits. Compare RetransSegs in /proc/net/snmp before and after.
  • While the leak was active it took 2.5–9s, with tens to over a hundred retransmits.
  • Check the app host's ethtool -S eth0 | grep allowance counters too; they rule the AWS network allowance on this side in or out.
• npm run fix-probe -- <blog> (scripts/fix-probe/README.md) runs a sync/fix check (default tag-ghosts) read-only in a throwaway container.
  • It pings the shared connection and a separate control connection every 100ms.
  • It samples node-redis' queue, the socket's Node and kernel (/proc/net/tcp) state, and retransmits.
  • Shared pings stalling while control pings stay fast, with an empty Node buffer and a full kernel send queue, means the network path is losing data.

Don't respond to this by loosening the lock (a longer TTL, swallowing ECOMPROMISED). The crash is the alarm, and loosening it hides a Redis-wide problem: every container's large reads and writes are slowed too.

Pitfalls when gathering this evidence:

• Use ssh -n (or redirect stdin) for remote commands that might read standard input. A grep whose file variable came out empty, or a backgrounded node -, will otherwise hang or silently do nothing.
• docker run <image tag> pulls the image if the tag isn't on the host under that name, even when the containers are running it.
• Don't write scratch files on the production hosts. Pull the logs locally and analyse them there.