Hermes S6 Container Supervision

Hermes s6-overlay Container Supervision

When to use this skill

Load this skill when you're working on:

• Adding or removing a static service in the Hermes Docker image (something that should be supervised at every container start, like the dashboard)
• Diagnosing why a per-profile gateway isn't starting, restarting, or surviving docker restart
• Understanding why the container's CMD is /opt/hermes/docker/main-wrapper.sh and how leading-dash args reach the user's program
• Modifying cont-init.d boot scripts (UID remap, volume seeding, profile reconciliation)
• Changing the rendered run-script for per-profile gateways (Phase 4)

If you're just running the Hermes Agent and want to use Docker, see website/docs/user-guide/docker.md instead.

Architecture at a glance

/init                                  ← PID 1 (s6-overlay v3.2.3.0)
├── cont-init.d                        ← oneshot setup, runs as root
│   ├── 01-hermes-setup                ← docker/stage2-hook.sh
│   │   ├── UID/GID remap
│   │   ├── chown /opt/data
│   │   ├── chown /opt/data/profiles (every boot)
│   │   ├── seed .env / config.yaml / SOUL.md
│   │   └── skills_sync.py
│   └── 02-reconcile-profiles          ← hermes_cli.container_boot
│       ├── chown /run/service (hermes-writable for runtime register)
│       └── walk $HERMES_HOME/profiles/<name>/gateway_state.json
│           → recreate /run/service/gateway-<name>/
│           → auto-start only those with prior_state == "running"
│
├── s6-rc.d (static services, in /etc/s6-overlay/s6-rc.d/)
│   ├── main-hermes/run                ← exec sleep infinity (no-op slot)
│   └── dashboard/run                  ← if HERMES_DASHBOARD=1, runs `hermes dashboard`
│
├── /run/service (s6-svscan watches; tmpfs)
│   ├── gateway-coder/                 ← runtime-registered per-profile
│   │   ├── type        ("longrun")
│   │   ├── run         ("#!/command/with-contenv sh ... exec s6-setuidgid hermes hermes -p coder gateway run")
│   │   ├── down        (marker — present means "registered but don't auto-start")
│   │   └── log/run     (s6-log → $HERMES_HOME/logs/gateways/coder/current)
│   └── ...
│
└── CMD ("main program")               ← /opt/hermes/docker/main-wrapper.sh
    └── routes user args: bare exec | hermes subcommand | hermes (no args)
        — exec'd by /init with stdin/stdout/stderr inherited (TTY for --tui)

Key files

Why Architecture B (CMD as main program, not s6-supervised)

The original plan (v1–v3) called for main hermes to run as a supervised s6-rc service. Two real s6-overlay v3 mechanics blocked that:

1. cont-init.d scripts receive no CMD args — so the stage2 hook can't parse docker run <image> chat -q "hi" to set HERMES_ARGS for a service run script to consume.
2. /run/s6/basedir/bin/halt does NOT propagate the exit code written to /run/s6-linux-init-container-results/exitcode. Containers always exit 143 (SIGTERM) regardless. Confirmed by skarnet (s6 author) in issue #477: "if you want a container shutdown, you need to either have your CMD exit, or, if you have no CMD, write the container exit code you want then call halt".

So we use the s6-overlay-native CMD pattern via the dispatcher: ENTRYPOINT ["/opt/hermes/docker/entrypoint-dispatch.sh"], which under PID 1 exec's /init /opt/hermes/docker/main-wrapper.sh "$@". The wrapper is prepended to user args automatically — so docker run <image> --version becomes /init main-wrapper.sh --version, and --version doesn't get intercepted by /init's POSIX shell. The wrapper drops to hermes via s6-setuidgid, then exec's the chosen program. The program's exit code becomes the container exit code, exactly matching the pre-s6 tini contract. When the entrypoint is NOT PID 1 (Fly Machines, docker run --init), the dispatcher skips /init entirely (it would abort with can only run as pid 1), restores the s6 helper PATH, runs stage2-hook.sh, and exec's main-wrapper.sh directly — no supervised services on that path (#38349).

Trade-off: main hermes is unsupervised under s6. That exactly matches its behavior under tini (the pre-s6 image). Dashboard supervision is the only new guarantee — and per-profile gateways under /run/service/ get full supervision.

Quick recipes

Verify s6 is PID 1 in a running container

docker exec <c> sh -c 'cat /proc/1/comm; readlink /proc/1/exe'
# Expect: s6-svscan or init / /package/admin/s6/.../s6-svscan

Inspect a profile gateway service

# /command/ isn't on docker-exec PATH — use absolute path
docker exec <c> /command/s6-svstat /run/service/gateway-<name>
# "up (pid …) … seconds"            → running
# "down (exitcode N) … seconds, normally up, want up, …" → s6 wants it up but the process keeps exiting (crash loop)
# "down … normally up, ready …"     → user stopped it

Bring a service up/down manually

docker exec <c> /command/s6-svc -u /run/service/gateway-<name>   # up
docker exec <c> /command/s6-svc -d /run/service/gateway-<name>   # down
docker exec <c> /command/s6-svc -t /run/service/gateway-<name>   # SIGTERM (restart)

Watch the cont-init reconciler log

docker exec <c> tail -n 50 /opt/data/logs/container-boot.log
# 2026-05-21T06:18:05+0000 profile=coder prior_state=running action=started
# 2026-05-21T06:18:05+0000 profile=writer prior_state=stopped action=registered

Add a new static service

1. Create docker/s6-rc.d/<name>/type with longrun\n and docker/s6-rc.d/<name>/run (use #!/command/with-contenv sh + # shellcheck shell=sh).
2. Drop to hermes via s6-setuidgid hermes at the top of run (unless you specifically need root).
3. Create empty docker/s6-rc.d/<name>/dependencies.d/base so it waits for the base bundle.
4. Create empty docker/s6-rc.d/user/contents.d/<name> so it joins the user bundle.
5. The COPY docker/s6-rc.d/ in the Dockerfile picks it up automatically — no other changes.

Change the per-profile gateway run command

Edit S6ServiceManager._render_run_script in hermes_cli/service_manager.py. The function is also called by hermes_cli/container_boot.py::_register_service during boot reconciliation, so it's the single source of truth. Update the corresponding assertion in tests/hermes_cli/test_service_manager.py::test_s6_register_creates_service_dir_and_triggers_scan.