Hermes S6 Container Supervision: Debug and Modify s6 Services

This skill is for anyone who needs to modify or debug the s6 service supervision inside the Hermes Docker image. You would reach for it when adding a static service that should run at every container start, diagnosing why a per-profile gateway won't start or survive a restart, or understanding how the container's CMD and entrypoint dispatch work. It is an optional skill, installed on demand, and it assumes you are already comfortable with Docker and the Hermes Agent runtime.

What it does

Hermes uses s6-overlay v3.2.3.0 as its process supervisor inside the Docker image. This skill gives you the map of that supervision tree: what runs at boot, which services are static versus runtime-registered, and how the container's main program is launched. With it you can inspect the state of any supervised service, bring services up or down manually, add a new static service, change how per-profile gateway run scripts are rendered, and understand the common failure modes that come up when working with s6 in this image.

The architecture is split into three layers. First, cont-init.d scripts run once at boot as root and handle setup like UID remapping, volume seeding, and profile reconciliation. Second, static services live in /etc/s6-overlay/s6-rc.d/ and are defined in the Dockerfile. Third, per-profile gateway services are registered at runtime under /run/service/ on a tmpfs, so they disappear on restart and are recreated by the reconciler. The container's CMD is /opt/hermes/docker/main-wrapper.sh, which routes user arguments to the right program and drops privileges to the hermes user.

Before you start

This skill is optional and installed on demand. It targets Linux platforms only. You need a running Hermes Docker container to inspect, and you need the ability to run docker exec and docker build commands. The skill's path is optional-skills/devops/hermes-s6-container-supervision, version 1.0.0, authored by Hermes Agent, licensed under MIT. The related skill hermes-agent is listed as a companion.

Architecture at a glance

The source provides a tree view of the supervision hierarchy. Reproduced here exactly:

/init                                  ← PID 1 (s6-overlay v3.2.3.0)
├── cont-init.d                        ← oneshot setup, runs as root
│   ├── 01-hermes-setup                ← docker/stage2-hook.sh
│   │   ├── UID/GID remap
│   │   ├── chown /opt/data
│   │   ├── chown /opt/data/profiles (every boot)
│   │   ├── seed .env / config.yaml / SOUL.md
│   │   └── skills_sync.py
│   └── 02-reconcile-profiles          ← hermes_cli.container_boot
│       ├── chown /run/service (hermes-writable for runtime register)
│       └── walk $HERMES_HOME/profiles/<name>/gateway_state.json
│           → recreate /run/service/gateway-<name>/
│           → auto-start only those with prior_state == "running"
│
├── s6-rc.d (static services, in /etc/s6-overlay/s6-rc.d/)
│   ├── main-hermes/run                ← exec sleep infinity (no-op slot)
│   └── dashboard/run                  ← if HERMES_DASHBOARD=1, runs `hermes dashboard`
│
├── /run/service (s6-svscan watches; tmpfs)
│   ├── gateway-coder/                 ← runtime-registered per-profile
│   │   ├── type        ("longrun")
│   │   ├── run         ("#!/command/with-contenv sh ... exec s6-setuidgid hermes hermes -p coder gateway run")
│   │   ├── down        (marker — present means "registered but don't auto-start")
│   │   └── log/run     (s6-log → $HERMES_HOME/logs/gateways/coder/current)
│   └── ...
│
└── CMD ("main program")               ← /opt/hermes/docker/main-wrapper.sh
    └── routes user args: bare exec | hermes subcommand | hermes (no args)
        — exec'd by /init with stdin/stdout/stderr inherited (TTY for --tui)

This tree shows the separation of concerns. The cont-init.d scripts are oneshot and run as root. The static services are defined in the image and are always present. The per-profile gateways are dynamic, created from persistent state on each boot. The CMD is the user-facing entry point.

Key files

The source lists the files that matter. Here they are with their roles:

Path	Role
Dockerfile	s6-overlay install + cont-init.d wiring + ENTRYPOINT ["/opt/hermes/docker/entrypoint-dispatch.sh"]
docker/entrypoint-dispatch.sh	PID-1 dispatcher: exec's /init + main-wrapper when the image owns PID 1; on wrapped runtimes (Fly Machines, docker run --init) falls back to stage2-hook + main-wrapper directly, restoring the s6 helper PATH first (#38349).
docker/stage2-hook.sh	The "old entrypoint logic", UID remap, chown, seed, skills sync. Runs as cont-init.d/01-hermes-setup.
docker/cont-init.d/02-reconcile-profiles	Calls hermes_cli.container_boot on every boot to restore profile gateway slots from the persistent volume.
docker/main-wrapper.sh	The container's CMD. Routes user args, drops to hermes via s6-setuidgid, exec's the chosen program.
docker/s6-rc.d/main-hermes/run	No-op sleep infinity, slot exists so the s6-rc user bundle is valid; main hermes runs as the CMD, not as a supervised service.
docker/s6-rc.d/dashboard/run	Conditional service, exec sleep infinity unless HERMES_DASHBOARD is truthy.
docker/entrypoint.sh	Back-compat shim that execs the stage2 hook. External scripts that hard-coded the old entrypoint path still work.
hermes_cli/service_manager.py	S6ServiceManager: register_profile_gateway, unregister_profile_gateway, start/stop/restart/is_running, list_profile_gateways.
hermes_cli/container_boot.py	reconcile_profile_gateways(), walks persistent profiles, regenerates s6 slots, emits container-boot.log.
hermes_cli/gateway.py::_dispatch_via_service_manager_if_s6	Intercepts hermes gateway start/stop/restart and routes to s6 when running in a container.

These files are the touchpoints for any modification. The S6ServiceManager is the central class for runtime service operations. The container_boot.py module handles the boot-time reconciliation.

Why Architecture B (CMD as main program, not s6-supervised)

The original plan (v1, v3) called for main hermes to run as a supervised s6-rc service. Two real s6-overlay v3 mechanics blocked that:

1. cont-init.d scripts receive no CMD args, so the stage2 hook can't parse docker run chat -q "hi" to set HERMES_ARGS for a service run script to consume.
2. /run/s6/basedir/bin/halt does NOT propagate the exit code written to /run/s6-linux-init-container-results/exitcode. Containers always exit 143 (SIGTERM) regardless. Confirmed by skarnet (s6 author) in issue #477: "if you want a container shutdown, you need to either have your CMD exit, or, if you have no CMD, write the container exit code you want then call halt".

So we use the s6-overlay-native CMD pattern via the dispatcher: ENTRYPOINT ["/opt/hermes/docker/entrypoint-dispatch.sh"], which under PID 1 exec's /init /opt/hermes/docker/main-wrapper.sh "$@". The wrapper is prepended to user args automatically, so docker run --version becomes /init main-wrapper.sh --version, and --version doesn't get intercepted by /init's POSIX shell. The wrapper drops to hermes via s6-setuidgid, then exec's the chosen program. The program's exit code becomes the container exit code, exactly matching the pre-s6 tini contract. When the entrypoint is NOT PID 1 (Fly Machines, docker run --init), the dispatcher skips /init entirely (it would abort with can only run as pid 1), restores the s6 helper PATH, runs stage2-hook.sh, and exec's main-wrapper.sh directly, no supervised services on that path (#38349).

Trade-off: main hermes is unsupervised under s6. That exactly matches its behavior under tini (the pre-s6 image). Dashboard supervision is the only new guarantee, and per-profile gateways under /run/service/ get full supervision.

This design decision is worth understanding before you start debugging. The main hermes process is not supervised, so if it crashes, the container exits. That is intentional and matches the previous tini behavior. The supervision you get is for the dashboard and the per-profile gateways.

Quick recipes

The source provides several ready-to-run commands. Each is reproduced exactly.

Verify s6 is PID 1 in a running container

docker exec <c> sh -c 'cat /proc/1/comm; readlink /proc/1/exe'
# Expect: s6-svscan or init / /package/admin/s6/.../s6-svscan

This confirms that s6-overlay is actually running as the init process. If you see something else, the container may be using a wrapped runtime.

Inspect a profile gateway service

# /command/ isn't on docker-exec PATH — use absolute path
docker exec <c> /command/s6-svstat /run/service/gateway-<name>
# "up (pid …) … seconds"            → running
# "down (exitcode N) … seconds, normally up, want up, …" → s6 wants it up but the process keeps exiting (crash loop)
# "down … normally up, ready …"     → user stopped it

The output of s6-svstat tells you the state and the desired state. The comments in the source explain how to interpret the three common cases.

Bring a service up/down manually

docker exec <c> /command/s6-svc -u /run/service/gateway-<name>   # up
docker exec <c> /command/s6-svc -d /run/service/gateway-<name>   # down
docker exec <c> /command/s6-svc -t /run/service/gateway-<name>   # SIGTERM (restart)

These commands let you control a service without going through the Hermes CLI. Useful for testing or recovering a stuck gateway.

Watch the cont-init reconciler log

docker exec <c> tail -n 50 /opt/data/logs/container-boot.log
# 2026-05-21T06:18:05+0000 profile=coder prior_state=running action=started
# 2026-05-21T06:18:05+0000 profile=writer prior_state=stopped action=registered

The reconciler writes a log line per profile, showing what action it took. This is the first place to look when a gateway is missing after a restart.

Add a new static service

The source gives a five-step procedure. Follow it exactly:

1. Create docker/s6-rc.d//type with longrun\n and docker/s6-rc.d//run (use #!/command/with-contenv sh + # shellcheck shell=sh).
2. Drop to hermes via s6-setuidgid hermes at the top of run (unless you specifically need root).
3. Create empty docker/s6-rc.d//dependencies.d/base so it waits for the base bundle.
4. Create empty docker/s6-rc.d/user/contents.d/ so it joins the user bundle.
5. The COPY docker/s6-rc.d/ in the Dockerfile picks it up automatically, no other changes.

The placeholders in the paths are intentional; you replace them with your service name. The key is that the Dockerfile already copies the whole docker/s6-rc.d/ directory, so any new service directory you add there is picked up automatically.

Change the per-profile gateway run command

Edit S6ServiceManager._render_run_script in hermes_cli/service_manager.py. The function is also called by hermes_cli/container_boot.py::_register_service during boot reconciliation, so it's the single source of truth. Update the corresponding assertion in tests/hermes_cli/test_service_manager.py::test_s6_register_creates_service_dir_and_triggers_scan.

This is the central place to change how gateway run scripts are generated. Because it is used both at runtime registration and at boot reconciliation, a change here affects both paths consistently.

Run the docker test harness

docker build -t hermes-agent-harness:latest .
HERMES_TEST_IMAGE=hermes-agent-harness:latest scripts/run_tests.sh tests/docker/ -v
# Expect 19 passed, 0 xfailed against the s6 image

The harness lives in tests/docker/ and skips when Docker isn't available. The per-test timeout is bumped to 180s (see tests/docker/conftest.py).

This is how you validate any changes you make to the s6 setup. The expected result is 19 passed and 0 xfailed.

Common pitfalls

The source lists several pitfalls that you are likely to hit. Each is explained here in more detail.

"command not found" via docker exec

/command/ (where s6-overlay puts its binaries) is on PATH only for processes spawned by the supervision tree, services, cont-init.d, main-wrapper.sh. docker exec s6-svstat … will fail with "command not found"; always use the absolute path /command/s6-svstat. The hermes binary works because the Dockerfile adds /opt/hermes/.venv/bin to the runtime ENV PATH.

This is a common first stumble. The fix is simple: use the full path.

Profile directory ownership

The cont-init reconciler runs as hermes (s6-setuidgid hermes in 02-reconcile-profiles). If a profile dir ends up root-owned (e.g. because docker exec hermes profile create … ran as root by default), the reconciler can't read SOUL.md and fails with PermissionError. Mitigation: stage2-hook.sh chowns $HERMES_HOME/profiles to hermes on every boot, idempotently. Don't remove that block.

This is a permission trap. The reconciler runs as the hermes user, so any profile directory that is root-owned will cause a failure. The stage2 hook is designed to fix ownership on every boot, so keep that block intact.

Files written by docker exec are root-owned

docker exec defaults to root. Either pass --user hermes or rely on the stage2 chown sweep next reboot. Don't write files under $HERMES_HOME/profiles// as root manually, the next reconcile pass will sweep them but in-flight operations may hit perm errors.

If you create or edit files inside a running container, they will be owned by root unless you specify --user hermes. That can cause permission errors for the reconciler or the gateway processes. The stage2 sweep will fix ownership on the next boot, but it is safer to use the correct user from the start.

Service slot exists but s6-svstat says "s6-supervise not running"

The service directory is on tmpfs and was wiped on container restart. Either the cont-init reconciler hasn't run yet (give it a moment after docker restart) or it failed. Check docker logs | grep '02-reconcile'.

Because /run/service is tmpfs, it is empty after a restart. The reconciler recreates the slots, but that takes a moment. If the slot is missing, the reconciler may have failed; the logs will show why.

Gateway starts then immediately exits (down (exitcode 1) in svstat)

Most likely the profile has no model or auth configured. The service slot is correct, the gateway itself is unconfigured. Run hermes -p setup first. The s6 supervisor will keep restarting it; that's the desired behavior (when you fix the config, the next attempt succeeds and stays up).

A crash loop is often a configuration problem, not a supervision problem. The supervisor is doing its job by restarting the service. Fix the profile configuration and the gateway will stay up.

Reconciler skipped a profile

The reconciler keys on the presence of SOUL.md as the "real profile" marker. hermes profile create always seeds it. If a profile dir is missing SOUL.md (stray directory, partial restore, backup-in-progress), the reconciler skips it intentionally. Add a SOUL.md (even empty) to opt back in.

This is a deliberate design choice. The reconciler only manages profiles that have a SOUL.md file. If you have a profile directory that is being ignored, check for that file.

"Help, the container exits 143!"

Check whether something is invoking s6-svscanctl -t or /run/s6/basedir/bin/halt, both cause /init to begin stage 3 shutdown but return 143 (SIGTERM) rather than the desired exit code. This was the Phase 2 architecture pivot from A to B. For container shutdown with a real exit code, you must let the CMD (main-wrapper.sh) exit normally; do not try to control exit from a finish script.

Exit code 143 is a symptom of the s6 shutdown path not propagating the intended exit code. The fix is to let the main program exit normally. Do not attempt to use finish scripts to control the container exit code.

When not to use it

If you are just running the Hermes Agent and want to use Docker, the source points you to website/docs/user-guide/docker.md instead. This skill is for modifying or debugging the s6 supervision itself, not for everyday Docker usage.

Limits and gotchas

The main limitation is that main hermes is not supervised by s6. That is a deliberate trade-off to preserve the exit-code contract. Also, the supervision tree only exists when the container runs with s6 as PID 1. On wrapped runtimes like Fly Machines or docker run --init, the dispatcher skips /init entirely, so there are no supervised services on that path. The per-profile gateway slots live on tmpfs and are recreated at boot, so any manual changes to /run/service/ are lost on restart. The reconciler only manages profiles that have a SOUL.md file, so a missing SOUL.md means the profile is skipped. Finally, the docker exec PATH does not include /command/, so you must use absolute paths for s6 binaries.

Related skills

The source lists two related skills. hermes-agent-dev is for general hermes-agent codebase navigation, which is useful when you need to trace how the s6 stack interacts with the rest of the code. hermes-tool-quirks covers specific Hermes-tool workarounds (sed/grep/etc.) and is recommended when debugging the s6 stack's interaction with hermes built-in tools.