What happened
On the Ailoos server, a value in .env.production was changed and the coordinator container was restarted with docker restart. The new value did not take effect. It could not: docker restart restarts the process inside the existing container, which keeps the environment it was created with. Environment variables are fixed at container creation. To apply a change you must recreate: docker compose up -d --no-deps <service> (or down and up).
Code changes on bind mounts, by contrast, are picked up by a plain restart — the process rereads the files from disk on start. Two different mechanisms, easily confused because one of them works.
The second lesson
Recreating the container (1 August 2026) caused a real production crash loop. The container had been alive for months. In earlier sessions, Python packages had been installed into it by hand with docker exec pip install … — never captured in the compose file or a Dockerfile. Recreation discarded the container's writable layer, and with it those packages. The process died on an ImportError for email-validator.
The rule
- Env var change → recreate, not restart.
- Before recreating any long-lived production container, check that its build definition installs every dependency the process actually needs. Assume a container that has lived for months may carry hand-installed packages nobody recorded, and be ready to diagnose a missing-module error immediately after recreation.
- The permanent fix is the boring one: every dependency in the versioned build, never in a shell.
Related rules from the same server
- The coordinator's code is mounted, not baked into the image; deploy is copy then restart.
- Before deploying a central file,
scpthe remote copy and diff it against local: one file diverged from production twice in a single session because local accumulated unshipped work.
From the Ailoos node-network work, 2026-08-01.


