NOA Labs

Production Orchestration Patterns

Health checks, restart policies, resource limits, graceful shutdown, and log rotation for containers in production.

Beyond "it starts"

Getting a container running is the easy part — Docker Compose already showed a healthcheck on the database service. Running containers reliably in production means answering a few more questions for every service: how does the platform know it's actually healthy (not just running), what happens when it crashes, how much of the host's CPU and memory can it consume, and how does it shut down without dropping in-flight work. This page covers all four, in plain Docker and in Compose.

Health checks: "running" isn't "ready"

A container can be in the Up state from docker ps while the application inside it is still starting up, stuck in a crash loop it keeps restarting from, or wedged and no longer actually serving requests — none of which "the process hasn't exited" detects on its own. A health check runs a command inside the container on an interval and tracks whether it succeeds, giving Docker (and anything watching Docker) a real signal of application-level health.

In a Dockerfile:

Dockerfile
FROM node:20-alpine
WORKDIR /app
COPY . .
RUN npm ci --omit=dev

HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \
    CMD wget --quiet --tries=1 --spider http://localhost:3000/health || exit 1

CMD ["node", "server.js"]
  • --interval=30s — how often to run the check.
  • --timeout=5s — how long the check itself is allowed to take before counting as a failure.
  • --start-period=10s — a grace period after container start during which failures don't count against the retry limit, for apps that take a moment to warm up.
  • --retries=3 — how many consecutive failures before the container is marked unhealthy.
Bash
$ docker ps
CONTAINER ID   IMAGE      STATUS
a1b2c3d4e5f6   my-app     Up 2 minutes (healthy)

The same thing in a docker-compose.yml, applied to a plain HTTP service rather than the database example already covered in the Compose page:

YAML
services:
  app:
    build: .
    ports:
      - "3000:3000"
    healthcheck:
      test: ["CMD", "wget", "--quiet", "--tries=1", "--spider", "http://localhost:3000/health"]
      interval: 30s
      timeout: 5s
      start_period: 10s
      retries: 3

Health checks matter beyond just docker ps output: a load balancer or orchestrator that's healthcheck-aware stops routing traffic to a container reporting unhealthy, and (as shown in the Compose page) depends_on: { condition: service_healthy } lets other services wait for a real ready signal instead of merely "the container process started."

Restart policies

A restart policy tells the Docker daemon what to do when a container's main process exits — whether that exit was a crash, an intentional stop, or the daemon itself restarting (e.g., after a host reboot).

Bash
docker run -d --restart unless-stopped my-app:1.0
Policy Behavior
no (default) Never automatically restart, regardless of exit code.
on-failure[:max-retries] Restart only on a non-zero exit code; optionally cap the number of retries.
always Always restart, no matter the exit code — including after a manual docker stop, if the daemon itself restarts.
unless-stopped Like always, but respects an explicit manual docker stop — won't restart a container someone deliberately stopped, even across a daemon restart.
YAML
services:
  app:
    build: .
    restart: unless-stopped

  worker:
    build: .
    command: node worker.js
    restart: on-failure:5

unless-stopped is the sensible default for most long-running services — it recovers automatically from a crash or host reboot, without fighting you when you deliberately stop something for maintenance. on-failure with a retry cap fits a job that's expected to sometimes legitimately finish and exit zero (a one-shot worker or migration task) where endless restarting on a genuine failure would just spin uselessly instead of surfacing the problem.

Resource limits

Without limits, a single container with a memory leak or a runaway loop can starve every other container — and the host itself — of CPU and memory. Setting explicit limits contains that blast radius to one container.

Bash
docker run -d \
  --memory="512m" \
  --memory-swap="512m" \
  --cpus="1.5" \
  my-app:1.0
  • --memory="512m" caps the container's memory usage; exceeding it triggers the kernel's OOM killer for that container specifically, terminating it rather than letting it degrade the whole host.
  • --memory-swap="512m" equal to --memory disables swap for the container entirely (swap plus memory would otherwise allow it to exceed the memory limit by swapping) — the right choice for most latency-sensitive services, where swapping is a performance cliff you'd rather avoid than tolerate.
  • --cpus="1.5" limits the container to at most 1.5 CPU cores' worth of processing time, enforced by the kernel's CFS scheduler quota — it can still use multiple cores simultaneously, just never more than 1.5 cores' total worth at once.

In Compose (the modern syntax, under deploy.resources, honored by plain docker compose up since Compose v2 even outside Swarm):

YAML
services:
  app:
    build: .
    deploy:
      resources:
        limits:
          cpus: "1.5"
          memory: 512M
        reservations:
          cpus: "0.5"
          memory: 256M

limits is the hard ceiling; reservations is a soft guarantee used mainly by orchestrators (Swarm, Kubernetes-style schedulers) when deciding which host has room to schedule a container — plain single-host docker compose up mostly cares about limits.

Graceful shutdown

When a container is stopped, Docker sends SIGTERM first (the same signal covered for regular processes in essential commands) and waits a grace period before escalating to SIGKILL:

Bash
docker stop my-app                  # SIGTERM, wait, then SIGKILL if still running
docker stop -t 30 my-app            # give it 30 seconds instead of the 10s default
YAML
services:
  app:
    build: .
    stop_grace_period: 30s

An application that doesn't handle SIGTERM at all gets forcibly killed the moment the grace period expires — dropping in-flight HTTP requests, leaving a database transaction uncommitted, or losing an in-progress job. Handling the signal (closing the HTTP server to new connections, letting in-flight requests finish, then exiting) is application-level work, but the grace period Docker gives it to do that work is configured here.

Logging drivers, briefly

By default, docker logs reads from the json-file logging driver, which writes container stdout/stderr to disk with no rotation unless configured — an easy way to quietly fill a disk on a long-running host:

YAML
services:
  app:
    build: .
    logging:
      driver: json-file
      options:
        max-size: "10m"
        max-file: "3"

This caps total log storage per container to 3 files of 10MB each, rotating automatically — a small but easy-to-forget setting until a host runs out of disk space from months of unbounded log growth.

Common mistakes

  • Adding a healthcheck that only checks "is the process alive" (e.g., CMD ["true"]) instead of an endpoint that actually exercises the application (like /health hitting a real code path) — this reports healthy even when the app is wedged and not actually serving anything.
  • Leaving restart: no (the default) on a production service and being surprised it doesn't come back after a crash or host reboot.
  • Setting a memory limit without also matching --memory-swap to it — the container can still balloon past the intended limit by swapping, which trades an OOM kill for a much worse, harder-to-diagnose performance cliff.
  • Not setting any resource limits at all on a shared host, so one runaway container can starve every other container (and the host itself) of CPU or memory.
  • Ignoring SIGTERM in application code and relying on the default (short) grace period, resulting in dropped requests or interrupted transactions on every single deploy or restart.
  • Forgetting log rotation on a long-running host — json-file logs grow unbounded by default and are a common, easily-avoidable cause of a host quietly running out of disk space.