Graceful shutdown: handling SIGTERM in services

Goal

Make deployments and scaling events invisible to clients: no dropped requests, no half-written jobs, no orphaned locks.

Prerequisites

The process receives signals directly (PID 1 in the container is the service or a proper init, not a shell that swallows signals), and the orchestrator's grace period is known (the cited Docker documentation describes SIGTERM followed by SIGKILL after a timeout).

Steps

1. On SIGTERM, mark readiness as failing so load balancers stop sending new requests; keep liveness passing.
2. Stop accepting new connections; keep serving in-flight requests up to a deadline shorter than the grace period.
3. For background workers: stop pulling new jobs, finish the current one if it fits in the deadline, otherwise release it (negative acknowledgement) for another worker.
4. Flush logs and metrics, close database connections and pools, release advisory locks.
5. Exit with status 0; log the shutdown duration.
6. Configure the grace period (stopGracePeriod, terminationGracePeriodSeconds, TimeoutStopSec) to exceed the longest expected in-flight work.

Expected result

Rolling deploys show no 5xx spikes; queues show no duplicated or lost jobs around restarts.

Limits and test basis

Long-running requests (uploads, streams) need application-level checkpoints or must be tolerated as failures. Test by sending SIGTERM under load in a staging environment and watching error rates.