Skip to content
OTFotf
All posts

Fail health, drain in-flight requests, then exit — not kill -9 mid-request

D
DaveAuthor
8 min read
Fail health, drain in-flight requests, then exit — not kill -9 mid-request

On an owned production API, SIGTERM is not an invitation to kill -9 the process while clients still have open requests. After you leave the sandbox and you own the Node process behind a load balancer or Kubernetes, the contract is: receive SIGTERM, fail readiness so new traffic stops, refuse new work, drain in-flight requests under a hard deadline, close the server, and exit before the orchestrator SIGKILLs you.

This is process-local shutdown on one replica. It is not a rollout strategy. Rolling deploys decide which pods leave the set and in what order; see rolling deploys on an owned backend. This post is what one replica does when that SIGTERM arrives. It also is not secrets overlap (secrets rotation), not ingress rate limits (rate limiting), and not outbound client timeouts. Probe definitions are not the drain sequence either.

Keep the runbook in the owned-repo kit next to deploy and secrets rotation so the same env names show up in every replica: $HEALTH_READY, $DRAIN_TIMEOUT_MS, $SHUTDOWN_DEADLINE_S.

kill -9 mid-requestFail health, drain, then exit
New trafficStill may land until LB notices death$HEALTH_READY=false then stop accept
In-flight requestsCut mid-responseFinish or abort under $DRAIN_TIMEOUT_MS
LB / kube viewSurprise 502/resetReady failed, then target gone on schedule
OrchestratorSIGKILL was the planExit before terminationGracePeriodSeconds
Rollback storyGuess which write committedLogs: ready false → drain → close → exit

The sequence, locked

SIGTERM
  |
  v
$HEALTH_READY=false   (probes/LB stop new work)
  |
  v
stop accept           (no new sockets / queue pulls)
  |
  v
drain in-flight  <= $DRAIN_TIMEOUT_MS
  |
  v
server.close          (Node / Fastify / Nest hooks)
  |
  v
exit before $SHUTDOWN_DEADLINE_S
  |                   (< terminationGracePeriodSeconds)
  v
orchestrator SIGKILL only if you failed to exit

Do these steps in this order. Skipping readiness while you still accept is how you keep receiving new work while you think you are draining. Skipping drain and jumping to close is how you reset in-flight clients. Skipping the hard deadline is how you hang until SIGKILL anyway.

1. Receive SIGTERM

Node surfaces process signals, including SIGTERM, on the process object. Register a handler once. On Kubernetes, pod termination sends SIGTERM to the container's main process, then SIGKILL after terminationGracePeriodSeconds if you are still alive. Your app must start the drain on SIGTERM. SIGKILL cannot be handled.

Confirm the Node process receives SIGTERM — a wrapper that swallows it means kube still SIGKILLs on the grace period.

2. Flip readiness: $HEALTH_READY=false

The first side effect of the handler is not server.close. It is failing the readiness check so the load balancer and kube endpoints stop sending new work.

Set $HEALTH_READY=false in the same process that serves /ready (or whatever path your probes hit). Liveness can stay true while you drain: you are not dead; you are refusing new traffic. If you fail liveness at the same time as readiness, kube may restart you during drain, which is another SIGTERM/SIGKILL cycle on a process that was trying to finish requests.

Load balancers and kube poll on an interval, so a few new requests can still land after you fail ready — that is why the next step is still stop accepting.

3. Stop accepting new connections / new work

After readiness is false, stop taking new connections and new jobs:

  • HTTP: no new request handlers enter the in-flight set.
  • Fastify: return503OnClosing returns 503 to new requests while close is in progress.
  • Queues / in-process workers: stop pulling; in-flight handlers finish; new messages wait for another replica.
  • WebSockets / long-lived streams: stop accepting upgrades; existing sockets count as in-flight.

Existing requests stay until they finish or $DRAIN_TIMEOUT_MS fires; new work must not enter the in-flight set.

4. Drain in-flight under $DRAIN_TIMEOUT_MS; hard deadline $SHUTDOWN_DEADLINE_S

In-flight means HTTP requests already dispatched whose responses are not finished, plus work those handlers spawned that must complete for a correct response.

$DRAIN_TIMEOUT_MS is the budget for that set. When it elapses, you stop waiting. Hung clients are why you have a timeout.

$SHUTDOWN_DEADLINE_S is the hard wall for the entire sequence: SIGTERM → fail ready → stop accept → drain → close → process.exit. It must be less than Kubernetes terminationGracePeriodSeconds with headroom. If grace is 30s and your deadline is 30s, kube SIGKILLs while you are still in close.

If drain finishes early, do not sit idle until the deadline. Close and exit. The deadline is a cap, not a sleep.

Fail health and drain in-flight requests instead of a hard kill mid-request

5. Close the server

After the in-flight set is empty or the drain timeout fired, close the HTTP server.

  • Node http.Server: server.close(callback) stops accepting new connections and waits for existing ones. That wait is not a substitute for $DRAIN_TIMEOUT_MS; race close against your deadline so a stuck keep-alive cannot pin the process until SIGKILL.
  • Express: sits on the Node server. You still own close and the deadline.
  • Fastify: close() plus return503OnClosing. forceCloseConnections for idle keep-alives avoids idle sockets blocking close — use it as documented after drain, not as "kill everyone immediately" unless that is intentional post-timeout.
  • Nest: enableShutdownHooks() and the application shutdown lifecycle so SIGTERM reaches Nest's close path.

An unclosed server with open keep-alives still holds the event loop until kube SIGKILLs.

6. Exit cleanly before kube SIGKILL

When close has returned (or the hard deadline hits), exit the process. Do not start new work in exit handlers. Kubernetes will SIGKILL when terminationGracePeriodSeconds elapses regardless. A clean exit before that is the difference between a drained replica and a replica that vanished mid-write.

Kubernetes is the orchestrator contract, not the whole post

Pod termination sends SIGTERM, then SIGKILL after terminationGracePeriodSeconds. Your drain must complete inside that window.

preStop can sleep so the load balancer has time to see the pod leaving before SIGTERM hits the app. That is complementary to $HEALTH_READY=false, not a replacement. If preStop sleeps 5s, that 5s is inside the grace period — subtract it from the time the app has to drain.

Rule: app $SHUTDOWN_DEADLINE_S < terminationGracePeriodSeconds, with headroom for preStop and the kubelet. Do the arithmetic in the same runbook as the env vars.

11 production screens. Login, database, payments — all wired.

The SaaS Dashboard Kit ships everything already connected. Nothing to set up. Live demo at saas.otf-kit.dev.

See the live demo

Load balancer draining is parallel, not instead

AWS ALB connection draining / target deregistration delay is the balancer's wait after it stops choosing the target. It does not run your Node handler. You still fail $HEALTH_READY, still drain in-process, still close. Align deregistration delay, grace period, and $SHUTDOWN_DEADLINE_S so the LB is not sending to a PID that no longer exists, and the PID is not exiting while the LB still thinks the target is live.

SIGTERM to ready-false to drain to exit before SIGKILL

What this is not

  • Rolling deploys: maxUnavailable, surge, and which replica gets SIGTERM. This post is the handler inside that replica.
  • Secrets rotation: overlapping credentials. Shutdown must not drop in-flight requests that still use the old credential during overlap.
  • Rate limiting: 429 at ingress. A 503 during close is "this replica is leaving," not "you are over quota."
  • Outbound timeouts to AI backends: client-side. Drain is inbound in-flight on this process.
  • Health check articles: how /live and /ready are defined. Here /ready becomes false because SIGTERM happened.

Implementation notes that stay honest

Handle SIGTERM once. Re-entrancy should not start a second drain; no-op or shorten the deadline, not double-close.

$HEALTH_READY must be the same flag the probe reads. A memory boolean the handler flips is enough if the ready handler reads it.

Log signal received, ready false, accept stopped, in-flight count, drain vs timeout, close, exit — so a page is not a guess about SIGKILL timing.

Do not kill -9 yourself to "be sure." If the hard deadline fires, exit. Let the orchestrator SIGKILL only if you failed to exit — that is a bug in the handler, not a feature.

Idle keep-alives can outlive in-flight work — drain is for work; $SHUTDOWN_DEADLINE_S is for the process. If a cluster master receives SIGTERM, stop forking and propagate the same sequence to workers.

Checklist

  1. SIGTERM handler registered on the process that owns the server.
  2. Handler sets $HEALTH_READY=false first.
  3. Stop new connections and new queue work.
  4. Wait for in-flight up to $DRAIN_TIMEOUT_MS.
  5. server.close (Node / Express), Fastify close + return503OnClosing, Nest shutdown hooks.
  6. Exit before $SHUTDOWN_DEADLINE_S, less than terminationGracePeriodSeconds with headroom for preStop.
  7. Align LB deregistration delay with that same budget.

If any step is missing, you do not have graceful shutdown — you have hope that the load balancer forgives a mid-request death.

Sources

architecturebackendagents
OTF SaaS Dashboard Kit

Ship the product, not the setup.

  • 11 production screens — auth, billing, team, analytics, settings
  • Real database, payments, and login — all wired on day 1
  • AI configs pre-tuned so your agent extends instead of regenerates
See the live demoView pricing

On this page

The sequence, locked1. Receive SIGTERM2. Flip readiness: $HEALTH_READY=false3. Stop accepting new connections / new work4. Drain in-flight under $DRAIN_TIMEOUT_MS; hard deadline $SHUTDOWN_DEADLINE_S5. Close the server6. Exit cleanly before kube SIGKILLKubernetes is the orchestrator contract, not the whole postLoad balancer draining is parallel, not insteadWhat this is notImplementation notes that stay honestChecklistSources