# Fail health, drain in-flight requests, then exit — not kill -9 mid-request

> On an owned backend, handle SIGTERM by failing health, draining in-flight work under a deadline, then exiting — not kill -9 mid-request.
> By Dave · 2026-09-23
> Source: https://otf-kit.dev/blog/graceful-shutdown-drain-owned-backend

On an owned production API, SIGTERM is not an invitation to `kill -9` the process while clients still have open requests. After you leave the sandbox and you own the Node process behind a load balancer or Kubernetes, the contract is: receive SIGTERM, fail readiness so new traffic stops, refuse new work, drain in-flight requests under a hard deadline, close the server, and exit before the orchestrator SIGKILLs you.

This is process-local shutdown on one replica. It is not a rollout strategy. Rolling deploys decide which pods leave the set and in what order; see [rolling deploys on an owned backend](https://otf-kit.dev/blog/rolling-deploys-owned-backend). This post is what one replica does when that SIGTERM arrives. It also is not secrets overlap ([secrets rotation](https://otf-kit.dev/blog/secrets-rotation-owned-backend)), not ingress rate limits ([rate limiting](https://otf-kit.dev/blog/rate-limiting-owned-api-production)), and not outbound client timeouts. Probe definitions are not the drain sequence either.

Keep the runbook in the owned-repo kit next to deploy and secrets rotation so the same env names show up in every replica: `$HEALTH_READY`, `$DRAIN_TIMEOUT_MS`, `$SHUTDOWN_DEADLINE_S`.

| | kill -9 mid-request | Fail health, drain, then exit |
| --- | --- | --- |
| New traffic | Still may land until LB notices death | `$HEALTH_READY=false` then stop accept |
| In-flight requests | Cut mid-response | Finish or abort under `$DRAIN_TIMEOUT_MS` |
| LB / kube view | Surprise 502/reset | Ready failed, then target gone on schedule |
| Orchestrator | SIGKILL was the plan | Exit before `terminationGracePeriodSeconds` |
| Rollback story | Guess which write committed | Logs: ready false → drain → close → exit |

## The sequence, locked

```
SIGTERM
  |
  v
$HEALTH_READY=false   (probes/LB stop new work)
  |
  v
stop accept           (no new sockets / queue pulls)
  |
  v
drain in-flight  <= $DRAIN_TIMEOUT_MS
  |
  v
server.close          (Node / Fastify / Nest hooks)
  |
  v
exit before $SHUTDOWN_DEADLINE_S
  |                   (< terminationGracePeriodSeconds)
  v
orchestrator SIGKILL only if you failed to exit
```

Do these steps in this order. Skipping readiness while you still accept is how you keep receiving new work while you think you are draining. Skipping drain and jumping to `close` is how you reset in-flight clients. Skipping the hard deadline is how you hang until SIGKILL anyway.

### 1. Receive SIGTERM

Node surfaces process signals, including SIGTERM, on the process object. Register a handler once. On Kubernetes, pod termination sends SIGTERM to the container's main process, then SIGKILL after `terminationGracePeriodSeconds` if you are still alive. Your app must start the drain on SIGTERM. SIGKILL cannot be handled.

Confirm the Node process receives SIGTERM — a wrapper that swallows it means kube still SIGKILLs on the grace period.

### 2. Flip readiness: `$HEALTH_READY=false`

The first side effect of the handler is not `server.close`. It is failing the readiness check so the load balancer and kube endpoints stop sending new work.

Set `$HEALTH_READY=false` in the same process that serves `/ready` (or whatever path your probes hit). Liveness can stay true while you drain: you are not dead; you are refusing new traffic. If you fail liveness at the same time as readiness, kube may restart you during drain, which is another SIGTERM/SIGKILL cycle on a process that was trying to finish requests.

Load balancers and kube poll on an interval, so a few new requests can still land after you fail ready — that is why the next step is still stop accepting.

### 3. Stop accepting new connections / new work

After readiness is false, stop taking new connections and new jobs:

- HTTP: no new request handlers enter the in-flight set.
- Fastify: `return503OnClosing` returns 503 to new requests while close is in progress.
- Queues / in-process workers: stop pulling; in-flight handlers finish; new messages wait for another replica.
- WebSockets / long-lived streams: stop accepting upgrades; existing sockets count as in-flight.

Existing requests stay until they finish or `$DRAIN_TIMEOUT_MS` fires; new work must not enter the in-flight set.

### 4. Drain in-flight under `$DRAIN_TIMEOUT_MS`; hard deadline `$SHUTDOWN_DEADLINE_S`

In-flight means HTTP requests already dispatched whose responses are not finished, plus work those handlers spawned that must complete for a correct response.

`$DRAIN_TIMEOUT_MS` is the budget for that set. When it elapses, you stop waiting. Hung clients are why you have a timeout.

`$SHUTDOWN_DEADLINE_S` is the hard wall for the entire sequence: SIGTERM → fail ready → stop accept → drain → `close` → `process.exit`. It must be less than Kubernetes `terminationGracePeriodSeconds` with headroom. If grace is 30s and your deadline is 30s, kube SIGKILLs while you are still in `close`.

If drain finishes early, do not sit idle until the deadline. Close and exit. The deadline is a cap, not a sleep.

![Fail health and drain in-flight requests instead of a hard kill mid-request](https://cdn.otf-kit.dev/blog/graceful-shutdown-drain-owned-backend/inbody1-20260923d.png)

### 5. Close the server

After the in-flight set is empty or the drain timeout fired, close the HTTP server.

- **Node `http.Server`:** `server.close(callback)` stops accepting new connections and waits for existing ones. That wait is not a substitute for `$DRAIN_TIMEOUT_MS`; race `close` against your deadline so a stuck keep-alive cannot pin the process until SIGKILL.
- **Express:** sits on the Node server. You still own `close` and the deadline.
- **Fastify:** `close()` plus `return503OnClosing`. `forceCloseConnections` for idle keep-alives avoids idle sockets blocking close — use it as documented after drain, not as "kill everyone immediately" unless that is intentional post-timeout.
- **Nest:** `enableShutdownHooks()` and the application shutdown lifecycle so SIGTERM reaches Nest's close path.

An unclosed server with open keep-alives still holds the event loop until kube SIGKILLs.

### 6. Exit cleanly before kube SIGKILL

When `close` has returned (or the hard deadline hits), exit the process. Do not start new work in `exit` handlers. Kubernetes will SIGKILL when `terminationGracePeriodSeconds` elapses regardless. A clean exit before that is the difference between a drained replica and a replica that vanished mid-write.

## Kubernetes is the orchestrator contract, not the whole post

Pod termination sends SIGTERM, then SIGKILL after `terminationGracePeriodSeconds`. Your drain must complete inside that window.

`preStop` can sleep so the load balancer has time to see the pod leaving before SIGTERM hits the app. That is complementary to `$HEALTH_READY=false`, not a replacement. If `preStop` sleeps 5s, that 5s is inside the grace period — subtract it from the time the app has to drain.

Rule: app `$SHUTDOWN_DEADLINE_S` < `terminationGracePeriodSeconds`, with headroom for `preStop` and the kubelet. Do the arithmetic in the same runbook as the env vars.

## Load balancer draining is parallel, not instead

AWS ALB connection draining / target deregistration delay is the balancer's wait after it stops choosing the target. It does not run your Node handler. You still fail `$HEALTH_READY`, still drain in-process, still `close`. Align deregistration delay, grace period, and `$SHUTDOWN_DEADLINE_S` so the LB is not sending to a PID that no longer exists, and the PID is not exiting while the LB still thinks the target is live.

![SIGTERM to ready-false to drain to exit before SIGKILL](https://cdn.otf-kit.dev/blog/graceful-shutdown-drain-owned-backend/inbody2-20260923d.png)

## What this is not

- **[Rolling deploys](https://otf-kit.dev/blog/rolling-deploys-owned-backend):** maxUnavailable, surge, and which replica gets SIGTERM. This post is the handler inside that replica.
- **[Secrets rotation](https://otf-kit.dev/blog/secrets-rotation-owned-backend):** overlapping credentials. Shutdown must not drop in-flight requests that still use the old credential during overlap.
- **[Rate limiting](https://otf-kit.dev/blog/rate-limiting-owned-api-production):** 429 at ingress. A 503 during close is "this replica is leaving," not "you are over quota."
- **[Outbound timeouts to AI backends](https://otf-kit.dev/blog/api-timeouts-retries-ai-backends):** client-side. Drain is inbound in-flight on this process.
- **Health check articles:** how `/live` and `/ready` are defined. Here `/ready` becomes false because SIGTERM happened.

## Implementation notes that stay honest

Handle SIGTERM once. Re-entrancy should not start a second drain; no-op or shorten the deadline, not double-close.

`$HEALTH_READY` must be the same flag the probe reads. A memory boolean the handler flips is enough if the ready handler reads it.

Log signal received, ready false, accept stopped, in-flight count, drain vs timeout, `close`, exit — so a page is not a guess about SIGKILL timing.

Do not `kill -9` yourself to "be sure." If the hard deadline fires, exit. Let the orchestrator SIGKILL only if you failed to exit — that is a bug in the handler, not a feature.

Idle keep-alives can outlive in-flight work — drain is for work; `$SHUTDOWN_DEADLINE_S` is for the process. If a cluster master receives SIGTERM, stop forking and propagate the same sequence to workers.

## Checklist

1. SIGTERM handler registered on the process that owns the server.
2. Handler sets `$HEALTH_READY=false` first.
3. Stop new connections and new queue work.
4. Wait for in-flight up to `$DRAIN_TIMEOUT_MS`.
5. `server.close` (Node / Express), Fastify `close` + `return503OnClosing`, Nest shutdown hooks.
6. Exit before `$SHUTDOWN_DEADLINE_S`, less than `terminationGracePeriodSeconds` with headroom for `preStop`.
7. Align LB deregistration delay with that same budget.

If any step is missing, you do not have graceful shutdown — you have hope that the load balancer forgives a mid-request death.

## Sources

- Node.js signal events: https://nodejs.org/api/process.html#signal-events
- Node.js `http.Server.close`: https://nodejs.org/api/http.html#serverclosecallback
- Fastify `close`, `return503OnClosing`, `forceCloseConnections`: https://fastify.dev/docs/latest/Reference/Server/#close
- NestJS application shutdown / lifecycle: https://docs.nestjs.com/fundamentals/lifecycle-events#application-shutdown
- Express performance best practices: https://expressjs.com/en/advanced/best-practice-performance.html
- Kubernetes pod termination: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-termination
- Container lifecycle hooks (`preStop`): https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/
- Attach handlers to lifecycle events: https://kubernetes.io/docs/tasks/configure-pod-container/attach-handler-lifecycle-event/
- AWS ALB connection draining: https://docs.aws.amazon.com/elasticloadbalancing/latest/application/connection-draining.html
- AWS ALB target deregistration delay: https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-target-groups.html#deregistration-delay