Rolling deploys on an owned backend: health checks, drain, and rollback

A restart is not a deploy. If you own the process, the load balancer, and the artifact, a production ship is a cutover: new instances must prove they can serve, old instances must stop taking work without dropping it, and a bad binary must be reversible without a war room. Rolling deploys are that cutover, written down. Downtime theater is everything else — a green CI check, a process bounce, and a hope that keep-alive connections die quietly.

This is ops on a backend you can SSH to, or at least kubectl exec into. Sandboxes that hide the replica set do not give you these knobs. If you cannot name the health endpoint, the drain timeout, and the previous artifact, you do not have zero-downtime deploys. You have lucky restarts.

What a rolling deploy actually does

A rolling deploy replaces running instances in batches. One (or a few) new processes start, pass a readiness check, join the pool, and only then do old processes leave. Traffic should hit a healthy listener the entire time. That sentence hides three independent systems:

1. A readiness signal the load balancer or scheduler trusts, not a /health that returns 200 because the HTTP server booted.
2. A drain so in-flight requests and long-lived connections finish, or fail closed on purpose, before the old process exits.
3. A rollback path that puts the previous artifact back in the pool without rebuilding from a dirty working tree.

Kubernetes documents this as a Deployment rolling update: you control surge and unavailability, and Pods leave Service endpoints when readiness fails. See the Deployment rolling update and Pod lifecycle docs. Cloud load balancers document the same idea as connection draining or deregistration delay — AWS for Classic Load Balancers in connection draining, and for Application Load Balancer target groups in target group attributes (deregistration delay). Google Cloud documents connection draining the same way: stop new connections, wait, then drop the instance.

Health checks that mean ready, not merely alive

Three probes exist because they answer three different questions. Mixing them is how you get crash loops that look like “the platform is flaky.”

Startup (when you have it) covers slow boots: migrations, JIT, large caches. Without it, liveness fires during a long first start and you never join the pool.

Connection drain: stop new work, finish old work, then die

The sequence is:

1. Take the instance out of rotation (readiness fail, target deregister, nginx upstream removal).
2. Stop accepting new connections on that instance.
3. Let in-flight requests complete, or hit a stated timeout.
4. Close keep-alives so the client reconnects to a living peer.
5. Exit. Only then may the supervisor or kubelet SIGKILL.

Rollback is a deploy, not a meeting

Rollback is the same machinery in reverse: previous artifact, same health checks, same drain. It is not “git revert and hope CI is green.” You need:

• Pinned artifacts. Image digest or a tarball hash, not a latest tag, not “whatever was on main at 14:00.”
• The previous pin still in the registry. Garbage-collecting untagged images the day you ship is how rollback becomes a rebuild.
• A stop condition. Error rate, latency, saturation, or a synthetic check on the new instances — not a vibe after five minutes in Slack.
• No schema trap. If the new binary required a blocking migration that the old binary cannot read, rolling back the process is not enough. Expand/contract your schema so the old binary still runs, or accept that this release is not rollback-safe and treat it as a maintenance window. That is an honest choice. Lying that every ship is reversible is how you get a two-hour outage plus a forward-fix on a bad migration.

Do not wait for the full roll to finish before you look. Surge one instance (or maxUnavailable: 0 with a small maxSurge in Kubernetes terms — see rolling update parameters), watch that instance’s error rate and its readiness flapping, then continue. A canary that shares a log pile with the old version is useless unless every line carries a release id. Tie stack traces to the binary you rolled: crash triage that ships fixes for AI-built apps exist so a panic on the new replica is attributable to the new pin, not to “the backend.”

Feature flags are not a substitute for binary rollback. Flags can hide a path. They cannot un-break a native crash, a runaway memory limit, or a listen port that never binds. Keep both.

A sequence you can run on a box you own

Strip the platform names and the job is still the same. Write it as a checklist the on-call can execute without improvising.

Before traffic moves

• Artifact built and pinned. Record the digest next to the change ticket.
• Config and secrets for the new version already on the host or in the secret store. A roll that fetches config after it is in the pool will fail readiness in production only.
• Database expand-step applied, or confirmed unused. Contract-step waits until the old binary is gone.
• Drain timeout ≥ p99 in-flight + load balancer health interval + a small buffer. Measure, do not guess.
• Previous pin still pullable.

Cutover

• Start N new instances (N = surge). Wait for readiness, not for “process is up.”
• Confirm they receive a copy of synthetic traffic (or a slice of real traffic) and that error rate is not worse than the old pool.
• Fail readiness on one old instance. Wait drain. Confirm in-flight is zero or that remaining connections are the ones you chose to kill (WebSocket policy).
• Terminate that instance. Repeat until the old pool is empty.
• If error rate, latency, or readiness flaps on the new pin: stop the roll, drain the new instances, put the previous pin back through the same health/drain path.

After

• Leave the previous pin tagged until the next successful roll, not until midnight.
• Confirm mixed-version windows are closed. Two binaries serving the same session cookie is a class of bug you will only see if you log version on every request.
• If you own the repo and the runtime — the point of an owned project, not a rented workspace — put this checklist in the repo next to the Dockerfile or unit file. A deploy doc that lives only in a chat thread will not be there at 02:00.

What skipping a step looks like in production

You do not need a novel outage to diagnose a missing drain. The symptoms are repetitive:

• 502 / 504 spikes that last one health interval. The balancer still holds a dead target, or the process died before endpoints dropped.
• POST/PUT loss, GET looks fine. In-flight writes were killed; retries are not idempotent; clients do not retry.
• Sticky sessions that “logout” mid-deploy. The next request landed on a new instance that does not share the in-memory session you never stored.
• It heals after two minutes. That is not health. That is DNS, connection pools, and clients giving up. Users already saw the error.
• Rollback that rebuilds. You shipped latest. There is no previous artifact. You are compiling under load.

None of these require a special platform. They require a readiness URL that is honest, a SIGTERM handler that waits, and a pin you can put back.

Own the cutover or admit you bounce the process

Rolling deploys are not a product feature and they are not a vibe. They are three timeouts and one pin: probe interval, drain, grace period, previous digest. Set them from how the service actually behaves — boot time, p99, connection mix — and keep the old binary until the new one has served real traffic.

If you cannot fail readiness without killing the process, you do not have readiness. If SIGTERM immediately closes the listen socket, you do not have drain. If latest is the rollback plan, you do not have rollback. Fix those on the owned backend before you advertise zero downtime. The alternative is a restart with better marketing.

Sources

• Kubernetes, “Deployments” (rolling update, maxSurge / maxUnavailable): https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#rolling-update-deployment
• Kubernetes, “Pod lifecycle” (probes, termination, terminationGracePeriodSeconds): https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/
• Kubernetes, “Configure liveness, readiness and startup probes”: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/
• AWS, “Configure connection draining for your Classic Load Balancer”: https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/config-conn-drain.html
• AWS, “Target groups for your Application Load Balancers” (deregistration delay): https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-target-groups.html
• Google Cloud, “Enable connection draining”: https://cloud.google.com/load-balancing/docs/enabling-connection-draining
• Related: https://otf-kit.dev/blog/production-structured-logging-for-agents
• Related: https://otf-kit.dev/blog/sentry-crash-triage-ships-fixes
• Related: https://otf-kit.dev/blog/cursor-projects-owned-repo