Rotate production secrets with dual-key overlap, not an edit to .env

Sandbox credentials are disposable. Production credentials on a backend you own are not. Once the process is serving real traffic, rotating a database password, a webhook signing secret, a provider API key, or a session signing key by editing a file and hoping the next deploy "picks it up" is how you get a half-migrated fleet: some replicas still sign with the old key, some reject the new one, and the only recovery is a second emergency change. The owned-backend pattern is narrower and stricter. Keep a versioned secret source, run an overlap window where the app can read both the current value and the previous one, prove the new value under live traffic, then revoke the old one on a kill path you already wrote down.

This is not the same problem as an OpenAI project key hitting its maximum lifetime. Provider-imposed expiry is a clock you do not control. Owned-backend rotation is a procedure you do control: who holds the secret, which process reads it, how long the old value stays valid, and how you cut it. It is also not an EAS env-profile problem and not a Lambda-only wiring note. Those surfaces inject values at build or function configuration time. Here the runtime that accepts webhooks and opens database connections has to change what it trusts without a gap.

What must rotate, and what is only build-time

Inventory before you touch a store. Four classes almost always belong in the rotation set on an owned API: the password inside $DATABASE_URL; $WEBHOOK_SIGNING_SECRET (and $WEBHOOK_SIGNING_SECRET_PREV during overlap); outbound provider keys such as $STRIPE_SECRET_KEY with $STRIPE_SECRET_KEY_PREV when the vendor still accepts the previous key; and session or token signing keys, which invalidate cookies unless you verify with both keys for one TTL.

Build-time only values — a public site key, a non-secret feature-flag project id, or a CI token used solely at image build — do not belong in the same runbook as $DATABASE_URL. Write the inventory as a table: name, store path, consumers, overlap supported, revoke owner, and the proof you will accept before revoke. If you cannot name the proof, you are not ready to rotate. Not every vendor offers dual-accept for provider keys; when they do not, the overlap lives on your side: deploy the new key, confirm outbound calls succeed, then revoke at the vendor.

A versioned store, not a file on a laptop

The secret source has to answer three questions the .env file cannot. What is the current version? What was the previous version, and until when is it allowed? Who can read it, and is that read audited?

AWS Secrets Manager is built around that model. Rotation is a Lambda you configure, with the rotation overview and the Lambda rotation steps describing create, set pending, test, then finish by moving stage labels. During the test phase the app can still see the prior version — overlap as version stages, not two forgotten shell exports.

Doppler treats configs as the unit you promote; the Doppler docs are the contract. It will not magically dual-read inside your process — keep the previous value under a second name until cutover. HashiCorp Vault's dynamic secrets lease credentials that expire (Vault secrets docs). Useful for databases; for signing keys and webhooks you still need two trusted verifiers for a TTL. Pick one system of record and a $SECRET_VERSION every replica can report. If two replicas disagree after a rolling restart, you are not done.

The application pattern: dual-read, then revoke

Rotation fails in the app, not in the console. Read $SECRET_VERSION at startup and on a controlled reload. Log the version identifier, never the secret. Pair that with a rolling deploy on an owned backend so every replica loads the same pair without an SSH export.

For a provider key that supports overlap, load $STRIPE_SECRET_KEY as current and $STRIPE_SECRET_KEY_PREV as fallback. Outbound calls use current. Bound the fallback with time or $SECRET_VERSION, then delete previous. For material you verify rather than present, dual-read is the window: verify current, then previous, accept only inside the overlap TTL. Do not sign new tokens with the previous key. If the database can hold two passwords, ship $DATABASE_URL at the new one, canary, then roll and drop the old. If it cannot, say so in the runbook.

1. Create the new version in the store. Do not mark it current until the app build that understands it is ready.
2. Deploy the build that reads current plus previous. Confirm every replica reports the new $SECRET_VERSION and error rates stay flat.
3. Prove the specific call: a signed webhook accepted, a query executed, a session refreshed.
4. Revoke previous: remove $STRIPE_SECRET_KEY_PREV, $WEBHOOK_SIGNING_SECRET_PREV, and the old database password. Kill path with an owner — not eventual cleanup.

After revoke, rollback means rotating forward to a third version, not "put the old .env back."

store: stage AWSCURRENT=N, AWSPREVIOUS=N-1
        |
        v
rolling restart: each replica loads N and N-1
        |
        v
dual-read: sign/send with N, verify with N then N-1
        |
        v
prove: all replicas log SECRET_VERSION=N, dependent route healthy
        |
        v
revoke N-1: drop PREV env, disable old version, vendor revoke

Webhook signing is its own overlap

The sender and the receiver do not restart together. The provider may retry a delivery signed with the old secret for minutes after you rotate. If your verifier only knows the new $WEBHOOK_SIGNING_SECRET, those retries look like attacks. If you keep accepting the old secret with no TTL, a leaked previous secret stays valid.

Dual-verify for one TTL: try $WEBHOOK_SIGNING_SECRET, then $WEBHOOK_SIGNING_SECRET_PREV, accept only inside the overlap (and the provider's replay window). Log a boolean, the matched version label (current or prev), and the event id — never the digest next to the body. When the TTL expires, remove $WEBHOOK_SIGNING_SECRET_PREV in the same change that revokes at the provider, after current-key matches dominate.

Never log the value

Rotation day is when secret values leak into logs: an env dump to "confirm" the deploy, a debug middleware that prints Authorization, an error formatter that includes $DATABASE_URL.

Log $SECRET_VERSION, the key name, the replica id, and success or failure — never the value or a prefix. Scrub before the line leaves the process. The same egress path you use for PII scrub before logs leave the box should treat secret-shaped strings as a deny. Keep the production rate-limit setup on the routes those keys protect so a stolen webhook or session key cannot be replayed without bound during the TTL.

What this playbook is not

An OpenAI project key with a maximum lifetime expires because the vendor said so — the failure mode here is a mixed fleet, not a 401 from an aged-out key. EAS env profiles inject mobile-build values; wrong layer for a database password the API holds. A Lambda rotation function talks to the store; without the dual-read deploy it is half a rotation.

A runbook you can execute half asleep

• Inventory: which of $DATABASE_URL, $STRIPE_SECRET_KEY, $WEBHOOK_SIGNING_SECRET, and the session signing key is in scope today.
• Store: version id, who can read it, where $SECRET_VERSION surfaces in logs.
• Overlap: $STRIPE_SECRET_KEY_PREV, $WEBHOOK_SIGNING_SECRET_PREV, and the TTL.
• Deploy: rolling restart, replica count, log query proving every replica on the new version.
• Prove: one authenticated success per secret class, no sustained error rise.
• Revoke: disable N-1, delete _PREV variables, vendor revoke, timestamp.

If revoke has no proof gate, you either drop traffic early or leave two valid passwords forever. Owned-repo kits are where this runbook should live next to the deploy and the log scrub. Copy the steps into the repo you actually ship. A secret you can rotate on purpose is a secret you can survive losing.

Sources

• https://docs.aws.amazon.com/secretsmanager/latest/userguide/rotating-secrets.html
• https://docs.aws.amazon.com/secretsmanager/latest/userguide/rotate-secrets_lambda.html
• https://docs.doppler.com/docs
• https://developer.hashicorp.com/vault/docs/secrets
• https://otf-kit.dev/blog/rate-limiting-owned-api-production
• https://otf-kit.dev/blog/rolling-deploys-owned-backend
• https://otf-kit.dev/blog/pii-scrub-log-egress-before-leave-box