Park failed jobs in a DLQ with reason, snapshot, and replay — not silent drop

On an owned backend, a failed background job belongs in a dead-letter queue with a failure reason, a payload snapshot, an attempt count, timestamps, the original queue identity, and a human or agent replay path — not a silent drop, and not an infinite retry loop that burns the worker on the same poison message.

This is not process shutdown. Graceful shutdown and drain is what you do on SIGTERM so in-flight work finishes or returns to the queue before the process exits. A dead-letter queue is what you do after $JOB_ATTEMPTS (or SQS maxReceiveCount via $MAX_RECEIVE_COUNT) is exhausted and the job is still wrong. Shutdown is a lifecycle event. Dead letter is a failure policy.

Keep the runbook in the owned-repo kit next to the workers so $DLQ_NAME, $MAX_RECEIVE_COUNT, $JOB_ATTEMPTS, and $RETRY_BACKOFF_MS show up in one env example with the same store you actually run.

	Silent drop	Infinite retry	DLQ with reason + snapshot + replay
After budget	Ack and delete	Keep looping	Park in `$DLQ_NAME` / failed set / `status=dead`
Payload	Gone	Same poison forever	Snapshot as received
Worker	Looks healthy	Slot burned each cycle	Free for healthy jobs
Operator	Support ticket, no job id	Same stack forever	Inspect → fix → redrive
Rollback story	Guess	Logs of the loop	Record: reason, attempts, queue, times

fail in worker
  |
  v
retry <= $JOB_ATTEMPTS
  (backoff $RETRY_BACKOFF_MS)
  |
  v
budget exhausted
  |
  v
park in $DLQ_NAME
  (reason + payload + attempts + times + source queue)
  |
  v
human / agent inspect
  |
  v
fix cause or patch payload
  |
  v
redrive / replay onto primary

Finite budget, then stop

Set $JOB_ATTEMPTS and $RETRY_BACKOFF_MS in the worker config, not as magic numbers inside each handler. The handler throws or returns a failure. The queue runtime counts attempts and waits.

BullMQ models this as job attempts plus backoff (retrying failing jobs). When the count is exhausted, the job lands in the failed set unless you asked for auto-removal — wiping failed jobs on a timer is a silent drop with extra steps. Stop retrying jobs ends the loop when a retry cannot succeed. Workers run the processor. Stalled jobs (lock expired mid-job) are a different failure mode; stall recovery is not a DLQ substitute.

Amazon SQS does not have a job-level attempts field on the message. A consumer receives, fails to delete, and the message becomes visible again after the visibility timeout. RedrivePolicy.maxReceiveCount is the budget — set it from $MAX_RECEIVE_COUNT. When receive count exceeds that value, SQS moves the message to $DLQ_NAME. The RedrivePolicy API is deadLetterTargetArn plus maxReceiveCount. Overview and setup live in the dead-letter queues chapter and configure a dead-letter queue.

A Postgres job table uses the same numbers as columns. attempts < $JOB_ATTEMPTS and next_run_at <= now() selects work. On failure you increment attempts, store the error text, and set next_run_at from $RETRY_BACKOFF_MS. When the next increment would pass the budget, set status=dead instead of scheduling another run. Do not delete the row.

Park exhausted jobs in a DLQ instead of silent drop or infinite retry

What the dead-letter record must hold

A name in a dashboard is not a record. Whoever replays the job needs five fields.

Failure reason. The error message and, when you have it, the error name or code. "Job failed" is not a reason. SQS does not attach your exception text when it redrives; persist the handler error yourself (tied by message id) or accept that the DLQ body is payload-only. BullMQ stores the failure on the job. A Postgres row should write last_error in the same transaction as the attempt increment.

Payload snapshot as received. Copy the body at consume time, not a later mutation. If the handler half-updated a struct and you snapshot that, replay sends the corrupted version. SQS already keeps the original body. BullMQ keeps job data — do not overwrite job.data with a "cleaned" object and expect the failed set to show what arrived. On Postgres, keep original JSON in payload and put any patch only at replay time.

Attempt count. $JOB_ATTEMPTS or $MAX_RECEIVE_COUNT next to how many times this message was tried. BullMQ exposes failed jobs through job getters. SQS exposes approximate receive count. A table stores attempts as an integer.

Timestamps. Enqueued, last attempt, and dead-letter time — enough to line a spike up with a deploy or outage.

Original queue identity. Queue, worker group, job name. A shared DLQ only works if each record says where to put it back.

11 production screens. Login, database, payments — all wired.

The SaaS Dashboard Kit ships everything already connected. Nothing to set up. Live demo at saas.otf-kit.dev.

See the live demo

Replay is a path, not a hope

Parking without replay is a slower drop. The path is inspect, fix the root cause or patch the payload, then redrive onto the primary.

For BullMQ that is the failed set via getters, then manual retry once the cause is fixed — deliberate, not another automatic loop. If you built an explicit DLQ queue named $DLQ_NAME, the replay path removes from that queue and adds to the source with attempts reset. Document which of the two you use.

SQS redrive is first-class. Dead-letter queue redrive and using a dead-letter queue redrive move messages back to the source or another destination. Redrive does not fix a bad payload. If the body is wrong, fix it before you send it, or the same maxReceiveCount will park it again.

Postgres replay is a reviewed UPDATE (status back to pending, attempts reset, next_run_at now, optional patched payload) or an INSERT with the dead row marked replayed. Pick one so you do not double-apply. Idempotency keys on the job still matter — replay is another delivery.

Lambda on SQS still obeys the queue redrive policy (SQS error handling). Partial batch failure keeps one bad record from failing the whole batch. Async invocation error destinations are a dead-letter for the invoke — not a substitute for a job table's status=dead.

Alert on depth. CloudWatch alarms for SQS dead-letter queues page when $DLQ_NAME has visible messages. BullMQ and Postgres need an equivalent gauge — failed-set size or count(*) where status=dead. An alarm on every retry is noise. An alarm when the dead letter grows is the page you want.

Fail, retry under budget, park with reason, inspect, then redrive

Same contract, three stores

Be honest about which system you run. Copying an SQS tutorial into a BullMQ worker produces a runbook nobody can execute.

SQS. Source queue has a redrive policy. maxReceiveCount comes from $MAX_RECEIVE_COUNT. The target is $DLQ_NAME. You do not delete the message to "give up"; you let the receive count trip the policy. Replay is SQS redrive, or a consumer that reads $DLQ_NAME and sends to the source after a payload fix. The handler error is yours to log.

BullMQ. attempts and backoff use $JOB_ATTEMPTS and $RETRY_BACKOFF_MS. Exhausted jobs sit in the failed set — treat that set as the dead letter unless you explicitly move failures into $DLQ_NAME. Auto-removal that deletes failed jobs defeats the policy.

Postgres job table. Columns for payload, attempts, last error, queue name, timestamps, and status. Status moves to dead when attempts hit $JOB_ATTEMPTS. Replay is a reviewed UPDATE or INSERT, not a cron that flips every dead row back to pending.

Idempotency stays on the handler. A replayed job will run again. The dead-letter record does not make the handler safe — it makes the failure visible.

What this is not

AI production background jobs puts model work on a queue with retries and idempotency. This post starts after that budget is spent — do not invent an infinite loop because the call is "just an LLM."

API timeouts and retries on AI backends is the outbound HTTP client budget, not $JOB_ATTEMPTS. Nested client retries inside one job attempt multiply load; set both budgets on purpose.

Webhook idempotency (Stripe webhook write-up) survives at-least-once delivery. A dead letter stops retrying work that cannot succeed until code or data changes. An idempotency key does not repair a payload that will never validate.

Rate limits, secret rotation, and rolling deploys can burst failures — they are not the parking policy. A client offline outbox is not $DLQ_NAME on the server.

Runbook next to the worker

Name $DLQ_NAME, $MAX_RECEIVE_COUNT, $JOB_ATTEMPTS, and $RETRY_BACKOFF_MS in one env example. State which store you use, the five fields on the record, the replay command or query, and that replay is manual until an agent is explicitly allowed to redrive. Alarm when dead-letter depth stays non-zero long enough to matter. Then a failed job is a record you can read — not a log line you will not find, and not a loop that eats the pool.

Sources

OTF SaaS Dashboard Kit

Ship the product, not the setup.

11 production screens — auth, billing, team, analytics, settings
Real database, payments, and login — all wired on day 1
AI configs pre-tuned so your agent extends instead of regenerates

See the live demo View pricing