# Park failed jobs in a DLQ with reason, snapshot, and replay — not silent drop

> Exhausted retries belong in a dead-letter queue with reason, payload snapshot, and a human/agent replay path — not silent drop or infinite loops.
> By Dave · 2026-09-23
> Source: https://otf-kit.dev/blog/dead-letter-queue-background-jobs

On an owned backend, a failed background job belongs in a dead-letter queue with a failure reason, a payload snapshot, an attempt count, timestamps, the original queue identity, and a human or agent replay path — not a silent drop, and not an infinite retry loop that burns the worker on the same poison message.

This is not process shutdown. [Graceful shutdown and drain](https://otf-kit.dev/blog/graceful-shutdown-drain-owned-backend) is what you do on SIGTERM so in-flight work finishes or returns to the queue before the process exits. A dead-letter queue is what you do after `$JOB_ATTEMPTS` (or SQS `maxReceiveCount` via `$MAX_RECEIVE_COUNT`) is exhausted and the job is still wrong. Shutdown is a lifecycle event. Dead letter is a failure policy.

Keep the runbook in the owned-repo kit next to the workers so `$DLQ_NAME`, `$MAX_RECEIVE_COUNT`, `$JOB_ATTEMPTS`, and `$RETRY_BACKOFF_MS` show up in one env example with the same store you actually run.

| | Silent drop | Infinite retry | DLQ with reason + snapshot + replay |
| --- | --- | --- | --- |
| After budget | Ack and delete | Keep looping | Park in `$DLQ_NAME` / failed set / `status=dead` |
| Payload | Gone | Same poison forever | Snapshot as received |
| Worker | Looks healthy | Slot burned each cycle | Free for healthy jobs |
| Operator | Support ticket, no job id | Same stack forever | Inspect → fix → redrive |
| Rollback story | Guess | Logs of the loop | Record: reason, attempts, queue, times |

```
fail in worker
  |
  v
retry <= $JOB_ATTEMPTS
  (backoff $RETRY_BACKOFF_MS)
  |
  v
budget exhausted
  |
  v
park in $DLQ_NAME
  (reason + payload + attempts + times + source queue)
  |
  v
human / agent inspect
  |
  v
fix cause or patch payload
  |
  v
redrive / replay onto primary
```

## Finite budget, then stop

Set `$JOB_ATTEMPTS` and `$RETRY_BACKOFF_MS` in the worker config, not as magic numbers inside each handler. The handler throws or returns a failure. The queue runtime counts attempts and waits.

BullMQ models this as job attempts plus backoff ([retrying failing jobs](https://docs.bullmq.io/guide/retrying-failing-jobs)). When the count is exhausted, the job lands in the failed set unless you asked for [auto-removal](https://docs.bullmq.io/guide/queues/auto-removal-of-jobs) — wiping failed jobs on a timer is a silent drop with extra steps. [Stop retrying jobs](https://docs.bullmq.io/patterns/stop-retrying-jobs) ends the loop when a retry cannot succeed. [Workers](https://docs.bullmq.io/guide/workers) run the processor. [Stalled jobs](https://docs.bullmq.io/guide/jobs/stalled) (lock expired mid-job) are a different failure mode; stall recovery is not a DLQ substitute.

Amazon SQS does not have a job-level `attempts` field on the message. A consumer receives, fails to delete, and the message becomes visible again after the visibility timeout. `RedrivePolicy.maxReceiveCount` is the budget — set it from `$MAX_RECEIVE_COUNT`. When receive count exceeds that value, SQS moves the message to `$DLQ_NAME`. The [RedrivePolicy API](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_RedrivePolicy.html) is `deadLetterTargetArn` plus `maxReceiveCount`. Overview and setup live in the [dead-letter queues](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html) chapter and [configure a dead-letter queue](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-configure-dead-letter-queue.html).

A Postgres job table uses the same numbers as columns. `attempts < $JOB_ATTEMPTS` and `next_run_at <= now()` selects work. On failure you increment attempts, store the error text, and set `next_run_at` from `$RETRY_BACKOFF_MS`. When the next increment would pass the budget, set `status=dead` instead of scheduling another run. Do not delete the row.

![Park exhausted jobs in a DLQ instead of silent drop or infinite retry](https://cdn.otf-kit.dev/blog/dead-letter-queue-background-jobs/inbody1-20260923e.png)

## What the dead-letter record must hold

A name in a dashboard is not a record. Whoever replays the job needs five fields.

**Failure reason.** The error message and, when you have it, the error name or code. "Job failed" is not a reason. SQS does not attach your exception text when it redrives; persist the handler error yourself (tied by message id) or accept that the DLQ body is payload-only. BullMQ stores the failure on the job. A Postgres row should write `last_error` in the same transaction as the attempt increment.

**Payload snapshot as received.** Copy the body at consume time, not a later mutation. If the handler half-updated a struct and you snapshot that, replay sends the corrupted version. SQS already keeps the original body. BullMQ keeps job data — do not overwrite `job.data` with a "cleaned" object and expect the failed set to show what arrived. On Postgres, keep original JSON in `payload` and put any patch only at replay time.

**Attempt count.** `$JOB_ATTEMPTS` or `$MAX_RECEIVE_COUNT` next to how many times this message was tried. BullMQ exposes failed jobs through [job getters](https://docs.bullmq.io/guide/jobs/getters). SQS exposes approximate receive count. A table stores `attempts` as an integer.

**Timestamps.** Enqueued, last attempt, and dead-letter time — enough to line a spike up with a deploy or outage.

**Original queue identity.** Queue, worker group, job name. A shared DLQ only works if each record says where to put it back.

## Replay is a path, not a hope

Parking without replay is a slower drop. The path is inspect, fix the root cause or patch the payload, then redrive onto the primary.

For BullMQ that is the failed set via getters, then [manual retry](https://docs.bullmq.io/patterns/manual-retrying) once the cause is fixed — deliberate, not another automatic loop. If you built an explicit DLQ queue named `$DLQ_NAME`, the replay path removes from that queue and adds to the source with attempts reset. Document which of the two you use.

SQS redrive is first-class. [Dead-letter queue redrive](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues-redrive.html) and [using a dead-letter queue redrive](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-using-dlq-redrive.html) move messages back to the source or another destination. Redrive does not fix a bad payload. If the body is wrong, fix it before you send it, or the same `maxReceiveCount` will park it again.

Postgres replay is a reviewed `UPDATE` (status back to pending, attempts reset, `next_run_at` now, optional patched payload) or an `INSERT` with the dead row marked `replayed`. Pick one so you do not double-apply. Idempotency keys on the job still matter — replay is another delivery.

Lambda on SQS still obeys the queue redrive policy ([SQS error handling](https://docs.aws.amazon.com/lambda/latest/dg/services-sqs-errorhandling.html)). Partial batch failure keeps one bad record from failing the whole batch. [Async invocation error destinations](https://docs.aws.amazon.com/lambda/latest/dg/invocation-async-errors.html) are a dead-letter for the invoke — not a substitute for a job table's `status=dead`.

Alert on depth. [CloudWatch alarms for SQS dead-letter queues](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues-cloudwatch-alarms.html) page when `$DLQ_NAME` has visible messages. BullMQ and Postgres need an equivalent gauge — failed-set size or `count(*) where status=dead`. An alarm on every retry is noise. An alarm when the dead letter grows is the page you want.

![Fail, retry under budget, park with reason, inspect, then redrive](https://cdn.otf-kit.dev/blog/dead-letter-queue-background-jobs/inbody2-20260923e.png)

## Same contract, three stores

Be honest about which system you run. Copying an SQS tutorial into a BullMQ worker produces a runbook nobody can execute.

**SQS.** Source queue has a redrive policy. `maxReceiveCount` comes from `$MAX_RECEIVE_COUNT`. The target is `$DLQ_NAME`. You do not delete the message to "give up"; you let the receive count trip the policy. Replay is SQS redrive, or a consumer that reads `$DLQ_NAME` and sends to the source after a payload fix. The handler error is yours to log.

**BullMQ.** `attempts` and `backoff` use `$JOB_ATTEMPTS` and `$RETRY_BACKOFF_MS`. Exhausted jobs sit in the failed set — treat that set as the dead letter unless you explicitly move failures into `$DLQ_NAME`. Auto-removal that deletes failed jobs defeats the policy.

**Postgres job table.** Columns for payload, attempts, last error, queue name, timestamps, and status. Status moves to `dead` when attempts hit `$JOB_ATTEMPTS`. Replay is a reviewed `UPDATE` or `INSERT`, not a cron that flips every dead row back to pending.

Idempotency stays on the handler. A replayed job will run again. The dead-letter record does not make the handler safe — it makes the failure visible.

## What this is not

[AI production background jobs](https://otf-kit.dev/blog/ai-production-background-jobs) puts model work on a queue with retries and idempotency. This post starts after that budget is spent — do not invent an infinite loop because the call is "just an LLM."

[API timeouts and retries on AI backends](https://otf-kit.dev/blog/api-timeouts-retries-ai-backends) is the outbound HTTP client budget, not `$JOB_ATTEMPTS`. Nested client retries inside one job attempt multiply load; set both budgets on purpose.

Webhook idempotency ([Stripe webhook write-up](https://otf-kit.dev/blog/stripe-webhook-idempotency-saved-us)) survives at-least-once delivery. A dead letter stops retrying work that cannot succeed until code or data changes. An idempotency key does not repair a payload that will never validate.

Rate limits, secret rotation, and rolling deploys can burst failures — they are not the parking policy. A client offline outbox is not `$DLQ_NAME` on the server.

## Runbook next to the worker

Name `$DLQ_NAME`, `$MAX_RECEIVE_COUNT`, `$JOB_ATTEMPTS`, and `$RETRY_BACKOFF_MS` in one env example. State which store you use, the five fields on the record, the replay command or query, and that replay is manual until an agent is explicitly allowed to redrive. Alarm when dead-letter depth stays non-zero long enough to matter. Then a failed job is a record you can read — not a log line you will not find, and not a loop that eats the pool.

## Sources

- https://docs.bullmq.io/guide/retrying-failing-jobs
- https://docs.bullmq.io/patterns/stop-retrying-jobs
- https://docs.bullmq.io/patterns/manual-retrying
- https://docs.bullmq.io/guide/jobs/getters
- https://docs.bullmq.io/guide/queues/auto-removal-of-jobs
- https://docs.bullmq.io/guide/workers
- https://docs.bullmq.io/guide/jobs/stalled
- https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html
- https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-configure-dead-letter-queue.html
- https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues-redrive.html
- https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-using-dlq-redrive.html
- https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_RedrivePolicy.html
- https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues-cloudwatch-alarms.html
- https://docs.aws.amazon.com/lambda/latest/dg/services-sqs-errorhandling.html
- https://docs.aws.amazon.com/lambda/latest/dg/invocation-async-errors.html