Background jobs for AI features: queues, retries, and idempotency

A production AI feature should not make the browser wait for the model, the database, the email provider, and the payment system in one HTTP request. Return a job ID, put the slow work behind a queue, and make every retry safe to run again.

The key distinction is simple: a request tells the system what the user wants; a background worker does the work. That separation gives you room for retries, progress, cancellation, and recovery when a process disappears halfway through a run.

It also forces you to answer the uncomfortable question demos hide: what happens if the worker completed the side effect but the client never received the response?

Why a long AI request fails differently

A short request often has a clear boundary. The server either returns a response or the client sees an error. A generation workflow may call a model, store output, create an embedding, charge an account, and send an email. Those operations can cross several services and take longer than a browser or proxy is willing to wait.

Three failure cases are enough to justify a job system:

1. A duplicate charge. The payment request succeeds, but the connection drops before the worker sees the response. A retry without an idempotency key can create a second charge.
2. A duplicate email. The mail provider accepts the message just before the worker times out. The retry sends the same notification again.
3. A half-finished run. The model output is stored, but the embedding or audit record is not. A restarted worker needs to know where to resume.

These are not model-quality problems. They are distributed-systems problems attached to a slower workflow.

Start with a queue and a job record

The request handler should validate the user, create a job record, enqueue work, and return 202 Accepted. It should not call the model or a payment provider while the user is waiting.

type GenerateJob = {
  id: string
  ownerId: string
  prompt: string
}

export async function POST(request: Request) {
  const { prompt } = await request.json()
  const ownerId = await requireUserId(request)
  const id = crypto.randomUUID()

  await db.jobs.insert({
    id,
    ownerId,
    status: "queued",
    prompt,
  })

  await queue.enqueue("generate", { id, ownerId, prompt })

  return Response.json({ id, status: "queued" }, { status: 202 })
}

The client can poll a status endpoint or subscribe to progress updates. The worker owns the slow path:

while (true) {
  const message = await queue.receive("generate")
  if (!message) continue

  try {
    await runGeneration(message.body)
    await queue.delete(message)
  } catch (error) {
    await recordFailure(message.body.id, error)
    await queue.retryOrDeadLetter(message, error)
  }
}

The queue is not a guarantee that work runs only once. Amazon SQS documents visibility timeouts and at-least-once delivery: a message becomes visible again if it is not deleted before the timeout, and standard queues can deliver a message more than once. That is why the worker must be safe when it sees the same job again. See the SQS visibility timeout documentation.

Set the visibility timeout longer than the normal processing window, extend it for work that is still active, and use a dead-letter path for repeated failures. The exact queue product can vary. The requirement does not: assume duplicate delivery.

Give every side effect an idempotency key

An idempotency key identifies one intended operation across retries. The worker derives it from the durable job ID and the specific side effect:

const chargeKey = `job:${job.id}:charge`
const emailKey = `job:${job.id}:summary-email`

Do not create a new random key inside each retry. That would identify every retry as a new operation and defeat the protection.

Your own database needs a unique constraint or an equivalent atomic operation. A simplified helper looks like this:

async function once<T>(key: string, action: () => Promise<T>) {
  const claimed = await db.idempotency.insertIfMissing({ key })
  if (!claimed) return db.idempotency.readResult<T>(key)

  try {
    const result = await action()
    await db.idempotency.saveResult(key, result)
    return result
  } catch (error) {
    await db.idempotency.recordFailure(key, error)
    throw error
  }
}

The claim and result-writing details need a real transaction strategy, but the invariant is clear: only one execution owns a key, and later attempts can retrieve the recorded result.

For payment calls, use the provider’s own mechanism as well. Stripe’s API documentation says an idempotency key lets a client safely retry a request, and that Stripe saves the first request’s status code and body for subsequent requests with the same key. It also recommends high-entropy keys and says keys can be up to 255 characters. Read the Stripe idempotent requests documentation before choosing retention and key-reuse rules.

The key should describe the business operation, not the network attempt. job:abc:charge is useful. retry-3 is not.

Checkpoint before moving to the next step

A long workflow should be a sequence of durable transitions, not one large function that keeps all state in memory. Store the output of each completed step before starting the next one.

async function runGeneration(job: GenerateJob) {
  let state = await db.jobs.readState(job.id)

  if (state.step < 1) {
    const output = await model.generate(job.prompt)
    await db.jobs.checkpoint(job.id, {
      step: 1,
      output,
    })
    state = { ...state, step: 1, output }
  }

  if (state.step < 2) {
    await assertJobStillAllowed(job.id, job.ownerId)
    const embedding = await embedder.create(state.output)
    await db.documents.upsert({ jobId: job.id, embedding })
    await db.jobs.checkpoint(job.id, { step: 2 })
    state = { ...state, step: 2 }
  }

  if (state.step < 3) {
    await assertJobStillAllowed(job.id, job.ownerId)
    await once(`job:${job.id}:summary-email`, () =>
      mailer.sendSummary(job.ownerId, state.output)
    )
    await db.jobs.checkpoint(job.id, {
      step: 3,
      status: "completed",
    })
  }
}

If the process exits after step one, the next attempt reads the checkpoint and skips the model call. If it exits during step two, the document write must be an upsert or have its own idempotency key. Checkpointing reduces repeated expensive work; it does not remove the need to protect writes.

Keep state explicit enough that an operator can inspect it. A useful job record includes queued, running, completed, failed, and cancelled states, an attempt count, timestamps, the current step, and a safe error summary. Never store a raw prompt or model response in logs if it may contain private customer data.

Keep the outbox boundary honest

A database transaction cannot atomically include a remote email or payment API. This creates two dangerous orderings:

• the database commits, the remote call fails, and the system forgets to retry;
• the remote call succeeds, the database transaction fails, and the system repeats the call later.

An outbox gives the worker a durable handoff. In the same database transaction as the business state change, insert an outbox row describing the intended external action. A separate dispatcher claims outbox rows, calls the provider with a stable idempotency key, stores the result, and retries failures.

The outbox is not magic exactly-once delivery. It gives you a durable intent plus a place to record attempts. The remote operation still needs idempotency because the dispatcher can crash after the provider accepts a request but before the dispatcher records success.

Cancellation and authorization belong in the worker

A cancel button should update the job record. The worker should check that state at safe boundaries, usually before each expensive step and before each external side effect.

await assertNotCancelled(job.id)
await assertJobStillAllowed(job.id, job.ownerId)

Check authorization again in the worker. The user may have been deleted, downgraded, or removed from a project after the request was accepted. Request-time authorization answers whether the job may be created. Worker-time authorization answers whether it may still perform this side effect.

Cancellation cannot always interrupt a model call already in flight. Document that behavior in the UI: cancellation stops the next step, and the job status changes when the current operation reaches a safe boundary.

A production checklist for the first implementation

Before shipping a background AI workflow, verify each item:

• the request returns a job ID instead of waiting for the full workflow;
• the job record has an owner and a state machine;
• duplicate queue delivery is expected and tested;
• every external side effect has a stable business key;
• checkpointed steps can resume after a process exit;
• writes use unique constraints or upserts where appropriate;
• cancellation is checked between steps;
• authorization is checked in the worker;
• repeated failures reach a dead-letter or operator-visible state;
• logs include job ID, step, attempt, and duration without exposing customer content;
• the status endpoint does not reveal another user’s job;
• an operator can retry a failed job without manually editing production rows.

The production checklist for choosing a web app template is a useful companion because background work exposes the gaps around authentication, data ownership, deployment, and operations. If payment webhooks are part of the same feature, compare the idempotency pattern for repeated Stripe webhooks as well.

The durable pattern

Queues handle time. Checkpoints handle progress. Idempotency keys handle retries. Authorization and cancellation handle changing user intent. The outbox handles the boundary between your database and services you do not control.

You do not need to predict every failure. You need to make the common failures boring: a worker disappears, a message is delivered twice, a connection drops after a successful API call, or a user cancels halfway through. With durable state and repeatable side effects, the next worker can inspect what happened and continue without charging twice or sending the same message twice.

That is the production difference between an AI feature that merely returns text and one that can survive real users, retries, and partial failure. If you want a starting point with the surrounding application concerns already visible, browse the OTF production-ready kits and evaluate the boundaries you still need to implement before your first background job ships.

Sources

• AWS Builders’ Library: Making retries safe with idempotent APIs — stable request identifiers and semantically equivalent retry responses.
• Amazon SQS visibility timeout — visibility windows, retries, and at-least-once delivery.
• Stripe idempotent requests — provider-side retry protection and key behavior.