Why AI in Production Requires solid Background Jobs and Idempotency

The AWS Builders' Library entry on making retries safe with idempotent APIs is the single highest-use read for anyone shipping AI features past the demo. The pattern it codifies — every retried operation carries a stable key, the system stores the outcome, retries return the cached result — is the same idea that has quietly powered production payments, queues, and durable workflows for the better part of two decades. It is also the idea almost every AI demo skips.

Then it ships. Then it bills the customer twice.

The demo-to-production gap

A 300-millisecond HTTP request is easy to reason about: it ran or it didn't, and the client knows which. A 30-second AI workflow — generate, embed, store, email, write back to the DB — is not. It runs across three processes, four retries, and one database transaction. Each retry is an opportunity to double-bill, double-email, or leave a half-written row.

Three concrete failure modes show up the week after launch.

1. The duplicate charge. The model returns 200 OK but the network drops before the client reads the body. The client retries. The payment processor creates a second charge. The user disputes both.
2. The phantom email. "Send summary" times out at 28 seconds. The worker actually finished at 31 and sent the email. The client times out and retries. Two emails arrive.
3. The orphaned run. Generation A completes the model call, generation B completes the database write. They share a job id. Half-finished state lives forever.

None of this is exotic. It is just the cost of treating a long-running, retried, multi-step workflow like a single HTTP round-trip.

The three primitives production AI actually needs

You can build the whole thing on three ideas, and once you have them you can stack anything else on top.

• A queue. Decouple "the user pressed the button" from "the work happened." The request returns a job id immediately; the worker does the slow part.
• An idempotency key. Every external write carries a stable token. Retries with the same key return the same outcome.
• A checkpoint. Long work is broken into steps. Each step's result is stored before the next one runs. A crashed worker resumes from the last checkpoint.

Everything else — retries with backoff, cancellation, observability, authorization — is decoration on these three.

The queue: don't make the request do the work

The request handler's only job is to enqueue and return. It does not call the model. It does not touch the database. It does not call the payment processor.

// POST /api/generate
export async function POST(req: Request) {
  const { prompt, userId } = await req.json()
  const jobId = crypto.randomUUID()

  await queue.enqueue('generate', {
    jobId,
    userId,
    prompt,
    createdAt: Date.now(),
  })

  return Response.json({ jobId }, { status: 202 })
}

Status 202 Accepted. The client polls GET /api/generate/:jobId for status. The worker does the slow part:

// worker.ts — runs in a separate process, not in the request
while (true) {
  const job = await queue.dequeue('generate', { visibilityTimeout: 60_000 })
  if (!job) continue

  try {
    await runGeneration(job)
  } catch (err) {
    await queue.requeueOrDeadLetter(job, err)
  }
}

The visibility timeout is the lease. If the worker dies mid-run, the job reappears after 60 seconds and another worker picks it up. That is the entire retry story for the queue itself. Visibility timeouts have been the canonical queue primitive for years — SQS has shipped with them since it launched — and every other retry policy leans on them.

Idempotency keys: making retries safe

The queue's retry is only safe if the worker's side effects are idempotent. The pattern: every external write carries a key derived from the job id, and the side-effect target stores the result keyed by that id.

async function chargeForJob(jobId: string, amount: number, customerId: string) {
  const key = `stripe:charge:${jobId}`

  const existing = await db.insertIdempotent(key, async () => {
    return await stripe.charges.create({
      amount,
      customer: customerId,
      idempotency_key: key, // payment processor's own dedup
    })
  })

  return existing
}

The idempotency table is dead simple: a unique key column, a payload column, a created-at timestamp. The insert is INSERT ... ON CONFLICT DO NOTHING RETURNING *. If the row already exists, you read it back and return it. No second charge. No second email. No second row.

The same key flows through every step:

const emailKey = `email:summary:${jobId}`
await db.insertIdempotent(emailKey, () =>
  mailer.send({ to, template: 'summary', idempotencyKey: emailKey })
)

The rule is mechanical: any external side effect whose duplicate would be observable to a human must take a key.

Checkpoints: state for long work

A 30-second generation is not one operation. It is generate, embed, write summary, send email, write audit row. The worker should checkpoint after each step.

async function runGeneration(job: Job) {
  const ckpt = await db.checkpoints.get(job.jobId) ?? { step: 0 }

  // step 0: generate
  if (ckpt.step <= 0) {
    const output = await model.generate(job.prompt)
    await db.checkpoints.set(job.jobId, { step: 1, output })
    ckpt.output = output
    ckpt.step = 1
  }

  // step 1: embed + store
  if (ckpt.step <= 1) {
    const vec = await embedder.embed(ckpt.output)
    await vectors.upsert(job.jobId, vec)         // upsert, not insert
    await db.checkpoints.set(job.jobId, { step: 2 })
    ckpt.step = 2
  }

  // step 2: side effects with idempotency keys
  if (ckpt.step <= 2) {
    await chargeForJob(job.jobId, 100, job.userId)
    await sendSummaryEmail(job.jobId, job.userId, ckpt.output)
    await db.checkpoints.set(job.jobId, { step: 3, completedAt: Date.now() })
  }
}

If the worker crashes between step 1 and step 2, the next worker reads step: 1 and resumes. No re-generation. No double charge. No half-written row. The model call — the expensive part — ran exactly once.

Cancellation, auth, and the outbox

Three things still bite teams that already have the three primitives above.

Cancellation. Users hit "cancel" mid-run. The worker must check, on every checkpoint, whether the job was cancelled — and stop cleanly. A flag in the job record plus a check after each step is enough:

if (ckpt.step <= 1) {
  if (await isCancelled(job.jobId)) return  // exit cleanly
  // ... do work
}

Authorization at the worker, not just the request. The request checks "can this user submit this job?" The worker checks "is this job still owned by this user?" — because the user could have been deleted, downgraded, or banned between enqueue and execution. The worker must re-validate before every side effect. The request is a guest at this point; the worker is the gate.

The outbox. The worker should never write to the database and call an external system in the same statement. If the DB write commits and the external call fails, you have lost the side effect. If the external call succeeds and the DB write fails, you double-bill on retry. The pattern: write the intent into a local outbox table inside the same transaction as the checkpoint, and a separate dispatcher drains the outbox to the external system, with its own retry and idempotency key.

What this gets us

The same AI feature, the same model, the same prompt — but the difference between a demo that survives 100 users and a system that survives 100,000.

• A failed payment call retries forever without double-charging, because the key is the job id.
• A crashed worker resumes from the last checkpoint, so the model call runs once.
• A cancelled job exits at the next step boundary, not mid-generation.
• A user who downgraded mid-run is rejected at the worker before any side effect.

The pieces are not novel. Visibility timeouts, idempotency-keyed writes, and step-level checkpoints are the primitives most production queue systems expose in some form. The enable is treating them as one coherent pattern instead of three separate tickets.

Where OTF fits

The Stripe + auth + billing boundary is the one that bites every AI builder the first week. It is exactly the boundary the SaaS Dashboard kit ships already wired — auth, billing, DB, and the Stripe webhook flow are in place, with project conventions your coding agent can extend rather than regenerate.

Every kit ships with AI-tool configs — CLAUDE.md, .cursorrules, and twenty-plus tested prompts under ai/prompts/. When you ask Claude Code to add a background generation feature, it has the conventions, the Stripe shape, and the existing endpoint styles to extend — not a green-field clean slate to invent from.

The model underneath churns. The infra pattern does not. A payment processor expects an idempotency key on charges today; it will tomorrow. A queue with a visibility timeout is a primitive that outlasts any specific vendor. That is the part worth buying once: the conventions, the AI-tool configs, and the already-wired boundary — so the worker you bolt on inherits the same shape as the code you started with.