# API timeouts and retries when your app calls AI backends you own

> Own connect and total timeouts, bounded retries with jitter, and idempotency keys when your service calls AI HTTP APIs after leaving sandbox hosting.
> By Dave · 2026-09-20
> Source: https://otf-kit.dev/blog/api-timeouts-retries-ai-backends

When your owned app calls a model or AI HTTP API, the client timeout budget and retry policy live in *your* code — not in a sandbox dashboard. Sandbox builders (Lovable, Bolt, and similar) often hide connect/read latency behind a platform proxy; once the same feature ships from your repo, every hung `/v1/chat` call can pin a worker, cascade into user-facing stalls, or silently burn tokens on duplicate POSTs. The production answer is explicit: set connect and total timeouts, classify which failures are retryable, back off with jitter, and attach an idempotency key when you retry a POST.

This post is about **outbound** HTTP from your service to an AI backend you configure with `$AI_API_BASE`. It is not a job-queue tutorial ([background jobs for AI features](/blog/ai-production-background-jobs) already cover queues, worker retries, and job idempotency). It is also not Expo push receipt retries. Pair this client contract with [structured production logs](/blog/production-structured-logging-for-agents) so agents can triage which attempt failed, and keep product claims [citation-ready](/blog/ai-citation-ready-product-docs) when you document the policy for your team.

## What sandbox latency hides

In a sandbox, a “call the model” button often feels bounded because the host terminates, retries, or times out behind a UI toast you never configured. After export, your process opens TCP/TLS to `$AI_API_BASE`, waits for headers, streams tokens, and holds a worker slot until something ends.

Skip client timeouts and a slow upstream holds connections until the reverse proxy limit kills the user request while your handler still waits. Concurrent users amplify the pile-up. Own a timeout budget that fails closed before the edge does — plus bounded retries that do not turn one blip into a stampede.

![Sandbox opaque latency vs owned client timeout budget](https://cdn.otf-kit.dev/blog/api-timeouts-retries-ai-backends/inbody1-20260920a.png)

## Set a timeout budget before you retry

A timeout budget answers three questions for every outbound AI call:

1. **Connect** — how long to establish TCP/TLS to `$AI_API_BASE` (usually a few seconds).
2. **First byte / headers** — how long to wait for an HTTP status after the request is sent.
3. **Total / overall** — hard ceiling for the whole attempt, including stream read.

Vendor SDKs often ship generous defaults (minutes) because generation can be slow — fine for batch, dangerous for a synchronous user request. Split budgets by path: interactive chat ~15–45s; short classify/embed ~5–15s; long document work prefers a queue, or a dedicated 60–300s path if you must stay sync.

OpenAI documents `APITimeoutError` as “request took too long” and recommends a brief wait before retry ([Error codes](https://developers.openai.com/api/docs/guides/error-codes)). Treat that as a signal to *own* the ceiling, not wait forever. Log `attempt`, `timeout_ms`, `status`/`timeout`, and a correlation ID so [agent-readable logs](/blog/production-structured-logging-for-agents) show connect vs read hangs.

```ts
// ai-http.ts — owned outbound client (path-only; base from env)
const AI_API_BASE = process.env.AI_API_BASE!; // e.g. your proxy origin — no hardcoded host in source
const CONNECT_MS = 5_000;
const TOTAL_MS = 30_000;

export async function postChat(
  body: unknown,
  opts: { idempotencyKey?: string; signal?: AbortSignal } = {},
): Promise<Response> {
  const ctrl = new AbortController();
  const timer = setTimeout(() => ctrl.abort("total-timeout"), TOTAL_MS);
  const onOuter = () => ctrl.abort("outer-abort");
  opts.signal?.addEventListener("abort", onOuter, { once: true });

  try {
    return await fetch(`${AI_API_BASE}/v1/chat`, {
      method: "POST",
      headers: {
        "content-type": "application/json",
        authorization: `Bearer ${process.env.AI_API_KEY}`,
        ...(opts.idempotencyKey
          ? { "Idempotency-Key": opts.idempotencyKey }
          : {}),
      },
      body: JSON.stringify(body),
      signal: ctrl.signal,
      // If your runtime supports connect timeouts separately, set CONNECT_MS there.
    });
  } finally {
    clearTimeout(timer);
    opts.signal?.removeEventListener("abort", onOuter);
  }
}
```

Prefer path-only routes (`/v1/chat`, `/v1/embeddings`) in application code and inject the origin via `$AI_API_BASE`. That keeps reviews honest and avoids baking a vendor hostname into every handler.

## Classify failures, then bound retries with jitter

Not every error is retryable. Auth failures, validation errors, and spend/quota denials will not heal if you hammer them. Transient network failures, timeouts, `408`, many `429`s, and `5xx` often will — if you wait and limit attempts.

AWS SDK retry guidance is a clean general model: classify transient vs throttling vs non-retryable, use exponential backoff with **full jitter**, and cap max attempts ([Retry behavior](https://docs.aws.amazon.com/general/latest/gr/api-retries.html)). Full jitter spreads retries so clients do not stampede. Honor `Retry-After` when present — OpenAI’s guide says the same for `429` and overloaded `503` ([Error codes](https://developers.openai.com/api/docs/guides/error-codes)).

Owned interactive policy:

1. **Max attempts = 3** (1 initial + 2 retries) unless a batch path says otherwise.
2. **Retry** timeout/abort, connection reset, `408`, `429`, and `500`/`502`/`503`/`504`.
3. **Do not retry** `401`/`403`/`400`, or billing/spend codes that need config changes.
4. **Backoff** = `random(0, min(cap, base * 2^attempt))` — base ~200–500ms transient; higher for throttling.
5. **Stop** when the user-facing budget is exhausted even if attempts remain.

```ts
// retry.ts — bounded retries with full jitter (interactive path)
function sleep(ms: number) {
  return new Promise((r) => setTimeout(r, ms));
}

function fullJitterDelay(attempt: number, baseMs: number, capMs: number) {
  const exp = Math.min(capMs, baseMs * 2 ** attempt);
  return Math.floor(Math.random() * exp);
}

function shouldRetry(status: number | null, err: unknown): boolean {
  if (err) return true; // abort/timeout/network — classified upstream
  if (status == null) return true;
  if (status === 408 || status === 429) return true;
  if (status >= 500 && status <= 504) return true;
  return false;
}

export async function withRetries(
  run: (attempt: number) => Promise<Response>,
  opts: { maxAttempts?: number; baseMs?: number; capMs?: number } = {},
): Promise<Response> {
  const maxAttempts = opts.maxAttempts ?? 3;
  const baseMs = opts.baseMs ?? 400;
  const capMs = opts.capMs ?? 8_000;
  let lastErr: unknown;

  for (let attempt = 0; attempt < maxAttempts; attempt++) {
    try {
      const res = await run(attempt);
      if (!shouldRetry(res.status, null) || attempt === maxAttempts - 1) {
        return res;
      }
      const retryAfter = Number(res.headers.get("retry-after"));
      const wait = Number.isFinite(retryAfter) && retryAfter > 0
        ? retryAfter * 1000
        : fullJitterDelay(attempt, baseMs, capMs);
      await sleep(wait);
    } catch (err) {
      lastErr = err;
      if (attempt === maxAttempts - 1) throw err;
      await sleep(fullJitterDelay(attempt, baseMs, capMs));
    }
  }
  throw lastErr ?? new Error("retry-exhausted");
}
```

![Request timeout to backoff retry to give-up](https://cdn.otf-kit.dev/blog/api-timeouts-retries-ai-backends/inbody2-20260920a.png)

Verify fail-closed behavior against a local stub before real traffic:

```bash
# Stub sleeps > TOTAL_MS → client aborts; stop at maxAttempts
# Stub returns 429 + Retry-After → wait, then retry once
AI_API_BASE=http://127.0.0.1:8080
curl -sS -D - -o /dev/null -X POST "$AI_API_BASE/v1/chat" \
  -H "content-type: application/json" \
  -d '{"messages":[{"role":"user","content":"ping"}]}'
```

## Idempotency when you retry POSTs

Retries without identity turn one user action into N billable completions. Chat POSTs are usually **not** safe to replay blindly: a timeout after the upstream accepted the body can still produce tokens while your client thinks nothing happened.

Stripe documents the mutating-HTTP pattern: send an `Idempotency-Key` on POST so a safe repeat returns the original result ([Idempotent requests](https://docs.stripe.com/api/idempotent_requests)). Do the same on your AI gateway: one UUID per user-visible attempt (not per TCP try), reuse it on transport retries, and store the first response (or in-flight lock) for a TTL you own.

- **One key per logical action** — reuse across retries; mint a new key only after a final failure when the user clicks try-again.
- **No PII in keys** — avoid emails or personal identifiers ([Idempotent requests](https://docs.stripe.com/api/idempotent_requests)).
- **GET needs no key**; if the vendor has no header, put the key on your proxy in front of `$AI_API_BASE`, or move the call into a job with a unique ID ([background jobs](/blog/ai-production-background-jobs)).

Wire it into the client you already own:

```ts
import { randomUUID } from "node:crypto";

export async function chatOnce(messages: unknown[]) {
  const idempotencyKey = randomUUID();
  return withRetries((attempt) =>
    postChat(
      { messages, stream: false },
      { idempotencyKey }, // same key on every attempt
    ),
  );
}
```

## Ship checklist for owned AI HTTP clients

1. `$AI_API_BASE` and secrets from env — no fake hosts in source.
2. Connect + total timeouts **below** your edge/proxy limit.
3. Retries bounded, jittered, skipping non-retryable codes.
4. Chargeable POSTs carry an idempotency key across retries.
5. Logs: correlation ID, attempt, status/timeout, latency ([logging](/blog/production-structured-logging-for-agents)).
6. Interactive paths fail to a user-visible error before workers pile up.

Fold this into the wider [ship checklist](/blog/ship-ai-mvp-to-production-checklist) when you leave sandbox. For full-stack kits that assume owned backends, start at [otf-kit.dev/templates](https://otf-kit.dev/templates).

## Sources

- [OpenAI API — Error codes](https://developers.openai.com/api/docs/guides/error-codes) — timeout, rate-limit, overload, and Retry-After guidance
- [AWS — Retry behavior](https://docs.aws.amazon.com/general/latest/gr/api-retries.html) — error classification, exponential backoff with full jitter, max attempts
- [Stripe API — Idempotent requests](https://docs.stripe.com/api/idempotent_requests) — Idempotency-Key for safe POST retries