API timeouts and retries when your app calls AI backends you own

When your owned app calls a model or AI HTTP API, the client timeout budget and retry policy live in your code — not in a sandbox dashboard. Sandbox builders (Lovable, Bolt, and similar) often hide connect/read latency behind a platform proxy; once the same feature ships from your repo, every hung /v1/chat call can pin a worker, cascade into user-facing stalls, or silently burn tokens on duplicate POSTs. The production answer is explicit: set connect and total timeouts, classify which failures are retryable, back off with jitter, and attach an idempotency key when you retry a POST.

This post is about outbound HTTP from your service to an AI backend you configure with $AI_API_BASE. It is not a job-queue tutorial (background jobs for AI features already cover queues, worker retries, and job idempotency). It is also not Expo push receipt retries. Pair this client contract with structured production logs so agents can triage which attempt failed, and keep product claims citation-ready when you document the policy for your team.

What sandbox latency hides

In a sandbox, a “call the model” button often feels bounded because the host terminates, retries, or times out behind a UI toast you never configured. After export, your process opens TCP/TLS to $AI_API_BASE, waits for headers, streams tokens, and holds a worker slot until something ends.

Skip client timeouts and a slow upstream holds connections until the reverse proxy limit kills the user request while your handler still waits. Concurrent users amplify the pile-up. Own a timeout budget that fails closed before the edge does — plus bounded retries that do not turn one blip into a stampede.

Set a timeout budget before you retry

A timeout budget answers three questions for every outbound AI call:

1. Connect — how long to establish TCP/TLS to $AI_API_BASE (usually a few seconds).
2. First byte / headers — how long to wait for an HTTP status after the request is sent.
3. Total / overall — hard ceiling for the whole attempt, including stream read.

Vendor SDKs often ship generous defaults (minutes) because generation can be slow — fine for batch, dangerous for a synchronous user request. Split budgets by path: interactive chat ~15–45s; short classify/embed ~5–15s; long document work prefers a queue, or a dedicated 60–300s path if you must stay sync.

OpenAI documents APITimeoutError as “request took too long” and recommends a brief wait before retry (Error codes). Treat that as a signal to own the ceiling, not wait forever. Log attempt, timeout_ms, status/timeout, and a correlation ID so agent-readable logs show connect vs read hangs.

// ai-http.ts — owned outbound client (path-only; base from env)
const AI_API_BASE = process.env.AI_API_BASE!; // e.g. your proxy origin — no hardcoded host in source
const CONNECT_MS = 5_000;
const TOTAL_MS = 30_000;

export async function postChat(
  body: unknown,
  opts: { idempotencyKey?: string; signal?: AbortSignal } = {},
): Promise<Response> {
  const ctrl = new AbortController();
  const timer = setTimeout(() => ctrl.abort("total-timeout"), TOTAL_MS);
  const onOuter = () => ctrl.abort("outer-abort");
  opts.signal?.addEventListener("abort", onOuter, { once: true });

  try {
    return await fetch(`${AI_API_BASE}/v1/chat`, {
      method: "POST",
      headers: {
        "content-type": "application/json",
        authorization: `Bearer ${process.env.AI_API_KEY}`,
        ...(opts.idempotencyKey
          ? { "Idempotency-Key": opts.idempotencyKey }
          : {}),
      },
      body: JSON.stringify(body),
      signal: ctrl.signal,
      // If your runtime supports connect timeouts separately, set CONNECT_MS there.
    });
  } finally {
    clearTimeout(timer);
    opts.signal?.removeEventListener("abort", onOuter);
  }
}

Prefer path-only routes (/v1/chat, /v1/embeddings) in application code and inject the origin via $AI_API_BASE. That keeps reviews honest and avoids baking a vendor hostname into every handler.

Classify failures, then bound retries with jitter

Not every error is retryable. Auth failures, validation errors, and spend/quota denials will not heal if you hammer them. Transient network failures, timeouts, 408, many 429s, and 5xx often will — if you wait and limit attempts.

AWS SDK retry guidance is a clean general model: classify transient vs throttling vs non-retryable, use exponential backoff with full jitter, and cap max attempts (Retry behavior). Full jitter spreads retries so clients do not stampede. Honor Retry-After when present — OpenAI’s guide says the same for 429 and overloaded 503 (Error codes).

Owned interactive policy:

1. Max attempts = 3 (1 initial + 2 retries) unless a batch path says otherwise.
2. Retry timeout/abort, connection reset, 408, 429, and 500/502/503/504.
3. Do not retry 401/403/400, or billing/spend codes that need config changes.
4. Backoff = random(0, min(cap, base * 2^attempt)) — base ~200–500ms transient; higher for throttling.
5. Stop when the user-facing budget is exhausted even if attempts remain.

// retry.ts — bounded retries with full jitter (interactive path)
function sleep(ms: number) {
  return new Promise((r) => setTimeout(r, ms));
}

function fullJitterDelay(attempt: number, baseMs: number, capMs: number) {
  const exp = Math.min(capMs, baseMs * 2 ** attempt);
  return Math.floor(Math.random() * exp);
}

function shouldRetry(status: number | null, err: unknown): boolean {
  if (err) return true; // abort/timeout/network — classified upstream
  if (status == null) return true;
  if (status === 408 || status === 429) return true;
  if (status >= 500 && status <= 504) return true;
  return false;
}

export async function withRetries(
  run: (attempt: number) => Promise<Response>,
  opts: { maxAttempts?: number; baseMs?: number; capMs?: number } = {},
): Promise<Response> {
  const maxAttempts = opts.maxAttempts ?? 3;
  const baseMs = opts.baseMs ?? 400;
  const capMs = opts.capMs ?? 8_000;
  let lastErr: unknown;

  for (let attempt = 0; attempt < maxAttempts; attempt++) {
    try {
      const res = await run(attempt);
      if (!shouldRetry(res.status, null) || attempt === maxAttempts - 1) {
        return res;
      }
      const retryAfter = Number(res.headers.get("retry-after"));
      const wait = Number.isFinite(retryAfter) && retryAfter > 0
        ? retryAfter * 1000
        : fullJitterDelay(attempt, baseMs, capMs);
      await sleep(wait);
    } catch (err) {
      lastErr = err;
      if (attempt === maxAttempts - 1) throw err;
      await sleep(fullJitterDelay(attempt, baseMs, capMs));
    }
  }
  throw lastErr ?? new Error("retry-exhausted");
}

Verify fail-closed behavior against a local stub before real traffic:

# Stub sleeps > TOTAL_MS → client aborts; stop at maxAttempts
# Stub returns 429 + Retry-After → wait, then retry once
AI_API_BASE=http://127.0.0.1:8080
curl -sS -D - -o /dev/null -X POST "$AI_API_BASE/v1/chat" \
  -H "content-type: application/json" \
  -d '{"messages":[{"role":"user","content":"ping"}]}'

Idempotency when you retry POSTs

Retries without identity turn one user action into N billable completions. Chat POSTs are usually not safe to replay blindly: a timeout after the upstream accepted the body can still produce tokens while your client thinks nothing happened.

Stripe documents the mutating-HTTP pattern: send an Idempotency-Key on POST so a safe repeat returns the original result (Idempotent requests). Do the same on your AI gateway: one UUID per user-visible attempt (not per TCP try), reuse it on transport retries, and store the first response (or in-flight lock) for a TTL you own.

• One key per logical action — reuse across retries; mint a new key only after a final failure when the user clicks try-again.
• No PII in keys — avoid emails or personal identifiers (Idempotent requests).
• GET needs no key; if the vendor has no header, put the key on your proxy in front of $AI_API_BASE, or move the call into a job with a unique ID (background jobs).

Wire it into the client you already own:

import { randomUUID } from "node:crypto";

export async function chatOnce(messages: unknown[]) {
  const idempotencyKey = randomUUID();
  return withRetries((attempt) =>
    postChat(
      { messages, stream: false },
      { idempotencyKey }, // same key on every attempt
    ),
  );
}

Ship checklist for owned AI HTTP clients

1. $AI_API_BASE and secrets from env — no fake hosts in source.
2. Connect + total timeouts below your edge/proxy limit.
3. Retries bounded, jittered, skipping non-retryable codes.
4. Chargeable POSTs carry an idempotency key across retries.
5. Logs: correlation ID, attempt, status/timeout, latency (logging).
6. Interactive paths fail to a user-visible error before workers pile up.

Fold this into the wider ship checklist when you leave sandbox. For full-stack kits that assume owned backends, start at otf-kit.dev/templates.

Sources

• OpenAI API — Error codes — timeout, rate-limit, overload, and Retry-After guidance
• AWS — Retry behavior — error classification, exponential backoff with full jitter, max attempts
• Stripe API — Idempotent requests — Idempotency-Key for safe POST retries