Queue backpressure on an owned backend: refuse the burst before the pool melts

Agent runs, vendor webhook fan-out, and retry storms arrive as a sudden pile of HTTP calls on an API you operate. The database pool and the worker processes behind that API are a fixed budget. Once every extra call is allowed to pin a connection, the burst becomes latency for every tenant, including the ones that sent a normal amount of traffic.

Backpressure is the refusal at that edge. Cap in-flight work. Cap how many requests may wait, and for how long. When both caps are exhausted, answer with a status code and a Retry-After value the caller can honor. A process that keeps an ever-growing list of HTTP requests in memory is betting the pool will catch up. Under this load, it will not.

What owned capacity means here

Owned capacity is the part you restart and pay for: database connections, worker processes, and the concurrency budget of the next service you run. The caller does not know those numbers. An agent loop will keep issuing work. Admission turns your budget into a yes, a short wait, or a reject on the backend that holds the pool.

Matthew Palma's notes on HTTP API admission control describe that gate as a concurrency cap, a bounded queue with a maximum wait, and a reject that carries 503 and Retry-After. Readiness fails when the process should leave rotation. Size the knobs from your pool: $DB_POOL_SIZE, $MAX_INFLIGHT, $MAX_QUEUE_DEPTH, $QUEUE_WAIT_MS, and $RETRY_AFTER_SEC.

API gateway guidance from API7 separates two knobs dashboards often mash together. A rate limit counts arrivals in a window. Concurrency counts work still inside the system. A client under its rate limit can still occupy every database connection when each call is slow. Rate limiting an owned API is the arrival window. This piece is occupancy. You want the pair.

Keep the other queues in their own posts

This series already uses "queue" for three different jobs. Mixing them buffers the wrong thing.

Circuit breakers on an owned backend fail fast on outbound calls when a vendor is already failing. Inbound admission decides whether a new request may enter your database at all.

Dead letter queues for background jobs quarantine a message after its retries are spent. That storage sits later than the HTTP gate. A request you have not admitted yet has nothing to dead-letter.

A client-side offline queue holds mutations on the device and replays them when the network returns. The replay still has to pass the server gate, or the reconnect wave becomes the burst this post is about.

An unbounded in-process queue stores the overload until memory, file descriptors, or the database pool fail together. Bounded concurrency plus a short wait, then 429 or 503 with Retry-After, spends the pool on work already admitted and tells everyone else to come back.

Admit, wait briefly, or reject

Per process, or per worker behind a load balancer, take a semaphore of size $MAX_INFLIGHT. Derive it from $DB_POOL_SIZE and any downstream budget the handler also consumes. If each admitted request holds one pool connection for the life of the call, $MAX_INFLIGHT must stay inside what that pool can finish. Extra handlers wait inside the driver, and the reject you wanted becomes a timeout.

When several workers share one database, the fleet total is the real cap. Four processes each sized at the full $DB_POOL_SIZE admit four times what the pool can serve. Divide the pool across the processes, and leave a remainder for migrations, health checks, and admin work.

In front of the semaphore, a wait queue is optional and short. Cap it with $MAX_QUEUE_DEPTH and $QUEUE_WAIT_MS. When the queue is full, or a waiter exceeds the wait budget, reject without checking out a connection.

API7 frames that choice as bounded delay or reject, and names load shedding for the moment shared capacity is already gone.

429 is the caller or the tenant. This API key, this agent, this customer has too much in flight relative to the share you assigned them. RFC 6585 section 4 defines 429 Too Many Requests and allows a Retry-After header so the client knows when another attempt is reasonable.

503 is shared capacity. The process, the pool, or the worker fleet is full no matter who called. RFC 9110 specifies Retry-After as either a delay in seconds or an HTTP-date. For this gate, send Retry-After: $RETRY_AFTER_SEC on both 429 and 503. Callers that honor the header back off. Callers that ignore it still hit the same cheap reject on the next try.

Do not answer the shed with a 200 whose body claims the work was queued while the process has nowhere to run it. That hides the overload and invites the client to poll.

Occupancy moves when latency moves

API7 points at Little's Law for a first estimate: average concurrency is about arrival rate multiplied by time in the system. When a dependency slows and handler latency doubles, the same arrival rate doubles how many calls are in flight. The rate limit still says the tenant is fine. The semaphore sees the extra occupancy and starts to wait or reject.

That is why $MAX_INFLIGHT is tied to $DB_POOL_SIZE, then revised when handler time changes. Keep the rate limit beside the semaphore: rate stops one tenant from filling $MAX_QUEUE_DEPTH with cheap arrivals; concurrency stops a slow dependency from holding $DB_POOL_SIZE until every other request times out.

Metrics, readiness, and the path

Emit queue_depth and admission_rejected. Split rejects by 429 and 503 if both exist. Shedding is the gate working. A dashboard that only alerts on 500 stays green while you turn traffic away.

Fail readiness when queue_depth or processing lag crosses the threshold for this process. Readiness is what the platform checks before it sends new work. Liveness can remain true so the process finishes requests it already admitted. Palma includes readiness in the admission story for that reason: a live, saturated process should drop out of rotation until the wait queue recedes.

Draining an owned backend on shutdown is the same refusal with a different trigger. Shutdown stops new work, then waits for in-flight handlers to finish. Backpressure stops new work while the process is supposed to stay up, because the pool is already committed.

The path is a straight line. In-flight below $MAX_INFLIGHT means the handler runs. In-flight full, and queue depth below $MAX_QUEUE_DEPTH, means the request waits up to $QUEUE_WAIT_MS. Wait expired, or queue full, means 429 or 503 with Retry-After, and the pool is never checked out.

A gate in front of the handler

One process-wide semaphore, one bounded waiter list, one reject response. A per-tenant cap uses the same shape under the process cap. Scope "tenant" maps to 429. Scope "process" maps to 503. Call release in a finally. Skip the database until admit returns ok.

// Pseudocode: semaphore + bounded waiters + reject
async function admit(scope: "tenant" | "process"): Promise<Admit> {
  if (inflight < Number(process.env.MAX_INFLIGHT)) {
    inflight += 1;
    return { ok: true, release };
  }
  if (queueDepth >= Number(process.env.MAX_QUEUE_DEPTH)) {
    return {
      ok: false,
      status: scope === "tenant" ? 429 : 503,
      retryAfterSec: Number(process.env.RETRY_AFTER_SEC),
    };
  }
  // wait up to QUEUE_WAIT_MS, then same reject shape
}

Increment admission_rejected whenever admit returns ok: false, and return that status with Retry-After: $RETRY_AFTER_SEC.

Broker backlog vs HTTP buffer

A durable job broker with retention and a consumer concurrency setting is a real queue. Consumer count still has to respect $DB_POOL_SIZE, or the jobs pin the same pool the HTTP gate just protected. An array on the API process has no retention and no bound until you add $MAX_QUEUE_DEPTH.

Agent clients retry on their own timer. Every extra attempt still gets a cheap reject: no pool checkout. Log admission_rejected and the scope.

Checklist

• Set $MAX_INFLIGHT per process from $DB_POOL_SIZE. Across workers, the sum still has to fit the pool.
• Cap any wait queue with $MAX_QUEUE_DEPTH and $QUEUE_WAIT_MS.
• On a full gate, return 429 for a caller or tenant budget and 503 for shared capacity, both with Retry-After: $RETRY_AFTER_SEC.
• Emit queue_depth and admission_rejected. Fail readiness when depth or lag crosses the threshold.
• Keep a rate limit on arrivals. Keep HTTP request buffering bounded.

Scaffolding the API, the worker, and the repo you deploy is a separate decision from this gate. OTF ships full-stack kits you own — Booking, Fitness, and SaaS Dashboard at $99 each, or the Everything Bundle at $149 — plus a free SDK under MIT. The templates live at otf-kit.dev/templates.

Protect owned capacity under load with a hard in-flight cap, a short wait budget, and an honest 429 or 503 with Retry-After at your edge. Infinite buffering is not backpressure. It is a deferred outage.

Sources

• RFC 6585 §4, 429 Too Many Requests — defines 429 and allows Retry-After on that response.
• RFC 9110, Retry-After — delay in seconds or an HTTP-date; the time to wait before the next request.
• Matthew Palma, HTTP API admission control — concurrency caps, a bounded queue with maxWaitMs, 503 plus Retry-After, and readiness.
• API7, API gateway concurrency control — rate versus concurrency, Little's Law as an estimate, 429 for a caller or tenant versus 503 for shared capacity, bounded delay or reject, and load shedding.