·12 min read
Rate Limits Are an Architecture Problem
Rate limits are a throughput constraint, and the fix is a queue, a concurrency cap, and a degradation path. A longer backoff only resubmits work the provider has already refused.
LLM / AI / Architecture / Backend / Rate Limiting / Node.js
The first time an LLM API returns a 429, you add a retry with exponential backoff and move on. The second time, you raise the retry count. The third time, you add jitter. By the fourth incident someone asks why the retry loop ran for ninety seconds and the user saw a loading spinner the whole time, and the answer is that retrying was never the right response to a rate limit. It was just the fastest thing to type.
A rate limit is a throughput ceiling. It tells you how many requests or tokens per minute the provider will accept from your key, and everything above that number is rejected. That makes it a capacity constraint rather than a transient fault, and the correct response to a capacity constraint is the same in every system: queue the work, control admission, and decide what to do when demand exceeds supply.
Two limits, and neither is fixed
LLM rate limits are unusual because they operate on two axes simultaneously. Most providers enforce both a requests-per-minute (RPM) cap and a tokens-per-minute (TPM) cap, and hitting either one triggers the 429. A short prompt sent a thousand times hits RPM. A single prompt with a hundred thousand tokens of context hits TPM. Your system probably hits both, at different times, for different reasons.
The limits are also not fixed. They depend on the model, the tier your account is on, and sometimes the time of day. Some providers publish them, some return them in response headers, and some do both with numbers that disagree. The headers are the truth:
// Headers you will see on a 200 or a 429
x-ratelimit-limit-requests: 500
x-ratelimit-limit-tokens: 200000
x-ratelimit-remaining-requests: 12
x-ratelimit-remaining-tokens: 34820
x-ratelimit-reset-requests: 8s
x-ratelimit-reset-tokens: 22s
Read them on every response, not just on errors. They are the provider's real-time view of your remaining budget, and they are far more useful than a hardcoded constant in your config.
Why retrying harder makes things worse
The naive retry loop has a specific failure mode, and it is worse than no retry at all. When the rate limit is hit, all in-flight requests fail at roughly the same time. They all wait the same backoff interval. They all retry at roughly the same time. They all hit the limit again. This is the thundering herd, and jitter reduces it but does not solve it, because timing is not the cause. You are submitting more work than the provider will accept, and retrying submits the same work again.
Every retry consumes a request against your RPM cap, even though it will be rejected. A retry storm can keep your effective throughput at zero for the entire reset window, because every slot that opens is immediately consumed by a retry rather than by new work. Meanwhile, the user sees a spinner, or worse, a timeout after the retry budget is exhausted.
Retries have one correct use in this context: recovering from a transient failure (a network blip, a 500, a connection reset). A 429 says “you are sending too fast; slow down.” Honouring that means admission control, not persistence.
A queue in front of the provider
The fix is a queue that sits between your application code and the provider SDK. Every model call goes through the queue. The queue drains at a rate that respects the limit. When the queue is full, the caller gets a signal immediately rather than after sixty seconds of retrying.
import { setTimeout } from "node:timers/promises";
interface PendingCall<T> {
execute: () => Promise<T>;
resolve: (value: T) => void;
reject: (err: unknown) => void;
estimatedTokens: number;
}
class ModelQueue {
private queue: PendingCall<unknown>[] = [];
private inflight = 0;
private tokensThisWindow = 0;
private requestsThisWindow = 0;
private windowStart = Date.now();
constructor(
private maxConcurrency: number,
private rpmLimit: number,
private tpmLimit: number,
private windowMs: number = 60_000,
) {}
async enqueue<T>(
execute: () => Promise<T>,
estimatedTokens: number,
): Promise<T> {
if (this.queue.length >= this.maxConcurrency * 4) {
throw new Error("model_queue_full");
}
return new Promise<T>((resolve, reject) => {
this.queue.push({
execute: execute as () => Promise<unknown>,
resolve: resolve as (v: unknown) => void,
reject,
estimatedTokens,
});
this.drain();
});
}
private async drain(): Promise<void> {
this.resetWindowIfNeeded();
while (this.queue.length > 0 && this.canAdmit(this.queue[0]!)) {
const item = this.queue.shift()!;
this.inflight++;
this.requestsThisWindow++;
this.tokensThisWindow += item.estimatedTokens;
this.run(item).finally(() => {
this.inflight--;
this.drain();
});
}
}
private canAdmit(item: PendingCall<unknown>): boolean {
if (this.inflight >= this.maxConcurrency) return false;
if (this.requestsThisWindow >= this.rpmLimit) return false;
if (this.tokensThisWindow + item.estimatedTokens > this.tpmLimit) {
return false;
}
return true;
}
private async run(item: PendingCall<unknown>): Promise<void> {
try {
const result = await item.execute();
item.resolve(result);
} catch (err) {
item.reject(err);
}
}
private resetWindowIfNeeded(): void {
if (Date.now() - this.windowStart >= this.windowMs) {
this.windowStart = Date.now();
this.requestsThisWindow = 0;
this.tokensThisWindow = 0;
}
}
}
This is a simplification for clarity. A production version reads the rate-limit headers from each response and adjusts the remaining budget rather than tracking a fixed window, because the provider's window and yours will drift. But the pieces are right: a bounded queue, a concurrency cap, and token accounting that prevents submission when the budget is spent.
The critical design decision is what happens when the queue is full. Throwing immediately is the right default. The caller, whether it is an API handler or a pipeline stage, can then decide: serve a cached result, skip the optional enrichment, or tell the user the system is busy. That decision is product logic, and burying it inside a retry loop means it never gets made.
Concurrency is the limit that bites first
RPM and TPM are the published limits. The limit that actually matters in most systems is concurrency: how many requests are in flight at the same time. A model call takes one to thirty seconds. If the average is five seconds and your RPM is 500, the maximum useful concurrency is roughly 42 (500 ÷ 60 × 5). Sending 200 concurrent requests makes nothing faster: 158 of them wait in the provider's queue or get rejected, and your process holds 158 open connections doing nothing.
Concurrency control also protects your own system. Each in-flight model call holds a connection, a promise, and the memory for the response stream. At high concurrency, that is meaningful memory pressure in a Node.js process. The event loop post covered why that matters: anything that backs up the event loop blocks everything else.
The concurrency cap also interacts with timeouts. Without a cap, a burst of slow calls can exhaust the RPM limit before any of them complete, which means the next window starts with a queue full of stale work. With a cap, the queue drains at a steady rate, and timeouts are meaningful because you know roughly when each queued call will be dispatched.
// The number to start with
const MAX_CONCURRENCY = Math.floor(
(RPM_LIMIT / 60) * EXPECTED_LATENCY_SECONDS
);
// For a 500 RPM limit and 4s average latency: ~33
// Then tune down from there, leaving headroom for bursts
Degradation under quota, not failure
The queue gives you a choke point, and the choke point gives you a place to make decisions. When the queue depth crosses a threshold, or when the remaining-tokens header says the window is nearly spent, the system should change its behaviour rather than slow down uniformly.
The decisions are specific to the product, but the pattern is general:
- Drop optional work first. In a pipeline with required and optional stages, the optional ones (a summary polish, an alternative phrasing, a confidence cross-check) are the first to skip. The degradation contracts from the pipeline post make this possible: every stage already declares what it returns when it cannot do its job, so skipping it under quota pressure produces a typed degradation rather than a crash.
- Route to a smaller model. If the task has been validated on a cheaper, faster model with acceptable quality for non-critical paths, route there when the primary model's quota is under pressure. This requires per-stage eval scores, which the model-call-owner post already described: you can only downgrade safely when you know the quality delta per stage.
- Serve from the completion cache. An exact-hash cache keyed on the rendered prompt, as described in the Postgres AI stack post, turns a cache miss into a model call and a cache hit into a database read. Under quota pressure, a stale cache hit is better than a rejected request. If your cache has a TTL, extending it dynamically when the rate limit is tight is a valid degradation strategy.
- Shed load at the edge. If your own API is receiving more requests than the provider can serve, return a 503 with a
Retry-Afterheader to your callers. They learn at once that you are overloaded, and the cascade stops there. The worst outcome is accepting the request, queueing it for minutes, and then failing it anyway.
The common thread: every degradation path is a value, not an exception. The system produces a worse result, but it produces a result. A system that retries until it times out produces nothing and costs more.
Batching where the API allows it
Some providers offer a batch API that trades latency for throughput: you submit a set of requests together, they process them within a window (typically hours), and you get the results back in bulk at a lower per-token price with separate, higher rate limits. That rules it out for user-facing calls, but it is the correct choice for offline work: nightly pipeline runs, bulk re-scoring, backfill jobs, and the kind of embedding revalidation that runs when the codebase changes.
The architectural implication is that your system needs to know which calls are latency-sensitive and which are not. That is a scheduling decision rather than a code change. In a pipeline with attributed stages, the distinction already exists: the stage name tells you whether the user is waiting.
type CallPriority = "interactive" | "background" | "batch";
function priorityForStage(stage: Stage): CallPriority {
switch (stage) {
case "chat_response":
case "search_rerank":
return "interactive"; // user is waiting, use live API
case "nightly_assess":
case "backfill_embed":
return "batch"; // no user, use batch API
default:
return "background"; // live API, but can be queued
}
}
A batch API also has its own rate limits and quotas. The difference is that they are expressed in requests per day rather than per minute, and the headroom is usually much larger. Moving offline work to the batch path frees live quota for user-facing calls, which is a throughput improvement with no code change on the interactive path.
Multiple providers and key pools
The simplest way to increase total throughput is a second API key, possibly on a second account. Each key has its own rate limit, so two keys at 500 RPM each give you 1000 RPM with a round-robin dispatcher.
class KeyPool {
private keys: ApiKey[];
private index = 0;
constructor(keys: ApiKey[]) {
this.keys = keys;
}
next(): ApiKey {
const key = this.keys[this.index % this.keys.length]!;
this.index++;
return key;
}
/** Move a key to the back when it receives a 429 */
deprioritise(key: ApiKey): void {
this.keys = [
...this.keys.filter((k) => k !== key),
key,
];
}
}
The step beyond key rotation is a second provider. If the same task runs acceptably on two different models from two vendors, a fallback from the primary to the secondary when the primary's rate limit is hit gives you the combined quota of both. The quality trade-off needs to be measured per stage (the same eval discipline that guards any model change), but the architectural pattern is simple: the queue tries the primary key pool first, and on a 429 or queue-full signal, routes to the fallback.
Two cautions. First, each provider has different rate-limit semantics, different header names, and different reset behaviour; the queue must understand both, and hiding those differences is what the abstraction layer is for. Second, some providers count tokens differently (especially for tool-calling and system prompts), so the token estimate that was accurate for one provider may be off for the other. Track actual versus estimated tokens and correct the estimator.
When this is not your problem
Not every system needs a rate-limit architecture. These are the cases where a simpler approach is correct:
- Low volume, single user. A developer tool, an internal admin panel, or a personal project that makes a handful of calls per minute is never going to hit a rate limit. A simple retry with backoff is fine here, because the cause of a 429 is transient: a brief spike rather than sustained pressure. The queue adds complexity for a problem that does not exist.
- The provider SDK already does it. Some SDKs include built-in rate-limit handling with configurable retries and backoff. If you are making straightforward calls without a pipeline or fan-out, the SDK's built-in behaviour may be enough. Check what it actually does (some are just retry loops), and replace it when you outgrow it.
- Latency tolerance is high. A batch job that runs overnight and can take four hours does not need tight admission control. A simple semaphore limiting concurrency to a comfortable number, with a sleep on 429, is sufficient and easier to reason about.
- The money runs out before the rate limit. If your budget constrains you more tightly than the provider's RPM cap, the cost budgets from the model-call-owner post are the binding constraint, and rate-limit handling is secondary. Build the cost controls first.
The short version
An LLM rate limit is a throughput ceiling, not a transient error, so a retry loop is the wrong response: it resubmits work the provider already refused and keeps your effective throughput at zero during a storm. Put a bounded queue in front of the provider with a concurrency cap derived from your RPM limit and average call latency. Read the rate-limit headers on every response and adjust admission accordingly. When the queue fills, degrade rather than wait: drop optional stages, route to a cheaper model, serve from the completion cache, or shed load at the edge with a 503. Move offline work to the batch API to free live quota for interactive calls. If one key's rate limit is still not enough, pool keys or fall back to a second provider, with per-stage eval scores guarding the quality trade-off. Build all of this only when the rate limit is a real constraint. A handful of calls per minute needs nothing more than the SDK's built-in retry.