Handling LLM API Rate Limits in AI Workflows

LLM API rate limits become a workflow reliability problem when several jobs share one provider quota. Each job can look harmless on its own while their combined requests and token demand trigger 429 responses. If every worker retries independently, those retries create more pressure. Interactive requests end up behind batch work, the queue gets older, and a temporary limit spreads through the service.

A larger retry count will not fix this. Model calls need one admission and scheduling layer that accounts for requests and tokens before dispatch. That layer also needs a concurrency cap and an explicit way to prioritize jobs. Retries come later, after the scheduler has reserved capacity. The implementation below includes the checks needed to test the scheduler under load.

Why per-request retries fail

Providers do not enforce only one simple request counter. OpenAI documents rate limits across request and token dimensions and exposes remaining-limit and reset information in response headers through its rate limits guide. Anthropic similarly documents request, input-token, and output-token limits, along with acceleration controls and retry timing in its rate limits documentation.

A workflow can therefore fail while its request count still looks modest. A burst of long documents may exhaust the input-token allowance. Jobs that reserve large outputs put pressure on a different limit. Many short tasks may have plenty of token capacity left but exceed requests per minute. One integer called max_requests_per_minute cannot describe these cases.

Independent retries add a second problem. The official OpenAI Cookbook example for handling rate limits warns that unsuccessful requests still contribute to limits and recommends exponential backoff with random jitter. Backoff is necessary, but it is not admission control. If fifty workers all discover the limit by failing first, the system has already created unnecessary load.

The system has to make separate decisions about admission, dispatch, and recovery:

1. Should this job enter the model-call queue now?
2. When is there enough request, token, and concurrency capacity to dispatch it?
3. If the dispatched call fails, should it retry, wait for a reset, or stop?

Keeping those decisions separate prevents a retry policy from pretending to be a capacity policy.

Model capacity as several budgets

Keep a capacity record for each provider, model, and credential scope. Two API keys do not necessarily create independent quotas, so only separate them when the provider contract defines separate capacity. The scheduler should track at least these values:

• request capacity and its refill rate;
• estimated input-token capacity and refill rate;
• reserved output-token capacity and refill rate;
• maximum in-flight calls;
• the latest provider reset or retry time;
• queue age and rejection policy by workload class.

Use provider headers as observations, not as the only source of truth. A response arrives after a dispatch decision, so headers cannot prevent the first burst. Maintain a local conservative budget, then reconcile it with the provider's reported remaining capacity and reset timestamps.

Token estimates do not need to be perfect to be useful. Estimate input tokens with the tokenizer supported by the selected model when available. Reserve the configured output maximum or a smaller bound justified by the task. When the response returns, replace the reservation with actual usage. If estimation is uncertain, add a safety margin and tune it from observed deltas rather than assuming every character maps to the same token count.

Concurrency deserves its own limit because provider quota is not the only bottleneck. Open sockets, memory, downstream tools, and worker slots can fail before a minute-level token bucket empties. Temporal's worker performance guidance exposes poller and maximum concurrent activity controls, illustrating why work acquisition and active execution need explicit bounds. The same principle applies whether the queue runs on Temporal, a message broker, or a custom service.

Build a shared admission path

Route every model call through one logical admission service. The service might be a small library backed by shared storage, a dedicated gateway, or a queue consumer. Whichever form it takes, workers that share a quota must read and update the same counters.

Represent each pending call with fields that support scheduling and recovery:

ModelJob {
  job_id
  workflow_id
  provider
  model
  credential_scope
  priority_class
  enqueued_at
  input_tokens_estimated
  output_tokens_reserved
  attempt
  deadline_at
  idempotency_key
}

The credential_scope identifies the quota boundary. The priority_class separates interactive, scheduled, and bulk work. The deadline prevents a stale request from being dispatched after its result is no longer useful. The idempotency key protects any workflow transition around the call, although it cannot make a model response deterministic.

Admission should reject impossible jobs early. If one document is larger than the model or provider limit, waiting will never help. Route it to chunking, summarization, or human review. If the deadline is shorter than the current queue delay, fail with a specific capacity error instead of accepting work that cannot finish on time.

Schedule for tokens, requests, and fairness

The dispatcher must reserve every required budget atomically. If one is unavailable, the job stays queued. A request reservation without a matching token reservation can still hit the token limit. Token capacity reserved without an in-flight slot can sit unused while another job waits.

The core loop can remain simple:

function dispatch_next(now):
  reconcile_provider_resets(now)
  job = choose_eligible_job(now)
  if job is none:
    return sleep_until_next_event()

  needed = {
    requests: 1,
    input_tokens: job.input_tokens_estimated,
    output_tokens: job.output_tokens_reserved,
    concurrency: 1
  }

  if not budgets.can_reserve(job.quota_key, needed):
    return sleep_until_budget_refill(job.quota_key, needed)

  budgets.reserve_atomically(job.quota_key, needed)
  mark_dispatched(job)
  call_provider(job)

choose_eligible_job should not be plain FIFO when interactive and bulk traffic share capacity. Strict priority can starve batch work forever, while pure FIFO can place a customer-facing request behind a large import. Use weighted fair scheduling or reserve a fraction of capacity for each class. Add aging so a low-priority job gradually becomes eligible after waiting too long.

Do not split one provider allowance among workers with static fractions unless worker count never changes. Static division wastes capacity when one worker is idle and overloads the provider when autoscaling adds workers. A shared atomic reservation lets every worker use available capacity without exceeding the combined budget.

Handle 429 responses without a retry storm

A 429 response usually points to stale scheduler state, an optimistic estimate, or another client on the same quota. Feed that response back into the shared budget instead of handling it only inside the worker that received it.

First, record the provider, model, quota scope, estimated tokens, attempt number, and returned limit headers. Do not log prompts or credentials to diagnose capacity. Then update the shared next-eligible time from a valid provider reset or retry value. If no usable value is available, calculate exponential backoff with random jitter as shown in the OpenAI Cookbook guidance.

Bound retries by both count and deadline. A retry after the workflow deadline is wasted load. Also classify errors before retrying. A malformed request, authentication failure, unsupported model, or oversized context will not improve after sleeping. Only transient provider errors and genuine quota responses belong on the capacity retry path.

Avoid retry multiplication across layers. If the HTTP client retries three times, the model adapter retries three times, and the workflow engine retries the activity three times, one logical call can produce many attempts. Choose one layer to own 429 retries. Configure outer workflow recovery to see a clear terminal capacity error after that bounded policy finishes.

Example: protect interactive work from a nightly batch

Assume an internal support assistant and a nightly document-classification job use the same model quota. At midnight the batch enqueues thousands of documents. Without workload classes, those jobs occupy every worker and consume the available token budget. A support agent's request enters the same FIFO queue and may wait until the batch drains.

Define two classes: interactive and bulk. Reserve enough concurrency for interactive work while giving bulk work the remaining capacity. Use weighted fairness so idle interactive capacity can temporarily serve bulk jobs, but reclaim it when an interactive request arrives. Set a short interactive deadline and a longer bulk deadline. Estimate each document's input tokens before admission, rather than treating every job as equal.

When the provider reports less remaining token capacity than the local scheduler expected, reduce the shared budget immediately. The dispatcher pauses new bulk calls. Already running calls finish, interactive requests use the protected allowance, and bulk jobs remain durable in the queue. Nothing needs to hammer the provider to discover that capacity has returned.

This design also makes degraded behavior explicit. If queue age crosses the agreed batch threshold, pause new batch ingestion or move overflow to another scheduled window. Do not silently borrow all interactive capacity to make a throughput chart look healthy.

Observe the scheduler, not just provider errors

Fewer 429 responses do not prove the scheduler works. It could eliminate those responses simply by holding work forever. Measure what happens to each job:

• admitted, queued, dispatched, rejected, expired, and cancelled jobs;
• queue age by provider, model, and priority class;
• estimated versus actual input and output tokens;
• reserved and observed capacity by dimension;
• in-flight calls and dispatch rate;
• 429 responses by attempt and quota scope;
• retry delay and deadline exhaustion;
• starvation indicators for each workload class.

Alert on queue age and expired deadlines before alerting only on 429 totals. A provider can lower capacity, a new application can begin sharing the quota, or one workload can suddenly submit larger prompts. Each change may hurt latency before the error rate rises.

Keep diagnostic telemetry separate from sensitive content. Job identifiers, estimates, actual usage, queue times, and status codes are enough to debug most capacity incidents. Prompt bodies and API credentials are not required.

Verify the design under controlled overload

Test the scheduler against a fake provider before relying on production limits. The fake provider should enforce separate request and token budgets, return reset information, inject 429 responses, and vary latency. Cover these cases:

1. A request burst exhausts request capacity while token capacity remains.
2. A few large prompts exhaust token capacity while request capacity remains.
3. Interactive jobs arrive during a sustained bulk queue.
4. The provider reduces an advertised budget during the test.
5. Several workers start at once and compete for the same shared counters.
6. A 429 response lacks a usable retry time.
7. A queued job reaches its deadline before dispatch.
8. One worker crashes after reserving capacity.

For each case, assert more than eventual completion. Confirm that reservations never exceed each budget, interactive queue age stays within its target, bulk work continues without starvation, expired jobs are not dispatched, and retries remain within one configured layer.

Crash recovery needs a lease around reservations. If a worker dies before dispatch, the lease should expire and return capacity. If it dies after sending the request, the outcome may be unknown. Record that state and let the workflow's broader recovery policy decide whether the model call can be repeated safely. This is where capacity control hands off to failure recovery rather than replacing it.

Common implementation mistakes

Do not key limits only by model name. Quotas may also depend on provider, organization, project, account tier, region, or credential scope. Use the boundary documented for the account and check it against response metadata.

Request counting alone misses token-heavy workloads. Reserve input and output tokens separately, then reconcile the estimate after every response.

Adding workers makes a quota bottleneck worse because they compete for the same provider allowance. Scale dispatch capacity only when both the external quota and local resources can support it.

Strict priority protects interactive work but can starve maintenance jobs. Use weighted service and aging, then monitor queue age for both classes.

Backoff handles pressure after a failure. It cannot replace admission, shared reservations, or fair scheduling, all of which prevent avoidable failures before dispatch.

Put one gate in front of every model call

Treat LLM API rate limits as shared capacity. Before dispatch, account for requests, tokens, and concurrency across every worker using the quota. Keep retries bounded and jittered, and use them only for recovery. Queue-age and deadline metrics show whether the scheduler is avoiding errors by delaying work too long.

Start by listing every workflow and service that uses the same provider quota. Route one model and credential scope through a shared admission path, add request and token reservations, and run the eight overload cases above. Once the measurements match the scheduler's decisions, extend the same pattern to the remaining models and providers.

References

• OpenAI rate limits guide: Supports the request and token limit dimensions plus response-header guidance.
• Anthropic rate limits: Supports request, token, acceleration, and retry timing behavior.
• OpenAI Cookbook rate-limit handling: Supports exponential backoff with jitter and the warning about unsuccessful retries.
• Temporal worker performance: Supports explicit poller and concurrent activity controls for bounded work execution.

---