LLM Load Testing: Find Your Workflow's Safe Capacity

LLM load testing fails when it treats every request as equal and measures only HTTP success. Ten short classifications and ten long document summaries impose different token demand, queue time, model time, and downstream work. A test can report no 429 errors while jobs sit in a queue past their deadlines or continue running after callers give up.

The test needs to produce a safe capacity envelope, meaning the highest offered load your complete workflow can accept while meeting latency, quality, side effect, and cost rules. This guide shows how to model the workload, generate arrivals without hiding overload, measure the full execution path, stop before the test becomes expensive, and turn the first failed invariant into a production admission limit.

Define capacity for the complete workflow

Provider throughput is only one boundary. An internal support workflow may validate a request, retrieve account history, call a model, run a policy check, save a proposed reply, and open a review task. Any stage can saturate first. More model concurrency can make the system slower if it floods the retrieval store or fills the worker pool.

OpenAI documents rate limits across request and token dimensions in its rate limits guide. Anthropic documents request, input-token, output-token, and acceleration limits in its rate limits documentation. Those limits matter, but they do not state how many complete business tasks your application can finish safely.

Write a capacity claim in application terms:

> The support workflow can accept 30 new tickets per minute, including the expected mix of short and long tickets, while 99 percent finish within five minutes, fewer than 1 percent require a capacity retry, every accepted ticket reaches one terminal state, and model spend stays below the test budget.

The test now has an arrival rate, workload mix, deadline, retry ceiling, terminal-state invariant, and cost limit. Replace the sample values with requirements from the workflow owner. Do not copy them into production unchanged.

Build a representative workload manifest

A flat list of identical prompts produces a clean graph and a weak conclusion. Build classes from the work the system will actually receive. Payloads may be synthetic or sanitized, but their size and control path should remain representative.

For each class, record:

• expected share of arrivals;
• input-token distribution, not only the mean;
• allowed output-token range;
• retrieval depth and document count;
• tools or business systems invoked;
• interactive or background priority;
• end-to-end deadline;
• quality assertion and terminal-state rule;
• maximum model calls and side effects per execution.

A support workflow might use four classes. Short classification requests make up half the workload. Ordinary draft requests make up 30 percent. Long account-history cases make up 15 percent. Escalations that require a review task make up 5 percent. Keep that mix stable during the baseline test, then run separate skew tests for plausible peaks such as a burst of long cases.

Use recorded metadata to shape the distribution, but do not copy customer text, credentials, or personal data into fixtures. Token counts, attachment size, language, route, and observed duration are usually enough to build a useful generator. Store the manifest with a version so the same workload can qualify later changes.

Generate offered load without hiding saturation

A closed-loop test waits for one response before its virtual user sends the next request. When the service slows, the generator sends fewer requests. The chart can therefore improve while the system is failing to accept the intended arrival rate. This is coordinated omission: slow responses suppress the very traffic that should reveal overload.

Use an open-loop generator for the main capacity test. Schedule arrivals independently of completion, retain their intended timestamps, and let queue delay become visible. Closed-loop tests remain useful for a fixed pool of interactive users, but they answer a different question.

The generator needs stable execution identities and a hard scope. It should submit only fixture tenants or isolated test accounts. Tool adapters should use sandboxes or dry-run modes unless the test explicitly checks a bounded side effect. A prompt that asks the model to pretend it called a tool does not exercise queue, network, storage, or idempotency behavior.

A minimal test plan can look like this:

workload_version: support-capacity-v3
stages:
  - arrivals_per_minute: 5
    duration_minutes: 5
  - arrivals_per_minute: 15
    duration_minutes: 10
  - arrivals_per_minute: 30
    duration_minutes: 10
  - arrivals_per_minute: 45
    duration_minutes: 10
mix:
  short_classification: 0.50
  normal_draft: 0.30
  long_history: 0.15
  escalation: 0.05
limits:
  maximum_model_cost: 120
  maximum_total_runs: 800
  maximum_test_minutes: 45
  maximum_nonfixture_writes: 0
abort_if:
  - duplicate_review_task
  - unknown_side_effect
  - deadline_breach_rate_over_0.05
  - model_cost_over_limit

Warm the service before recording the baseline so startup effects are not mistaken for sustained capacity. Keep a short cold-start test as a separate scenario if deployments or scale-to-zero behavior matter.

Instrument arrivals, queues, attempts, and outcomes

Edge latency alone cannot locate the bottleneck. Carry one execution ID from scheduled arrival to terminal outcome. Record intended arrival, actual admission, queue start and end, every dependency attempt, model token usage, first token, final token, tool completion, and terminal state.

The OpenTelemetry generative AI semantic conventions provide common attributes for model operations. Add application fields for workload class, manifest version, workflow version, priority, queue, deadline, attempt owner, quality result, and side effect count. Avoid capturing prompt bodies unless the environment and policy explicitly allow it.

Track these measurements by stage and workload class:

• offered, admitted, rejected, queued, dispatched, completed, expired, and cancelled runs;
• queue age at dispatch and terminal completion;
• time to first token and total model duration;
• end-to-end latency percentiles;
• estimated and actual input and output tokens;
• provider 429 and 5xx responses;
• retry calls per original execution;
• active worker, connection, and tool slots;
• quality pass, review, and fail counts;
• jobs still running after their parent deadline;
• model and infrastructure cost for each completed outcome.

Temporal's worker performance guidance separates task polling and task-slot behavior. The same distinction matters in any orchestrator. An empty worker slot does not prove a provider request or downstream database connection is available.

Preserve quality and side effect rules under load

Throughput is not useful if the workflow starts producing weaker results at saturation. Run the same deterministic assertions and sampled model evaluations used in release testing. Compare pass rates by workload class and stage. A faster result caused by truncated context, skipped retrieval, or an emergency fallback is not equivalent to the normal result.

Keep hard invariants outside aggregate percentages. Each accepted execution should produce one terminal outcome. Each fixture ticket should create at most one review task. No test should write outside the fixture scope. A cancelled run should release capacity or enter a documented reconciliation state.

A current practitioner report in OpenClaw issue 131 describes timeout and cancellation not reaching an in-flight model request. It is one author-reported defect, not a general failure rate. It illustrates why a load test must count work that survives its caller's deadline. Otherwise abandoned calls occupy capacity while the edge records a completed timeout.

Increase load in stages and stop on evidence

Begin below expected production load and hold each stage long enough for queues and token budgets to reach a stable condition. A two-minute spike can miss a queue that grows slowly. Move to the next stage only when the current stage completes its observation window without a hard invariant failure.

Stop immediately for a duplicate or unauthorized side effect, a nonfixture write, unknown tool outcomes beyond the defined reconciliation budget, or an uncontrolled cost increase. These are safety failures, not reasons to collect more samples.

For capacity failures, record the first stage where any service rule breaks:

• queue age rises continuously rather than returning to baseline;
• the target latency percentile crosses its deadline;
• quality drops outside the accepted band;
• retry amplification exceeds its limit;
• unfinished work remains after the stage ends;
• provider or local concurrency stays pinned;
• the cost per successful outcome exceeds its ceiling.

Google's SRE guidance on addressing cascading failures explains how overload, retries, and work completed after deadlines can compound pressure. The test should expose that transition before production traffic does.

Turn the saturation point into an admission limit

Do not set production capacity equal to the highest stage that barely passed. Choose a lower limit that preserves headroom for traffic variance, slower dependencies, background work, and model response variance. State the reasoning rather than applying a universal percentage.

Suppose 30 arrivals per minute passed for every class, while 45 caused queue growth and deadline failures in long-history cases. Inspect the bottleneck first. If the model token budget was exhausted, extra workers will not help. If tool connections saturated while provider capacity remained available, model routing will not help. Fix the actual boundary, rerun the same manifest, and compare the curve.

The capacity record should contain:

1. workflow, deployment, model, and manifest versions;
2. environment and quota differences from production;
3. highest fully passing stage;
4. first failed stage and failed invariant;
5. bottleneck evidence;
6. selected production admission limit;
7. reserved headroom and priority allocation;
8. rejection, queueing, or deferred-work behavior above the limit;
9. date and event that require a retest.

The existing Fire in Belly rate-limit guide explains how to enforce runtime admission, token reservations, fair queues, and bounded retries. The load test supplies the evidence used to configure those controls. It does not replace them.

Avoid the common false passes

Testing only the model endpoint measures the provider client, not the workflow. Exercise the actual queue, retrieval, validation, and tool adapters in a controlled environment.

Counting HTTP 200 responses misses wrong outputs, late results, duplicate actions, and work abandoned below the edge. Require terminal-state and quality evidence.

Using average prompt size hides token-heavy cases. Preserve a distribution and report every metric by class.

Letting the generator slow down with the service hides overload. Use scheduled open-loop arrivals for the capacity claim.

Adding retries during a failing stage can make completion look better while total calls and cost explode. Report retries per original run and assign one retry owner.

Running against unlimited mocks creates a local benchmark that says nothing about provider quotas. Use fakes while developing the test runner, then run a bounded qualification against the intended quota and production-shaped dependencies.

Run one bounded qualification test

Export one hour of non-sensitive workflow metadata: task class, input and output tokens, route, tool path, priority, and elapsed times. Turn it into a versioned workload manifest. Define latency, quality, side effect, retry, and cost rules before generating traffic.

Run four increasing arrival stages in an isolated scope. Stop at the first failed invariant, identify the saturated boundary, and choose a production admission limit below the highest fully passing stage. Record the manifest and evidence beside the release. Repeat the same qualification after a material model, prompt, queue, retrieval, tool, or quota change.

References

1. OpenAI rate limits guide: supports request and token rate-limit dimensions.
2. Anthropic rate limits: supports request, token, acceleration, and retry constraints.
3. Temporal worker performance: supports separate worker polling, task-slot, and concurrency controls.
4. OpenTelemetry GenAI semantic conventions: supports standard model-operation telemetry fields.
5. Google SRE: Addressing Cascading Failures: supports overload, deadline, retry, and load-shedding guidance.
6. OpenClaw issue 131: provides the cited practitioner report about cancellation propagation.
7. Fire in Belly LLM API rate-limit guide: establishes the existing runtime admission-control baseline.

---