Back to Blog
Warehouse racking filled with rows of boxed goods waiting to be processed in bulk

LLM Batch Processing: Building Reliable Production Workflows

9 min read

LLM batch processing can cut the cost of nonurgent inference, but replacing a synchronous loop with one upload and a polling job creates a different failure model. Results may arrive out of order. A completed job can contain failed records. A job can expire after processing only part of its input. A large result download can fail near the end. If the recovery path resubmits everything, downstream systems may receive duplicate writes.

A more persistent polling loop does not fix this. Build a small batch controller that owns an immutable input manifest, provider capability checks, job state, per-record reconciliation, selective retries, and idempotent delivery. This article defines that controller and a verification plan for production workloads.

Decide whether the workload belongs in a batch

Batch inference is appropriate when the result can arrive later and each input can be processed independently. Common examples include document classification, nightly extraction, embedding generation, evaluation suites, catalog enrichment, and recurring report preparation. Interactive support replies, approval screens, and tool calls that must react to live state usually belong on a synchronous path.

The provider contract reinforces that split. The OpenAI Batch API offers asynchronous processing with a 24 hour completion window. Anthropic Message Batches similarly processes requests asynchronously and exposes results when all requests finish or the 24 hour window ends. The Gemini Batch API describes batch jobs as suitable for large, nonurgent work such as preprocessing and evaluations.

Use three admission rules:

  1. The caller does not need the answer in the current request.
  2. Every record has a stable business identity and can reach a terminal state independently.
  3. Applying a successful result can be retried without duplicating the business effect.

If any rule fails, keep that workload synchronous or redesign the business operation first. A batch discount does not compensate for an ambiguous completion contract.

Treat the batch as a durable state machine

A reliable controller separates the provider job from the business batch. The provider owns inference. Your system owns what was requested, which outputs were accepted, and whether every record reached a final disposition.

Use explicit states rather than a boolean done field: 

prepared -> submitted -> running -> results_available
results_available -> reconciling -> completed
running -> expired | cancelled | provider_failed
reconciling -> retry_required | completed_with_errors | completed

Persist every transition with a timestamp and a reason. Store the provider job identifier only after the submission response is durable. If the process crashes between provider acceptance and local persistence, recover through an idempotency key when the provider supports one, or quarantine the submission for operator reconciliation. Blindly submitting again can create two jobs for the same manifest.

The business batch should contain at least these fields: 

batch_id
manifest_version
manifest_digest
provider
model
request_schema_version
provider_job_id
state
submitted_at
provider_deadline_at
expected_record_count
reconciled_record_count
retry_batch_id

Keep provider status values in a separate field. Map them into your states, but do not erase the raw value. That evidence matters when a provider adds a new terminal state or when an incident must be reconstructed.

Freeze an immutable input manifest

Use the manifest to determine whether processing is complete. Give each input a stable record_id derived from the business object and operation, not from its line number. Include the prompt or request payload, model settings, schema version, and a digest of the normalized request. Once submitted, never mutate the manifest.

OpenAI requires a unique custom_id for each request, and Anthropic uses a similar custom identifier. Use these IDs to join input and output because response order is not a safe contract. Amazon states directly that Bedrock batch output order may differ from input order.

A minimal record envelope can look like this: 

{
  "record_id": "invoice:7842:extract:v3",
  "business_key": "invoice:7842",
  "operation": "extract",
  "schema_version": 3,
  "request_digest": "stored-digest",
  "payload": {
    "model": "configured-model",
    "input": "stored-or-referenced-input"
  }
}

Do not place reusable credentials or unnecessary personal data in the manifest. Keep large source documents in controlled storage when the provider supports references, and record the exact source version. If an input changes after preparation, create a new record version rather than silently changing the submitted request.

Before upload, verify unique IDs, accepted file size, record count, model availability, and JSON schema. Then calculate a digest for the complete manifest. Store that digest beside the provider job so a later worker can prove it is reconciling the same inputs.

Gate provider capabilities before submission

A provider can support a feature synchronously but not in batch mode. That difference should fail during preparation, not after a long queue wait.

For example, Amazon Bedrock batch inference processes JSONL records independently but does not support tool calling or structured output in batch inference. Anthropic documents unsupported parameters for Message Batches. Current SDK reports also show that schemas can behave differently in batch mode, which makes a capability test part of deployment rather than an optional check.

Maintain a small capability matrix keyed by provider, model, endpoint, and tested date. Include:

  • accepted input format and maximum job size
  • supported model parameters
  • structured output support
  • tool or function calling support
  • provider deadline and result retention
  • cancellation behavior
  • output and error file format
  • rate and concurrency limits

Reject an unsupported job with a clear reason. Do not strip a required schema or tool definition just to make submission pass. That produces syntactically successful output that no longer meets the business contract.

Run a canary batch whenever the provider, model, SDK, or request schema changes. The canary should contain valid records, one deliberate validation failure, Unicode, a maximum practical payload, and a record whose output exercises every required field.

Poll without turning polling into the workflow

Polling should observe state, not carry business logic. Schedule the next check with bounded backoff and jitter, persist the next poll time, and allow any worker to resume. One process should not sleep for hours while holding the only knowledge of a job.

Stop polling when the provider reaches any terminal state, not only success. OpenAI documents completed, failed, expired, cancelling, and cancelled states. Anthropic exposes request counts for succeeded, errored, canceled, and expired outcomes. A provider job ending successfully does not mean every record succeeded.

Set an internal deadline earlier than the provider expiry. That gives the controller time to inspect a stalled job, cancel it if appropriate, and prepare recovery. Alert on lack of progress, but avoid interpreting a quiet interval as failure when the provider does not promise continuous count updates.

Store the last observed counts and status. Useful operational metrics include queue age, running age, completed records, failed records, polls per job, result download bytes, reconciliation lag, retry rate, and unreconciled record count.

Download and reconcile every record

Treat result retrieval as a data transfer job. Stream large files to controlled storage, calculate a digest while downloading, and only mark the artifact available after the stream closes and validation passes. A current OpenAI Python issue reports a large result download failing after substantial transfer, so keeping the entire file in memory or assuming one download call is enough is a fragile design.

Reconciliation must perform a full outer comparison between manifest IDs and observed result IDs. For each result:

  1. Reject an unknown ID.
  2. Reject a duplicate ID unless the duplicate is byte-identical and the policy explicitly permits deduplication.
  3. Validate the response envelope and expected output schema.
  4. Classify the disposition as succeeded, retryable failure, permanent failure, cancelled, or expired.
  5. Persist the raw result and classification before applying a business write.

After reading all output and error artifacts, identify manifest IDs with no result. Never infer that a missing ID succeeded because the overall job says completed. The completion invariant is simple: 

expected IDs = succeeded + permanent failures + retryable failures + cancelled + expired

The sets must be disjoint, and their union must equal the manifest. If that equality does not hold, the batch remains in reconciling or moves to an incident state. It is not complete.

Retry records, not whole jobs

Build a retry manifest only from records classified as retryable. Keep each original record ID in lineage metadata, but assign a new attempt identity so outputs cannot be confused across jobs. Set a retry limit by error class rather than applying one blanket count.

Retry transport errors, transient provider failures, and expiry when the request remains valid. Do not automatically retry invalid JSON, unsupported features, policy refusals, or business validation errors. Those require correction or a terminal failure disposition.

A cancelled batch may still have usable partial output. Anthropic explicitly notes that cancelled batches can contain results produced before cancellation. Reconcile those results first, then place only unresolved retryable records in the next batch. Resubmitting the original file wastes inference and creates duplicate-delivery risk.

Apply successful outputs through an idempotent consumer. A useful key combines the business operation, record ID, request digest, and accepted result version. Insert an application ledger entry and the business change in one transaction when possible. If the key already exists, return the previous disposition instead of applying the action again.

Model generation and business delivery are different stages. A record can have a valid model result while its database update fails. Preserve that result and retry delivery. Do not spend another model call to repair a local transaction problem.

Verify the complete lifecycle

Test the controller with a fake provider or recorded fixtures before using a costly live batch. Cover these cases:

  • outputs arrive in a different order from inputs
  • one output ID is duplicated
  • one expected ID is missing
  • one unknown ID appears
  • the job completes with individual failures
  • the job expires with partial output
  • cancellation returns some successful results
  • the result download stops midway
  • the reconciler crashes after persisting a result
  • the business write succeeds but its acknowledgement is lost
  • a retry batch receives a late result from the original job
  • a model or schema capability changes between deployments

The final acceptance test should submit a small live canary and prove three facts from stored evidence: every manifest ID has one terminal disposition, every accepted output passes the business schema, and replaying reconciliation plus delivery causes no additional business writes.

Success requires complete reconciliation against the immutable manifest, not merely provider_job.status == completed. Provider-specific submit and poll examples usually stop before this check.

Put the controller in front of one real backlog

Start with one low-risk workload whose outputs can be reviewed, such as nightly classification or an evaluation suite. Prepare stable IDs, capability gates, a manifest digest, terminal-state handling, streamed downloads, and the reconciliation invariant before scaling the record count. Then force one partial failure and replay delivery.

If the system cannot prove what happened to every input, it is not ready for a larger batch. The next action is to implement the manifest and reconciliation tables first, then connect one provider behind that contract.

References

About Fire In Belly: Independent senior engineering from Tallinn, Estonia. We design and build AI workflow automation with the immutable manifests, per-record reconciliation, and idempotent delivery described above, at published fixed prices. Schedule a call to discuss your next project.