Back to Blog
Close-up of a network patch panel with dense rows of red cables carrying live connections

LLM Streaming API: Recovering Interrupted Streams Without Duplicates

10 min read

An LLM streaming API may deliver several paragraphs and then close before the application receives a terminal event. The browser is left with an answer that looks finished but is not. A tool call can end with half of its JSON missing. Retrying at that point may repeat the generation, its cost, or a downstream action. A generic retry loop cannot tell these outcomes apart. Instead, treat the model provider, application server, and client as separate participants. Track semantic completion apart from transport closure, cancel work that has no listener, and base recovery on what the operation already exposed.

Use a completion state machine, not a socket flag

A closed connection proves only that bytes stopped arriving. It does not prove that the model completed, that every structured block assembled correctly, or that the client received the final bytes. Anthropic's documented stream has an ordered event lifecycle ending in message_stop, and it may also send an error event inside an otherwise successful HTTP stream. Your adapter should map provider-specific events into a small internal state machine: 

created -> connected -> receiving -> terminal
                       |          -> cancelled
                       |          -> provider_error
                       |          -> idle_timeout
                       |          -> truncated
                       -> parse_error

Only a recognized provider terminal event can move the operation to terminal. End of file, a client disconnect, a parser exception, or an idle deadline moves it to a different state. Many implementations miss this distinction. A normal transport close says nothing about whether the response is semantically complete.

Store an operation record before opening the provider stream. Include an operation ID, user and tenant scope from trusted server context, the provider request identifier when available, model and prompt version, timestamps, last event type, terminal state, visible character count, structured block status, and cancellation reason. Debugging the transport does not require copying sensitive prompts into this record. Apply the redaction rules already used for application telemetry.

Why interrupted streams produce ambiguous outcomes

A stream can fail at several boundaries, and each one leaves different evidence.

The provider connection can fail. Anthropic documents named Server-Sent Events, incremental content blocks, ping events, in-stream errors, and a final message_stop. Its tool inputs can arrive as partial JSON deltas that must be accumulated before parsing. A current Anthropic TypeScript SDK issue reports a connection ending during a partial tool payload without the expected terminal events. That issue is a practitioner report, not a universal provider guarantee, but it demonstrates the state your code must handle: bytes arrived, a tool block started, and completion was never proven.

The application-to-client connection can fail while the provider connection remains healthy. A user closes a tab, a mobile network changes, or a reverse proxy reaches its own timeout. The server may continue reading and paying for tokens even though no client can consume them. If generated text triggers no side effect, continuing may only waste cost. If the stream is part of an auditable workflow, stopping immediately may lose useful evidence. The policy must be explicit rather than inherited from whatever the HTTP library does by default.

Automatic client reconnection creates a separate ambiguity because it is easy to mistake delivery replay for model-stream resumption. The WHATWG Server-Sent Events standard defines reconnection behavior and the Last-Event-ID mechanism for an SSE source that supports replay. It does not make an arbitrary upstream model request resumable. Your application can resume delivery only if it assigned durable event IDs and retained the emitted events. Otherwise a reconnect creates a new delivery session, not a continuation of the original generation.

Build the LLM streaming API boundary in six steps

Normalize provider events

Create one adapter per provider. The adapter should emit internal events such as text_delta, structured_delta, usage, provider_error, terminal, and unknown. Keep the raw provider event type for diagnostics. Unknown events should be recorded and ignored unless they affect a block you cannot safely interpret. Anthropic specifically advises clients to handle new event types gracefully.

Do not let provider event names leak into business logic. The application should ask whether the internal stream reached terminal, not whether one provider sent message_stop or another sent a differently named completion event. This also makes provider migration and testing easier because recorded fixtures share one internal contract.

Separate display text from structured data

Text deltas can be displayed as they arrive because an incomplete sentence is visible but not executable. Tool arguments, JSON objects, citations, and other structured blocks need a stricter rule. Buffer each block by its provider index or stable block identifier. Do not parse or execute it until the provider closes that block and the complete payload passes schema validation.

This rule prevents a partial string such as {"customer_id":"42","action":"refu from reaching an execution layer. It also keeps a parser failure from destroying text that was already useful to the user. Text delivery and structured execution have different risk, so they need different commit points.

Assign delivery sequence numbers

Wrap every client-facing event in an envelope with an operation ID and monotonically increasing sequence number: 

{
  "operation_id": "server-generated-id",
  "sequence": 17,
  "kind": "text_delta",
  "payload": "next visible fragment",
  "provider_terminal": false
}

Sequence numbers let the client deduplicate replayed delivery. If you choose to retain a short event buffer, a reconnect can request events after the last acknowledged sequence. This is your application protocol. The browser's generic SSE reconnect behavior is not enough unless your endpoint implements durable IDs and replay. The MDN SSE guide explains named events, event IDs, retry intervals, explicit closure, and server-side disconnect detection that can support this layer.

Keep replay retention bounded. An interactive response may need only a few minutes of event history. After the retention window, return an explicit cannot_resume result and let the user decide whether to restart. Do not silently regenerate and splice new output onto an old prefix.

Propagate cancellation in both directions

Connect the client request's cancellation signal to the provider SDK's abort mechanism. When the client leaves, mark the local reason first, then request provider cancellation, then wait for bounded cleanup. Treat a cancellation exception as an expected terminal path when it matches your own signal. Log it as cancelled, not as an unexplained application crash.

Cancellation also needs a business rule. For ordinary chat text, abort when the last client disconnects. For a workflow whose result must be stored regardless of the browser, detach client delivery from generation and continue in a durable worker. Make that choice before production. Mixing both models produces jobs that sometimes survive disconnects and sometimes disappear.

Add an idle watchdog

A total request timeout does not detect a stream that connects and then sends no meaningful events for a long period. Record last_event_at for every valid event, including provider pings if the provider defines them as liveness. Set an idle deadline based on observed provider behavior and your user-facing latency target. When it expires, abort the upstream request and mark idle_timeout.

Reset the watchdog only after a complete protocol event passes basic validation. Arbitrary bytes that fail parsing must not extend the deadline, or a damaged stream can remain open indefinitely. Keep the idle limit separate from the total generation deadline because they answer different questions: is the stream alive, and has the operation taken too long overall?

Commit completion evidence

When the terminal event arrives, finish all open text blocks, require every structured block to be closed, validate accumulated tool arguments, and persist usage data that arrived with or before completion. Then mark the operation terminal in one durable write. Send a final client envelope containing the terminal status and last sequence.

If the transport closes before that write, classify the operation as truncated even if the visible text looks complete. The user can still copy the partial answer, but the application must not present a completion badge or execute a pending tool call.

Choose recovery by external effect

Recovery depends on what escaped the operation, not on the exception class alone.

For read-only text with no durable delivery log, show the partial response as incomplete and offer a full restart. An automatic retry behind the same visible message may diverge and repeat the opening text, so the restart needs a new operation.

For text with sequenced delivery and a retained event buffer, replay only already generated events after the client's last acknowledged sequence. Continue the original provider stream only if it is still active. If upstream generation ended without a terminal event, close the delivery as truncated and offer a new operation.

For incomplete structured output, discard the open block. Never repair partial tool JSON by guessing the missing suffix. A new model request may regenerate the proposal, but it must pass the normal validation, authorization, approval, and idempotency controls before execution.

For a tool action already committed before delivery failed, return the stored action result on retry. Key the write by the business operation and request identity so a second generation cannot repeat it. The stream record should link to the action ledger without treating streamed text as proof that the action happened.

For an in-stream provider error, retain the provider error type and any safe retry hint. Retry only before a consequential action and within the remaining deadline. Otherwise surface an explicit failure for operator or user review.

Verify disconnect and truncation behavior

Test the complete chain through the same proxy and client libraries used in production. A unit test around the provider iterator cannot reveal buffering, proxy timeouts, or missing cancellation propagation.

Run these cases with a fake provider stream or recorded fixtures:

  1. Send text deltas and a valid terminal event. Assert one terminal operation and one final client envelope.
  2. Close after several text deltas with no terminal event. Assert truncated, no completion marker, and no structured execution.
  3. End during partial tool JSON. Assert the block is discarded and the parser never sends arguments to the tool layer.
  4. Send a named provider error after an HTTP success. Assert provider_error, not terminal.
  5. Stop all events without closing the socket. Assert the idle watchdog aborts the provider request.
  6. Disconnect the client. Assert the configured policy either cancels upstream work or detaches it into a durable worker.
  7. Reconnect with the last sequence. Assert retained events replay once and expired buffers return cannot_resume.
  8. Deliver the final text, then lose the client's acknowledgement. Assert replay does not duplicate visible events.
  9. Commit a tool action, then drop delivery. Assert retry returns the existing action result rather than executing again.
  10. Send an unknown event type between valid events. Assert it is recorded without corrupting event order.

Track terminal-state counts, truncation rate, client disconnects, provider cancellations, idle timeouts, parse failures, replayed events, replay misses, incomplete structured blocks, and operations with no terminal evidence. Alert on changes relative to normal traffic rather than treating every user cancellation as an incident.

Shortcuts that break stream recovery

Marking success in a finally block confuses cleanup with completion. That block runs after failure, cancellation, and parser errors too. A verified protocol event and closed structured blocks are the only completion evidence.

Parsing partial tool arguments on every delta is safe only for a preview. Preview data cannot cross the execution boundary. The completed block still needs schema and policy validation.

SSE reconnect is a delivery feature, not provider resume. Resuming a model operation requires server-side state, sequence numbers, retained events, and an upstream request that is still alive. If any piece is missing, start a new operation and label it as such.

Put one streamed path through failure injection

Start with the highest-volume interactive path that has no irreversible tool action. Add the operation record, normalized event adapter, separate structured buffers, client sequence numbers, cancellation wiring, idle watchdog, and terminal write. Then inject a disconnect after visible text and another during partial tool JSON.

Both tests must end in an explicit non-success state. No structured action should run, and retrying must not duplicate delivered events. After those checks pass, apply the same LLM streaming API contract to tool-enabled streams. Keep automatic recovery disabled until the action ledger can return an earlier result instead of running the action again.

References

About Fire In Belly: Independent senior engineering from Tallinn, Estonia. We design and build AI workflow automation with the completion state machines, cancellation wiring, and failure injection described above, at published fixed prices. Schedule a call to discuss your next project.