Agent Trace Replay Testing: Turn Production Failures Into CI Gates

A curated offline regression suite can stay green while a diagnosed production agent failure returns on the next prompt, tool schema, model, or policy change. The suite often never absorbed the real multi-step trajectory: which tool ran, which middleware rewrote the arguments, which clock or random seed shaped the path, and which frozen model output made the bad branch reachable. Replaying only the user message against live models and live tools turns the gate flaky. Waiting for the next shadow-mode evaluation or canary releasing run re-exposes users to a failure class you already paid to understand.

Hermetic trace replay answers a narrower question than a general AI workflow regression suite. It captures a diagnosed production failure as an immutable boundary envelope, redacts it into a portable fixture, replays the candidate against frozen non-deterministic boundaries, asserts behavioral invariants rather than exact prose, and fails CI when the same failure signature returns. Optional cut-point replay lets changed code run live while recorded envelopes serve the complementary boundaries. This guide is the operating contract for that production-to-CI transaction on one internal agent workflow. It begins after offline harness design and privacy-safe fixture transforms, and it stops before user-visible canary or shadow traffic.

What hermetic replay answers, and what it does not

OpenAI's evaluation best practices treat production logs as a primary source for continuous evaluation and warn that vibe-based checks are not a release system. Hermetic replay takes that advice one step further for agents: the unit under test is not a single reply string. It is the trajectory across model calls, tool calls, middleware transforms, and final disposition.

Hermetic replay answers whether a candidate still violates a diagnosed invariant when the same frozen boundaries are present. It can prove that a refund still fires before approval, that a date parser still swallows an impossible day, or that a hook still strips an error field the model needed. It cannot invent novel futures. It is not a substitute for red-teaming, chaos testing, load tests, or live shadow observation. Replay catches known failures. Shadow and canary still matter for the live distribution.

Capture a diagnosed failure as a boundary envelope

Start from a reviewed incident, not from a firehose of interesting traces. Braintrust's production-failure workflow frames the durable fix as capture, label, promote, score, and gate. For an agent, capture must include every non-deterministic boundary the run crossed:

1. User and system inputs that entered the agent.
2. Model outputs at each decision step, including tool-call proposals.
3. Tool arguments after any middleware rewrite, and tool results before they return to the model.
4. Clocks, random seeds, feature flags, prompt versions, model versions, and retrieval snapshot identifiers.
5. Final user-visible response and side-effect disposition.

Store those fields as an immutable envelope keyed by trace_id, failure_signature, and review metadata. The Chronicle record-and-replay approach describes the same idea for agent decision graphs: reproduce a production failure as a committed regression test and re-run a fix without live LLM calls. Your envelope schema can be JSONL or a small fixture directory. What matters is that the recorded boundaries do not change when the candidate changes.

Instrument both sides of middleware

Middleware that rewrites tool inputs or outputs outside the recorded boundary makes a seemingly complete production trace silently incomplete for replay. A common silent failure is incomplete capture around hooks. As one practitioner comment on a trace-native CI write-up notes, middleware that mutates tool inputs before the call or rewrites tool outputs afterward can leave a "complete" tool-boundary trace silently incomplete. Replay then exercises the post-hook world and never proves whether the bug lived in model selection or in the transform nobody tests.

Record pre-hook input and post-hook output as separate envelope fields for every tool crossing. When a PreToolUse hook normalizes a date, strips an error, or injects a default, the fixture must retain both the model-proposed arguments and the arguments the tool actually saw. Without that split, hermetic replay can certify the wrong layer.

Redact before the fixture leaves review

Production traces are not portable fixtures until privacy work is finished. Fire in Belly's privacy-safe evaluation dataset guide owns the transform lifecycle: drop, generalize, synthesize, and restrict fields before sampling into CI. Apply that contract before any envelope is committed to the repository, shared with a hosted grader, or attached to a pull request.

Minimum redaction for agent replay:

• Replace direct identifiers in user text, tool payloads, and metadata.
• Preserve task relationships that the invariant needs, such as "amount above limit" rather than the real customer value when a band is enough.
• Quarantine raw diagnostic copies under short retention when a parser defect still needs the original bytes.
• Disable body capture in secondary systems that would re-store the fixture during grading.

how2's replay harness outline keeps redaction as an explicit first step before candidate execution. Treat an unredacted production export as a blocked promotion, not as a temporary convenience.

Assert invariants, not exact prose

Agents fail productively while still returning fluent text. The DEV travel-assistant example that accepted "July 32" and apologized instead of correcting the date still looked like a successful HTTP response. Encode the violated behavior in the gate itself.

Use three assertion tiers, matching the release-contract split in the regression-testing baseline:

1. Hard invariants - schema validity, allowed tools, forbidden writes, required approval before side effects, maximum tool-call count, and required correction language for impossible inputs.
2. Trajectory constraints - partial order of required reads before writes, no duplicate side effects for the same idempotency key, and no silent swallowing of tool errors the model must see.
3. Optional semantic checks - sampled judges only after deterministic tiers pass, and only for criteria that truly need judgment.

Pin temperature, judge versions, and fixture versions in the run manifest. Prefer relative gates against the pinned champion envelope when a score is noisy. Exact string match on the final reply is almost always the wrong primary assertion.

Choose full hermetic replay or cut-point replay

Pick the replay mode that matches the question you need answered.

Full hermetic replay serves every recorded model and tool boundary from the envelope. It is bit-stable, cheap, and ideal for proving that a fixed invariant still holds when the world is frozen. Use it as the default pull-request gate.

Cut-point replay serves a chosen subset of boundaries from the record and executes the complementary subset live with the candidate code. That is how you test a prompt, router, or guard change against a recorded incident without re-calling every historical model step. Chronicle's public description centers on this cut-point operation: turn a recorded incident into a regression test that runs in continuous integration while changed code actually executes at the boundaries under test.

Prefer cut-point proof over full stubs

Cut-point replay must execute the changed code path live while serving frozen envelopes at other boundaries; full-stub replay can miss mutants that re-enable an unsafe action. If every boundary is stubbed, a test that only checks the final assertion may pass even when the new code would have allowed the unsafe action under a slightly different recorded prefix. Cut-point replay exists to catch that class of mutant: keep the recorded unsafe opportunity visible at earlier steps, then execute the changed guard live and demand that it still blocks. Do not treat "all stubs green" as proof that a new approval check works. Prove the check on a cut-point that still presents the recorded dangerous proposal.

Be explicit about cut-point mode. Name which boundaries are live, which are frozen, and which side effects remain stubbed. Never let a cut-point replay call a production write tool.

Promote one fixture per failure signature

Promote one representative fixture per failure signature, not every unique incident, or the suite overfits historical wording and still misses near-duplicates. Promoting every unique incident creates a suite that overfits historical wording and still misses near-duplicates. Cluster failures by signature first: same violated invariant, same tool, same middleware stage, same policy boundary. Keep one representative fixture per signature, store related trace_ids in metadata, and add a second fixture only when a distinct branch is required. Braintrust's guidance on overfitting evals to individual traces makes the same point from the dataset side. A small representative set that fails closed on the signature is more valuable than a large set that engineers learn to ignore.

Also keep a few previously successful traces in the suite. A fix that stops one failure can weaken a path that already worked. Positive envelopes protect against that trade.

Wire the CI gate

Promptfoo and similar runners show that evaluation belongs in CI as an exit code, not as a dashboard hobby. For hermetic replay, the pull-request job should:

1. Load only redacted fixtures for the touched workflow.
2. Run deterministic invariant checks first and fail fast.
3. Run full hermetic replay for the signature set that matches the changed components.
4. Run cut-point replay only for the boundaries the change claims to fix or touch.
5. Publish a compact report with trace_id, signature, violated assertion, and whether the boundary was frozen or live.
6. Exit non-zero on any hard-invariant failure or on any signature regression versus the pinned champion result.

Keep the PR suite small enough that engineers will not bypass it. Run the broader signature catalog nightly. An expensive gate that engineers skip is just documentation.

Worked example: support agent refund path

Suppose production shows this sequence for ticket T-1042:

1. User asks to refund order A184 for 750 GBP.
2. Model calls get_order correctly.
3. A PostToolUse hook strips a policy_limit_gbp field from the tool result.
4. Model calls issue_refund without creating an approval.
5. User receives a polite confirmation. Observability records HTTP 200.

Offline golden prompts that only check the final sentence may still pass after a prompt tweak. Hermetic replay encodes:

• Envelope fields for model-proposed tool args, post-hook tool results, and the write that occurred.
• Hard invariant: issue_refund is forbidden when amount exceeds limit unless an approval token is present.
• Trajectory constraint: approval creation must precede any refund write.
• Failure signature: refund-over-limit-missing-approval@post-tool-hook-strip.

A candidate that restores the stripped field and adds an approval guard must pass full hermetic replay on the frozen dangerous proposal. A cut-point replay that freezes steps 1-3 and executes the new guard live must still refuse the write. Either failure blocks merge.

Promote, hold, or discard

After replay:

• Promote the fixture into the required PR set when the signature is recurring or high risk and the redaction review is complete.
• Hold in a review queue when the failure is ambiguous, the envelope is incomplete, or privacy review is unfinished.
• Discard one-off noise after labeling it, without deleting the production trace retention copy your incident process requires.

Do not claim a signature is covered until a deliberate break test fails the gate: reintroduce the missing approval check or the hook strip in a branch and confirm CI goes red.

Put the first hermetic gate in place

Pick one agent workflow with at least three diagnosed production failures. For each, write a boundary envelope, redact it, assign a failure signature, and encode one hard invariant plus one trajectory constraint. Run full hermetic replay on every pull request that touches that workflow's prompts, tools, or middleware. Add cut-point replay for the next guard you intend to ship. Link escaped production failures back into signatures until the suite reflects the workflow's real risk history rather than a synthetic demo set.

The first gate will not prove the agent is safe on unseen traffic. It will prove a failure you already understood cannot quietly return in the next change.

References

1. OpenAI evaluation best practices - Eval-driven development, production-log mining, continuous evaluation, and agent nondeterminism.
2. Promptfoo documentation - Open-source CI/CD evaluation workflow for prompts and LLM apps.
3. Braintrust: turn LLM production failures into regression tests - Capture, label, promote, score, and gate from production traces.
4. how2: agent replay testing from production traces - Redaction, stubs, frozen clocks, deterministic assertions, and CI exit codes.
5. DEV: trace-native CI/CD for agent failures - Practitioner report on multi-step replay and incomplete middleware capture.
6. Chronicle record-and-replay - Record-and-replay for agent decision graphs without live LLM calls.
7. Fire in Belly AI workflow regression testing - First-party offline harness baseline that this hermetic gate extends.

---