Shadow Mode LLM Evaluation: Evidence Before Any User Exposure

A prompt, model, retrieval setting, or tool schema can pass offline regression tests and still fail on live traffic. Production brings long-tail inputs, concurrent load, and tool states that no curated fixture set fully captures. Sending the candidate to even a small user cohort creates real exposure before you know whether the change crashes, times out, chooses different tools, or spends multiples more tokens.

Shadow-mode evaluation answers a narrower question than a canary. It mirrors or samples production requests into a pinned candidate while the champion remains on the user-visible path. The candidate output is logged and compared, never returned. The goal is real-distribution evidence with zero user exposure before any bounded canary begins.

This guide gives AI and platform engineers an operating contract for that pre-canary gate on one internal workflow. It covers version pins, request mirroring, side-effect isolation, failure-isolated awaits, paired traces, tiered comparison, a worked support-triage example, and a promote or hold decision. It begins after offline regression testing and stops before user-visible canary cohorts.

What shadow mode answers, and what it does not

Arize's shadow deployment glossary defines the pattern clearly: real requests are mirrored to a candidate, outputs are logged and compared against the incumbent, and nothing the candidate produces is returned to a user. Shadow answers questions about behavior, latency, cost, error rate, schema validity, and tool selection on the real input distribution. It cannot answer questions about clicks, resolutions, escalations, or satisfaction, because nobody experienced the candidate response.

That limitation is not a defect. It is the reason shadow sits before canary. Google's SRE workbook defines a canary as a partial, time-limited deployment evaluated against a control. A canary spends user risk to learn outcomes. Shadow spends inference cost to learn behavior without that risk. Treat them as sequenced instruments, not interchangeable labels.

The Agent Patterns Catalog shadow canary pattern applies the same idea to agents: dual-route a fraction of real traffic through champion and challenger, return only the champion reply, and diff the challenger on agreed metrics. Fire in Belly's existing canary deployment guide already notes that shadow is related but narrower and unsafe for irreversible writes unless tools are replaced with recorders. This article owns that narrower contract end to end.

Shadow promotion criteria must exclude user outcome metrics because nobody saw the candidate response. If a dashboard uses thumbs-up rate, ticket resolution, or revenue as a shadow gate, the metric is fabricated. Use only signals available from paired outputs and telemetry: errors, timeouts, schema failures, policy violations, latency, token cost, tool-call diffs, and sampled judge scores.

Pin the release unit before you mirror anything

Shadow evidence is useless if you cannot name what ran. Freeze a release unit that includes:

• Prompt and policy version identifiers
• Model name and pinned revision or snapshot where the provider allows it
• Retrieval index alias, embedding model, and chunking version when RAG is involved
• Tool schema versions and allowed tool set
• Workflow code or graph version
• Feature-flag or router configuration that selects champion versus candidate
• Evaluation rubric version used for comparison

Write the champion and candidate pins into a shadow run record before the first mirrored request. If either pin drifts mid-run, stop and start a new record. Otherwise you cannot tell whether a regression came from the intended change or from an unrelated dependency move.

Decide sampling up front. Mirroring 100% of traffic doubles inference cost and can saturate provider quotas. Many teams sample 5% to 20% of eligible requests, stratified by case class when volume allows. Record the sampling rule in the same run record so a later reader does not treat a thin sample as full coverage.

Isolate side effects or do not shadow

For a pure generation or classification step, discarding the candidate output is enough. For an agent that can write to a CRM, send email, post to Slack, create calendar events, issue refunds, or call internal mutating APIs, discarding the final text is not enough. The dangerous part is the tool call.

Build an explicit side-effect matrix before enabling mirrors:

Default every mutating tool to blocked. Prefer returning a structured shadow_blocked tool result to the candidate so the agent can continue and reveal later decisions, rather than crashing the loop in a way that hides behavior. Sandboxes and recorded replays help for deeper trajectory comparison, but they are still not a substitute for authorization controls on high-stakes actions.

If privacy policy forbids sending production payloads to a second model path, shadowing may be disallowed for that workflow. Record that as an applicability failure and fall back to offline fixtures plus a tightly bounded canary, rather than quietly shipping without either gate.

Keep shadow off the production await boundary

Latency isolation is as important as write isolation. If the user request waits for both champion and candidate, a slow or failing shadow path becomes a user outage.

A shadow path that shares the production await boundary can turn observation into a user-facing outage when the candidate is slow or fails. That failure mode is not theoretical. agent-safe-pipeline issue 68 describes a shadow comparison that waited on production execution and authority evaluation through one shared promise, so a slow or rejected hypothetical decision could delay or reject the caller after the production action already ran.

Implement these rules:

1. Return the champion result as soon as the champion path completes.
2. Enqueue candidate work asynchronously with its own timeout, concurrency limit, and budget.
3. Never let candidate errors change the champion HTTP status, callback count, or authorization path.
4. Label candidate decisions so they cannot be reused as executable gate decisions.
5. Emit a structured shadow status such as received, timed_out, invalid, or unavailable instead of rejecting a successful champion result.

Add a hard spend ceiling for the shadow run: maximum candidate requests per hour, maximum tokens, and maximum judge calls. When the ceiling trips, stop mirroring and keep the evidence already collected. Shadow is optional observation. Production continuity is not optional.

Log paired traces you can actually compare

Store one paired record per mirrored request. A practical schema looks like this:

{
  "shadow_run_id": "shadow-2026-09-21-support-triage",
  "correlation_id": "req-7f3a",
  "case_class": "billing-question",
  "champion_version": "wf-support-triage@12",
  "candidate_version": "wf-support-triage@13",
  "champion": {
    "latency_ms": 1840,
    "input_tokens": 2200,
    "output_tokens": 310,
    "tool_calls": ["read_ticket", "draft_reply"],
    "final_status": "ok"
  },
  "candidate": {
    "latency_ms": 2510,
    "input_tokens": 2200,
    "output_tokens": 480,
    "tool_calls": ["read_ticket", "update_crm", "draft_reply"],
    "final_status": "ok",
    "blocked_tools": ["update_crm"]
  },
  "checks": {
    "schema_ok": true,
    "policy_ok": true,
    "tool_diff": ["update_crm"],
    "judge_sample": false
  }
}

Use comparable telemetry fields on both sides. OpenTelemetry GenAI semantic conventions exist so teams do not invent private vocabularies for model operations, token counts, and related spans. Keep raw prompts out of long-lived shadow logs when they contain secrets or personal data. Store redacted payloads or pointers to a restricted store with retention limits.

Compare in tiers so cost stays bounded

Exact string equality is a weak signal for generative systems. Two different replies can both be acceptable. Use a tiered comparison plan:

1. Deterministic checks on 100% of pairs. Schema validity, required fields, forbidden content, tool-name allowlists, argument-policy checks, and hard latency or error budgets.
2. Cheap similarity or embedding checks on a sample. Useful for spotting large drift without paying for a judge on every request.
3. Sampled semantic judges on 1% to 5% of pairs. Prefer pairwise judging of champion versus candidate against a fixed rubric when the task allows it. Calibrate the judge separately; do not invent a new evaluator in the same change you are shadowing.
4. Human review of disagreement clusters. Group by case class and tool-diff pattern. Convert durable failures into offline regression fixtures.

OpenAI's evaluation guide documents task-specific evals, graders, and continuous evaluation as part of an improvement loop. Use that machinery for the comparison layer. Do not treat shadow logs as self-explanatory evidence without graders and thresholds written before the run.

Pre-register promote and hold rules before mirroring starts. Example rules for a support triage shadow:

• Hold if candidate error or timeout rate exceeds champion by more than an agreed absolute margin.
• Hold if blocked mutating tool attempts exceed a threshold, even when the block worked.
• Hold if schema or policy failure rate rises.
• Hold if sampled pairwise judge preference for champion exceeds an agreed margin on safety or correctness slices.
• Promote to a tiny canary only when the observation window covers weekday and weekend traffic for the workflow and no hold rule fired.

Handle multi-turn agents without fooling yourself

Multi-turn shadow sessions diverge after the first differing reply, so single-turn paired comparison is the trustworthy unit of evidence. After turn one, the champion and candidate conversations are no longer the same state machine. Continuing to shadow later turns as if they shared context produces directional anecdotes, not paired evidence.

Practical rules:

• Shadow the decision unit you can pair: one user message plus the tools available at that turn.
• For session-level behavior, rely on canary exposure after shadow clears single-turn gates.
• If you replay historical transcripts through a candidate, label the result as replay evidence, not live shadow evidence, and keep tool backends sandboxed.

Worked example: support triage prompt change

Suppose the champion workflow reads a support ticket, drafts an internal classification and customer reply, and waits for a human to send. The candidate prompt asks the model to update a CRM priority field before drafting. Offline fixtures are green.

Shadow plan:

1. Pin wf-support-triage@12 as champion and @13 as candidate.
2. Sample 10% of billing and account tickets for five business days plus one weekend.
3. Allow candidate read_ticket and draft_reply. Block update_crm and send_reply.
4. Keep candidate execution asynchronous with an 8 second independent timeout.
5. Run deterministic checks on every pair. Sample 2% for pairwise judging on correctness and tone.
6. Hold if blocked update_crm attempts appear on more than 2% of pairs, or if candidate p99 latency exceeds champion by 40%, or if schema failures rise.

During the run, the candidate repeatedly attempts update_crm on ambiguous billing tickets. The block prevents production writes, but the attempt rate itself is a regression signal: the new prompt increased mutating intent on uncertain cases. The team holds, adds fixtures for those tickets, and revises the prompt before any canary.

That is the point of shadow mode. The failure was visible on real traffic without changing a single customer-visible reply.

Decision record and handoff to canary

Close every shadow run with a short decision record:

• Run ID, window, sampling rule, and spend consumed
• Champion and candidate pins
• Hold rules and observed results by slice
• Notable disagreement clusters converted into fixtures
• Decision: promote to canary, hold for repair, or discard
• Owner and next review time

Only after promote should you enter the canary contract owned by the existing canary guide: stable cohorts, user-visible exposure, outcome metrics, and automatic rollback. Do not skip from offline green to broad exposure because shadow felt expensive. The expensive outcome is learning about a bad prompt from customer escalations.

Implementation checklist

• Champion and candidate pins frozen in a run record
• Sampling and spend ceilings declared
• Side-effect matrix reviewed; mutating tools blocked or sandboxed
• Candidate path asynchronous and failure-isolated from the user response
• Paired trace schema emitting comparable telemetry fields
• Deterministic checks on all pairs; sampled judges calibrated separately
• Promote and hold rules pre-registered without user-outcome metrics
• Multi-turn limits documented
• Privacy review for payload duplication completed
• Disagreements filed as offline fixtures before the next attempt

References

1. Arize shadow deployment glossary
2. Google SRE canarying releases
3. Agent Patterns Catalog shadow canary
4. OpenAI evaluation guide
5. OpenTelemetry GenAI semantic conventions
6. agent-safe-pipeline issue 68
7. Fire in Belly AI workflow canary guide

---