AI Chaos Engineering: Fault Injection Before Production

AI chaos engineering exposes failure paths that ordinary test fixtures leave untouched. A workflow can pass its output evaluations while retrying one write twice, continuing after its caller has timed out, or reporting success after a fallback returned stale data. Those defects surface when model APIs throttle, tools respond late, streams end halfway through an event, and workers crash near a side effect.

A controlled fault experiment makes those conditions repeatable before production. State what must remain true, inject one defined failure, limit the blast radius, and prove from stored evidence that the workflow contained the fault and recovered correctly. Random breakage has no place in that process.

Why regression tests do not prove resilience

Regression testing and fault injection answer different questions. A regression suite checks whether a candidate release produces acceptable decisions, outputs, and tool trajectories for a known dataset. A chaos experiment checks whether the complete system preserves its business invariants when a dependency behaves badly.

A mocked timeout is often implemented as a clean exception raised before any work happens. A real tool can commit a record and lose its response. A model stream can deliver valid text followed by half of a tool-call argument. A queue can redeliver an event while the first worker is still running. The response body alone cannot tell you whether the workflow handled those states safely.

The Principles of Chaos Engineering frame an experiment around a measurable steady state, realistic events, production-like conditions, and a deliberately constrained blast radius. Apply that discipline to orchestration state, queues, retrievers, tools, approval steps, and final writes across the full AI workflow.

Existing regression tests remain the entry gate. A fault experiment is useful only after the normal path is known to work. It then tests the recovery mechanisms that otherwise sit idle until an outage.

Define steady state as a business contract

Infrastructure health is too weak as the experiment oracle. An invoice workflow can return HTTP 200 while creating two approval tasks. A support workflow can complete while sending a draft based on stale policy. Define steady state in terms of observable business outcomes and side effects.

For one invoice-routing workflow, the contract might require:

• every accepted invoice reaches one terminal state within its deadline;
• each invoice creates at most one approval task;
• no payment or ledger write occurs without the required approval;
• an unresolved tool outcome remains visible rather than being labelled failed;
• fallback data carries its source version and freshness status;
• retries, model calls, and tool calls stay inside fixed budgets;
• every stopped or recovered run retains enough evidence for diagnosis.

These are invariants, not averages. A small mean latency does not offset one duplicate payment. Separate hard invariants from service targets. Hard invariants must hold for every experiment. Latency or completion targets can use a stated threshold across several runs.

Name the evidence for each condition before injecting anything. If the test cannot count approval objects, inspect workflow state, and connect tool calls to one run, it cannot prove that the system recovered.

Map dependencies and side effects first

Create a dependency map from entry event to terminal write. Include the component, request deadline, retry owner, side effect, idempotency mechanism, checkpoint boundary, fallback, and evidence source. This prevents the team from testing only the obvious model call.

A useful inventory for each boundary contains:

boundary: create-approval-task
dependency: approval-service
request_deadline_ms: 3000
retry_owner: workflow-engine
max_attempts: 3
side_effect: creates-task
idempotency_key: invoice-id-plus-action-version
checkpoint_before: approval-intent-saved
checkpoint_after: remote-task-id-saved
unknown_outcome_action: query-by-idempotency-key
fallback: none
evidence:
  - workflow-event-log
  - approval-service-record
  - retry-counter

Do not accept three retry layers. If an HTTP client, adapter, and workflow engine each retry, one injected timeout can multiply into many requests. Assign one owner, then configure the other layers to return a classified error.

The map should also show shared dependencies. Two apparently separate tools may use the same database or credential. Breaking that shared service can invalidate the assumption that fallback A is independent from primary path B.

Write a bounded experiment contract

An experiment needs more than a fault name. Record the hypothesis, scope, trigger, expected transitions, abort conditions, cleanup, and evidence query. Azure Chaos Studio describes faults as controlled experiments against selected targets and capabilities in its service overview. AWS Fault Injection Service similarly uses experiment templates, actions, targets, and stop conditions in its official documentation. Those concepts apply even when the injector is a small test adapter rather than a cloud product.

Use a contract like this:

experiment_id: invoice-model-throttle-v1
environment: isolated-staging
steady_state:
  terminal_state: completed
  approval_tasks_per_invoice: 1
fault:
  boundary: model-classification
  trigger: attempts-1-and-2
  response: http-429
blast_radius:
  fixture_tenant: chaos-test
  max_runs: 20
  max_duration_seconds: 180
abort_if:
  - any-nonfixture-write
  - more-than-one-approval-task
  - model-calls-over-budget
expected:
  - retries-owned-by-workflow-engine
  - bounded-backoff-observed
  - no-tool-call-before-valid-classification
cleanup:
  - remove-fixture-records
  - verify-no-running-leases

Use isolated accounts and fixture tenants for side-effecting workflows. A production-like environment means realistic architecture and policies, not permission to create arbitrary customer records. If the experiment must run against production infrastructure, begin with read-only paths, internal subjects, and a kill condition that can stop dispatch immediately.

Inject faults at controlled boundaries

Put the fault controller beside the dependency adapter. The workflow should call the same interface used in normal operation, while test configuration selects a deterministic response for a particular experiment, run, boundary, and attempt.

Useful fault classes include:

1. Admission failures: return 429 or 503 before the provider accepts work.
2. Slow responses: exceed a child deadline but remain inside the experiment's total deadline.
3. Unknown outcomes: commit a tool action, then drop the response before the worker saves the remote identifier.
4. Malformed success: return HTTP 200 with invalid JSON, a missing field, or a type change.
5. Partial streams: close after a complete text event but before a tool call or terminal event is complete.
6. Worker loss: terminate after a checkpoint, before a side effect, or after the effect but before the next checkpoint.
7. Redelivery: send the same queue event concurrently or after its visibility timeout.
8. Stale fallback: make the primary retriever unavailable while the fallback holds an expired snapshot.

Avoid random faults in the first pass. Deterministic triggers make a failed assertion reproducible. Random scheduling can come later, after every named state transition has a direct test.

A practitioner report in OpenClaw issue 131 describes timeout and cancellation behavior not propagating through the execution path as expected. Treat that issue as an author report, not a measured failure rate. It illustrates why a caller-side timeout is not enough evidence that lower-level work stopped.

Build a runner that can stop safely

The runner should reserve the fixture scope, verify the baseline, activate one fault, execute bounded runs, watch abort conditions, and always perform cleanup. It should never depend on a human noticing a dashboard quickly enough.

def run_experiment(spec, workflow, injector, evidence):
    assert evidence.steady_state_holds(spec)
    lease = evidence.reserve_fixture_scope(spec.experiment_id)

    try:
        injector.activate(spec.fault, scope=lease.scope)
        results = []

        for case in spec.cases[: spec.max_runs]:
            if evidence.abort_condition_met(spec):
                injector.deactivate()
                return evidence.fail_experiment("abort condition met")

            result = workflow.execute(
                case.input,
                run_id=case.stable_run_id,
                deadline=spec.deadline,
            )
            results.append(result)

        return evidence.evaluate(spec, results)
    finally:
        injector.deactivate()
        evidence.cleanup_fixture_scope(lease)
        evidence.assert_no_active_runs_or_leases(lease.scope)

Keep the control plane separate from model instructions. A prompt saying "simulate a timeout" does not test network cancellation, queue leases, or tool idempotency. The injector belongs in transport adapters, fake providers, a service mesh, a workflow test harness, or a managed fault tool.

Require a stable run ID and action-level idempotency keys. Retrying an experiment with fresh identities can hide duplicates because each attempt appears legitimate. The same logical case must retain its identity across worker restarts and queue redelivery.

Test an invoice workflow with a fault matrix

Start with a small matrix that crosses important failures with observable invariants. Do not multiply every fault by every component immediately.

Run one normal control beside the faulted case. The control confirms that fixture data, credentials, and the environment were healthy during the experiment. Without it, a configuration error can look like successful fault containment.

Check records in both systems. Workflow telemetry can say approval_created while the approval service contains duplicates. The remote system is the authority for its own side effects. Join its records back to the stable experiment and action identities.

Handle unknown outcomes instead of guessing

The hardest fault is a lost response after a write. A timeout says that the caller stopped waiting. It does not prove whether the remote service committed the action.

Represent this state explicitly as unknown. The recovery decision is then mechanical:

• if the remote service supports idempotency, repeat with the same key;
• if it supports lookup by a stable request reference, reconcile before retrying;
• if compensation is safe, confirm the original outcome before applying it;
• if no reliable check exists, pause for operator review rather than issuing another write.

A chaos test should force this branch and assert that the workflow does not relabel it as an ordinary failure. It should also prove that the operator view contains the run ID, intended action, key, attempts, deadlines, and remote evidence needed to decide.

This is where fault injection extends the current Fire in Belly workflow regression-testing guide. A regression fixture can assert that recovery code exists. The experiment verifies that checkpoints, remote idempotency, cancellation, and evidence collection cooperate when execution is interrupted at the worst boundary.

Verify containment, recovery, and cleanup separately

Do not reduce the result to pass or fail. Produce three decisions.

Containment asks whether the blast radius stayed inside the declared fixture tenant, run count, duration, call budget, and side-effect budget. Any write outside scope is a hard failure even if the workflow later recovers.

Recovery asks whether every run reached the expected terminal state with the correct business outcome. Count remote records, not just internal events. Confirm attempt limits, backoff ownership, fallback freshness, stable identities, compensation results, and operator-visible unknown states.

Cleanup asks whether the injector was disabled, test records were removed or archived according to policy, queues contain no test messages, and no workers, leases, or scheduled retries remain active. A passing outcome with a retry scheduled five minutes later is not complete.

Store the experiment specification, workflow release ID, injector version, fixture revision, event trace, assertion results, and cleanup receipt. That packet should let another engineer explain what failed without rerunning the test immediately.

Avoid common chaos-testing mistakes

Randomly killing components without a hypothesis produces noise. Start from one business invariant and one controlled boundary.

Testing only the model provider misses the side effects that make internal workflows risky. Include queues, state stores, retrieval, identity services, and business tools.

Letting fault code reach production by an unprotected flag creates a new incident path. Restrict activation to a dedicated identity and environment, validate scope server-side, log every change, and default the injector to off.

Checking only eventual completion can hide duplicate actions, stale fallback data, and runaway cost. Pair every positive assertion with negative checks on forbidden writes, extra calls, orphaned work, and scope escape.

Running every fault in one experiment destroys diagnosis. Inject one primary fault at a time until the expected transitions work. Combine faults only when testing a documented dependency or cascading-failure hypothesis.

Run the first AI chaos engineering experiment

Choose one workflow with a meaningful external action. Map its boundaries and write five hard steady-state invariants. Add a deterministic 429 fault at the model adapter and an unknown-outcome fault at one idempotent tool adapter. Limit the run to an isolated tenant, twenty fixtures, three minutes, and a fixed call budget.

Execute the control and faulted cases, then verify the remote record count, workflow terminal states, retries, active leases, and cleanup receipt. Do not promote the next release until another engineer can reproduce the experiment and explain its evidence.

That first exercise closes the gap between having recovery code and knowing it works. Expand the catalogue one production incident or dependency boundary at a time.

References

1. Principles of Chaos Engineering: supports steady-state hypotheses, realistic failure events, production-like experimentation, and constrained blast radius.
2. Microsoft Azure Chaos Studio overview: supports controlled fault experiments, targets, capabilities, and safety boundaries.
3. AWS Fault Injection Service overview: supports experiment templates, actions, targets, and stop conditions.
4. OpenClaw issue 131: provides the cited practitioner author report about timeout and cancellation propagation.
5. Fire in Belly AI workflow regression testing: establishes the existing fixed-fixture and release-gate baseline that this fault-experiment method extends.

---