AI Workflow Failure Recovery Without Duplicate Actions

AI workflow failure recovery has to do more than retry a failed API call. A workflow may create a ticket, update a customer record, and then lose its model connection before sending an approval request. Restarting it can create a second ticket or repeat the update. Abandoning it leaves an incomplete business process.

Design the recovery path with the workflow. Persist progress around side effects. Give each action a stable idempotency key. Retry errors only when another attempt has a chance of succeeding, and define compensation for changes that cannot be repeated safely. The implementation and tests below put those rules into practice.

Why ordinary retries fail

The model call gets attention because its output is probabilistic and its provider sits outside your system. The more troublesome failures occur between steps. A process may crash after an external service accepts a request but before the worker records the response. A timeout proves that the caller stopped waiting. It says nothing about whether the remote operation happened.

Call that ambiguity an "unknown outcome." Consider an accounts workflow that reads an invoice, asks a model to classify it, creates an approval task, and posts a message. The task service might commit its change while the HTTP response disappears. A blind retry then creates another task. Skipping the retry can leave the workflow stuck even though the first task exists.

Durable workflow tools store enough state for a process to continue after a failure. LangGraph saves graph state as checkpoints organized into threads for fault-tolerant execution and later inspection, as described in its persistence documentation. Checkpoints solve only part of the problem. An external write still needs a clear repeat contract.

Retries need boundaries too. Temporal's retry policy documentation separates intervals, backoff, maximum intervals, maximum attempts, and non-retryable errors. It also notes that Temporal Activities retry by default while Workflow Executions do not. Other engines use different controls, but the decision should still be explicit. An HTTP client should not repeat every failure on its own.

Define the recovery contract first

Start with a table of workflow steps. Record the input, output, side effect, repeat behavior, and recovery action for each one. The team can then review decisions that might otherwise remain assumptions.

Step	Side effect	Safe to repeat?	Recovery rule
Read invoice	None	Yes	Retry with bounded backoff
Classify invoice	Provider usage and generated output	Usually	Reuse saved result when available
Create approval task	New task in another system	No	Use an idempotency key and query by key
Notify approver	Message delivery	Not always	Save provider message ID or use an outbox
Post ledger entry	Financial record	No	Require a stable transaction reference and compensation plan

Classify every side-effecting call with one of these outcomes:

1. succeeded: the system has durable proof of the result.
2. failed: the system has durable proof that no change occurred.
3. unknown: the request may have succeeded, but the caller cannot prove either outcome.

Do not relabel unknown as failed to simplify the state machine. Query the target system with the idempotency key, or send the same request to an endpoint that honors that key. If neither option exists, route the run to an operator because the software cannot resolve the outcome safely.

Implement AI workflow failure recovery step by step

1. Give the run a stable identity

Create one workflow ID when the business event enters your system. Keep that ID across process restarts, queue redeliveries, and manual resumes. Do not derive it from a worker process or a timestamp generated on each attempt.

Derive a separate key for each side effect. A practical format combines the workflow ID, the logical step, and a version:

workflow_id = "invoice:tenant-42:inv-1847"
approval_key = hash(workflow_id + ":create-approval:v1")
notification_key = hash(workflow_id + ":notify-approver:v1")

The version belongs to the action contract, not the retry count. If an operator clicks retry five times, all five attempts must send the same key. Change the version only when the intended business action changes.

The receiving service should store the key with the first result and return that result for duplicates. When you cannot change the receiver, add an adapter that records the key and remote object ID in your own database. Protect that adapter with a unique database constraint. A preliminary "does this exist?" query without a constraint has a race between the query and insert.

2. Separate decisions from effects

Keep model output away from direct writes. First ask the model for a proposed action in a validated schema. Then let deterministic application code check permissions, business rules, and current state before performing the action.

def run_invoice_workflow(state, services):
    if not state.classification:
        proposal = services.model.classify(state.invoice_text)
        state.classification = validate_classification(proposal)
        services.checkpoints.save(state)

    if not state.approval_task_id:
        key = stable_key(state.workflow_id, "create-approval", version=1)
        task = services.approvals.create_or_get(
            idempotency_key=key,
            payload=approval_payload(state.classification),
        )
        state.approval_task_id = task.id
        services.checkpoints.save(state)

    return state

During replay, the workflow can reuse the accepted classification instead of asking the model for another proposal. If policy requires a fresh decision, record the reason and create a new version. A worker restart should not change the decision by accident.

3. Checkpoint around side effects

Save validated inputs before a side effect, then save the remote result as soon as it returns. The first checkpoint records intent. The second records the observed outcome.

Use a transaction when local state and an outbox record share one database. Write the business state and an event to the outbox in the same transaction. A separate publisher can retry delivery. This avoids the gap where the database commits but the process crashes before publishing a message.

A graph checkpoint should include the workflow version, step status, model result or its durable reference, idempotency keys, remote object IDs, attempt counters, and the last classified error. Keep credentials and unnecessary sensitive text out of checkpoint state. Recovery needs identifiers and decisions, not a copy of every secret available to the worker.

4. Classify errors before retrying

Create a small error taxonomy and make the default conservative:

• Retry temporary transport errors, rate limits, and service unavailability with bounded exponential backoff and jitter.
• Do not retry invalid input, failed authorization, policy rejection, or an unsupported operation until something changes.
• Reconcile unknown outcomes before issuing another non-idempotent write.
• Pause repeated model format failures after a small attempt budget. Preserve the invalid output for diagnosis when policy permits.
• Send exhausted or unclassified failures to a dead-letter queue or operator review view.

Temporal exposes the policy controls needed for intervals, backoff, attempt limits, and non-retryable errors in its official retry policy reference. Even if your queue or orchestrator uses different names, put equivalent values in code or configuration and test them. An unlimited retry policy can turn a permanent permission error into a noisy, expensive loop.

5. Compensate instead of pretending to roll back

A database transaction cannot roll back an email already delivered or a task already created in another product. For distributed changes, define a compensating action that restores an acceptable business state. Microsoft's Saga pattern guidance describes a sequence of local transactions whose earlier changes can be undone by compensating transactions when a later step fails.

Compensation is a business operation, not always a literal reversal. Deleting an approval task might erase an audit trail, so "cancel task with reason" can be safer. A sent email cannot be unsent, so the compensation may be a correction message plus an incident marker. A ledger entry may require a reversing entry rather than deletion.

For each compensatable action, specify who may trigger it, whether it is itself idempotent, what evidence it records, and what happens if compensation fails. Run compensations in reverse dependency order when later actions depend on earlier ones. Do not automatically compensate after every technical error. First determine whether the original action succeeded and whether business policy permits reversal.

6. Make operator recovery explicit

Software cannot resolve every outcome. Give operators a view of the workflow ID, current step, intended action, idempotency key, remote references, last error, attempt count, and available controls. They may need to resume from a checkpoint, reconcile remote state, retry one step, compensate a completed step, or terminate the run with a reason.

Each control should call the same application functions used by automatic recovery. A manual retry must not bypass idempotency or permission checks. Record the operator, timestamp, reason, prior state, and resulting state. This turns an emergency button into an auditable workflow transition.

Test failures rather than only happy paths

A successful demonstration says little about recovery. Inject failures at every boundary, then inspect the workflow state and the external effects.

1. Crash before the external request. The resumed run should perform the action once.
2. Crash after the receiver commits but before the worker saves the response. The resumed run should reconcile or repeat with the same key and still produce one external object.
3. Return a rate limit twice, then succeed. Confirm backoff, attempt counts, and eventual completion.
4. Return a permanent authorization error. Confirm the workflow stops without consuming the temporary-error retry budget.
5. Save a model decision, then restart. Confirm replay uses the accepted decision rather than silently generating another.
6. Fail a late step after two earlier effects. Confirm the defined compensation order and audit records.
7. Fail a compensation action. Confirm the run remains visible and does not report a clean rollback.
8. Deliver the same queue event concurrently. Confirm the unique constraint allows one logical run and one copy of each side effect.

Include assertions for negative outcomes. Count approval tasks, messages, and ledger entries after each test. Verify that no step remains in running beyond its lease, no completed step loses its remote reference, and no terminated run can resume without an explicit transition.

Common implementation mistakes

A random idempotency key on every attempt labels duplicate work as new work. Saving state only at the end leaves no durable boundary for a crash. Treating a timeout as proof of failure is equally dangerous because the remote system may already have committed the change.

Another mistake is wrapping the whole workflow in one broad retry policy. Reads, model calls, messages, and financial writes do not share the same risk. Policies belong to individual steps. Retrying should also have a budget based on elapsed time and attempts, followed by a visible terminal or review state.

Call compensation a rollback only when it truly restores the previous state. A compensation can fail, create new records, or leave the business process in an acceptable alternative state. Operator labels should make that distinction clear during an incident.

A practical next action

Choose one production workflow and list every external write it can make. For each write, add a stable idempotency key, an unknown outcome path, a checkpoint before and after the call, and either a reconciliation query or a compensating action. Then run the eight failure tests above in a non-production environment.

This exercise exposes recovery gaps without requiring a change of orchestration framework. AI workflow failure recovery is ready when a crash leads to a tested state transition instead of an improvised replay of the entire process.

References

• Temporal retry policies supports the retry interval, backoff, maximum-attempt, and non-retryable error controls discussed above.
• Microsoft Azure Architecture Center: Saga pattern supports the use of compensating transactions after partial completion across services.
• LangGraph persistence supports the checkpoint, thread, state inspection, and fault-tolerant execution concepts used in the recovery design.

---