AI Agent Dead Letter Queue: A Recovery Contract for Stuck Jobs

When an AI agent job exhausts retries and lands in a classic dead-letter queue, the work is not done. You still have partial side effects, burned tokens, and an operator who must decide what happens next. Amazon SQS dead-letter queues and Azure Service Bus dead-letter queues solve the broker problem: stop redelivering a message forever after maxReceiveCount or MaxDeliveryCount is exceeded. They do not solve the agent problem. An agent task is a stochastic multi-step trajectory rather than a static payload. Blind redrive can succeed or fail for reasons unrelated to any fix you shipped. A poison attempt can also burn thousands of tokens looping before it fails again.

This guide is the post-exhaustion recovery contract for one internal queued agent workflow. It complements AI workflow failure recovery, which owns in-flight retries, checkpoints, and compensating actions. It stops before hermetic production-trace replay gates, which turn a diagnosed failure into a CI fixture after the operational recovery path is clear. Here the job is narrower: classify the dead-lettered item, bound spend, reconcile UNKNOWN effects, resume safely, and keep the DLQ visible to an owning team.

What a broker DLQ assumes, and where agents break it

Broker DLQs assume the unit of work is bytes you can redeliver identically. SQS moves a message after a configurable receive count. Service Bus dead-letters after a delivery count and records a reason such as MaxDeliveryCountExceeded. Later, an operator redrives or resubmits. That mental model is sound for deterministic handlers.

Agent work breaks it in two places that dreaming.press documents for AI agent tasks:

1. Redrive is a fresh sample. Replaying a message re-runs the same handler on the same bytes. Re-running an agent draws a new path through model calls and tools. "It worked on retry" may be variance, not proof that you fixed the cause.
2. Poison cost is unbounded. A classic poison message wastes one handler execution per receive. A poison agent can re-plan, re-call tools, and burn tokens on every attempt. Receive count alone is queue hygiene. For agents it must also be a spend circuit-breaker.

If your platform already uses Temporal-style durable execution with non-retryable error types, permanent failures may surface as inspectable failed workflows rather than broker messages. The contract below still applies whenever agent jobs ride a plain queue, job runner, or cloud task system that inherited message-era DLQ semantics.

Build a decision-ready dead-letter record

Do not park only the original enqueue payload. Keep the facts an operator needs before any button labeled "replay" is safe to press. A minimal record looks like this:

global_id: 01J8DEADLETTER0001
run_id: run_4821
job_id: job_support_triage_91
attempt: 4
failure_class: UNKNOWN
status: DEAD_LETTERED
first_failed_at: 2026-09-21T11:00:00+03:00
last_failed_at: 2026-09-21T11:12:00+03:00
tokens_in: 18420
tokens_out: 6120
steps_completed: 5
steps_attempted: 7
spend_usd_estimate: 1.84
effect_ids:
  - effect: create_ticket
    provider_request_id: zendesk_req_771
    idempotency_key: run_4821:create_ticket:v1
    disposition: UNKNOWN
payload_digest: sha256:4f2c...
checkpoint_id: cp_step_5
policy_version: policy-42
credential_version: github-token-v7
prompt_version: support-triage@3.2.1
model: gpt-4.1-mini
trace_id: tr_9f3a
tenant_id: acme
next_action: NEEDS_RECONCILIATION

Keep the row immutable except for audited fields such as status, operator decision, reconciliation result, and next action. Link job_id → run_id → trace_id so on-call can jump from the DLQ UI into the exact tool span. Redact PII in the operator view even when the encrypted payload snapshot remains available for authorized recovery.

Praesidia's agent DLQ guidance frames the same requirement as answering three questions without a database console: why it failed, what else already happened, and who is waiting. If the console cannot answer those, it is a storage bucket, not a recovery surface.

Classify before you choose an automatic action

Derive retry behavior from an explicit failure class, not from the last exception string. A practical taxonomy, aligned with the DEV recovery-contract write-up, is:

Tag the class at write time when the worker dead-letters the job. Ambiguous errors should stay ambiguous rather than being forced into TRANSIENT so an automated drain can replay them. Tian Pan's framing of silent async agent failures treats the DLQ as a control plane: routing by failure class also becomes telemetry for prompt drift, injection attempts, and input-distribution shifts.

Bound quarantine with a spend circuit-breaker

Receive-count thresholds are the wrong meter for poison agent tasks; quarantine must also bound tokens and steps as a spend circuit-breaker. Implement three meters together:

1. Receive or attempt count for broker compatibility.
2. Cumulative tokens and estimated USD across attempts for the same job_id.
3. Step or tool-call count so a tight loop cannot hide behind a low receive count.

When any meter trips, move the job to DEAD_LETTERED with failure_class set from the terminal error, and refuse further automatic retries until an operator decision lands. A second trip after a replay must mark replay_failed and block another automated drain. An automated drain pointed at a DLQ full of poison messages is a cost incident, not a recovery feature.

This is the control most teams skip because broker defaults already "have a DLQ." Those defaults meter delivery, not inference spend.

Separate reconciliation from replay

UNKNOWN is not a failure verdict: if the worker stopped during an effect window, reconcile the external system before creating any new attempt. Give operators two different actions:

• Reconcile: ask the external system whether the effect happened, using a stable effect ID or provider request ID.
• Replay: create a new attempt only after the system has established that replay is safe.

A useful state machine:

DEAD_LETTERED
  ├─ TRANSIENT            → RETRY_ELIGIBLE → NEW_ATTEMPT
  ├─ UNKNOWN              → RECONCILIATION_REQUIRED
  │                         ├─ effect confirmed → COMPLETE
  │                         ├─ effect absent + safe → RETRY_ELIGIBLE
  │                         └─ no answer → UNKNOWN_HOLD
  ├─ POLICY_DENIED        → APPROVAL_REQUIRED
  ├─ POISONED_INPUT       → INPUT_REPAIR_REQUIRED
  ├─ CREDENTIAL_EXPIRED   → REAUTH_THEN_REEVALUATE
  └─ NON_RETRYABLE        → OPERATOR_DECISION

If the provider cannot answer, keep the item in UNKNOWN_HOLD. Turning uncertainty into a duplicate ticket, charge, or calendar event is worse than leaving the work visibly blocked.

Every recovery decision needs its own idempotency key so a double-click or two operators cannot create two replay attempts:

def start_recovery(item_id: str, decision, operator_id: str):
    key = f"recover:{item_id}:{decision.version}"
    with transaction():
        row = insert_decision_if_absent(
            key=key,
            item_id=item_id,
            decision=decision.kind,
            operator_id=operator_id,
        )
        if not row.created:
            return row.result
        assert current_status(item_id) == "DEAD_LETTERED"
        assert decision_matches_current_versions(item_id)
        return create_recovery_attempt(item_id, key)

Re-check authority at recovery time. A dead-letter item may sit for hours. Tenant scope, policy version, credential lease, and tool capability can all change while the row waits. Never assume the original authorization is still valid.

Resume from the checkpoint; do not restart from zero

Blind broker redrive of an agent trajectory is a fresh stochastic sample, not a deterministic retry, so resume-from-checkpoint with per-step idempotency must beat restart-from-zero. A job that failed at step seven may already have created a ticket, posted a Slack message, or spent budget on enrichment. Restarting from step one duplicates those effects.

Require:

1. A durable checkpoint after each side-effecting step.
2. Per-step idempotency keys derived from run_id and step identity.
3. A side-effect ledger keyed by (job_id, action_type, idempotency_key).
4. An operator choice between resume from checkpoint and new run with a fresh idempotency scope after input or policy repair.
5. Documented compensating actions for irreversible tools before any production replay path ships.

This is where the DLQ contract meets the in-flight recovery work from the failure-recovery guide. The DLQ does not replace checkpoints. It refuses to pretend that a parked trajectory is a clean message you can safely re-enqueue unchanged.

When resume is impossible because intermediate state is gone or the world changed, the honest options are compensate-then-restart or discard-with-reason. Discard-with-reason must be a first-class disposition. If the only way to clear a stuck alert is a bulk purge, the record of what was lost goes with it.

Alert on depth, rate, and age with an owner

An unwatched DLQ is silent data loss. Start with these alerts:

• Depth greater than zero routed to the owning team, not a general channel where every service competes.
• Dead-letter rate above baseline to catch active incidents rather than isolated bad payloads.
• Age of oldest message to catch work that sat through a weekend without triage.

Retain DLQ payloads long enough to diagnose and act, and remember they may contain personal data subject to retention and erasure rules. Do not give the DLQ its own DLQ. Terminal state plus monitoring beats another parking lot.

Worked example: support-triage agent on a queue

A support-triage agent dequeues a customer email, classifies urgency, creates a ticket, drafts a reply, and posts an internal summary.

1. Attempt 1 times out after create_ticket accepts the request but before the worker persists the provider ID. Class: UNKNOWN. Effect disposition: unknown.
2. Attempts 2-3 hit rate limits on the draft step. Tokens accumulate. Class remains mixed; spend meter approaches the budget.
3. Attempt 4 exceeds the token budget while retrying a malformed attachment parse. Worker dead-letters with POISONED_INPUT for the attachment path and UNKNOWN still open for the ticket effect.
4. Operator console refuses one-click redrive. It requires reconciliation of zendesk_req_771. Ticket exists → mark effect COMPLETE, do not recreate.
5. Operator repairs the attachment validator, chooses resume from checkpoint after ticket creation, and issues recovery decision recover:01J8...:v2.
6. New attempt inherits a fresh idempotency scope for remaining write tools, skips completed ticket creation via the ledger, and finishes the draft under the spend budget.
7. If the same poison signature returns, mark replay_failed and block automated drain until code or input changes again.

The same flow would be unsafe if the console only exposed "redrive to source queue."

Implementation checklist

1. Confirm every agent queue has a DLQ and an owner.
2. Emit decision-ready dead-letter records with trajectory, spend, effect IDs, versions, and digests.
3. Classify failures at write time; keep UNKNOWN distinct from TRANSIENT.
4. Bound automatic retries by attempts, tokens, steps, and estimated USD.
5. Implement reconcile and replay as separate operator actions with recovery idempotency keys.
6. Resume from checkpoints with per-step idempotency; document compensation for irreversible tools.
7. Alert on depth, rate, and age; support discard-with-reason.
8. Add failure-injection tests: stop after provider accept, expire credentials before replay, dual-operator clicks, and provider-no-answer reconciliation.
9. Link recovered or discarded items into the path that can later become a hermetic CI fixture when the failure class is worth preventing at merge time.

Verification

You have a working contract when an operator can pick a dead-lettered agent job and, without a database console, state the failure class, whether any external effect already happened, whether replay is within budget, which checkpoint to resume from, and which authority versions were re-checked. Broker-only redrive of the original payload should be impossible or strongly gated for agent queues. Automated drains should replay only clearly transient classes with a loop guard. Depth alerts should fire on the first parked job, not on the thousandth.

Common mistakes

• Treating the broker DLQ as sufficient because "we already have redrive."
• Metering only receive count while poison loops burn tokens.
• Offering a single retry button for every failure class.
• Restarting multi-step agent jobs from zero after partial side effects.
• Leaving DLQ depth unowned until finance notices the spend spike.
• Purging without discard-with-reason so nobody can reconstruct what was abandoned.
• Expecting the DLQ to catch semantic success-with-wrong-output; that still needs quality gates and judges.

Next action

List every side-effecting tool your queued agent can call. For each one, define the effect ID, idempotency key, reconciliation query, and compensation or human cleanup path. Do not ship a DLQ replay button until that inventory exists. The recovery contract is only as strong as the side-effect ledger behind it.

References

1. Amazon SQS dead-letter queues - Official broker contract for parking repeatedly failed messages via maxReceiveCount and later redrive.
2. Azure Service Bus dead-letter queues - Official documentation of dead-letter subqueues, MaxDeliveryCount, reasons, and operator resubmission.
3. Dead Letter Queues for AI Agent Tasks (dreaming.press) - Why agent redrive is a fresh stochastic sample and poison thresholds must bound tokens and steps.
4. Your AI Agent's Dead-Letter Queue Needs a Recovery Contract (DEV) - Failure classes, reconcile-before-replay, recovery idempotency, and authority re-checks.
5. Dead-Letter Queues for Failed Agent Jobs (Praesidia) - Depth alerts, poison-versus-transient triage, resume-not-restart, and loop guards.
6. Silent Async Agent Failures (Tian Pan) - DLQ as a control plane with failure-class routing and per-step idempotency.
7. Fire in Belly AI workflow failure recovery - First-party in-flight recovery baseline that this DLQ contract extends.

---