Back to Blog
A dim operations room representing incident response for a compromised AI agent

AI Agent Incident Response: A Practical Runbook

11 min read

AI agent incident response becomes chaotic when a suspicious workflow can keep calling tools while responders argue about evidence, credentials, and stored memory. Stopping the process too early can destroy volatile context. Waiting can allow more unauthorized actions. Rotating every secret can widen the outage without removing a poisoned instruction. The runbook must put those decisions in a fixed order. First stop new authority, then preserve the user-to-model-to-tool chain, revoke exposed access, quarantine untrusted state, and require adversarial tests before restart. Security and platform teams can rehearse this procedure before an internal agent touches a consequential system.

Why a standard incident plan leaves gaps

A conventional application incident usually starts with a host, account, service, or network boundary. An agent introduces another chain of influence. A user message, retrieved document, tool response, model output, policy decision, and downstream API call may all contribute to one business action. The process can be technically healthy while making unauthorized decisions.

NIST SP 800-61 Rev. 3 integrates incident response into Cybersecurity Framework 2.0 risk management rather than treating it as an isolated cleanup task. That lifecycle still applies, but an agent runbook needs concrete procedures for model inputs, tool authority, persistent memory, and workflow replay.

An agent also accepts more than ordinary application requests. OWASP's Agentic AI threats and mitigations addresses systems where generative AI is integrated with autonomous components and business tools. Responders must investigate the software path and the content that influenced the agent.

During triage, answer each of these questions separately:

  • What input or state influenced the decision?
  • What identity and policy allowed the action?
  • What side effects reached systems of record?
  • What information is needed to reproduce the decision safely?
  • Which component can be contained without erasing that information?

A generic "restart the service" step answers none of them. The agent may reload the same poisoned memory, receive the same retrieved document, or reuse the same overbroad credential after restart.

Define incident triggers before deployment

Do not wait for an operator to decide whether strange output counts as an incident. Name observable triggers and attach a severity. Useful triggers include an attempted denied tool call, a destination outside the allowlist, a credential canary in model output, cross-tenant retrieval, repeated approval bypass attempts, an unexplained write in a system of record, or a durable memory item created from untrusted content.

Separate a quality defect from a security event. A poorly worded draft with no side effect may need evaluation and release rollback. A model that tries to read environment variables, changes a customer record outside its tenant, or sends data to an unapproved host requires containment even if the attempt fails. A failed attack is still evidence about intent and control effectiveness.

Assign four roles in the runbook:

  1. The incident commander owns severity, scope, and the restart decision.
  2. The platform responder controls workflow admission, execution, and deployment state.
  3. The security responder owns evidence handling, identity scope, and threat analysis.
  4. The business-system owner verifies downstream records and decides whether actions need reversal.

One person may fill several roles on a small team. The responsibilities still need names. Otherwise the platform team may terminate a run while security expects it to remain available for evidence, or security may revoke an account without knowing which business process depends on it.

Use five ordered containment decisions

Start with the narrowest reversible control that stops new harm while preserving enough state to investigate.

1. Stop new actions without destroying evidence

Disable admission of new workflow runs first. Then block or require manual approval for consequential tool calls from active runs. That removes authority while leaving execution records available, unlike immediately deleting containers, queues, or session stores.

Your platform should expose more than one stop control:

  • pause new workflow admission;
  • deny a specific tool or operation;
  • disable one agent or workflow version;
  • suspend one tenant or subject;
  • block outbound traffic to one destination;
  • terminate an individual run when it is still causing harm.

A single global kill switch is useful for a fast emergency stop, but it is a blunt instrument. It can interrupt unaffected workflows and create partial business transactions. Prefer a scoped pause when identity and tool boundaries remain trustworthy. Use the global stop when scope is unknown, the policy layer may be compromised, or the agent is continuing unauthorized writes.

Record the exact time and control applied. Do not claim that pausing the orchestrator stopped a downstream job unless the target system confirms it. A queued email, payment, or account change may continue after the agent process stops.

2. Preserve the decision chain

Capture volatile evidence before cleanup. The minimum chain links the initiating subject, tenant, input reference, retrieved context identifiers, prompt and workflow versions, model request identifier, normalized tool proposal, authorization decision, approval record, workload identity, tool request identifier, destination, and outcome.

The OpenTelemetry generative AI agent span specification provides fields and conventions for agent operations. Use trace context to connect diagnostic spans with a smaller security timeline. Do not assume telemetry contains everything needed for accountability. Approval, authorization, tenant, and business-object fields often belong in a dedicated audit record.

Preserve references before copying content. Raw prompts, retrieved documents, and model outputs may contain secrets or personal data. Store immutable object identifiers, hashes where they have a defined verification purpose, collection times, and access controls. Export the sensitive payload only when the investigation requires it, and keep that evidence in a restricted case store.

A compact incident record can look like this: 

{
  "incident_id": "inc_204",
  "detected_at": "2026-08-05T13:14:00Z",
  "workflow_run_id": "run_913",
  "trace_id": "trace_551",
  "subject_id": "employee_42",
  "tenant_id": "org_7",
  "agent_version": "invoice-agent-18",
  "policy_version": "tool-policy-12",
  "suspected_input_ref": "document_818",
  "tool_action": "invoice.update_bank_details",
  "target_ref": "invoice_301",
  "containment": "tool-denied-and-admission-paused",
  "credential_scope": "accounts-payable-write",
  "downstream_status": "reconciliation-required"
}

This schema is only an implementation example. Its fields must let a responder move from detection to the input, decision, authority, and side effect without searching unstructured logs for guessed phrases.

3. Revoke only the exposed authority

Map the suspected path to credentials before rotating anything. Identify the human session, workload identity, OAuth token, API key, lease, certificate, and destination involved. Ask whether the agent could read the credential, merely request an operation through a broker, or influence a service that held the credential.

The OWASP Secrets Management Cheat Sheet treats rotation, revocation, auditing, and lifecycle management as related controls. Apply them according to exposure. Revoke a readable reusable key immediately. End a compromised user session. Disable a workload identity when its policy boundary cannot be trusted. If the agent used a broker and never received the secret, block the affected operation and inspect broker decisions before disrupting every integration.

Broad rotation can create a second incident. Unrelated workflows may fail, retries may accumulate, and responders may lose access to evidence services. Document dependencies and rotation order for each credential class. Include a test that confirms old authority no longer works without printing secret values into its output.

4. Quarantine memory and retrieved state

Do not delete suspected memory records during initial containment. Move them to a quarantined namespace or deny retrieval while preserving provenance, writer identity, creation time, source reference, and prior reads. The investigation needs to know which runs consumed the item and whether it crossed a user or tenant boundary.

Quarantine all state derived from the suspect source, not the entire memory database by default. That may include embeddings, summaries, cached tool results, conversation checkpoints, and generated instructions. Mark affected indexes for rebuild if deletion from the source store would not remove derived copies.

Test whether the behavior reproduces from a clean session with the suspect item excluded. Then test the quarantined item in an isolated environment with all external tools disabled. If the bad action appears only with that state present, you have stronger scoping evidence. If it persists, inspect prompt versions, tool descriptions, policies, and model routing rather than assuming memory was the sole cause.

5. Gate the restart with evidence

A successful process start does not justify restoring traffic. Require an incident-specific gate:

  • the suspected input or state is blocked or corrected;
  • exposed credentials are revoked and replacements are confined;
  • affected business records are reconciled;
  • the vulnerable workflow or policy version cannot receive traffic;
  • denied-action tests fail closed;
  • allowed control cases still complete correctly;
  • telemetry and audit events capture the full test chain;
  • an owner accepts the remaining risk and rollback plan.

Use a new run, new session, and clean memory namespace for the first test. Route it to a nonproduction target or a simulator. Then canary a small, identified cohort with consequential actions held for approval. Do not restore every tenant because one synthetic prompt produced the expected answer.

Work through one invoice-agent incident

Suppose an invoice agent retrieves a supplier document containing an instruction to replace bank details. The model proposes an update, policy allows the tool, and the accounts-payable API returns success. An anomaly rule notices that the destination account differs from the verified supplier record.

The incident commander classifies this as a suspected unauthorized write. The platform responder pauses new invoice-agent runs and denies invoice.update_bank_details across active runs. The business owner places affected invoices on payment hold. Security captures the source document reference, retrieval event, workflow and prompt versions, model and tool spans, policy decision, service identity, API receipt, and every other run that retrieved the document.

The service credential was held by a broker and was not visible to the agent. Responders therefore block the operation and revoke the issued short-lived token rather than rotate unrelated API credentials. They quarantine the document, its extracted text, embeddings, and summaries. The accounts-payable owner compares every target touched by the affected runs with the verified supplier register and reverses unauthorized changes through the system's normal controlled process.

For scoping, the team can use MITRE ATLAS as a knowledge base for adversary tactics and techniques involving AI-enabled systems. It does not replace evidence from this incident, but it helps responders ask whether the observed behavior suggests input manipulation, credential access, discovery, collection, or impact beyond the first alert.

Before restart, the team adds a rule that bank-detail changes require a verified source record and human approval. In an isolated replay, the malicious document still influences the model, but the normalized action is denied. A clean invoice succeeds. Both paths produce linked audit and trace records. The team then canaries the corrected version with writes held for review.

Plan for failure during response

Every containment step needs confirmation. A pause request may affect only one worker pool. A tool deny rule may not cover an alternate route. Revocation may race with an in-flight request. Evidence export may time out or copy sensitive data to the wrong store.

Build confirmation into each step. Query the orchestrator for active runs after pausing admission. Check the egress gateway and downstream API for calls after the containment timestamp. Test revoked authority against a harmless endpoint and record only the result. Compare systems of record with tool completion events instead of trusting the agent's final message.

If telemetry is incomplete, preserve what exists and label the gap. Do not invent a clean timeline from model output. If the policy service itself may be compromised, stop all consequential tools and move approvals to an independent path. If quarantine cannot reliably exclude derived state, rebuild from a known clean source rather than attempting selective reuse.

There is a tradeoff between evidence preservation and immediate termination. Ongoing harmful actions win that decision: terminate first. When authority has been removed and the run is stable, preserve state before destructive cleanup. Put that rule in writing so responders do not debate it during the event.

Rehearse the runbook and verify the controls

Run a tabletop exercise with a synthetic prompt injection, a canary credential, and a reversible tool action. The platform responder should pause admission and deny the target operation without editing code. Security should reconstruct the subject, model, policy, tool, credential, and target chain from stored identifiers. The business owner should find the side effect in the target system and reverse it through an approved procedure.

Then test negative cases. Try an alternate tool path, another tenant, a replayed request, stale memory, a redirected destination, and a run already in progress when admission stops. Each test should have an expected containment point and observable denial. Add any unobserved path to the runbook and architecture backlog.

Measure the exercise with outcomes rather than a generic pass label: time to stop new writes, percentage of affected runs identified, number of unexplained downstream actions, time to revoke exposed authority, and whether the restart suite caught the original path. These are local operational measurements, not industry benchmarks.

After the exercise, hold a lessons review and update trigger definitions, ownership, evidence fields, and restart checks. That feedback loop follows the risk-management approach in NIST SP 800-61 Rev. 3. A runbook that never changes after an exercise is documentation, not an operating control.

Start with one consequential tool

Choose the agent action with the largest irreversible or customer-facing effect. Write its incident trigger, scoped stop control, evidence chain, credential map, state-quarantine path, downstream reconciliation owner, and restart tests. Rehearse a malicious-input scenario with that tool in a safe environment. Do not add another consequential action until the team can stop the first one, explain exactly what happened, and prove the repaired workflow fails closed.

References

About Fire In Belly: Independent senior engineering from Tallinn, Estonia. We design and build AI workflow automation with the scoped containment, evidence preservation, and restart gating described above, at published fixed prices. Schedule a call to discuss your next project.