Webhook Security for AI Workflows: Verify Every Trigger

Webhook security breaks before the model runs. A public callback can look valid while being forged, replayed, delivered twice, or delivered after a newer event. If your handler parses and queues that request before it authenticates the exact bytes, one bad callback can spend model tokens, leak record context, send a message, or change a business system. The fix is an admission controller that verifies the sender and tenant, rejects stale attempts, reserves one durable event identity, validates the event, and only then queues AI work. This guide builds that boundary and shows how to prove that rejected callbacks never reach a model or tool.

Why ordinary request handling is not enough

A webhook is a public HTTP request initiated by another service. Session cookies and an interactive login do not protect it. The receiver needs a separate trust mechanism, usually a signature created with an endpoint key or a provider private key. The Standard Webhooks specification treats authenticity as a receiver requirement and signs the message ID, delivery timestamp, and body together.

Signature verification solves only part of the problem. A captured valid request is still valid cryptographically when an attacker sends it again. A provider may also retry after your server completed the work but lost the response. Two workers can receive the same event at once. Events about one resource can arrive out of order. A valid webhook therefore proves who sent specific bytes. It does not prove that the event is fresh, new, ordered, authorized for a tenant, or safe to execute.

AI makes the same failure more expensive. A duplicate callback might update an ordinary service row twice. In an AI workflow, it can also start another model call, produce a different proposal, retrieve sensitive context, and invoke several tools. Keep the callback outside the AI trust boundary until the receiver records a durable admission decision.

Define the webhook security contract

Write the receiver contract before implementing provider adapters. The contract should answer these questions for every endpoint:

• Which provider, provider account, internal tenant, and environment can send here?
• Which exact request bytes and headers are covered by the signature?
• Which key version verifies the request, and how does rotation overlap work?
• How old may a delivery timestamp be?
• Which provider value is the stable event or delivery ID?
• Which event types and schema versions are accepted?
• When is an event safely acknowledged?
• How are duplicates, stale events, and gaps reconciled?
• Which durable record proves why an event was admitted or rejected?

Do not derive tenant identity from an unverified JSON field. Route the callback through a provider and endpoint identifier that maps to server-side configuration. That configuration selects the provider adapter, tenant, environment, allowed event types, and active verification keys. The payload can confirm the expected account after verification, but it cannot choose the key used to verify itself.

Store endpoint keys in a secrets manager and expose them only to the ingress service. Model prompts, traces, job payloads, and general application logs do not need them. When symmetric keys rotate, accept the old and new key for a short planned overlap, record which version matched, and remove the old key after provider delivery has switched. Standard Webhooks permits multiple signatures specifically to support rotation without an abrupt outage.

Authenticate the exact request bytes

Capture the raw body before JSON parsing, normalization, decompression, or character conversion. Stripe warns that signature verification requires the raw request body and that framework manipulation can make verification fail in its webhook receiver guide. The same rule applies to any scheme that signs serialized bytes: parsing and reserializing JSON can alter spaces, key order, escapes, or Unicode representation.

Verification belongs in a provider adapter because header names and signed inputs differ. The shared ingress interface can require a result with these fields:

VerifiedEnvelope {
  provider
  endpoint_id
  tenant_id
  environment
  event_id
  event_type
  delivery_timestamp
  key_version
  raw_body_digest
}

The adapter should reject a missing signature, unknown key version, malformed timestamp, invalid encoding, body over the configured limit, or failed digest comparison. GitHub's webhook validation documentation specifies HMAC over the payload and warns against a plain equality operator. Use the provider SDK or a maintained cryptographic library, then compare signatures with a constant-time function.

Check the delivery timestamp only after its signature is valid. Otherwise an attacker can supply any timestamp. Reject attempts outside the endpoint's replay window, allowing for a documented amount of clock skew. Keep clocks synchronized and monitor drift because a broken clock can reject legitimate events or widen the effective replay window.

A timestamp does not replace event deduplication. A legitimate provider retry normally has a fresh attempt time or may retain the same delivery identity, depending on its contract. Preserve the provider's stable ID across attempts and use it as the durable idempotency key.

Reserve the event before acknowledging it

The receiver has one commit point between cryptographic verification and its successful HTTP response. Insert an ingress record with a unique key such as (provider, endpoint_id, event_id). Create or link the durable queue job in that same database transaction. Return success only after the commit completes.

The shape can remain small:

{
  "provider": "example_saas",
  "endpoint_id": "ep_42",
  "tenant_id": "tenant_17",
  "event_id": "evt_9041",
  "event_type": "case.updated",
  "delivery_timestamp": "2026-08-26T08:30:00Z",
  "admission": "queued",
  "key_version": "v3",
  "body_digest": "sha256:<digest>",
  "job_id": "job_551"
}

The values are illustrative. Do not log the signature, verification key, or full body merely to make debugging easier. Store the minimum event metadata and a body digest. If the job needs the body, encrypt it under the tenant's normal data controls or fetch current state from the provider with a scoped API credential.

A unique constraint, not a cache lookup, decides whether the event is new. Two concurrent requests can both miss a cache check and both enqueue work. Let one insert win. The other reads the existing admission record and returns the provider-appropriate success response without creating another job.

Acknowledgement and processing split at this point. Stripe says to return a successful status before complex business logic. GitHub's webhook best practices recommend a quick response and asynchronous queue processing. Acknowledge after durable admission, rather than on socket receipt or after model inference. An early response can lose work if the queue write fails. A late one invites retries while an expensive model call is still running.

Validate before any AI work begins

Cryptographic authenticity does not make every provider event relevant. After verification, parse JSON with explicit size and depth limits. Validate the event type, action, account identity, and schema version against the endpoint configuration. Reject or quarantine unknown types instead of passing a loose object to a prompt.

Convert the provider payload into a narrow internal command. For example, a CRM event might become RefreshLeadAssessment(tenant_id, lead_id, provider_version). The command should contain identifiers and trusted routing metadata, not arbitrary instructions copied from the event. Fetch the current record through the provider API when the workflow needs authoritative state.

The translation matters because webhook text can contain customer prose, ticket bodies, repository content, or attacker-controlled instructions. Treat that text as untrusted data. It must not alter system prompts, select tools, choose credentials, or authorize side effects. The worker should load its fixed workflow definition, scoped tenant credentials, and permitted actions independently of the callback body.

Tool authorization still runs at execution time. An admitted event may trigger evaluation, but it does not authorize a payment, deletion, publication, or outbound message by itself. Validate tool arguments against current policy and require human approval where the action needs it. Webhook admission establishes provenance, not business authority.

Handle duplicate and out-of-order events

Deduplication stops the same stable event ID from creating two jobs. It does not solve two different events about the same resource arriving in the wrong order. Store the provider's resource version, sequence, or event creation time when available. Before mutation, compare it with the last applied version for that resource.

Use one of three rules:

1. If the provider exposes a monotonic version, apply only a newer version under a row lock or conditional update.
2. If the event is a thin notification, fetch current provider state and calculate the action from that state.
3. If neither is available, serialize events per resource and reconcile ambiguous transitions before allowing a side effect.

Never use arrival time as proof of business order. Network retries can make an older event arrive later. For destructive or externally visible actions, bind the proposal to the resource version used during inference and recheck that version before execution. If the state changed, discard the proposal or send it through a fresh evaluation.

Manual redelivery must use the same path as automatic delivery. GitHub documents that a redelivery keeps the same X-GitHub-Delivery value, which lets the receiver recognize the original event. A bypass endpoint that ignores the unique key turns recovery tooling into a duplicate-action control failure.

Decide each failure response deliberately

Provider retry rules differ, so keep response policy in the adapter. The internal admission states should still be consistent:

• Invalid signature, unknown endpoint, wrong account, or stale timestamp: reject without queueing.
• Valid signature but malformed or disallowed event: reject or quarantine according to a documented contract.
• Duplicate event with a committed job: acknowledge without creating work.
• Verified new event with successful record and queue commit: acknowledge.
• Database or queue transaction failure: return a retryable failure and create no partial admission.
• Admitted event whose worker later fails: retry the durable job, not the original public request.
• Older resource version: record it as stale and perform no side effect.

Do not return success after placing work in an in-memory queue. A process crash between acknowledgement and persistence loses the event. Do not return a retryable error after committing a job unless a duplicate delivery is guaranteed to hit the unique constraint safely.

Test the boundary with negative cases

A passing happy-path event does not test the dangerous branches. Build fixtures from each provider's documented signing process and run them through the real ingress code. The Svix verification guide provides a maintained example of receiver-side signature and timestamp handling, but your fixtures must match the provider connected to the workflow.

Cover these cases:

• One byte changes after signing, so verification fails and no ingress row exists.
• JSON middleware parses the body before verification, and the integration test catches the configuration error.
• A valid event uses a timestamp outside the replay window.
• The right signature reaches the wrong tenant endpoint.
• Old and new rotation keys work during overlap, then the old key stops working.
• Two concurrent copies of one event create one ingress row and one queue job.
• The database commit fails, so the handler does not acknowledge success.
• A provider redelivery receives success but does not start another model call.
• A newer resource version is applied before an older event arrives.
• An event body contains tool-like instructions, but the worker treats them only as data.
• An admitted job proposes a side effect after the resource changed, so execution is blocked.
• Logs and traces contain no signature, verification key, or unrestricted payload.

Instrument admission outcomes by provider, endpoint, tenant, event type, and reason code. Alert on signature-failure spikes, timestamp rejections, unknown event types, queue commit failures, old rotation-key use after cutover, duplicate rates, stale events, and admitted jobs that never reach a terminal state. Keep payload content out of metric labels.

Start with one real callback

Pick one webhook that currently starts an AI workflow. Trace it from the public route to the first model call or tool execution. Insert the admission controller before parsing, then add the durable unique event record and queue transaction. Replay one signed fixture concurrently, send one stale fixture, alter one body byte, and deliver an older resource event after a newer one.

Webhook security is working when only the valid new event creates a job, duplicates resolve to the original admission, stale or forged requests create no work, and every tool call can be traced to a verified event and current business authorization. Run that drill on the real ingress path before connecting the next provider.

References

1. Standard Webhooks specification supports timestamped signatures, stable event identity, replay checks, constant-time comparison, key rotation, retries, and delivery recovery.
2. Stripe webhook guide supports raw-body signature verification, quick acknowledgement, endpoint configuration, and receiver testing.
3. GitHub webhook validation supports HMAC payload verification, UTF-8 handling, and constant-time signature comparison.
4. GitHub webhook best practices supports HTTPS, minimal subscriptions, asynchronous processing, event checks, redelivery, delivery identity, and replay protection.
5. Svix payload verification supports maintained receiver-side signature and timestamp verification patterns.

---