AI Data Quality Gates: Stop Bad Input Before the LLM Acts

AI data quality failures rarely look like broken JSON. A renewal workflow can receive a valid CRM record, a valid billing balance, and a valid support summary, then reason from facts captured on different days. The model still writes a confident recommendation. The failure appears later as a bad discount, a needless escalation, or an approval request built on evidence that was never true at one point in time.

Stop that failure before the model call. Build a deterministic gate that freezes the source evidence, checks whether it is fit for one named decision, and routes defects without asking the model to judge its own inputs. This guide defines the contract, outcomes, replay path, and tests.

Why valid records can still make a bad decision

A schema validator answers structural questions. It can require an account ID, constrain a balance to a number, and reject an invalid date. The JSON Schema guide shows how required properties, types, and nested constraints form a machine-readable contract. That contract cannot tell you whether a balance is current enough for a credit decision or whether the support cases belong to the same account version.

API contracts have the same limit. The OpenAPI Specification describes operations, parameters, responses, and schemas. It helps an adapter reject malformed provider responses. A workflow can nevertheless combine three individually conforming responses into a misleading whole.

Consider an internal renewal-risk workflow:

• The CRM says the customer is on an annual enterprise plan.
• Billing says the account is overdue, based on a snapshot from yesterday morning.
• Support says a priority incident is open, based on a ticket read five minutes ago.
• The CRM plan changed to monthly after billing produced its snapshot.

Every field parses. Each source may even be within a generic freshness threshold. The assembled evidence still describes two business states. Sending it to an LLM turns a temporal join error into persuasive prose.

The gate must answer a decision-specific question: is this exact bundle of evidence complete, fresh, related, and mutually consistent enough to support the renewal action?

Define a decision contract before a data contract

Start with the business decision, not the source payloads. A decision contract names the outcome the workflow may produce and the facts required to justify it. For a renewal-risk recommendation, the contract might require:

• a trusted tenant and account identity
• the active subscription version and effective date
• current balance, currency, and payment status
• open high-priority support incidents
• the last customer-contact timestamp
• a maximum allowed age for each fact
• consistency rules between account, subscription, invoice, and ticket identities
• the policy version that determines whether partial evidence is permitted

This list is narrower than a source schema. It contains only facts that can change the decision, authorization, routing, money, or an external side effect. Keep descriptive fields out unless the workflow actually uses them.

Attach a version to the contract. Store that version with every gate result and model request. If a freshness limit or required fact changes, operators can identify which historical decisions used the old rule. Versioning also makes replay explicit: a replay either applies the original contract for diagnosis or a newer contract for correction.

JSON Schema can enforce the structural portion. Deterministic code should enforce temporal, relational, and business invariants. Do not put a prompt between raw data and the acceptance decision. An LLM may help explain a rejection to an operator, but it must not waive a missing account identity or decide that yesterday's balance is probably good enough.

Freeze one decision snapshot

A quality gate should validate one immutable decision snapshot, not a collection of independently fresh fields.

Source adapters normally return current objects independently. The gate needs a manifest that binds every fact to the version the workflow observed. The W3C PROV model separates entities, activities, and agents, which is a useful basis for identifying source versions and the assembly activity. OpenLineage uses event-oriented jobs, runs, datasets, and extensible facets, which can connect the gate outcome to a workflow run without copying sensitive payloads into telemetry.

A snapshot manifest can look like this:

{
  "snapshot_id": "renewal/acme/2026-09-14T09:15:00Z",
  "decision": "renewal_risk_recommendation",
  "contract_version": "renewal-risk-v4",
  "assembled_at": "2026-09-14T09:15:00Z",
  "sources": [
    {"role": "account", "id": "crm:account-24", "version": "108", "observed_at": "2026-09-14T09:14:57Z"},
    {"role": "billing", "id": "billing:customer-991", "version": "2026-09-14T09:10Z", "observed_at": "2026-09-14T09:14:58Z"},
    {"role": "support", "id": "support:org-24", "version": "8841", "observed_at": "2026-09-14T09:14:59Z"}
  ],
  "payload_ref": "evidence://protected/renewal/acme/0915",
  "payload_hash": "sha256:example-only"
}

The identifiers are illustrative. In production, use immutable source revisions where available. If a system exposes no revision, capture a canonical content hash and observation time. Keep raw records in a protected evidence store, not in ordinary logs. The manifest should hold references, versions, timestamps, classifications, and a digest needed for integrity checks.

Validate temporal coherence as well as individual age. A billing snapshot that is six hours old may be allowed, but not if the subscription changed during those six hours. Use source change tokens, effective dates, or a second lightweight version check immediately before accepting the snapshot. If a source changes during assembly, discard the bundle and start a bounded refresh rather than mixing versions.

Implement the AI data quality gate

Run checks in a stable order so the same evidence and contract produce the same result. Cheap checks should fail before expensive refreshes or model calls.

1. Verify trusted tenant, workflow, and decision identity.
2. Parse each source through its versioned adapter.
3. Validate required fields and types.
4. Check referential integrity across account, subscription, invoice, and ticket IDs.
5. Apply per-fact freshness limits.
6. Check effective dates and cross-source consistency.
7. Confirm that source versions did not change during assembly.
8. Persist the manifest and gate result before inference.

The gate returns a typed disposition and reason codes:

def evaluate_snapshot(snapshot, contract, now):
    structural = contract.validate_structure(snapshot)
    if structural.errors:
        return quarantine(structural.errors)

    missing = contract.required_facts_missing(snapshot)
    if missing:
        return clarify([f"missing:{fact}" for fact in missing])

    stale = contract.stale_facts(snapshot, now)
    if stale:
        return refresh([f"stale:{fact}" for fact in stale])

    conflicts = contract.cross_source_conflicts(snapshot)
    if conflicts:
        return quarantine([f"conflict:{rule}" for rule in conflicts])

    if source_versions_changed(snapshot):
        return refresh(["snapshot_changed_during_assembly"])

    return accept(snapshot.snapshot_id, contract.version)

The accept result should carry the snapshot ID and contract version into the model request. Any prompt, retrieval step, approval preview, and later write can then point back to the same accepted evidence.

Route each defect to the right recovery path

Gate outcomes should be typed as accept, refresh, clarify, or quarantine instead of collapsing every defect into retry.

A generic retry hides ownership and can make the failure worse. Repeatedly fetching a permanently missing customer field wastes capacity. Retrying inconsistent records without changing the read strategy may assemble a different mixture each time. Treat the four outcomes as workflow states:

• accept releases the immutable snapshot to inference.
• refresh starts a bounded reread because required facts are older than policy allows or changed during assembly.
• clarify requests a missing business fact from the responsible person or system.
• quarantine isolates malformed, contradictory, wrongly scoped, or untrusted evidence for operator review.

Give every non-accept result machine-readable reason codes, an owner, a service target, and a safe resume checkpoint. The model must not run while the state is unresolved. A refresh needs an attempt cap and deadline. When it cannot produce a coherent snapshot, it should become a clarification or quarantine item rather than loop.

Preserve the rejected manifest and protected payload reference. A practitioner implementation plan in the Besser-Bahn issue describes raw snapshots, drift classification, quarantine, monitoring, and regression fixtures. It is an author report, not a product guarantee, but the operational pattern is useful: without the exact rejected input, engineers cannot reproduce the defect or prove that a repair handles it.

Replay corrected evidence under a new run ID. Keep a link to the rejected snapshot and use the workflow's existing idempotency key for any eventual side effect. The gate itself should not send messages, alter records, or reserve money. Separating acceptance from execution lets a corrected snapshot resume without duplicating an action that another path already completed.

Test the gate against a realistic failure matrix

A happy-path fixture proves almost nothing. Build cases that target the join between systems and the time between reads:

• required account identity is absent
• billing currency differs from the subscription currency
• invoice belongs to another tenant
• a support ticket is duplicated under two source IDs
• one source is stale but refreshable
• a required fact does not exist and needs clarification
• two current sources disagree on subscription status
• the CRM version changes after billing is read
• a source adapter returns valid JSON with an unknown enum
• a quarantine item is repaired and replayed once
• an evidence pointer expires before review
• a refresh reaches its attempt or time limit

For every fixture, assert more than the disposition. Check the reason code, contract version, snapshot identity, protected evidence reference, metric, owner, resume checkpoint, and absence of model or tool execution. A thrown validation exception does not prove that inference was blocked.

Release tests should measure escaped invalid inputs and false rejections, including records assembled across source-version boundaries.

Teams often count rejected records and call the gate effective. That count cannot show whether the gate missed bad evidence or blocked valid work. Label a representative replay set with the expected disposition. Measure the share of invalid snapshots incorrectly accepted, the share of valid snapshots incorrectly blocked, and results by rule, source, tenant class, and decision type. Review samples from both sides.

Add a race test that changes a source version between reads. The accepted manifest must never contain the old billing state beside the new subscription state when the contract requires coherence. Also test the opposite risk: harmless metadata updates should not cause an endless refresh if they do not affect a decision-critical fact.

Start in observation mode. Build manifests and proposed outcomes, but do not block production work yet. Compare proposed rejections with operator decisions and downstream outcomes. Tighten rules that let invalid snapshots escape, and narrow rules that reject valid work. Then enforce one decision type for a small internal group with a deterministic fallback.

Operate the gate without creating a new blind spot

Track outcomes by decision type, contract version, rule, source, and environment. Useful operational measures include accepted snapshots, refresh attempts, clarification age, quarantine age, escaped-invalid rate from reviewed samples, false-rejection rate, and the share of model calls carrying an accepted snapshot ID.

Keep customer values out of metric labels. Use bounded identifiers or classifications. Raw records belong in the protected evidence store with tenant-aware access and retention. A lineage event can reference the gate result and snapshot without revealing the payload.

Alert on the first failure of a high-risk identity or tenant rule. Use rate-based alerts for lower-risk freshness defects. A sudden increase in refreshes may indicate a source outage, a delayed ingestion job, or a contract that became stricter than the source can satisfy. The response differs, so the reason code must survive aggregation.

Treat contract changes like code changes. Run the labeled replay set, compare old and new dispositions, inspect protected-field differences, and canary the new version. Keep the prior contract available for diagnosis, but do not allow two versions to release side effects for the same snapshot.

Start with five decision-critical facts

Choose one internal workflow that can update a durable record, route a high-value case, or trigger an external action. Write down the five facts it must know before it is allowed to call a model. For each fact, name the trusted source, identity link, allowed age, source version, and what should happen when the fact is missing or inconsistent.

Build the snapshot manifest and run the gate in observation mode against recent protected examples. Do not start by creating a universal data-quality platform. Prove that one decision cannot reach inference without a coherent, versioned evidence bundle, then turn the observed defects into contract rules and release fixtures.

References

1. JSON Schema getting started supports the structural contract for required fields, types, and nested records.
2. OpenAPI Specification supports machine-readable API operation and schema contracts.
3. W3C PROV overview supports the entity, activity, agent, usage, generation, and derivation model for source versions.
4. OpenLineage documentation supports event-oriented job, run, dataset, and facet metadata for connecting gate results to workflow executions.
5. Besser-Bahn issue 3 is a practitioner implementation plan for snapshots, drift classification, quarantine, monitoring, and regression fixtures.

---