LLM SLOs and Error Budgets for AI Workflows

An LLM SLO built only from API uptime can stay green while the workflow fails its users. The model may respond in 800 milliseconds, yet retrieval returns the wrong policy, a tool writes twice, or a low confidence case bypasses review. Infrastructure telemetry sees a successful request. The business sees a bad outcome.

A useful AI workflow SLO measures the complete user journey. It defines which executions are eligible, what counts as a good outcome, when delayed quality evidence arrives, and what the team does when the error budget burns too quickly. This guide provides the event contract, formulas, decision rules, and rollout sequence needed to make that target operational.

Start with the user journey, not the model API

An invoice workflow may authenticate a user, retrieve evidence, call a model, invoke tools, wait for approval, and commit a business record. Measuring each dependency helps diagnosis, but no component metric answers whether the user received a correct result without an unsafe side effect.

Google's SRE guidance defines a service level indicator as a quantitative measure of service behavior and an objective as a target for that measure. It also recommends choosing indicators from what users care about, rather than from whatever the monitoring stack already exposes. See Google's explanation of service level objectives.

Write the journey in one sentence before choosing metrics. For example:

> An eligible invoice enters the queue and produces one correctly coded voucher, or a review case with sufficient evidence, within 15 minutes, without creating a duplicate payment instruction.

That sentence identifies several independent requirements: completion, correctness, latency, safe escalation, and side effect integrity. If any critical requirement fails, the execution is not good even when every HTTP request returned 200.

Define one SLO per important journey. Do not combine invoice processing, support triage, and knowledge search into an average. Their risk, latency, and evidence differ. A blended number lets a high volume low risk journey hide failures in a smaller critical one.

Define eligible events before good events

The denominator controls the meaning of an SLO. Count only executions that the service contract claims it can handle. Exclude test traffic, operator drills, malformed requests rejected before admission, and upstream outages that your contract explicitly excludes. Keep those exclusions narrow and auditable. Otherwise the team can improve the percentage by moving failures out of the denominator.

Create one terminal outcome event for every admitted execution. The workflow should emit it after success, controlled escalation, terminal failure, or timeout. Retries must share the same execution identity so they cannot inflate both numerator and denominator.

A compact event can look like this:

{
  "execution_id": "run_01842",
  "journey": "invoice_to_voucher",
  "risk_class": "financial_write",
  "workflow_version": "2026-09-09.3",
  "admitted_at": "2026-09-09T09:10:00Z",
  "terminal_at": "2026-09-09T09:18:12Z",
  "terminal_state": "completed",
  "duplicate_side_effect": false,
  "evidence_complete": true,
  "online_quality_gate": "pass",
  "delayed_quality_label": null,
  "label_due_at": "2026-09-16T00:00:00Z"
}

Do not put prompts, retrieved passages, personal data, or model output in the SLO event. Store protected evidence separately and link it through access controlled identifiers. The OpenTelemetry generative AI conventions define common attributes for model operations, but the terminal journey event remains an application responsibility. Join operation spans to the execution identity for diagnosis.

Now define the good event as a Boolean rule. For the invoice example, an execution is good when all of these conditions hold:

1. It reaches completed or an allowed human_review state.
2. It reaches that state inside the journey deadline.
3. Its required evidence bundle is complete.
4. It creates no duplicate or unauthorized side effect.
5. Its online quality gate passes.
6. Its delayed label is not a confirmed failure once that label arrives.

Use explicit fields instead of parsing log text. Version the rule with the workflow because a change to the outcome schema can otherwise create an apparent reliability jump.

Separate online reliability from delayed quality

Availability and latency can be known at termination. Business correctness often cannot. A support reply may be marked wrong by an agent tomorrow. An invoice code may be corrected during month end. A retrieval answer may become a confirmed miss after the user opens another ticket.

Do not ignore delayed evidence, and do not rewrite historical dashboards without a trace. Maintain two views:

• Operational SLI: uses evidence available when the execution terminates. It supports fast incident response.
• Matured quality SLI: includes labels whose observation window has closed. It supports release and product decisions.

OpenAI's evaluation guide recommends task specific evaluation criteria and continuous evaluation as systems change. Apply that principle to production labels. Define the label source, observation window, adjudication rule, and coverage target for each journey. A quality SLI based on 8 percent labeled coverage should not carry the same confidence as one based on a representative 70 percent sample.

Keep pending labels out of the matured denominator until their due date. When a label arrives, append a correction event rather than editing the terminal record in place. This preserves the history needed to explain why a dashboard changed.

Track label coverage beside quality:

matured_quality_sli = good_matured_executions / labeled_matured_executions
label_coverage = labeled_matured_executions / label_eligible_executions

Set a minimum coverage gate. If coverage falls below it, mark the quality SLO as unknown and block quality sensitive releases. Unknown is not success.

Calculate the SLO and error budget

For a window containing eligible executions:

journey_sli = good_executions / eligible_executions
allowed_bad = eligible_executions * (1 - slo_target)
consumed_budget = bad_executions / allowed_bad
remaining_budget = 1 - consumed_budget

If the target is 99 percent and the window has 10,000 eligible executions, the budget allows 100 bad executions. After 40 bad executions, 40 percent is consumed and 60 percent remains. Calculate these values in the monitoring system rather than rounding intermediate ratios.

Choose the target from user impact and observed capability. A target copied from a model provider's availability agreement is not a workflow target. The provider agreement covers only one dependency. The Google SRE Workbook implementation chapter recommends starting from user journeys, selecting indicators, and iterating on achievable targets.

Avoid promising 100 percent. It leaves no budget for controlled change and usually drives teams to weaken the definition of failure. For high risk writes, use a separate near zero tolerance safety indicator for duplicate or unauthorized actions rather than hiding it inside a broad 99 percent completion SLO.

Alert on burn rate, not every bad execution

An error budget answers whether the current failure rate can continue for the rest of the window. Burn rate compares the observed bad event rate with the rate allowed by the target.

burn_rate = observed_bad_rate / allowed_bad_rate

A burn rate of 1 consumes budget exactly on schedule. A burn rate of 10 would spend a 30 day budget in about three days if it continued. Use both a short window and a long window. The short window detects sharp incidents. The long window prevents a brief spike from waking the team after the system has already recovered.

Route alerts by journey and risk class. A duplicate financial write needs immediate containment even if aggregate budget remains. A small latency regression in a background enrichment job may justify a ticket. Error budgets do not replace hard safety limits.

Record the reason each execution failed. Use one primary outcome classification so a single failed journey is counted once in the SLO. Attach component causes such as provider timeout, retrieval miss, tool denial, review timeout, or write conflict as diagnostic dimensions. Counting every component error as a separate bad event exaggerates correlated failures and makes the budget impossible to interpret.

Turn budget state into engineering policy

A dashboard without a decision rule becomes decoration. Define policy before an incident so product pressure cannot reinterpret the number.

Use four budget states:

• Healthy: budget consumption is below plan. Normal releases continue.
• Warning: burn is elevated or label coverage is weakening. Increase review sampling and restrict risky changes.
• Critical: projected consumption will exhaust the window. Pause model, prompt, retrieval, and tool changes that can affect the journey.
• Exhausted: route high risk work to a deterministic fallback or human queue until reliability recovers and a reviewed release addresses the cause.

The current practitioner proposal for cloud SLOs in an LLM data system asks for this connection between telemetry, provider governance, incidents, deployment gates, and error budget state. Treat it as one team's implementation requirement, not evidence that every organization uses the same thresholds.

Name the authority that can approve an exception, the maximum duration, and the compensating controls. Exceptions should appear in the SLO record. Silently excluding a troubled workflow version is not an exception process.

Microsoft's Well Architected guidance for AI workloads places AI design inside the same reliability, security, cost, operational, and performance disciplines used for other production systems. The workflow SLO is the mechanism that turns those design choices into an observable service promise.

Segment results without hiding the total

Always publish the overall journey SLI, then segment it by dimensions that drive different failure modes:

• workflow and prompt version
• model and provider route
• risk class
• tenant or business unit, subject to privacy limits
• language, document type, or task category
• automated completion versus human review
• new execution versus retry or resume

Segments reveal a bad route that a total conceals. They can also become noisy when sample sizes are small. Show event counts and confidence alongside percentages. Do not declare a segment fixed after two successful runs.

Keep version changes visible. If a new workflow version handles only easy cases while the old version receives difficult ones, a raw comparison is misleading. Use a stable evaluation set and comparable production slices before shifting traffic.

Verify the measurement before enforcing it

Test the SLO pipeline like a financial control. A broken denominator can make every later decision wrong.

Run these checks before connecting budget state to releases:

1. Admit a known set of synthetic executions and prove each produces exactly one terminal event.
2. Force timeout, tool denial, duplicate prevention, human review, and terminal failure paths.
3. Confirm retries retain one execution identity and one outcome count.
4. Submit a delayed label and prove the matured SLI changes while the original terminal event remains immutable.
5. Drop labels intentionally and verify coverage becomes unknown rather than green.
6. Trigger a shared provider outage and prove each user journey fails once, with component causes attached separately.
7. Replay an event and prove deduplication prevents a second count.
8. Change the good event rule and confirm historical data stays tied to its original rule version.
9. Exhaust a test budget and verify the release gate, fallback, and exception workflow behave as documented.

Compare dashboard totals with the workflow's authoritative execution ledger. Sample good and bad outcomes manually. If the two systems disagree, fix measurement before tuning targets.

Roll out one journey at a time

Start with a workflow that has clear terminal states and an identifiable owner. Write its journey sentence, eligible event rule, good event rule, label policy, target, budget window, and release response in one reviewed document. Instrument the terminal event and run it in observation mode for at least one representative business cycle.

Review false positives, exclusions, missing labels, and segment size with product and operations. Only then enable paging or deployment gates. The first target will probably need adjustment, but change it through a recorded decision. Do not move it merely to make the dashboard green.

Your next action is to select one production journey and classify its last 100 completed executions using the proposed good event rule. If engineers and operations cannot agree on those classifications, the SLO definition is not ready for automation.

References

1. Google SRE service level objectives supports user centered SLI and SLO definitions.
2. Google SRE Workbook: Implementing SLOs supports the practical journey, indicator, target, and error budget process.
3. OpenTelemetry GenAI semantic conventions supports standard telemetry for generative AI operations.
4. OpenAI evaluation guide supports task specific and continuous evaluation design.
5. Microsoft Azure Well Architected AI workloads supports applying production architecture disciplines to AI systems.
6. Cloud SLO and reliability issue 147 supplies one current practitioner's requested link between AI reliability data and engineering policy.

---