Fire In Belly

AI Project Acceptance Criteria for a Custom AI Workflow Build

13 min read

White plastic vernier caliper with inch and millimetre scales lying on a blue cloth

You commissioned a custom AI workflow: invoice extraction, ticket reply drafting, or a weekly report assembled from three systems. The demo went well. Now the builder says it is done, and someone on your team says it still gets things wrong.

Both may be right. The statement of work probably says "the system shall accurately extract invoice data." As Precision Federal puts it, a clause like that is unenforceable in both directions: the vendor cannot prove it and the buyer cannot disprove it. Adding "95% accuracy" sounds precise and is not, because it does not say on which data, labeled by whom, or across how many items.

This guide is for the buyer. It shows how to write AI project acceptance criteria for one workflow so that "done" is a measurement both sides agreed before the build. It covers what most templates include, and two things they mostly skip: how big the acceptance set must be, and how good the answer key must be.

Why ordinary acceptance clauses fail for AI workflows

Acceptance for deterministic software is binary: the record saves or it does not. A model-backed step can produce different outputs for the same input, so quality is a rate across many inputs. DevShopVault states the consequence plainly: a distribution cannot be accepted by watching one happy-path demo.

A Hacker News commenter describes what happens on a fixed-price build whose SOW "lacked meaningful acceptance criteria / objective tests": weekly status calls turned into blame sessions about who owned the time and cost. It is one person's account, but it is exactly the failure the guides warn about.

Public-sector buyers already have a template. OMB Memorandum M-25-22 requires federal AI contracts to include ongoing testing terms, says evaluation data "should not be accessible to the vendor" and should resemble deployment data, and requires contracts to detail the vendor's own testing procedures. A private buyer can write the same terms into an agreement.

Write each criterion as behavior, method, and pass condition

Start with the format. Most disputes begin with a criterion that mixes three ideas into one sentence. QueryNow's acceptance sheet uses a structure worth copying: every row has a behavior stated as one observable fact, a verification method that names the test data and who runs it, and a pass condition that is a number or a binary outcome. One behavior per row, so a failure points at one thing.

For a supplier-invoice extraction workflow, a usable row looks like this:

BehaviorVerificationPass condition
Extracts invoice total, currency, and supplier VAT numberRun the frozen acceptance set in the buyer's staging environment; compare field values to adjudicated labelsAt least 95% of invoices have all three fields exactly right
Routes invoices it cannot read to the review queueSame run; count invoices sent to reviewNo more than 10% of the set routed to review
Never posts an invoice to the ledger without approvalAdversarial and timeout cases in the setZero unapproved postings

Notice what the table avoids. There is no "accurately," no "fast," no "substantially complete." DevShopVault suggests writing two numbers per metric: a target the build aims for and a floor below which the result is a defect. The gap between them leaves room for normal run-to-run variation. It also tells you how precise your measurement has to be, and that is where most acceptance clauses quietly break.

The review-rate cap in the second row matters. Without it, a workflow can hit any accuracy number by sending every hard case to a person. Routing to review is a legitimate outcome, but it has to be counted and capped, or the automation you paid for becomes a sorting step.

Keep the acceptance set away from the builder

Give the builder a development set: real, labeled, representative cases, as many as you can spare. Keep a separate acceptance set they never see until the formal run. If the builder tunes prompts against the items that decide payment, the score measures fit to those items, not next month's work.

Precision Federal adds two rules. Name a freeze date after which the set and labels change only by written agreement. And treat the held-out set as something that wears out each time someone studies its failures and adjusts the system; if you expect several attempts, reserve a second sealed partition.

Draw both sets from the same recent period of real work, including the ugly cases: scans, credit notes, second languages. Brief and Hire makes the same point for chatbots: use questions in customers' own words, with typos and half-sentences, plus some with no answer in your documents.

Size the set to the decision

Most templates say to build an evaluation set and set a threshold. Some give a size: Brief and Hire suggests about 30 real questions, and DevShopVault's example passes at 90 percent of 40 inputs. Those sizes suit a smoke test. They are too small to decide payment between a target and a floor.

Size the acceptance set to the decision, because a measured 90% pass rate on 40 items is still consistent with a true rate below 77%, so a small set cannot separate a target from a floor.

The numbers come from the Wilson interval for a proportion, the method the NIST/SEMATECH e-Handbook presents for confidence intervals on proportions such as pass rates. For an observed 90% pass rate, the 95% interval is roughly:

Acceptance set size95% interval around an observed 90%
30 itemsabout 74% to 97%
40 itemsabout 77% to 96%
100 itemsabout 83% to 95%
200 itemsabout 85% to 93%
400 itemsabout 87% to 93%

These are arithmetic, computed for this article; the lower bounds in the 30 and 100 item rows match Precision Federal's published figures. If your target is 92% and your floor is 85%, a 40-item set scoring 90% cannot tell you which side of the floor you are on. At 400 items the interval is narrower than that seven-point gap.

What to do differently:

  • Decide the target and the floor first, then pick a set size whose interval at the target sits above the floor. If you cannot afford that many labeled items, widen the band or accept that the formal run is a smoke test and add a provisional acceptance period on live work.
  • Write the pass condition against the lower bound, not the point estimate, if the stakes justify it: "the lower end of the 95% Wilson interval is at or above 85%." That rewards a larger set instead of punishing it.
  • Put the hard gates on separate, smaller adversarial sets. A zero-tolerance gate does not need hundreds of items to be meaningful, because a single failure decides it.

Precision Federal notes that comparing two systems costs far more than measuring one: separating a true 90% from a true 85% at conventional confidence and power takes on the order of 690 items per arm. If your acceptance criterion is "better than our current manual process," budget for that or make the comparison on a narrower, high-volume slice.

Fix the answer key before you fix the threshold

Every acceptance score is a comparison between the system's output and a label someone on your team wrote. Most templates assume that label is right. It often is not. Northcutt, Athalye, and Mueller estimated an average of at least 3.3% label errors across ten widely used machine learning test sets, at least 6% in the ImageNet validation set, and showed that correcting the labels can change which model ranks best. Those are public research datasets, not your invoices, but there is no reason to expect a set labeled in a hurry by one busy clerk to be cleaner.

Label errors cut both ways: a wrong gold answer can fail a correct output or pass a wrong one that matches the mistake. On a 200-item set, a handful of bad labels moves the score by several points, the whole width of your band.

Have two of your own staff label every acceptance item independently and adjudicate disagreements before the set is frozen, and never set the pass threshold above the agreement rate your own people reach.

In practice:

  1. Two people who do the work today label each item without seeing each other's answers. Use the same field definitions you gave the builder.
  2. Compute how often they agree. If two experienced clerks agree on all three invoice fields 94% of the time, a contract that demands 97% from the system asks it to beat your own team's consistency. Set the target at or below that agreement rate, or tighten the field definitions and relabel.
  3. A third person, named in the agreement, adjudicates each disagreement and records the reason. Many disagreements expose an ambiguous rule ("is the total before or after the early-payment discount?"). Fix the rule in writing and share it with the builder, because the builder needs the same definition to succeed.
  4. Where a case is genuinely ambiguous, either list every acceptable answer or move it out of the scored set into a reviewed-only bucket. Do not leave it to be argued about on acceptance day.
  5. Freeze the labels and the written rules together.

This costs your team time, which is why it belongs in the buyer obligations below. It also makes the result defensible: a disputed item has two independent labels and an adjudication note behind it.

If you plan to score open-ended outputs, such as drafted replies, with a model acting as judge, the judge is a labeler too. Qualify it against the same adjudicated human labels before its score decides payment; our guide to calibrating LLM-as-a-judge evaluators covers that procedure.

Separate gates from scored metrics

Some failures should never be traded for a good average. Heinrich & Co draws the line clearly: task success is a metric, but security and policy compliance are gates, and a gate does not average. An agent that passes a prohibited-action test 98 times out of 100 has failed, however strong its task success looks.

For an internal workflow, typical gates are:

  • Zero writes to the system of record without the approval the design requires.
  • Zero answers that cite or leak records the requesting user cannot access.
  • Every failed or timed-out run is logged and lands in a queue a person sees.
  • The documented rollback works in a live drill.

Keep these as pass or fail rows. Report the scored metrics, such as field accuracy, review rate, latency at the 95th percentile, and cost per completed item, against their targets and floors.

List the buyer's obligations with dates

An acceptance set that depends on two of your clerks labeling 200 invoices is a deliverable your side owes. So are staging access, service accounts, sample data exports, and someone who answers rule questions within a day. QueryNow's sheet lists client obligations with names and dates, and treats a criterion that could not be verified because an agreed item was never provided as paused, not failed, with the clock extended by the days lost.

This is fair to both sides, and it changes how people behave once the clock is running. Write the labeling hours, the adjudicator, the freeze date, and the access dates into the agreement. If you miss them, the builder should not absorb the delay; if the builder misses theirs, you have a clean record.

Record what was accepted

When the formal run passes, record the model name and version, the prompt and configuration commit, the acceptance set version, the scoring script, and per-item outputs. DevShopVault recommends recording the model version so a later deprecation becomes a warranty or change-order question rather than a dispute, and pinning acceptance to a reviewed commit.

That record is the baseline after handover. Our regression testing guide shows how the acceptance set seeds the suite for later changes, and the production readiness checklist covers the separate go-live decision.

Decide what happens on a miss and on a model change

  • Cure period. If a scored metric misses, the builder gets a fixed window to fix and re-run on the sealed set or reserve partition at no extra cost. Unico Connect suggests also stating how many failed attempts allow you to reject the milestone.
  • Borderline results. Heinrich & Co calls borderline handling the clause people skip and later argue about. Writing the pass condition against the interval's lower bound removes most of it.
  • Provisional acceptance. Where live inputs drift, Precision Federal suggests a 30 to 90 day provisional period with live measurement and a numeric trigger with a named remedy.
  • Model changes. M-25-22 encourages requiring performance standards before a new version is deployed and rollback if it fails. Write the same rule: any model, prompt, or retrieval change re-runs the acceptance set, and a result below the floor rolls back.

A worked example: supplier invoice extraction

A finance team of four commissions an extraction workflow that pre-fills the accounts payable screen. The numbers below are illustrative.

They agree a target of 95% and a floor of 90% for "all three key fields correct," a 10% review-rate cap, and two gates: no posting without approval, and every unreadable invoice routed to review. The builder gets 300 labeled invoices from the last two quarters for development.

Two clerks independently label 400 fresh invoices from the latest quarter and agree on all three fields for 96% of them. Most disagreements involve early-payment discounts, so the controller writes a rule that the total is the pre-discount amount and adjudicates the rest. Twelve ambiguous invoices move to a reviewed-only bucket. The 96% agreement shows a 95% target is demanding but possible; 98% would have asked the system to beat the clerks.

With 388 scored items, an observed 95% would have a Wilson interval of roughly 92% to 97%, so the set can decide. The formal run gets 363 of 388 fully right, about 93.6%, with an 8% review rate, and passes both gates. The lower bound is near 90.7%, just above the floor, so the build passes and both sides can see it was close. The record pins the model, commit, and set version.

On a 40-invoice demo set, 37 of 40 right (92.5%) gives an interval reaching down to about 80%, and nobody could say whether the build cleared the floor.

Next action

Pick the one workflow you are about to commission or are building now. Write a single acceptance row for its most important output: the behavior, how it will be verified, and a target and a floor. Then check the table above to see how many labeled items that floor needs, and ask two people on your team to label the same twenty items independently. Their agreement rate is the first number your clause needs.

If you want help turning that into a full acceptance sheet before a build, our AI workflow audit ends with a costed roadmap whose pilot is defined as "one workflow automated, one KPI measured," and our AI consulting cost page explains our fixed quotes and engagement shapes. If you are still choosing who builds it, see AI consultant vs in-house.


About Fire In Belly: Independent senior engineering from Tallinn, Estonia. We design and build AI workflow automation, internal tools, and custom software with the sized acceptance sets, adjudicated labels, and hard gates described above, at published fixed prices. Schedule a call to discuss your next project.