Internal AI Tool Adoption: Measuring Real Use After Launch

The invoice-intake workflow passed its pilot. Extraction accuracy held up on real documents, the service stayed up, and the monthly model bill was small. Ten weeks later the finance team is keying most invoices by hand again, and nobody filed a complaint. Failed internal AI tool adoption usually looks like this, with the workflow still running while the work flows around it. The fix starts with measurement that can see the bypass. Count the eligible tasks, record which ones went through the workflow and which did not, capture why people overrode or skipped it, and review those numbers at 30, 60, and 90 days against thresholds you set before launch. Then you can tell whether the problem is placement, trust, ownership, or quality, and change the right thing.

This guide covers one custom workflow after handover: document intake, support triage drafts, report assembly, or an internal question-answering agent. It does not repeat the value math in our AI automation ROI guide. ROI asks whether the work that went through the workflow paid off. Adoption asks whether the work went through it at all.

Why usage fades while the workflow still works

Three mechanisms show up again and again in write-ups on abandoned internal AI tools.

The first is placement. AdvantageWorks describes most internal AI tools as added next to an existing workflow rather than replacing a step inside it, so every use is extra work that loses under deadline pressure. Big Sky Consulting Group makes the same argument from the user's chair: a dispatcher lives in the TMS and a finance analyst lives in the ERP or its spreadsheet export, so an assistant in a separate tab costs a context switch on every use.

The second is trust. One confidently wrong answer that a user cannot cheaply verify turns the tool into something they double-check, and checking can cost more than doing the task by hand. Kikoff's engineering team reports exactly this with its first internal data assistant, a standalone web app that gave confidently incorrect answers to real business questions. A Hacker News commenter reports a sharper version for coding agents: the team soured on them within about a week because outputs needed significant time to understand and were often scrapped. Both are author reports rather than measured studies, and they describe the same mechanism.

The third is ownership. The delivery team disbands at go-live and nobody reads the logs, fixes the prompt, or refreshes the source data when the underlying process changes. AdvantageWorks calls this the post-launch ownership vacuum.

The broader survey data points the same way. Deloitte's State of AI in the Enterprise 2026 reports that fewer than 60% of workers with sanctioned AI access use it in their daily workflow and that 84% of companies have not redesigned jobs around AI. Its press release notes that the survey covers more than 3,000 director to C-suite leaders and that 37% use AI at a surface level with little or no change to underlying processes. Those are leader survey responses, not usage logs, so treat them as context rather than a benchmark for your workflow.

None of these causes shows up on an uptime dashboard, so the numbers have to describe the work itself.

Count eligible tasks instead of active users

Most adoption dashboards count people: seats activated, weekly active users, sessions. For a process workflow those numbers mislead. The Google HEART paper by Rodden, Hutchinson, and Fu points out that a seven-day active-user count could still rise in a worst case of 100% weekly turnover, and it recommends normalizing raw counts into ratios or per-user averages. A rising user count can hide a falling share of the actual work.

Measure adoption as the share of eligible tasks in the system of record that went through the workflow, because active-user counts cannot see the work that bypassed it. If 400 invoices arrived in the AP inbox last week and 120 were processed through the extraction workflow, routed share is 30%, no matter how many people logged in. The denominator comes from the ERP, the ticketing system, or the queue where the work originates, never from the AI workflow's own logs. The workflow can only log what it saw.

Defining "eligible" is the hard part, and it is a scoping decision you should write down:

• Which work-item types the workflow was built for (for example, PDF supplier invoices in EUR and USD, excluding credit notes).
• Which source systems and channels count (the shared AP inbox, not a manager's personal mailbox).
• Which items are excluded on purpose, such as amounts above an approval threshold that always need a human.
• The time window and the business calendar used for weekly comparisons.

Our ROI guide already uses eligible items and terminal outcomes to measure value. Reuse the same definition so adoption and ROI share one denominator; two definitions of "eligible" will produce two arguments in the review meeting.

People still matter as a second cut. GitHub's Copilot usage metrics separate active users, who request or apply something, from passive users, who only had the tool assigned to them, and they group users into adoption cohorts to see whether usage deepens or stalls at trial. Apply the same split by role. A workflow that three AP clerks use daily and two never touch tells you where to look next.

Instrument routing, bypass, and override events

Once the denominator exists, every eligible task needs one routing record. A minimal event shape looks like this:

task_id: inv-2026-10-0142
source_system: erp_ap_inbox
eligible: true
eligibility_rule: supplier_invoice_pdf_v2
route: workflow            # workflow | bypassed | excluded
handled_by_role: ap_clerk
workflow_outcome: accepted_with_edits   # accepted | accepted_with_edits | overridden | abandoned
override_reason: wrong_vat_rate         # null unless overridden or bypassed
bypass_reason: null                     # e.g. tool_slow, distrust_after_error, not_in_erp, unaware
fields_edited: [vat_rate]
seconds_to_first_output: 14
recorded_at: 2026-10-05T09:12:00Z

Two fields carry most of the diagnostic value. bypass_reason explains why an eligible task skipped the workflow, and override_reason explains why a human replaced or corrected its output. Keep the reason list short (six to ten codes) and let people pick one in a single click inside the tool they already use. A free-text box nobody fills in is no better than no field.

Log every bypass and every human override with a reason code and feed those cases into the evaluation set, because the tasks people route around are the ones your quality metrics never see. 1up's write-up on abandoned assistants describes the effect: users who were let down by a category of question stop asking that category, so the assistant's real coverage shrinks while its measured coverage looks stable. The HEART paper gives the measurement reason, noting that failure signals such as task abandonment and undo events are often easier to detect than success. Our recommendation, as a synthesis of those two sources, is to treat bypass and override records as labeled failure cases. Each week, sample them, add the representative ones to the regression set, and fix the category before you try to win the users back.

Override rate also works as a trust signal. GitHub uses acceptance rate to judge whether suggestions are relevant and trusted. For a custom workflow, the equivalent is the share of routed tasks accepted without edits, accepted with edits, or overridden. Our pilot to production guide already asks you to review human overrides and their reasons after launch; the routing record makes that review possible without reading every trace.

Read the numbers together before acting

No single number tells you what to fix. Read routed share, override rate, repeat use by role, and time to first output side by side.

Compare routed share against total eligible volume every time. The same caution applies to exception rates in human review queues: a falling count means little if volume also fell.

Put the workflow where the work already happens

Placement problems are usually the cheapest to diagnose and fix. Kikoff's second attempt, an agent that lived in Slack, met people where they were already working and, in the team's own report, became a daily habit across engineering, product, ops, and data. Weekly questions to its agents went from 30 to 45 at first, to about 75 after internal communication, to almost 120 after a specialized redesign. Those figures are self-reported by the team that built it.

For a process workflow, "where the work happens" usually means one of three places:

1. Inside the system of record, as a button, a pre-filled draft, or a field populated before the human opens the item.
2. In the queue or inbox where the item arrives, so the AI output is waiting when the person picks it up.
3. In the team's chat tool, for question-answering workflows where the alternative is messaging a colleague.

A separate web app is the weakest option for routine work. If the workflow must live outside the main system for technical reasons, measure seconds_to_first_output from the moment the item lands, not from the moment someone opens the AI tool. That keeps the context-switch cost visible.

TechTarget's reporting on employee AI adoption quotes the same principle from the governance side: make the sanctioned path the easiest path, because people will otherwise turn to unapproved tools.

Name an owner before the delivery team leaves

The ownership vacuum is a contract problem. Write the owner into the handover, with a weekly time budget, before the consultant or project team rolls off. The owner's job is small and specific:

• Read the routing summary and the week's bypass and override samples.
• File or fix the top failure category, and add its cases to the evaluation set.
• Tell the users what changed, in the channel where they work.
• Keep the eligibility rule current when the process changes.

GitHub's Copilot enablement guide adds two practices that transfer well to custom workflows: recruit champions from the pilot group who help others get started, and follow up directly with people who have access but have not used it. For a small team, the champion and the owner can be the same person, as long as someone has the job in writing.

Set the 30, 60, and 90 day thresholds before launch

GitHub's example rollout timeline starts onboarding 45 days before rollout by defining success metrics and training champions. Agree the thresholds before launch, while nobody has a usage chart to defend.

A workable template, with the numbers chosen by you and the process owner:

• Day 30: routed share at or above a floor (for example 40% of eligible items), every bypass and override carrying a reason code, and no reason code above a set share.
• Day 60: routed share at the target for each role that the workflow was built for, override rate below its ceiling, and the top failure category from day 30 fixed and covered by the evaluation set.
• Day 90: routed share stable for four consecutive weeks, and a decision on the old path.

If a threshold is missed, the routing data says which kind of fix to try. Sometimes the right call is to narrow scope. Big Sky argues that a tool with committed users in one function and none elsewhere can be a correctly sized departmental tool rather than a failed rollout. Shrinking the eligibility rule to the work that actually flows through the workflow is a legitimate outcome. Whether the narrowed workflow still earns its keep is a value question for the scale, repair, or stop gate in the ROI guide.

Retire the old path last

Teams that see low adoption often reach for a forcing function: turn off the manual path. Keep the old path open until the workflow wins on routed share and override rate; closing it earlier turns quiet bypass into side channels and shadow tools. This is our recommendation from first principles, supported by the shadow AI pattern TechTarget describes. If the workflow is still slower or less trusted than the manual route, removing the manual route does not remove the reason people avoided the workflow. It only moves the bypass somewhere you cannot measure it: a personal spreadsheet, a private chatbot account, or a colleague who "just does it."

We suggest one retirement rule. Close the old path for a work-item type only after routed share for that type has held at target for four weeks and its override rate is below the ceiling you set before launch. Announce the date in advance, keep a documented manual fallback for outages, and keep logging bypass reasons afterwards so a new problem shows up in the data before it shows up as a workaround.

A worked example: invoice intake

The numbers below are illustrative, chosen to show the method rather than to report a real deployment.

A finance team of five receives roughly 400 eligible supplier invoices a week. The extraction workflow pre-fills fields inside the AP module. At day 30 the routing records show:

• Routed share of 31%, against a day-30 floor of 40%.
• Two clerks route nearly all their invoices through the workflow; three route almost none.
• Of the bypassed invoices with a reason code, most say distrust_after_error, and the overrides cluster on vat_rate.

The owner pulls a sample of the VAT overrides and finds that invoices from one country use a reduced rate the extraction prompt never handled. The fix is a rule plus twelve new evaluation cases. The owner posts the change in the AP channel with two before-and-after examples. By day 60 the three low-use clerks are routing most invoices, the VAT override share has dropped, and the next reason code on the list is tool_slow for multi-page invoices, which becomes the next fix. At day 90, with routed share at target for four weeks, the team schedules the date to stop the manual entry path for standard supplier invoices and keeps it for credit notes, which were excluded from the start.

The ROI ledger would have reported this workflow's per-invoice value accurately all along. It would not have shown that most invoices were never reaching it.

Mistakes that make adoption numbers lie

• Counting seats or logins and calling it adoption.
• Taking the denominator from the workflow's own logs, which cannot contain bypassed work.
• Collecting free-text feedback instead of one-click reason codes, then never reading it.
• Reading a stable override rate as success while routed share falls.
• Mandating use before the workflow beats the manual path, which hides bypass instead of ending it.
• Changing the eligibility rule silently, which makes week-over-week routed share meaningless.

Next action

Pick one workflow you launched in the last six months. Pull last week's count of eligible items from the system of record and count how many went through the workflow. If you cannot produce both numbers, you do not yet know your internal AI tool adoption, and the routing record above is the first thing to build. If you want a second pair of eyes on the eligibility rule and the 30/60/90 thresholds before your next launch, that is part of what our AI workflow audit covers.

---