AI Platform Engineering for Internal Workflows: What the Platform Owns Before Teams Ship Agents
Three product teams each stood up their own model keys, prompt folders, and evaluation scripts. Six months later the company has three gateways, three ways to version a system prompt, and no shared answer when security asks who can call which model. Someone says "we need AI platform engineering." That can be right, and it is still easy to waste the budget: fund a catalog nobody asked for, copy a classic ML platform org chart onto GenAI work, or hire a platform group before enough teams need one.
This guide is for operations and engineering leaders who must decide what the shared layer owns before more internal agents and workflows ship. It assumes you already know that an LLM gateway can centralize credentials and quotas, that cost attribution needs explicit contracts, and that multi-agent orchestration is a separate design choice. The question here is narrower: which shared components earn a place on the paved road, in what order, and when a standing platform team is the right owner.
What current AI platform guides already cover
The role definition is settled enough to reuse. aiArch's AI platform engineer guide describes the job as building the shared platform other teams ship on: model serving, an AI gateway, retrieval infrastructure, evaluation and observability, CI/CD and infrastructure-as-code, plus cost controls. It is platform craft aimed at non-deterministic, token-metered workloads, not a product-feature role, and it is distinct from AgentOps (keeping specific agents healthy) and from the AI engineer who builds the product workflow.
Org charts and founding teams are covered as well. Simor Consulting argues for a dedicated platform team only once at least three teams are building or deploying AI, describes a small founding set of platform, data, and ML-tooling responsibilities, and states clearly that the platform team should not gate product deployments by hand. Modern AI Guide's enterprise org chart frames the AI platform team's OKRs around paved-road adoption and time-to-first-eval, not feature shipping.
Component catalogs exist too. DS Stream lists a model gateway, prompt registry, evaluation harness, ingestion template, guardrail pack, orchestration runtime, and project template, and ties the idea to CNCF platform language about common use cases. Official cloud guidance supplies the architectural spine: AWS Prescriptive Guidance describes four layers for an enterprise generative AI platform, ending in repeatable application patterns so teams do not rebuild the same RAG, assistant, document, or agentic flows; Microsoft's Foundry gateway guide explains when inserting a gateway is justified and when it would jeopardise reliability or security objectives; Google Cloud's IDP overview defines golden paths and treats platform engineering as a product discipline for developers.
If you need the role, the org chart, or the full catalog, those pages are the right starting points. The rest of this article covers three commissioning decisions they leave open for an internal AI workflow program.
Run the second-team test before you fund a catalog
Promote a component into the shared platform only when a second team needs it in the same form, because a forecast catalog built before demand becomes ticket-ops and teams route around it. DS Stream's working definition is blunt: a platform is what the second team does not have to build. Everything else is documentation with a logo. That test matches the CNCF instinct they cite: prioritise the most common use cases across product teams over capabilities used by a single team.
Use the test as a funding gate:
- List every AI component each product team rebuilt in the last two quarters (keys, prompt stores, eval runners, retrieval loaders, default safety filters, orchestration helpers).
- Keep a component local when only one team uses it, or when two teams use it but need incompatible interfaces.
- Promote only when a second team needs the same interface: one endpoint and quota model, one prompt version scheme, one eval report format, one permission-aware ingestion template.
- Assign a named owner and an announced retirement path the day you promote it. A shared component without an owner is how platforms become catalogs of dead stubs.
Worked example: two finance workflows
Accounts payable builds a document-extraction flow with its own provider key and a notebook of golden invoices. Procurement then starts a contract-summary agent and asks for "the same key setup AP used." That is a second-team signal for model access with per-team identity and quota, not yet a signal for a full prompt registry or a shared orchestration runtime. When HR later wants the same evaluation report format AP already uses for extraction quality, the evaluation harness earns promotion. Until then, forcing a twelve-item catalog on AP alone produces a platform nobody asked to consume.
The same rule stops the opposite delay: waiting for a perfect strategy deck while three teams keep rebuilding the same gateway. The second request is enough to promote.
Start with the GenAI slice, not a classic ML platform inventory
Start the GenAI paved road with per-team model access and a project template with logging, then the evaluation harness and prompt registry, before you copy a classic ML platform inventory that leads with feature stores and training CI. Many founding-team guides still read like 2017 ML platform plans: serving stacks, feature stores, training pipelines, model registries. Those matter when you train and serve custom models. Most internal AI workflow programs in 2026 start on hosted foundation models, tools, and retrieval, which is why AWS's generative AI layered approach emphasises approved models, security and governance, and repeatable application patterns, and why DS Stream's recommended start order is model access with a per-team key and quota, plus a project template with logging, with the evaluation harness and prompt registry next.
A practical GenAI-first minimum paved road looks like this:
| Order | Shared component | What each team stops building | What stays with the product team |
|---|---|---|---|
| 1 | Model access through one governed endpoint | Provider SDKs, raw keys, ad-hoc retries | Which model to call for which task, within policy |
| 2 | Project or service template with default logging | Repo skeleton, basic tracing hooks, ownership record | Domain prompts, tools, and business logic |
| 3 | Evaluation harness | Custom runners and one-off report formats | Domain datasets, thresholds, and failure labels |
| 4 | Prompt registry | Prompt files buried in app repos without versions | Prompt content and rollback decisions for their flows |
| Later | Retrieval template, guardrail pack, orchestration runtime | Repeated loaders, default filters, retry/state helpers | Corpus design, policy exceptions, workflow graphs |
Microsoft's gateway guidance is a boundary check. A gateway can solve identity scope, rate pressure, cost attribution, and safe deployment coordination across clients. It can also become a single point of failure. Put a gateway on the paved road only when at least one of those cross-cutting problems is already real, and skip it if it would break service-level objectives you already promised. Our LLM gateway guide covers the cutover once that decision is made.
The same caution applies to orchestration. Shared retries and state helpers can belong on the platform later; choosing agents versus deterministic workflows for a given step remains a product design problem, covered in agentic step autonomy and deterministic fallbacks.
Hire a platform team only when pull needs a standing owner
Simor's three-team threshold is a useful default: with one or two AI product teams, shared tooling maintained by a single senior engineer beats a dedicated platform team. A consultant-built thin paved road with named owners can fill the same role. With three or more teams depending on the same interfaces, a dedicated platform team starts to pay for the coordination cost.
When you do staff it, keep the ownership line sharp:
- The platform owns the paved road: access, templates, harnesses, registries, default logging, and the self-service path to consume them.
- Product teams own prompts, tools, workflow logic, domain evaluations, and release decisions for their use cases.
- Automated quality gates can block a bad change. The platform team should not become a human approval queue for every deploy. Simor warns against that pattern, and DORA labels the same ticket-ops behaviour as an anti-pattern for internal platforms.
If a consultant builds the first slice, require the same artefacts you would demand from an internal platform team: component list with owners, interfaces, how to request access without a ticket, and a handover pack that names who runs the road after the engagement.
Measure the paved road by pull, not by features shipped
Measure the platform by voluntary pull and time-to-first-evaluated-release, and keep an exit from every golden path, because mandating exclusive use of an internal platform has been linked to lower delivery throughput. DS Stream cites DORA 2024 findings on internal developer platforms: higher reported productivity and team performance in some dimensions, alongside lower throughput and change stability, and an estimated throughput penalty where teams had to use the platform exclusively. Those figures are about developer platforms generally, not AI platforms specifically, but the operating lesson transfers: a golden path that cannot be left becomes a gate, and teams route around gates.
Use metrics the platform team can actually move:
| Metric | What good looks like | What it replaces |
|---|---|---|
| Time from first access request to first evaluated release | Days, not weeks, for a new team on the golden path | Feature counts the platform team shipped |
| Share of AI services on the golden path by choice | Rising voluntary adoption | Mandate compliance percentage |
| Support mix | Roughly more proactive platform work than reactive tickets | Unlimited concierge for every team |
| Platform-caused incident rate | Falling as self-service hardens | Vanity uptime of unused services |
Google Cloud's IDP framing matches this: treat the platform as a product, gather developer feedback, and make the right way the easy way. Pair that with adoption measurement for the workflows themselves (see measuring internal AI workflow adoption). Gateway availability is not the same as a business workflow people actually use.
A ninety-day commissioning sequence
Use this sequence when you are funding the first paved road, whether with an internal trio or a consultant.
Days 1-15 - Demand map. Inventory rebuilt components. Mark each as local, promote-now, or later. Write the second-team evidence for every promote-now item.
Days 16-45 - GenAI-first slice. Deliver governed model access with per-team identity and quota, a project template with default logging and an ownership record, and a thin evaluation harness that product teams can fill with their own datasets. Wire cost tags early enough that chargeback is possible later.
Days 46-75 - First two consumers. Onboard two real workflows end to end. Fix the interfaces they stumble on. Do not expand the catalog while those two are still blocked.
Days 76-90 - Pull review. Publish time-to-first-evaluated-release, golden-path adoption share, and support mix. Decide whether the next quarter needs a prompt registry, a retrieval template, a standing platform hire, or simply more consumer coaching. If shadow tools appeared while you built the road, fold them into a sanctioned backlog instead of pretending the catalog is complete.
Failure modes to refuse
- Catalog-first platform. A twelve-box architecture diagram with zero second-team evidence.
- ML platform cargo cult. Feature stores and training CI before anyone has a second consumer for model access.
- Gateway as the whole strategy. A reverse proxy without templates, evals, or ownership still leaves every team rebuilding the rest.
- Platform as change-advisory board. Human approval for every prompt or model bump recreates the ticket queue you meant to remove.
- Exclusive-use mandate without an exit. Teams will build side doors; you will lose both adoption and visibility.
Next action
Pick one internal AI workflow that a second team is already copying. Write the second-team evidence in one page, name the GenAI-first component it implies, and schedule a ninety-day pull review before you approve a broader platform budget. If you need help turning that slice into a maintainable internal tool, start with an AI audit of what teams already rebuilt, then commission the paved road through internal tools or workflow automation with owners named from day one.