Fire In Belly

AI Platform Engineering for Internal Workflows: What the Platform Owns Before Teams Ship Agents

10 min read

Rows of white server cabinets under black cable trays in a data centre

Three product teams each stood up their own model keys, prompt folders, and evaluation scripts. Six months later the company has three gateways, three ways to version a system prompt, and no shared answer when security asks who can call which model. Someone says "we need AI platform engineering." That can be right, and it is still easy to waste the budget: fund a catalog nobody asked for, copy a classic ML platform org chart onto GenAI work, or hire a platform group before enough teams need one.

This guide is for operations and engineering leaders who must decide what the shared layer owns before more internal agents and workflows ship. It assumes you already know that an LLM gateway can centralize credentials and quotas, that cost attribution needs explicit contracts, and that multi-agent orchestration is a separate design choice. The question here is narrower: which shared components earn a place on the paved road, in what order, and when a standing platform team is the right owner.

What current AI platform guides already cover

The role definition is settled enough to reuse. aiArch's AI platform engineer guide describes the job as building the shared platform other teams ship on: model serving, an AI gateway, retrieval infrastructure, evaluation and observability, CI/CD and infrastructure-as-code, plus cost controls. It is platform craft aimed at non-deterministic, token-metered workloads, not a product-feature role, and it is distinct from AgentOps (keeping specific agents healthy) and from the AI engineer who builds the product workflow.

Org charts and founding teams are covered as well. Simor Consulting argues for a dedicated platform team only once at least three teams are building or deploying AI, describes a small founding set of platform, data, and ML-tooling responsibilities, and states clearly that the platform team should not gate product deployments by hand. Modern AI Guide's enterprise org chart frames the AI platform team's OKRs around paved-road adoption and time-to-first-eval, not feature shipping.

Component catalogs exist too. DS Stream lists a model gateway, prompt registry, evaluation harness, ingestion template, guardrail pack, orchestration runtime, and project template, and ties the idea to CNCF platform language about common use cases. Official cloud guidance supplies the architectural spine: AWS Prescriptive Guidance describes four layers for an enterprise generative AI platform, ending in repeatable application patterns so teams do not rebuild the same RAG, assistant, document, or agentic flows; Microsoft's Foundry gateway guide explains when inserting a gateway is justified and when it would jeopardise reliability or security objectives; Google Cloud's IDP overview defines golden paths and treats platform engineering as a product discipline for developers.

If you need the role, the org chart, or the full catalog, those pages are the right starting points. The rest of this article covers three commissioning decisions they leave open for an internal AI workflow program.

Run the second-team test before you fund a catalog

Promote a component into the shared platform only when a second team needs it in the same form, because a forecast catalog built before demand becomes ticket-ops and teams route around it. DS Stream's working definition is blunt: a platform is what the second team does not have to build. Everything else is documentation with a logo. That test matches the CNCF instinct they cite: prioritise the most common use cases across product teams over capabilities used by a single team.

Use the test as a funding gate:

  1. List every AI component each product team rebuilt in the last two quarters (keys, prompt stores, eval runners, retrieval loaders, default safety filters, orchestration helpers).
  2. Keep a component local when only one team uses it, or when two teams use it but need incompatible interfaces.
  3. Promote only when a second team needs the same interface: one endpoint and quota model, one prompt version scheme, one eval report format, one permission-aware ingestion template.
  4. Assign a named owner and an announced retirement path the day you promote it. A shared component without an owner is how platforms become catalogs of dead stubs.

Worked example: two finance workflows

Accounts payable builds a document-extraction flow with its own provider key and a notebook of golden invoices. Procurement then starts a contract-summary agent and asks for "the same key setup AP used." That is a second-team signal for model access with per-team identity and quota, not yet a signal for a full prompt registry or a shared orchestration runtime. When HR later wants the same evaluation report format AP already uses for extraction quality, the evaluation harness earns promotion. Until then, forcing a twelve-item catalog on AP alone produces a platform nobody asked to consume.

The same rule stops the opposite delay: waiting for a perfect strategy deck while three teams keep rebuilding the same gateway. The second request is enough to promote.

Start with the GenAI slice, not a classic ML platform inventory

Start the GenAI paved road with per-team model access and a project template with logging, then the evaluation harness and prompt registry, before you copy a classic ML platform inventory that leads with feature stores and training CI. Many founding-team guides still read like 2017 ML platform plans: serving stacks, feature stores, training pipelines, model registries. Those matter when you train and serve custom models. Most internal AI workflow programs in 2026 start on hosted foundation models, tools, and retrieval, which is why AWS's generative AI layered approach emphasises approved models, security and governance, and repeatable application patterns, and why DS Stream's recommended start order is model access with a per-team key and quota, plus a project template with logging, with the evaluation harness and prompt registry next.

A practical GenAI-first minimum paved road looks like this:

OrderShared componentWhat each team stops buildingWhat stays with the product team
1Model access through one governed endpointProvider SDKs, raw keys, ad-hoc retriesWhich model to call for which task, within policy
2Project or service template with default loggingRepo skeleton, basic tracing hooks, ownership recordDomain prompts, tools, and business logic
3Evaluation harnessCustom runners and one-off report formatsDomain datasets, thresholds, and failure labels
4Prompt registryPrompt files buried in app repos without versionsPrompt content and rollback decisions for their flows
LaterRetrieval template, guardrail pack, orchestration runtimeRepeated loaders, default filters, retry/state helpersCorpus design, policy exceptions, workflow graphs

Microsoft's gateway guidance is a boundary check. A gateway can solve identity scope, rate pressure, cost attribution, and safe deployment coordination across clients. It can also become a single point of failure. Put a gateway on the paved road only when at least one of those cross-cutting problems is already real, and skip it if it would break service-level objectives you already promised. Our LLM gateway guide covers the cutover once that decision is made.

The same caution applies to orchestration. Shared retries and state helpers can belong on the platform later; choosing agents versus deterministic workflows for a given step remains a product design problem, covered in agentic step autonomy and deterministic fallbacks.

Hire a platform team only when pull needs a standing owner

Simor's three-team threshold is a useful default: with one or two AI product teams, shared tooling maintained by a single senior engineer beats a dedicated platform team. A consultant-built thin paved road with named owners can fill the same role. With three or more teams depending on the same interfaces, a dedicated platform team starts to pay for the coordination cost.

When you do staff it, keep the ownership line sharp:

  • The platform owns the paved road: access, templates, harnesses, registries, default logging, and the self-service path to consume them.
  • Product teams own prompts, tools, workflow logic, domain evaluations, and release decisions for their use cases.
  • Automated quality gates can block a bad change. The platform team should not become a human approval queue for every deploy. Simor warns against that pattern, and DORA labels the same ticket-ops behaviour as an anti-pattern for internal platforms.

If a consultant builds the first slice, require the same artefacts you would demand from an internal platform team: component list with owners, interfaces, how to request access without a ticket, and a handover pack that names who runs the road after the engagement.

Measure the paved road by pull, not by features shipped

Measure the platform by voluntary pull and time-to-first-evaluated-release, and keep an exit from every golden path, because mandating exclusive use of an internal platform has been linked to lower delivery throughput. DS Stream cites DORA 2024 findings on internal developer platforms: higher reported productivity and team performance in some dimensions, alongside lower throughput and change stability, and an estimated throughput penalty where teams had to use the platform exclusively. Those figures are about developer platforms generally, not AI platforms specifically, but the operating lesson transfers: a golden path that cannot be left becomes a gate, and teams route around gates.

Use metrics the platform team can actually move:

MetricWhat good looks likeWhat it replaces
Time from first access request to first evaluated releaseDays, not weeks, for a new team on the golden pathFeature counts the platform team shipped
Share of AI services on the golden path by choiceRising voluntary adoptionMandate compliance percentage
Support mixRoughly more proactive platform work than reactive ticketsUnlimited concierge for every team
Platform-caused incident rateFalling as self-service hardensVanity uptime of unused services

Google Cloud's IDP framing matches this: treat the platform as a product, gather developer feedback, and make the right way the easy way. Pair that with adoption measurement for the workflows themselves (see measuring internal AI workflow adoption). Gateway availability is not the same as a business workflow people actually use.

A ninety-day commissioning sequence

Use this sequence when you are funding the first paved road, whether with an internal trio or a consultant.

Days 1-15 - Demand map. Inventory rebuilt components. Mark each as local, promote-now, or later. Write the second-team evidence for every promote-now item.

Days 16-45 - GenAI-first slice. Deliver governed model access with per-team identity and quota, a project template with default logging and an ownership record, and a thin evaluation harness that product teams can fill with their own datasets. Wire cost tags early enough that chargeback is possible later.

Days 46-75 - First two consumers. Onboard two real workflows end to end. Fix the interfaces they stumble on. Do not expand the catalog while those two are still blocked.

Days 76-90 - Pull review. Publish time-to-first-evaluated-release, golden-path adoption share, and support mix. Decide whether the next quarter needs a prompt registry, a retrieval template, a standing platform hire, or simply more consumer coaching. If shadow tools appeared while you built the road, fold them into a sanctioned backlog instead of pretending the catalog is complete.

Failure modes to refuse

  • Catalog-first platform. A twelve-box architecture diagram with zero second-team evidence.
  • ML platform cargo cult. Feature stores and training CI before anyone has a second consumer for model access.
  • Gateway as the whole strategy. A reverse proxy without templates, evals, or ownership still leaves every team rebuilding the rest.
  • Platform as change-advisory board. Human approval for every prompt or model bump recreates the ticket queue you meant to remove.
  • Exclusive-use mandate without an exit. Teams will build side doors; you will lose both adoption and visibility.

Next action

Pick one internal AI workflow that a second team is already copying. Write the second-team evidence in one page, name the GenAI-first component it implies, and schedule a ninety-day pull review before you approve a broader platform budget. If you need help turning that slice into a maintainable internal tool, start with an AI audit of what teams already rebuilt, then commission the paved road through internal tools or workflow automation with owners named from day one.


About Fire In Belly: Independent senior engineering from Tallinn, Estonia. We design and build custom AI agents, AI workflow automation, and internal tools with the thin paved roads, shared gateways, and named platform owners described above, at published fixed prices. Schedule a call to discuss your next project.