MCP Gateway vs LLM Gateway for Internal AI Workflows: When Tool Calls Need Their Own Door
You already put model calls behind an LLM gateway: one place for keys, quotas, and provider failover. Then product teams wired Claude, Cursor, and a custom agent straight to three MCP servers over the warehouse, CRM, and ticketing system. Six months later nobody can answer which agent read the customer table last Tuesday without grepping five services. The LLM gateway is fine. The gap is tool traffic without a door.
This guide is for platform and security leaders who must decide when an MCP gateway earns a place next to the model gateway. It assumes you already know when an action should be an MCP server versus a direct API, how to harden a single MCP server, and when shared platform components deserve a paved road. The question here is narrower: when do tool calls need their own control plane, which placement should you choose, and what must you own before you buy or build plumbing.
The short answer
Keep model traffic on the LLM gateway. Treat tool traffic as a separate path. Promote a central MCP gateway only when more than one team shares upstream servers, an auditor will ask a cross-agent question, or revocation must be one action. Until then, prefer an in-process allowlist or a per-agent sidecar with structured logs.
| Layer | Governs | Typical controls | Blind without the other layer |
|---|---|---|---|
| LLM / AI gateway | Applications calling models | Provider routing, token budgets, prompt guardrails, model cost attribution | Which tools an agent can invoke after the model decides |
| MCP gateway | Agents calling tools over MCP | Catalogue, identity, tool allowlists, credential isolation, tool-call audit, kill switch | Model cost, provider failover, and prompt-level safety |
| In-process allowlist | One agent process | Local tool load list and local logs | Estate-wide audit and one-shot revocation |
| Per-agent sidecar | Calls leaving one agent | Local credential inject and per-agent policy | Cross-agent patterns on one upstream |
Tetrate's definitions put the split cleanly: an LLM gateway sits between apps and model providers; an MCP gateway sits between agents and MCP servers; "AI gateway" increasingly means both under one control plane. TrueFoundry's comparison makes the operational point: governing only the model leaves tool actions untracked, and governing only tools leaves spend and prompt safety unmanaged.
Why direct MCP connections stop scaling
The first MCP connection feels cheap. The fifth recreates the N×M problem MCP was meant to solve, this time as governance. Fifty One Degrees describes the pattern: every agent connects to every server on its own terms, with its own credentials, permissions, and logs. OmniDataTec names the trigger that usually ends the arrangement: risk or audit asks which agent touched a table, and nobody can answer without five greps and five retention policies.
Practitioner reports match the same wall. On Hacker News, the Armour Show HN author reports connecting IDE agents to nearly all internal services via MCP, then hitting all-or-nothing permissions: read-only made agents useless, write access made a bad prompt catastrophic. The Cordon Show HN frames the protocol gap bluntly as an author report: without a policy layer, an agent is off or full admin, and "trust the model" is the default answer. Treat those as practitioner reports, not measured demand. They still explain why teams look for an MCP gateway after the second or third production agent.
A gateway built for accountability owns identity-to-capability mapping, a reviewed tool catalogue, one audit trail keyed to a principal, a kill switch, and an egress boundary for which servers agents may reach at all. Built only for throughput, it is another hop with the same credential mess underneath.
Climb the placement ladder before you fund a central hop
Start with an in-process allowlist or per-agent sidecar before a central MCP gateway, because a shared hop with no catalogue owner becomes a rubber stamp that teams route around. OmniDataTec's placement table is the decision rule most definition pages skip:
- In-process allowlist. One team, two or three servers, no cross-agent audit question yet. Load only the tools that agent needs, emit structured logs, and keep credentials out of prompts. Cheap, and blind to estate-wide questions.
- Per-agent sidecar. Mixed runtimes or teams that need local autonomy. The sidecar injects credentials and enforces per-agent policy as calls leave the process. Still weak when you need one revocation point across agents.
- Centralised gateway. Shared upstreams, regulated data, or an auditor who will ask a question that spans agents. Identity, catalogue, egress, audit, and cost limits live in one place. Fails when nobody is staffed to run the catalogue.
Use this promote checklist. Answer yes to two or more before funding a central MCP gateway:
- More than one product team shares the same upstream MCP server.
- Someone outside engineering will ask which agent read or wrote a record last week.
- You need to revoke a tool or server for every agent in one action.
- Developers are connecting IDE agents (Cursor, Claude Code, and similar) to internal MCP servers with long-lived keys.
- You cannot explain tool-call cost or volume by team from one place.
Fifty One Degrees adds commercial signals (regulation, partner agents hitting your systems, unexplained AI bills). Keep those commercial signals, and still run the placement ladder before any strategy deck. A 40-person firm with one CRM and one assistant should switch on native connectors first, not stand up ContextForge.
Finish the MCP-versus-API choice before you front servers with a gateway
Do not put an MCP gateway in front of one or two servers you have not yet decided should be MCP at all; finish the MCP-versus-API choice and a named owner for each server first. A gateway multiplies whatever you put behind it. If an action should stay a deterministic workflow with a direct API call, wrapping it as MCP and then fronting it with a gateway adds two layers of indirection and none of the accountability you wanted.
Before gateway design:
- For each candidate action, apply the MCP versus API decision: assistant-driven discovery and multi-client reuse favor MCP; fixed triggers, webhooks, and batch syncs usually stay direct API.
- Give every retained MCP server a named owner, a read-versus-write policy, and a retirement path. Ownerless servers become shadow MCP, the tool-layer cousin of shadow AI.
- Prefer wrapping a finished workflow as one outcome-shaped tool over exposing raw tables. Fifty One Degrees calls this curation; without it the model explores until something looks plausible and your audit log fills with fishing expeditions.
- Only then ask whether those owned servers are shared enough to justify a central door.
Build or buy the custom MCP server when the action earns MCP. Add the gateway when several agents and several servers need one policy and one log. Build or buy the server when the action earns MCP. Add the gateway when several agents and several servers need one policy and one log.
What the MCP gateway must own on day one
Fund catalogue projection and a single kill switch before routing or caching features, because an agent that can see a tool will try it and a refused call becomes a retry loop. OmniDataTec's six-stage pipeline sets the order: catalogue projection and audit emission first, response inspection next, routing and caching last. The first production cut should not try to do all six cleverly.
Minimum viable control plane
| Stage | Own on day one? | Why |
|---|---|---|
| Admission (authenticate caller and acting principal) | Yes | Unattributable calls cannot be audited later |
| Identity resolution to a policy set | Yes | Policy must change without redeploying agents |
| Catalogue projection (return only allowed tools) | Yes | Visible tools get tried; late denies burn tokens and pollute traces |
| Call brokering with credential inject | Yes | Agents call tools by name and never hold upstream keys |
| Audit emission (one event per call, allowed or denied) | Yes | Pause and policy changes cannot be checked without a record |
| Response inspection | Soon | Tool output is untrusted input; cap size and strip instruction-shaped payloads |
| Fancy routing, caching, multi-cluster federation | Later | Useful after the catalogue and kill switch work |
Official and product docs show the same centre of gravity even when packaging differs. IBM ContextForge federates MCP, A2A, and REST behind one governed endpoint with auth, registry, and OpenTelemetry. Docker's MCP Gateway centralises orchestration of containerized MCP servers for developers and security teams. Microsoft's mcp-gateway describes a reverse proxy and lifecycle layer for MCP servers in Kubernetes. Buy or adopt plumbing when it helps. Own the curated tool layer, because that layer is your business logic written for machines (Fifty One Degrees).
Worked example: finance agents sharing a warehouse server
Accounts payable runs an invoice-triage agent. FP&A starts a close-checklist agent. Both need warehouse reads. Neither should hold the warehouse service key.
- Keep the LLM gateway for both agents' model calls and token budgets.
- Register one warehouse MCP server with a named data-platform owner, read-only tools only, and pinned tool descriptions.
- Until a third team arrives, a sidecar or tight allowlist may be enough if logs already answer the audit question.
- When HR asks for the same warehouse tools and security asks for one revocation path, promote a central MCP gateway: project a curated catalogue per principal, inject credentials at the gateway, emit one audit event per call, and keep a kill switch that disables
warehouse.*for every agent without nine deploys. - Leave write tools denied or behind human approval until the read path is boring.
Policy sketch
principals:
- id: "ap-invoice-agent"
acting_for: required
environments: ["prod"]
tools:
allow: ["warehouse.read_invoices", "cases.read_*"]
require_approval: ["cases.write_note"]
deny: ["*.delete_*", "payments.*", "warehouse.write_*"]
limits:
calls_per_minute: 60
max_response_bytes: 65536
kill_switch:
disable_tools: ["warehouse.*"] # one action, all agents
Ignore the YAML shape. What matters is that deny evaluates last, require_approval exists for writes, and disable is a config change rather than a scavenger hunt.
Failure modes that make a gateway worse than direct connections
A central MCP gateway holds credentials and sees every call. That makes it the highest-value target in the estate. OmniDataTec records the practical failures:
- Catalogue as a ticket queue. If registering a tool takes weeks, teams route around the gateway and you lose the visibility you paid for. Budget review capacity or do not centralise the catalogue.
- Description drift. An upstream server changes a tool description, agent behaviour changes, and no deployment happened in your repo. Pin and diff descriptions; treat a change as a prompt change. This pairs with signed skill and tool schema integrity when you need cryptographic pins.
- Upstream impersonation and supply chain. A gateway that does not pin upstream server identity centralises blast radius. Vendor and review the gateway package itself; do not pull an unreviewed dependency at build time because it claims to be an "AI gateway."
- Latency across many tool hops. Agents make many calls per task. A hop that looks cheap in isolation shows up across a forty-call trajectory. Measure end-to-end task latency before you celebrate the architecture diagram.
- Rubber-stamp ownership. An unowned gateway adds a hop and a false sense of control. Assign a named operator, a monthly log review, and an explicit "we will turn this off if nobody owns the catalogue" exit.
Skip the central gateway when you have one team and fewer than about three MCP servers, when nobody will own the catalogue, or when the real problem is over-broad grants inside a single agent. Fix the grants first (tool-call validation and MCP server security).
Verify before you call it production
Run these checks before any second team depends on the gateway:
- Principal attribution. Every allowed and denied call records agent identity, acting human or service owner, environment, tool name, upstream server identity, argument hash, and outcome.
- Catalogue projection. A principal denied
payments.*never receives those tool definitions intools/listresults, not merely a deny at call time. - Credential isolation. Agent logs and traces contain tool names and outcomes, never upstream API keys or OAuth refresh tokens.
- Kill switch drill. Disable a tool estate-wide and confirm every agent fails closed within your agreed window without redeploying agents.
- LLM gateway still owns model traffic. Token budgets, provider failover, and prompt guardrails remain on the model path; the MCP gateway does not become a second place to hide provider keys.
- Bypass closed. Direct agent-to-server routes for shared upstreams are removed or firewalled after cutover, the same way you close direct egress when deploying an LLM gateway.
- Owner and review cadence. A named human owns the catalogue, and a monthly review of unused tools, failed calls, and description diffs is on the calendar.
If check 2 or 4 fails, you have a proxy, not a gateway. If check 7 fails, pause the rollout.
Common mistakes
- Treating "AI gateway" marketing as proof you already cover MCP tool traffic. Ask which layer the product actually mediates.
- Fronting every MCP server on day one, including personal IDE experiments that should stay local.
- Exposing raw schemas and hoping call-time RBAC is enough. Catalogue projection is the behavioural control.
- Buying plumbing and outsourcing curation. The curated tool layer encodes how your business works; keep it yours.
- Skipping the platform ownership question. A gateway without a paved-road owner becomes ticket-ops with better diagrams.
What to do next
Inventory every MCP server, personal connector, and long-lived key your agents and IDE assistants already use. For each shared upstream, decide allowlist, sidecar, or central gateway using the promote checklist above. If two or more promote signals are true, fund catalogue projection, credential inject, audit emission, and a kill switch with a named owner before you touch routing features. If you want a second opinion on that inventory and the promote decision, start with an AI audit or the custom MCP server conversation for servers that are not ready to sit behind a door.