Fire In Belly

MCP Gateway vs LLM Gateway for Internal AI Workflows: When Tool Calls Need Their Own Door

12 min read

Labelled yellow fibre-optic cables plugged into a network switch in a server rack

You already put model calls behind an LLM gateway: one place for keys, quotas, and provider failover. Then product teams wired Claude, Cursor, and a custom agent straight to three MCP servers over the warehouse, CRM, and ticketing system. Six months later nobody can answer which agent read the customer table last Tuesday without grepping five services. The LLM gateway is fine. The gap is tool traffic without a door.

This guide is for platform and security leaders who must decide when an MCP gateway earns a place next to the model gateway. It assumes you already know when an action should be an MCP server versus a direct API, how to harden a single MCP server, and when shared platform components deserve a paved road. The question here is narrower: when do tool calls need their own control plane, which placement should you choose, and what must you own before you buy or build plumbing.

The short answer

Keep model traffic on the LLM gateway. Treat tool traffic as a separate path. Promote a central MCP gateway only when more than one team shares upstream servers, an auditor will ask a cross-agent question, or revocation must be one action. Until then, prefer an in-process allowlist or a per-agent sidecar with structured logs.

LayerGovernsTypical controlsBlind without the other layer
LLM / AI gatewayApplications calling modelsProvider routing, token budgets, prompt guardrails, model cost attributionWhich tools an agent can invoke after the model decides
MCP gatewayAgents calling tools over MCPCatalogue, identity, tool allowlists, credential isolation, tool-call audit, kill switchModel cost, provider failover, and prompt-level safety
In-process allowlistOne agent processLocal tool load list and local logsEstate-wide audit and one-shot revocation
Per-agent sidecarCalls leaving one agentLocal credential inject and per-agent policyCross-agent patterns on one upstream

Tetrate's definitions put the split cleanly: an LLM gateway sits between apps and model providers; an MCP gateway sits between agents and MCP servers; "AI gateway" increasingly means both under one control plane. TrueFoundry's comparison makes the operational point: governing only the model leaves tool actions untracked, and governing only tools leaves spend and prompt safety unmanaged.

Why direct MCP connections stop scaling

The first MCP connection feels cheap. The fifth recreates the N×M problem MCP was meant to solve, this time as governance. Fifty One Degrees describes the pattern: every agent connects to every server on its own terms, with its own credentials, permissions, and logs. OmniDataTec names the trigger that usually ends the arrangement: risk or audit asks which agent touched a table, and nobody can answer without five greps and five retention policies.

Practitioner reports match the same wall. On Hacker News, the Armour Show HN author reports connecting IDE agents to nearly all internal services via MCP, then hitting all-or-nothing permissions: read-only made agents useless, write access made a bad prompt catastrophic. The Cordon Show HN frames the protocol gap bluntly as an author report: without a policy layer, an agent is off or full admin, and "trust the model" is the default answer. Treat those as practitioner reports, not measured demand. They still explain why teams look for an MCP gateway after the second or third production agent.

A gateway built for accountability owns identity-to-capability mapping, a reviewed tool catalogue, one audit trail keyed to a principal, a kill switch, and an egress boundary for which servers agents may reach at all. Built only for throughput, it is another hop with the same credential mess underneath.

Climb the placement ladder before you fund a central hop

Start with an in-process allowlist or per-agent sidecar before a central MCP gateway, because a shared hop with no catalogue owner becomes a rubber stamp that teams route around. OmniDataTec's placement table is the decision rule most definition pages skip:

  1. In-process allowlist. One team, two or three servers, no cross-agent audit question yet. Load only the tools that agent needs, emit structured logs, and keep credentials out of prompts. Cheap, and blind to estate-wide questions.
  2. Per-agent sidecar. Mixed runtimes or teams that need local autonomy. The sidecar injects credentials and enforces per-agent policy as calls leave the process. Still weak when you need one revocation point across agents.
  3. Centralised gateway. Shared upstreams, regulated data, or an auditor who will ask a question that spans agents. Identity, catalogue, egress, audit, and cost limits live in one place. Fails when nobody is staffed to run the catalogue.

Use this promote checklist. Answer yes to two or more before funding a central MCP gateway:

  • More than one product team shares the same upstream MCP server.
  • Someone outside engineering will ask which agent read or wrote a record last week.
  • You need to revoke a tool or server for every agent in one action.
  • Developers are connecting IDE agents (Cursor, Claude Code, and similar) to internal MCP servers with long-lived keys.
  • You cannot explain tool-call cost or volume by team from one place.

Fifty One Degrees adds commercial signals (regulation, partner agents hitting your systems, unexplained AI bills). Keep those commercial signals, and still run the placement ladder before any strategy deck. A 40-person firm with one CRM and one assistant should switch on native connectors first, not stand up ContextForge.

Finish the MCP-versus-API choice before you front servers with a gateway

Do not put an MCP gateway in front of one or two servers you have not yet decided should be MCP at all; finish the MCP-versus-API choice and a named owner for each server first. A gateway multiplies whatever you put behind it. If an action should stay a deterministic workflow with a direct API call, wrapping it as MCP and then fronting it with a gateway adds two layers of indirection and none of the accountability you wanted.

Before gateway design:

  1. For each candidate action, apply the MCP versus API decision: assistant-driven discovery and multi-client reuse favor MCP; fixed triggers, webhooks, and batch syncs usually stay direct API.
  2. Give every retained MCP server a named owner, a read-versus-write policy, and a retirement path. Ownerless servers become shadow MCP, the tool-layer cousin of shadow AI.
  3. Prefer wrapping a finished workflow as one outcome-shaped tool over exposing raw tables. Fifty One Degrees calls this curation; without it the model explores until something looks plausible and your audit log fills with fishing expeditions.
  4. Only then ask whether those owned servers are shared enough to justify a central door.

Build or buy the custom MCP server when the action earns MCP. Add the gateway when several agents and several servers need one policy and one log. Build or buy the server when the action earns MCP. Add the gateway when several agents and several servers need one policy and one log.

What the MCP gateway must own on day one

Fund catalogue projection and a single kill switch before routing or caching features, because an agent that can see a tool will try it and a refused call becomes a retry loop. OmniDataTec's six-stage pipeline sets the order: catalogue projection and audit emission first, response inspection next, routing and caching last. The first production cut should not try to do all six cleverly.

Minimum viable control plane

StageOwn on day one?Why
Admission (authenticate caller and acting principal)YesUnattributable calls cannot be audited later
Identity resolution to a policy setYesPolicy must change without redeploying agents
Catalogue projection (return only allowed tools)YesVisible tools get tried; late denies burn tokens and pollute traces
Call brokering with credential injectYesAgents call tools by name and never hold upstream keys
Audit emission (one event per call, allowed or denied)YesPause and policy changes cannot be checked without a record
Response inspectionSoonTool output is untrusted input; cap size and strip instruction-shaped payloads
Fancy routing, caching, multi-cluster federationLaterUseful after the catalogue and kill switch work

Official and product docs show the same centre of gravity even when packaging differs. IBM ContextForge federates MCP, A2A, and REST behind one governed endpoint with auth, registry, and OpenTelemetry. Docker's MCP Gateway centralises orchestration of containerized MCP servers for developers and security teams. Microsoft's mcp-gateway describes a reverse proxy and lifecycle layer for MCP servers in Kubernetes. Buy or adopt plumbing when it helps. Own the curated tool layer, because that layer is your business logic written for machines (Fifty One Degrees).

Worked example: finance agents sharing a warehouse server

Accounts payable runs an invoice-triage agent. FP&A starts a close-checklist agent. Both need warehouse reads. Neither should hold the warehouse service key.

  1. Keep the LLM gateway for both agents' model calls and token budgets.
  2. Register one warehouse MCP server with a named data-platform owner, read-only tools only, and pinned tool descriptions.
  3. Until a third team arrives, a sidecar or tight allowlist may be enough if logs already answer the audit question.
  4. When HR asks for the same warehouse tools and security asks for one revocation path, promote a central MCP gateway: project a curated catalogue per principal, inject credentials at the gateway, emit one audit event per call, and keep a kill switch that disables warehouse.* for every agent without nine deploys.
  5. Leave write tools denied or behind human approval until the read path is boring.

Policy sketch

principals:
  - id: "ap-invoice-agent"
    acting_for: required
    environments: ["prod"]
    tools:
      allow: ["warehouse.read_invoices", "cases.read_*"]
      require_approval: ["cases.write_note"]
      deny: ["*.delete_*", "payments.*", "warehouse.write_*"]
    limits:
      calls_per_minute: 60
      max_response_bytes: 65536
kill_switch:
  disable_tools: ["warehouse.*"]   # one action, all agents

Ignore the YAML shape. What matters is that deny evaluates last, require_approval exists for writes, and disable is a config change rather than a scavenger hunt.

Failure modes that make a gateway worse than direct connections

A central MCP gateway holds credentials and sees every call. That makes it the highest-value target in the estate. OmniDataTec records the practical failures:

  • Catalogue as a ticket queue. If registering a tool takes weeks, teams route around the gateway and you lose the visibility you paid for. Budget review capacity or do not centralise the catalogue.
  • Description drift. An upstream server changes a tool description, agent behaviour changes, and no deployment happened in your repo. Pin and diff descriptions; treat a change as a prompt change. This pairs with signed skill and tool schema integrity when you need cryptographic pins.
  • Upstream impersonation and supply chain. A gateway that does not pin upstream server identity centralises blast radius. Vendor and review the gateway package itself; do not pull an unreviewed dependency at build time because it claims to be an "AI gateway."
  • Latency across many tool hops. Agents make many calls per task. A hop that looks cheap in isolation shows up across a forty-call trajectory. Measure end-to-end task latency before you celebrate the architecture diagram.
  • Rubber-stamp ownership. An unowned gateway adds a hop and a false sense of control. Assign a named operator, a monthly log review, and an explicit "we will turn this off if nobody owns the catalogue" exit.

Skip the central gateway when you have one team and fewer than about three MCP servers, when nobody will own the catalogue, or when the real problem is over-broad grants inside a single agent. Fix the grants first (tool-call validation and MCP server security).

Verify before you call it production

Run these checks before any second team depends on the gateway:

  1. Principal attribution. Every allowed and denied call records agent identity, acting human or service owner, environment, tool name, upstream server identity, argument hash, and outcome.
  2. Catalogue projection. A principal denied payments.* never receives those tool definitions in tools/list results, not merely a deny at call time.
  3. Credential isolation. Agent logs and traces contain tool names and outcomes, never upstream API keys or OAuth refresh tokens.
  4. Kill switch drill. Disable a tool estate-wide and confirm every agent fails closed within your agreed window without redeploying agents.
  5. LLM gateway still owns model traffic. Token budgets, provider failover, and prompt guardrails remain on the model path; the MCP gateway does not become a second place to hide provider keys.
  6. Bypass closed. Direct agent-to-server routes for shared upstreams are removed or firewalled after cutover, the same way you close direct egress when deploying an LLM gateway.
  7. Owner and review cadence. A named human owns the catalogue, and a monthly review of unused tools, failed calls, and description diffs is on the calendar.

If check 2 or 4 fails, you have a proxy, not a gateway. If check 7 fails, pause the rollout.

Common mistakes

  • Treating "AI gateway" marketing as proof you already cover MCP tool traffic. Ask which layer the product actually mediates.
  • Fronting every MCP server on day one, including personal IDE experiments that should stay local.
  • Exposing raw schemas and hoping call-time RBAC is enough. Catalogue projection is the behavioural control.
  • Buying plumbing and outsourcing curation. The curated tool layer encodes how your business works; keep it yours.
  • Skipping the platform ownership question. A gateway without a paved-road owner becomes ticket-ops with better diagrams.

What to do next

Inventory every MCP server, personal connector, and long-lived key your agents and IDE assistants already use. For each shared upstream, decide allowlist, sidecar, or central gateway using the promote checklist above. If two or more promote signals are true, fund catalogue projection, credential inject, audit emission, and a kill switch with a named owner before you touch routing features. If you want a second opinion on that inventory and the promote decision, start with an AI audit or the custom MCP server conversation for servers that are not ready to sit behind a door.


About Fire In Belly: Independent senior engineering from Tallinn, Estonia. We design and build MCP servers, custom AI agents, and internal tools with the MCP gateways, per-agent tool catalogues, and tested kill switches described above, at published fixed prices. Schedule a call to discuss your next project.