All posts

AI gateway vs LLM gateway: what is the difference?

An LLM gateway sits between an AI application and the language models it calls. An AI gateway may describe that same route, or a wider service that also handles connections to tools and other agents. There is no fixed feature boundary between the names. To understand what a gateway does, follow the requests it actually receives.

The name alone does not tell you which paths a product handles. These examples show where a gateway can sit. MCP (Model Context Protocol) connects AI clients to external tools and data sources.

RequestExample pathWhat to verify
Model callApplication → LLM gateway → modelRouting, token usage, provider access and checks on prompts or responses.
MCP tool callAI client → MCP gateway → MCP serverWho may invoke the tool, which action is allowed and how the downstream credential is used.
Direct actionAgent runtime → API or local toolWhether that path is routed through any enforceable control at all.
Agent handoffOne agent → anotherWho may invoke the second agent and how the handoff is recorded.

Consider a support assistant that drafts a reply and then updates a customer record. Asking a model to write the reply is one request. Updating the record is another. Routing the first request through a gateway does not automatically put the second under its control.

What does an LLM gateway handle?

An LLM gateway receives model requests from an application, sends them to the configured provider or model, and returns the response. Depending on its setup, it can record token usage, apply budgets, route around an unavailable model, cache responses and check prompts or outputs. Our article “What is an LLM gateway?” explains those functions in more detail.

This describes the model route. A model response can include a proposed tool call, and some gateways can inspect or withhold that proposal. Whether the proposed action is actually stopped depends on the integration with the application that runs the tool. Seeing a tool call in a model response is not the same as controlling the request that reaches the tool.

Why does the AI gateway label vary?

Cloudflare calls its model-facing product AI Gateway and documents routing, caching, spend limits and checks on prompts and responses. LiteLLM also uses “AI Gateway,” but its documented endpoint can handle model requests, MCP tools and agent connections. Kong describes its AI Gateway as covering three traffic types: LLM, MCP and agent-to-agent calls. These products use the same label for different scopes.

The word “AI” can also include model types beyond language, such as image or speech models. Supporting more kinds of model calls does not, by itself, establish that a gateway governs tool execution. Check the routes and features in the particular product and configuration.

Where does the difference become visible?

Suppose the support assistant asks a model to summarize a ticket. The model then proposes updating the customer record. The application may carry out that update through an MCP server. The model request and the tool request can travel through different routes, even though they belong to one user task.

One AI task, two request paths: a model request through an LLM gateway, and a separate tool call through an MCP gatewayA support assistant sends a model request through an LLM gateway to a model and receives the response back through the same gateway. The response may propose updating a customer record. Only if the application invokes the tool does it make a separate tool call through an MCP gateway. If the call is forwarded, it reaches an MCP server, which makes a downstream API call to the CRM. A product called an AI gateway may handle the model path, the tool path, or both.One AI task, two request pathsExample: a support assistant summarizes a ticket and may update a customer record.1. Model request and responseSupport assistantAI applicationLLM gatewayModelThe model response may propose an update.The application makes a separate call if it invokes the tool.2. Tool request, if invokedSupport assistantAI applicationMCP gatewayMCP serverCRMif forwardeddownstream API callIllustrative paths. An “AI gateway” may handle the model path, tool path, or both. Check actual coverage.

A gateway on the model route can see the prompt and response it receives. Some products can flag or block a proposed tool call before the application acts, if the application honors that decision. A gateway on the MCP route can evaluate the tool request itself before forwarding it. For a direct API call or a local action, the team needs to identify the control on that path as well. The product name alone does not tell you which of these checks exists.

What should you ask before choosing one?

  • Which requests go through the gateway? Include model calls, MCP tools, direct APIs, local tools and agent handoffs that matter to your workflow. Identify any direct routes around it.
  • Where can it stop a request? Ask whether a decision is enforced before the model call, before the client executes a proposed tool call, or before the downstream tool receives it.
  • Whose identity and credential does it use? Check whether controls distinguish the application, agent and person, and where the credential used for the downstream service resides.
  • Can you reconstruct an action? Check whether logs connect the model response, tool request, policy decision and downstream result.

What it comes down to

A model gateway may be enough for an application that only calls models. Once an agent acts on external systems, its tool route needs a control as well. One broader AI gateway may cover both routes; another setup may use an LLM gateway alongside an MCP gateway or existing API controls. The useful distinction is the route each product governs and the decision it can enforce there.

Related reading: What is an MCP gateway?; Tool-call policy: should a model decide what AI can do?

Frequently asked questions

Is an AI gateway always broader than an LLM gateway?

No. Some vendors use AI gateway for model traffic. Others use it for model, tool and agent connections. Check the actual routes rather than infer scope from the name.

Can a product be both an LLM gateway and an AI gateway?

Yes. A product can use the AI gateway label while providing LLM gateway functions, and it may support additional kinds of traffic. Its configuration determines which paths your applications actually use.

Does a model gateway control what an agent does with a tool?

It can inspect a proposed tool call in a model response. That does not establish control of the separate request the application sends to the tool. Check where a decision is enforced and whether the tool route is covered.

Sources

Security for what you can least afford to lose

Learn how Station70 protects keys, identities, and access.

Learn more