An LLM gateway sits between an AI application and the language models it calls. An AI gateway may describe that same route, or a wider service that also handles connections to tools and other agents. There is no fixed feature boundary between the names. To understand what a gateway does, follow the requests it actually receives.
The name alone does not tell you which paths a product handles. These examples show where a gateway can sit. MCP (Model Context Protocol) connects AI clients to external tools and data sources.
| Request | Example path | What to verify |
|---|---|---|
| Model call | Application → LLM gateway → model | Routing, token usage, provider access and checks on prompts or responses. |
| MCP tool call | AI client → MCP gateway → MCP server | Who may invoke the tool, which action is allowed and how the downstream credential is used. |
| Direct action | Agent runtime → API or local tool | Whether that path is routed through any enforceable control at all. |
| Agent handoff | One agent → another | Who may invoke the second agent and how the handoff is recorded. |
Consider a support assistant that drafts a reply and then updates a customer record. Asking a model to write the reply is one request. Updating the record is another. Routing the first request through a gateway does not automatically put the second under its control.
What does an LLM gateway handle?
An LLM gateway receives model requests from an application, sends them to the configured provider or model, and returns the response. Depending on its setup, it can record token usage, apply budgets, route around an unavailable model, cache responses and check prompts or outputs. Our article “What is an LLM gateway?” explains those functions in more detail.
This describes the model route. A model response can include a proposed tool call, and some gateways can inspect or withhold that proposal. Whether the proposed action is actually stopped depends on the integration with the application that runs the tool. Seeing a tool call in a model response is not the same as controlling the request that reaches the tool.
Why does the AI gateway label vary?
Cloudflare calls its model-facing product AI Gateway and documents routing, caching, spend limits and checks on prompts and responses. LiteLLM also uses “AI Gateway,” but its documented endpoint can handle model requests, MCP tools and agent connections. Kong describes its AI Gateway as covering three traffic types: LLM, MCP and agent-to-agent calls. These products use the same label for different scopes.
The word “AI” can also include model types beyond language, such as image or speech models. Supporting more kinds of model calls does not, by itself, establish that a gateway governs tool execution. Check the routes and features in the particular product and configuration.
Where does the difference become visible?
Suppose the support assistant asks a model to summarize a ticket. The model then proposes updating the customer record. The application may carry out that update through an MCP server. The model request and the tool request can travel through different routes, even though they belong to one user task.
A gateway on the model route can see the prompt and response it receives. Some products can flag or block a proposed tool call before the application acts, if the application honors that decision. A gateway on the MCP route can evaluate the tool request itself before forwarding it. For a direct API call or a local action, the team needs to identify the control on that path as well. The product name alone does not tell you which of these checks exists.
What should you ask before choosing one?
- Which requests go through the gateway? Include model calls, MCP tools, direct APIs, local tools and agent handoffs that matter to your workflow. Identify any direct routes around it.
- Where can it stop a request? Ask whether a decision is enforced before the model call, before the client executes a proposed tool call, or before the downstream tool receives it.
- Whose identity and credential does it use? Check whether controls distinguish the application, agent and person, and where the credential used for the downstream service resides.
- Can you reconstruct an action? Check whether logs connect the model response, tool request, policy decision and downstream result.
What it comes down to
A model gateway may be enough for an application that only calls models. Once an agent acts on external systems, its tool route needs a control as well. One broader AI gateway may cover both routes; another setup may use an LLM gateway alongside an MCP gateway or existing API controls. The useful distinction is the route each product governs and the decision it can enforce there.
Related reading: What is an MCP gateway?; Tool-call policy: should a model decide what AI can do?
Frequently asked questions
Is an AI gateway always broader than an LLM gateway?
No. Some vendors use AI gateway for model traffic. Others use it for model, tool and agent connections. Check the actual routes rather than infer scope from the name.
Can a product be both an LLM gateway and an AI gateway?
Yes. A product can use the AI gateway label while providing LLM gateway functions, and it may support additional kinds of traffic. Its configuration determines which paths your applications actually use.
Does a model gateway control what an agent does with a tool?
It can inspect a proposed tool call in a model response. That does not establish control of the separate request the application sends to the tool. Check where a decision is enforced and whether the tool route is covered.
