LLM gateway

An LLM gateway receives requests from all of a company’s AI applications and forwards each one to the right language model, whether that model runs at OpenAI, Anthropic, Google or on the company’s own servers. Along the way it checks whether the calling application is allowed to, records what the call cost and passes the answer back. It usually exposes a single interface for this, often in the format of the OpenAI API, so applications can switch models without code changes. AI gateway and LLM proxy are common alternative names, sometimes for products with a narrower or broader scope.

There is no formal standard for the term. The pattern comes from the classic API gateway and was adapted to how language models work: billing by token, frequent model changes, outages at individual providers and confidential content inside requests. The open-source project LiteLLM describes itself as a self-hosted server that gives applications one OpenAI-compatible endpoint for more than a hundred providers. Cloudflare offers an AI Gateway as a hosted service, and Microsoft has built comparable features into Azure API Management (as of November 2026).

For executives, an LLM gateway is an architecture decision that shapes cost, data protection and vendor dependence. Without a shared layer, every application has its own keys, contracts and invoices. Nobody sees at a glance which data goes to which provider or who drives which costs. With a gateway, there is one place where rules apply to every call.

How a call moves through the gateway.

The application sends its request to the gateway instead of the model provider and authenticates with its own key. LiteLLM calls these virtual keys: each user, team or project gets its own key with defined access to specific models. The providers’ actual credentials live only inside the gateway.

Several checks follow. Is this key allowed to use the requested model? Is the budget used up or the rate limit reached? The gateway can also inspect the content, for example to mask personal data before it leaves the company. A cache can answer a request that has been answered before without calling a model at all. Microsoft also describes semantic caching in Azure API Management, which recognizes similar requests.

Only then does the request reach a model. The answer travels back through the gateway, which books the token usage and cost to the key and logs the call. Each step adds latency. Developers route around a slow gateway, so response time is a key number to watch in production.

Model routing decides which model answers.

Model routing is the gateway function that picks a model for each request. The simplest form is an alias: the application asks for “default model,” and the gateway decides which specific model sits behind that name. If the company switches providers, it changes that one entry.

Rule-based routing goes further. Cloudflare’s documentation on dynamic routing describes sending different user segments to different models, switching to a fallback model once a cost quota per key and period is exceeded, and rolling out new models gradually or as an A/B test without touching application code. If a provider goes down, fallback routing switches to a backup model.

Researchers are working on learned routers. FrugalGPT (Chen, Zaharia and Zou, 2023) describes a cascade: a cheap model answers first, and a more expensive one steps in only if that answer falls short. RouteLLM (Ong et al., 2024) trains routers that choose between a stronger and a weaker model for each request. Both aim to use large models only where a task needs them. Whether a smaller model is good enough only shows up in tests on real cases.

Costs become visible before the invoice arrives.

Model providers bill by token, meaning the amount of text processed in input and output. A gateway counts those tokens on every call and books them to an application, a team or a project. LiteLLM tracks spend by key, team, user and custom tags. Azure API Management offers a policy that caps tokens per minute or sets a token quota per hour, day, week, month or year.

Per-key budgets stop runaway spending early: an agent stuck in a loop, a broken batch job, a test that accidentally points at the largest model. The OWASP project lists this risk in its 2025 Top 10 for applications built on language models as Unbounded Consumption. It includes an attack called Denial of Wallet, in which attackers exploit pay-per-use billing to drive up costs. OWASP recommends rate limits, quotas and ongoing monitoring of consumption.

A gateway doesn’t lower model costs by itself. It makes them traceable. Costs come down once someone reviews the numbers, sends simple tasks to smaller models and serves repeat requests from the cache.

Logging and data protection come down to configuration.

A gateway sees every request and every answer, which makes calls traceable. Azure API Management, for instance, can log prompts and completions for billing and auditing. At the same time, the log itself becomes a store of confidential and personal data.

So someone has to decide what gets stored (only metadata such as timestamp, application, model, tokens and cost, or the content too), for how long, and who can read it. Under the EU’s General Data Protection Regulation, Article 5 requires that personal data be kept no longer than necessary. If the gateway runs as a vendor’s hosted service, that vendor processes the data on the company’s behalf and needs a data processing agreement under Article 28 GDPR, in addition to the agreements with the model providers.

Configured well, the gateway helps enforce data rules. It can make sure that requests containing customer data go only to models covered by the right contract or to self-hosted models. That only covers calls that pass through the gateway. Anyone using a personal chat account bypasses every rule. A gateway does not replace a policy on which tools are allowed.

API gateway, LLM gateway and orchestration are different layers.

An API gateway manages any kind of interface. It handles authentication, load balancing and rate limiting, but it counts requests instead of tokens and knows nothing about models or their outages. An LLM gateway is built for language models. It bills by token, translates between the formats of different providers and switches models when one fails.

The line blurs when an API gateway adds features for language models. Microsoft states plainly that the AI gateway in Azure API Management extends the existing API gateway and is not a separate offering. Four questions show whether a product does the job. Does it count tokens per application? Can it set budgets in tokens or money? Does it switch models automatically when a provider fails? Does it speak the interfaces of several providers?

AI orchestration sits one layer above. It controls workflows made of multiple steps, tools and checks within an application. The gateway handles each individual model call. An agent that plans five steps sends five calls through the same gateway. An LLM proxy usually means a slimmer version that only forwards and logs calls.

What a good setup looks like.

In a good setup, every application has its own key, and the providers’ credentials live only in the gateway. Every call can be traced to an application and a department, with its cost. Switching models is a configuration change, tested first against fixed test cases. The fallback model gets tested regularly, because an entry in the configuration proves nothing. Log retention is defined and documented.

The gateway is a central dependency. If it goes down, every AI application in the company stops. It needs monitoring, alerts and an update plan like any other critical system. Whether self-hosted or bought as a service, a gateway costs hosting or licensing, maintenance and monitoring on top of model costs.

A gateway is overkill when one team runs a single application with a single provider. That provider’s console and key management are enough. The shared layer starts to pay off once several applications, teams or providers are involved.

Where the term stops working

An LLM gateway governs the path to the model, not whether the answer is right, which depends on the model, the context and expert review. It differs from a general API gateway, which knows neither tokens nor models, from AI orchestration, which runs multi-step workflows, and from guardrails, which can sit inside the gateway as checks. A model router is one component of a gateway, not a substitute for it.