As AI moves from experiments to maturity, a company needs its own AI platform, and an LLM gateway is the low-friction first layer
I see AI in companies moving from experimentation to maturity. That's when a proper AI platform strategy matters: governance, control, and the autonomy to choose your own vendors.
I've seen what happens without a gateway: AI arrives through dozens of teams at once, each with its own tools, models and keys, and every team sees only its own slice of who used what, and at what cost.
The platform in one picture
Agents use 5 kinds of things: models, tools, skills, knowledge bases and other agents. A platform governs all of them with a few pieces that fit together.
Identity. Company sign-in for people and agents.
The portal. One place that lists what the company has approved, who owns each item and who may use it.
The gateway. One entry point that every model and tool call passes through. It enforces the portal's rules and records every call.
The inventory. What's approved, checked against what's actually used.
The device. Security tools on the laptop find what the gateway can't see.
Analysts such as Bain call the whole set a control plane. The big AI suites each offer part of it, and a company can embrace one big suite or mix and match the pieces. My approach is to own the platform, so the company doesn't depend entirely on one vendor for it.
For each piece I'd start from what the company already runs: its identity provider, its cloud or API platform, its CMDB and its security tools.
In Redesign before you re-platform I split the work into perform and transform. The platform fits transform: a central team owns it and its rules, and the perform teams build on it.
One approach for every team
Software development. Agents write, test and review code across the software development life cycle, the Agentic SDLC.
Infrastructure. Agents help run IT operations, what's known as AIOps: they read alerts, check systems and propose changes.
Business processes. Agents work inside a process, such as handling an invoice or a customer request.
Knowledge workers. People use assistants and agents to research, write and analyze.
Because the company owns the rules, each team can choose the approved models and agents that suit its work, whatever the vendor, as long as its tools point at the gateway.
I'd build these pieces starting with the gateway.
Layer 1: the gateway
It's the first layer of the platform, and a low-friction one: one internal gateway for every model and tool call, so decisions on cost, security, performance and access rest on real usage.
For the teams, little changes: they point compatible tools there and keep working.
Cloudflare runs its own AI this way, through one gateway for thousands of its people. Its engineers built it so they could add "per-user attribution, model catalog management, and permission enforcement later without touching any client configs."
I spent years with proxies, load balancers and firewalls, and the gateway looks familiar. In Where AI lives: from gigafactory to inference I noted that CDNs grew into WAFs, API gateways and Zero Trust.
The analogy breaks here: the payload now carries meaning. A firewall can match a known attack, but a prompt attack can be rephrased endlessly, and the UK's National Cyber Security Centre says prompt injection "may never be totally mitigated in the way that SQL injection attacks can be".
Which model answers
Load balancers learned to read the URL, and model routers now read the request itself. One example is the vLLM Semantic Router, an open project that can send easy questions to a small model and hard ones to a strong model with reasoning turned on.
It sits behind the company gateway, which keeps the keys, limits and traffic policy.
Routing by meaning is still early. A peer-reviewed benchmark found that several routers fail to reliably beat a simple baseline.
The same model can come direct from its maker, through the company's cloud subscription or, for open models, from the company's own servers, each with different terms for cost, data and features. The gateway lets the company choose for each request.
Many AI tools now make the routing decision for you with an "Auto" model setting: the vendor picks which model answers each request. Left unmanaged, that hands the vendor control over cost and quality, and the vendor's incentives can differ from the company's. In the end it's a choice: outsource the decision to a vendor, or take control of it.
The gateway's usage data is the evidence for that choice: which work needs which model, and whether a router earns its place.
Tools through the same gateway
Agents call tools through MCP, the common protocol between agents and tools, and load skills, which are packaged instructions they reuse. A tool call is an action: it can change a record or move data. The Coalition for Secure AI says production deployments "should deploy a centralized proxy or agent gateway as the primary enforcement point".
Layer 2: the portal
Once the gateway shows what's in use, the portal is where the company sets the rules. I'd expect it to pay off the way a good shared service does. One team builds a tool, a skill or a knowledge base, and the next team finds it in the portal instead of building it again.
When an auditor asks what the company's agents depend on, there's one list to show. We've governed this kind of thing before, with package registries and company app stores.
Knowledge bases need the right access for each person, and with agents it matters more, because an agent doesn't apply common sense and uses everything it can reach.
In How my AI gets better at working with me I described the levels: my knowledge, my team's and my company's, plus what each business process needs. I'd expect one governed knowledge base at each level, shared by many agents, to cost less and stay more current than each team building its own copy.
Layer 3: the inventory
The portal holds what's approved, and the gateway's record shows what's actually used. Governance is checking one against the other. The joint guidance on agentic AI from national cyber agencies asks for the same: "periodically reconcile the registry against the live set of agents".
The 2 lists drift apart when people move faster than their company. McKinsey put it plainly this year: "Most employees are ready for AI. Most organizations aren't."
A KPMG and University of Melbourne report published in 2025 found that 48% of employees surveyed said they had uploaded company information into public AI tools.
Local skills and tools can bypass the gateway, so security tools on the device have to find them.
As the platform grows
Network controls moved out of the central box over the years, to regions, to the edge and next to every service. Gateways are taking the same path: one portal that sets the rules, and many gateways close to the traffic.
A single central gateway becomes, in Microsoft's words, "a global single point of regional failure". And a prompt holding customer data may have to stay in its region, so that region needs its own gateway.
Security tools already find the AI apps, local models and MCP servers on a laptop, and at least one decides which AI agents may run there. I expect the gateway to follow the Zero Trust path, with a small one on every device. For now most of the inspection stays central.
Which of your teams' AI calls can you see today, and through which gateway?