← All Thinking

Infrastructure & sovereignty

Build your AI platform one layer at a time

As AI in companies moves from experimentation to maturity, a company needs its own AI platform, built one layer at a time; an LLM gateway is the low-friction first layer: every model and tool call passes through one internal gateway, so decisions rest on real usage; with one portal that sets the rules for the models, tools, skills, knowledge bases and agents that agents use, governance becomes checking what is approved against what is used, and the company owns the platform instead of depending entirely on one vendor.

View on LinkedIn

As AI moves from experiments to maturity, a company needs its own AI platform, and an LLM gateway is the low-friction first layer

I see AI in companies moving from experimentation to maturity. That's when a proper AI platform strategy matters: governance, control, and the autonomy to choose your own vendors.

I've seen what happens without a gateway: AI arrives through dozens of teams at once, each with its own tools, models and keys, and every team sees only its own slice of who used what, and at what cost.

The platform in one picture

Agents use 5 kinds of things: models, tools, skills, knowledge bases and other agents. A platform governs all of them with a few pieces that fit together.

Identity. Company sign-in for people and agents.

The portal. One place that lists what the company has approved, who owns each item and who may use it.

The gateway. One entry point that every model and tool call passes through. It enforces the portal's rules and records every call.

The inventory. What's approved, checked against what's actually used.

The device. Security tools on the laptop find what the gateway can't see.

Diagram titled One portal, one gateway, under the eyebrow Governing what agents use, as stacked bands with small drawings. Identity: company sign-in for people and agents. Portal: what's approved, who owns it and who may use it. Inventory: what's approved, checked against what's used. Gateway: enforces the rules and records every call. What agents use: models, tools, skills, knowledge bases, agents. Beside the bands, a dashed column, the device: security tools find what the gateway can't see, with a drawing of a laptop, a helper robot and a magnifying glass on a table.
One portal, one gateway.

Analysts such as Bain call the whole set a control plane. The big AI suites each offer part of it, and a company can embrace one big suite or mix and match the pieces. My approach is to own the platform, so the company doesn't depend entirely on one vendor for it.

For each piece I'd start from what the company already runs: its identity provider, its cloud or API platform, its CMDB and its security tools.

Table titled Example blueprint, in 2 blocks: Microsoft, AWS and Google Cloud, then model vendors, API and security platforms and open source. Portal, which lists what's approved: Agent 365; AWS Agent Registry; Google Agent Registry; Claude library, ChatGPT plugins; Kong Konnect Catalog; MCP Registry, IBM ContextForge. Gateway: API Management AI gateway; AgentCore Gateway; Agent Gateway, Apigee; none from the model vendors; Kong AI Gateway, Cloudflare AI Gateway, Prisma AIRS AI Gateway; LiteLLM, agentgateway. Models, same model, different terms: Foundry Models; Bedrock; Model Garden; OpenAI API, Claude API; OpenRouter, Perplexity; vLLM, llm-d, Ollama. Knowledge bases: Foundry IQ, Work IQ; Bedrock Knowledge Bases; Agent Search; Claude Projects, ChatGPT company knowledge; Cloudflare AI Search; Onyx, Weaviate. Below, a band titled Already in the company: expand it to AI: identity, your identity provider, Microsoft Entra ID, Okta, Keycloak; inventory, your CMDB, ServiceNow CMDB, Jira Service Management Assets, GLPI; and in a dashed box, the device, your security tools, Microsoft Defender, CrowdStrike Falcon, Netskope. Small drawings of a person at a desk with a helper robot, a person reading ledgers with a helper robot, and a laptop with a helper robot on a table.
An example blueprint.

In Redesign before you re-platform I split the work into perform and transform. The platform fits transform: a central team owns it and its rules, and the perform teams build on it.

One approach for every team

Software development. Agents write, test and review code across the software development life cycle, the Agentic SDLC.

Infrastructure. Agents help run IT operations, what's known as AIOps: they read alerts, check systems and propose changes.

Business processes. Agents work inside a process, such as handling an invoice or a customer request.

Knowledge workers. People use assistants and agents to research, write and analyze.

Because the company owns the rules, each team can choose the approved models and agents that suit its work, whatever the vendor, as long as its tools point at the gateway.

I'd build these pieces starting with the gateway.

Layer 1: the gateway

It's the first layer of the platform, and a low-friction one: one internal gateway for every model and tool call, so decisions on cost, security, performance and access rest on real usage.

For the teams, little changes: they point compatible tools there and keep working.

Cloudflare runs its own AI this way, through one gateway for thousands of its people. Its engineers built it so they could add "per-user attribution, model catalog management, and permission enforcement later without touching any client configs."

I spent years with proxies, load balancers and firewalls, and the gateway looks familiar. In Where AI lives: from gigafactory to inference I noted that CDNs grew into WAFs, API gateways and Zero Trust.

Table titled Same controls, new payload, under the eyebrow We've built this before, with a strip illustration of a server room, a person at a monitor, a person walking and a helper robot. Network control and its job at the gateway: forward proxy, see which teams use which models; load balancer, retries and failover across models; web application firewall, guardrails block, mask or log; API gateway, keys, quotas and budgets per team; CDN, a cache answers repeated questions; security log, who called which model, and at what cost; data loss prevention, mask the personal data and secrets it detects; Zero Trust, verify the person or service behind each call. Panel, Where the analogy breaks: the payload carries meaning: a prompt attack can be rephrased endlessly.
The network controls we already know, and their job at the gateway.

The analogy breaks here: the payload now carries meaning. A firewall can match a known attack, but a prompt attack can be rephrased endlessly, and the UK's National Cyber Security Centre says prompt injection "may never be totally mitigated in the way that SQL injection attacks can be".

Which model answers

Load balancers learned to read the URL, and model routers now read the request itself. One example is the vLLM Semantic Router, an open project that can send easy questions to a small model and hard ones to a strong model with reasoning turned on.

It sits behind the company gateway, which keeps the keys, limits and traffic policy.

Illustration titled The path of a model request, under the eyebrow A router that reads the request: a 3-storey building in cross-section. 1, the company gateway, who's calling, what it costs, what's allowed; examples LiteLLM, Agent Router; a visitor at a reception desk, with the receptionist holding a paper. 2, the routing decision, which model fits this request; examples vLLM Semantic Router, LiteLLM Complexity Router (beta), Jev decision model (TypeSafe); a person with a paper chooses between 3 doors, a helper robot beside him. 3, the servers, which server runs it; examples vLLM, llm-d; a server room with a person at a desk and a helper robot. Footer: Examples, not recommendations.
The model-request path: the gateway, the router and the servers.

Routing by meaning is still early. A peer-reviewed benchmark found that several routers fail to reliably beat a simple baseline.

The same model can come direct from its maker, through the company's cloud subscription or, for open models, from the company's own servers, each with different terms for cost, data and features. The gateway lets the company choose for each request.

Many AI tools now make the routing decision for you with an "Auto" model setting: the vendor picks which model answers each request. Left unmanaged, that hands the vendor control over cost and quality, and the vendor's incentives can differ from the company's. In the end it's a choice: outsource the decision to a vendor, or take control of it.

Comparison titled Who picks the model?, under the eyebrow Auto in the tools. Left, Auto in each vendor's tool: a person hands a paper through a half-shuttered service window, a helper robot behind him. Right, the company's gateway: 2 people read a book together at a table in front of 3 doors, a helper robot beside them. Rules: the vendor's, or the company's. Cost and quality: set by the vendor, or managed by the company. Models: what the vendor offers, or what the company chooses. Closing line: Outsource the decision to a vendor, or take control of it.
Who picks the model: the vendor's Auto, or the company's gateway.

The gateway's usage data is the evidence for that choice: which work needs which model, and whether a router earns its place.

Tools through the same gateway

Agents call tools through MCP, the common protocol between agents and tools, and load skills, which are packaged instructions they reuse. A tool call is an action: it can change a record or move data. The Coalition for Secure AI says production deployments "should deploy a centralized proxy or agent gateway as the primary enforcement point".

Layer 2: the portal

Once the gateway shows what's in use, the portal is where the company sets the rules. I'd expect it to pay off the way a good shared service does. One team builds a tool, a skill or a knowledge base, and the next team finds it in the portal instead of building it again.

When an auditor asks what the company's agents depend on, there's one list to show. We've governed this kind of thing before, with package registries and company app stores.

Knowledge bases need the right access for each person, and with agents it matters more, because an agent doesn't apply common sense and uses everything it can reach.

In How my AI gets better at working with me I described the levels: my knowledge, my team's and my company's, plus what each business process needs. I'd expect one governed knowledge base at each level, shared by many agents, to cost less and stay more current than each team building its own copy.

Layer 3: the inventory

The portal holds what's approved, and the gateway's record shows what's actually used. Governance is checking one against the other. The joint guidance on agentic AI from national cyber agencies asks for the same: "periodically reconcile the registry against the live set of agents".

The 2 lists drift apart when people move faster than their company. McKinsey put it plainly this year: "Most employees are ready for AI. Most organizations aren't."

A KPMG and University of Melbourne report published in 2025 found that 48% of employees surveyed said they had uploaded company information into public AI tools.

Local skills and tools can bypass the gateway, so security tools on the device have to find them.

Diagram titled How the pieces connect, with plain lines between boxes and small drawings. Identity, a person at a desk with a badge and a helper robot, is joined by a line labelled sign-in to a dashed boundary, the device, which finds what bypasses the gateway and holds people and agents, with a drawing of a laptop and a helper robot on a table. From the device, a line labelled calls reaches the gateway, a person with a book at an open door. The portal, a person and a helper robot at a cabinet of index cards, is joined to the gateway by a line labelled rules. Below, the inventory, a person reading 2 open ledgers with a helper robot, is joined to the portal by a line labelled what's approved and to the gateway by a line labelled what's used. The gateway is joined by an unlabelled line to a dark box titled What agents use: models, tools, skills, knowledge bases, agents.
How the pieces connect.

As the platform grows

Network controls moved out of the central box over the years, to regions, to the edge and next to every service. Gateways are taking the same path: one portal that sets the rules, and many gateways close to the traffic.

Illustration titled From one gateway to many, under the eyebrow Where the gateway sits, with the line One portal sets the rules for every gateway. 1, one central gateway: simple to start, one point of failure; a single gatehouse on a hill by the water. 2, regional gateways: customer data may have to stay in its region; 3 gatehouses, each inside a dashed boundary. 3, next to every service: like the service mesh; a village of small buildings, each with its own door. 4, on every device: where I expect it to go next; 3 people at laptops, each laptop showing a small door, with a helper robot beside each.
From one gateway to many.

A single central gateway becomes, in Microsoft's words, "a global single point of regional failure". And a prompt holding customer data may have to stay in its region, so that region needs its own gateway.

Security tools already find the AI apps, local models and MCP servers on a laptop, and at least one decides which AI agents may run there. I expect the gateway to follow the Zero Trust path, with a small one on every device. For now most of the inspection stays central.

Which of your teams' AI calls can you see today, and through which gateway?

#WhereAILives #AIGovernance

Sources and links

Subscribe via RSSFollow on LinkedIn

Ask Daniel AI