The most expensive sentence in an AI architecture review is five words long. We inspect the traffic, so we’re covered.

It ends the meeting. Everyone nods, the LLM proxy goes in the diagram, and the risk item gets closed. The sentence is expensive because it is half true, and the half that is false is the half everyone stops thinking about.

An LLM proxy, or AI gateway, is the chokepoint every prompt and completion passes through on the way to a model provider. Route, cache, meter, filter, log. Everyone agrees a regulated shop needs one. Nobody agrees why, and the why is what picks the product. Send three teams into the evaluation and they come back with three different winners.

Three Teams, Three Winners

The platform team evaluates on functionality. The finance team evaluates on cost. The security team evaluates on inspection. They are all looking at the same two obvious options, MuleSoft Anypoint Omni Gateway and Microsoft Azure API Management, plus a field of specialists, and each lens ranks them differently. The interesting part is not who wins. It is that the security lens, the one that carries the most weight in a bank, is arguing for the right product using the wrong reason.

Let me take the lenses one at a time, because the disagreement between them is the actual decision.

Functionality: Azure Wins the Feature Fight

On raw LLM-proxy capability, Azure API Management is ahead, and it is not especially close.

Microsoft has been iterating its GenAI gateway capabilities since 2024. You get a token-limit policy that meters prompt, completion, and reasoning tokens per key, semantic caching, backend pools with a circuit breaker that honors the provider’s Retry-After header, content safety with Prompt Shields, and native Entra managed identity so the gateway holds no provider keys at all. It reads like a product that has already survived contact with production, because it has.

MuleSoft’s AI Gateway answers with semantic routing and a native PII detection policy, and then it does something Azure does not. It governs agents, not just model calls. MCP and A2A traffic sit under the same policy plane as the LLM traffic, which is the argument I made in Foundry and Fabric: the value is unifying the agent control plane, not winning a feature checklist. If your problem is agent sprawl, that matters more than another caching option.

Both have honest gaps. MuleSoft reaches Anthropic only through Bedrock, and its core token and PII policies require Connected Mode, a live dependency on the control plane that a disconnected or air-gapped deployment cannot satisfy. Azure counts tokens per gateway rather than across instances, so a multi-region estate has quota blind spots. I traced the rebrand and the federation story in Beyond the Rebrand. The short version: for LLM-proxy duty, Azure wins functionality. For agent governance in one plane, MuleSoft is the more ambitious bet.

Cost: The Third Gateway You Do Not Need

The finance lens is where the evaluation gets uncomfortable, because the honest answer is often that you should buy nothing new.

Azure API Management is priced on published tiers with no platform tax. MuleSoft’s AI Gateway is free only in the sense that it comes bundled inside premium Anypoint editions whose entry price is six figures. Both meter tokens on top. So the real question is not which gateway is cheaper. It is whether you need a dedicated one at all.

If your estate is Azure OpenAI at the center, you very likely already own two AI gateways and are shopping for a third. A dedicated LLM proxy earns its keep in exactly three situations: a genuine multi-provider or multi-cloud strategy, a hard air-gap requirement, or concentration-risk pressure to avoid a single vendor. That is where the specialists belong. Kong AI Gateway is the strongest provider-neutral option you can run yourself or as a service. LiteLLM is the one to hold up as the air-gap play and the price anchor in any negotiation, since it self-hosts and keeps the data inside your boundary.

And retire the latency objection while you are here. LiteLLM’s own benchmarks put proxy overhead around two milliseconds, noise against a multi-second inference call. The proxy does not cost you latency. It costs you a new single point of failure and a new thing to learn, which is the tax I wrote about in The Architecture Is Clear. The Investment Isn’t. The finance lens picks Azure, or picks nothing, and it is usually right.

Security: The Sentence That Does the Damage

Now the lens that carries the most weight in a bank, and the one that needs the hardest look. The security team wants to inspect the traffic. The question worth asking out loud is whether that is business value or reflex.

It is both, and separating them is the whole point.

What the Proxy Really Earns

Start with the half that is true, because it is genuinely load-bearing. Inspecting LLM traffic at a chokepoint delivers real, measurable, and in banking legally required value, none of which has anything to do with stopping a hacker.

It catches the careless insider, who is the dominant risk by a wide margin. Cyberhaven found that a meaningful share of employees have pasted confidential company data into ChatGPT. That is not an attacker with a jailbreak. That is a well-meaning analyst pasting a customer list into a chatbot to summarize it, and a PII policy at the gateway is exactly the control that stops it. MuleSoft ships one as a first-class PII detection policy; Azure pairs metering with content safety.

It produces evidence, and in a regulated institution the evidence is the product. The GLBA Safeguards Rule requires covered institutions to monitor and log authorized-user activity and to oversee the third parties that process customer data. A model provider is such a third party. The EU AI Act mandates automatic event logging over a high-risk system’s lifetime and names financial institutions specifically. Model-risk supervision under the Federal Reserve’s SR 11-7 lineage treats a vendor model as your responsibility, not the vendor’s. The gateway is frequently the only place that evidence can be produced at all. In that framing, efficacy against attackers is not even the deciding variable. The examiner is.

So the proxy earns its cost. Cost governance, data-loss prevention against careless insiders, audit trails, shadow-AI discovery, model allow-listing. All real, all deterministic, all mandated. If the security team asked for the gateway on those grounds, there would be nothing to interrogate.

What Routes Around It

The interrogation is for the other grounds, the ones that get said in the meeting. We inspect the traffic, so we would catch an attack. That is the half that is false, and the evidence is not close.

The guardrails do not hold against anyone trying. A peer-reviewed study showed that simple character tricks, homoglyphs, zero-width characters, emoji smuggling, reached near-total evasion against six commercial guardrail systems. Independent testing cut Microsoft’s own Prompt Shields and text moderation detection rates by most of their claimed value using nothing exotic. OWASP still ranks prompt injection as the number one LLM risk and says plainly that a fool-proof prevention may not exist. A probabilistic filter is a speed bump, not a wall, and treating it as a wall is the error.

The real attack surface is below the wire, where the proxy is structurally blind. MCP tool poisoning hides instructions inside a tool description that the model reads and acts on before the user ever types anything, and a proof of concept exfiltrated an SSH key through it. The EchoLeak vulnerability in Microsoft 365 Copilot was a zero-click exfiltration that evaded Microsoft’s own injection classifier. None of that shows up as suspicious content in a prompt your gateway can read. It is the argument I made in What witness.ai Doesn’t See: the network layer sees a slice of the surface and reports it as the whole.

Then there is the traffic the proxy never sees. Agents call providers directly. SaaS copilots make their own backend model calls your gateway is not on the path for. Encrypted Client Hello is closing the last window inspection relied on. And the proxy you built to watch everything becomes the single thing worth attacking. The LiteLLM supply-chain compromise was one package and every key at once. The detailed inspection log that satisfies the auditor is also a concentrated store of exactly the sensitive prompts an attacker wants, and even the semantic cache can be poisoned across tenants.

The theater is not the box. The box is useful. The theater is the sentence, because it converts a governance control into an imagined security boundary and then closes the risk item behind it.

Which Lens Should Win

Three lenses, three winners. Functionality picks Azure. Cost picks Azure, or nothing. Security, examined honestly, does not pick a product at all. It picks a purpose.

In a bank, the security lens should carry the most weight. But weight is not the same as a blank check, and the security case only earns its weight when it is honest about what it is buying. Buy the LLM proxy as an evidence-and-governance control. It catches the careless majority, it produces the logs the examiner requires, it governs cost and discovers shadow AI, and it is legally non-optional. Do not buy it, and above all do not sell it upward, as protection against a motivated attacker. It is not that, and the gap between what it is and what it gets described as is where real incidents live.

So the blunt version. If your estate is Microsoft, Azure API Management is the pick, on functionality and cost and native identity together. If you are standardizing agent governance on Anypoint, MuleSoft Omni Gateway is the more coherent bet because it unifies LLM and agent policy in one plane. If you are genuinely multi-provider or air-gapped, evaluate Kong or LiteLLM. And if you are Azure-centric and reaching for a third gateway, stop, because you already own the capability twice. Whichever you choose, remember the one thing no gateway does: it does not keep inference inside your perimeter. That is a deployment decision, in-boundary serving and regional models, not a policy you can attach at the edge.

What I Will Not Claim

The honest counterweight, because the provocative version can overreach. The guardrail-bypass numbers are best-case attacker results, not what a random prompt achieves, and a filter that catches the obvious ninety-five percent is still worth running as one layer of several. Judging a defense-in-depth control by a perfect-boundary standard is the same fallacy as calling a firewall useless because phishing exists. Inspection genuinely blocks the class of risk that actually dominates, and the regulatory floor means you will deploy it no matter what any blog post argues.

That is the point, not the objection to it. The proxy is a real control aimed at the real dominant risk. The thing to refuse is the false comfort bolted onto it. Keep the box. Delete the sentence.

Covered, or Governed

Go back to the five words that ended the meeting. We inspect the traffic, so we’re covered. Cross out the last word and write the true one. We inspect the traffic, so we’re governed. Governed is a real, defensible, examiner-ready claim, and it is most of why the gateway is worth its cost. Covered is the story we tell to stop looking at the part of the surface the gateway was never on.

The gateway is not the mistake. The full stop after covered is.

---

If this resonated, the companion read is What witness.ai Doesn’t See. That post was about the blind spot below the network layer. This one is about the sentence we say to avoid looking at it.

If you are running this evaluation and landing on a different winner, or you think I have been unfair to the security case, I want to compare notes. Find me on X @orestesgarcia or LinkedIn /in/setsero.