Feature
Enterprises incorporating AI services into development are running into a problem: How to take advantage of the rapidly changing field without rewriting applications for every new model. The alternative seems to be picking a vendor and getting locked in as the industry progresses.
But there’s another solution: AI middleware.
The Challenge: Adopting AI Without Getting Locked In
“So much is in flux right now,” said Sanjeev Mohan, founder of advisory company Sanjmo. “It’s difficult for organizations to pin down a prescribed stack architecture. They want to future-proof their stack, and they want some abstraction.” That way, he added, enterprises don’t need to worry about changing AI infrastructure.
The Gartner report "Innovation Insight: AI Gateways" defined the problem: “An AI gateway is a middleware component that intercepts API calls between applications and providers of AI services. This enables them to provide an abstraction layer for AI traffic, accepting API requests from a client, processing them based on defined policies and directing them to the appropriate LLMs [large language models] and other AI services. They also then intercept the response, applying policies before routing back to the calling client.”
Consequently, the report continued, software engineering leaders use AI gateways to apply security, multi-LLM routing and cost visibility, as well as adding data protection, scanning and risk mitigation controls to their AI usage.
By 2028, 70% of organizations building multi-LLM applications will use AI gateway capabilities
In fact, Gartner predicted that by 2028, 70% of organizations building multi-LLM applications will use AI gateway capabilities, particularly for optimizing cost performance outcomes, up from less than 5% in 2024.
AI gateways are similar to the more general API gateways that some organizations use, but are also different, Gartner said.
| Feature | Traditional API Gateway | AI Gateway |
|---|---|---|
| Primary Traffic Direction | Inbound (from external users to internal infrastructure) | Outbound (from internal applications to external AI services) |
| Main Focus | Managing incoming traffic and provider-side controls | Managing outbound traffic and access to third-party AI APIs |
| Use Case | General API management for applications and services | Specialized for AI service interactions |
| Key Challenges | Traffic control, rate limiting, authentication | Scaling, operations, security, troubleshooting for AI-specific traffic |
| Position in Architecture | Sits in front of infrastructure | Interfaces between internal apps and external AI services |
Because AI gateways manage AI service calls, they provide features such as AI tokens for billing, response caching to reduce calls and latency, AI services portals that enforce usage plans, AI service mediation to route calls to different AI providers and incorporate retrieval augmented generation (RAG), request and response guardrails to filter out inappropriate questions or answers and security, according to Gartner.
Overcoming AI Vendor Lock-In With Middleware
Vendors have announced or are working on AI middleware products, but an AI middleware product from a specific vendor doesn’t necessarily solve the lock-in problem, just moves it to a different place in the stack. But one promising example is the Model Context Protocol (MCP) announced by Claude producer Anthropic in November. While it was created by an AI vendor, the company made it open source, and AI vendors such as Cloudflare, OpenAI and Microsoft have announced support for it.
With Model Context Protocol, AI systems can "read the instructions" for an external system at runtime, rather than relying on developers, according to VKTR contributor and digital transformation expert Felipe Jaramillo.
"This shift decouples the AI from fixed integrations, allowing businesses to evolve their capabilities, plug in new tools or update data sources much faster, responding to change more quickly and significantly reducing development overhead," he explained. "In the long term, the MCP ecosystem envisions rich, composable AI applications and sophisticated agentic behaviors enabled by potentially bidirectional communication."
Despite typical challenges facing new standards, Jaramillo noted MCP is gaining significant traction due to strong enterprise demand and a growing developer community. "For business leaders, this represents a crucial shift requiring strategic action: audit your AI infrastructure, launch focused pilot projects, evaluate vendor commitments to interoperability and establish internal champions to explore implementation opportunities."
Related Article: The AI Vendor Evaluation Checklist Every Leader Needs
How IBM Is Positioning Itself in the AI Middleware Market
IBM could also play a role, especially for organizations focused on legacy systems and for those that use AI on premises rather than in the cloud.
“If we were to believe in what the pundits say, 60-80% of data is still on-premises,” Mohan said. And for a variety of reasons — perhaps size, latency or compliance — that data can’t be moved, he said. “I need my AI to go where my data is. If it’s on premises, IBM has a huge advantage.”
Other advantages IBM offers are several purpose-built LLMs intended for specific use cases rather than competing with more general AI systems such as OpenAI and Anthropic, and a massive AI ecosystem it built over decades, he added.
IBM’s weakness is that it is perceived as a legacy company (Mohan prefers “heritage” because it sounds like something to be respected). “Why would an AI-first cloud native company choose IBM when they could go to Google Cloud?” he asked. On the other hand, he pointed out that Microsoft has largely overcome its similar legacy reputation. Microsoft was also a legacy, stodgy company that missed out on mobile, social, cloud and AI, he said. Then they transformed, and are now deeply embedded in AI. “If we learn lessons from Microsoft, the sky is the limit.”
Risks and Limitations of Open-Source AI Middleware
Open-source organizations such as Hugging Face have also developed middleware platforms, but they come with their own disadvantages, said Emily Barnes, chief AI and innovation officer at Edapted and VKTR contributor.
That’s not a dig against open source. “Unix or open-source databases that are offered widely by many vendors have been built and improved over time by a community of developers,” Barnes said. “They review each other’s work, follow guidelines and keep things well-documented. The crowdsourced approach has helped make those systems more reliable, secure and easy to maintain.”
But AI is a different beast, she added. “It is more unpredictable and complex." Unlike an operating system, which does the same thing every time, AI systems learn from huge datasets that can be messy, biased or inaccurate. “Because of how they work, it is harder to understand what is happening inside these models or why they behave a certain way. That makes it tough to catch problems early, whether it is hidden bias, security issues or the risk of people using the models in harmful ways.”
That goes double for middleware. “Middleware hides a lot of what is going on under the hood,” Barnes said. “Developers using these tools might not know where the model’s training data came from or whether it has built-in biases. That can lead to ethical issues like models unintentionally discriminating or being used in harmful ways, because nobody took a close look at how they were built and where the data came from.”
So how to solve it? According to Barnes, we need to establish more rigorous vetting processes, require standardized documentation formats and create mechanisms for auditing and recall of harmful models. "Hugging Face has made initial strides in this area by promoting ethical AI guidelines, but sustained progress will require broader industry coordination and, potentially, regulatory involvement.”
Conference
Sep
16
HR Tech Conference Las Vegas 2025
Register
Conference
Oct
27
Gartner HR Symposium/Xpo Orlando 2025
Register
Related Article: Open-Source AI Is Changing Higher Ed For Better or Worse
Enterprise Strategies for Evaluating and Deploying AI Middleware
Another issue with AI middleware in general is that the field is nascent, according to Gartner, meaning products in the field are new and unproven and they don’t all provide the same features. But enterprises can take certain steps now.
The two biggest factors for AI middleware, said Mohan, are:
- Vendor Partners: It's important to have a robust and thriving partner ecosystem.
- Ease of Use: If your product is not being adopted, it's time to take a look at the user experience.
Other features to consider include maturity, support for open standards and their level of innovation.
Gartner also recommended that enterprises use processes and procedures such as funneling AI API traffic through one control point, managing the security credentials of any AI back end from one place and providing standard API interfaces that route requests to multiple AI back ends.
