Feature
Gartner recently raised the question: How are businesses safeguarding against wrong or errant AI outputs?
Its insight: the long-standing "human in the loop" model, once considered essential for AI governance, is collapsing under its own weight. As AI-generated output scales exponentially, humans simply can't keep up with the volume of oversight required.
As Gartner VP analyst Alicia Mullery put it: “AI can make mistakes faster than we humans can catch them.”
In response, some organizations are deploying AI models to check other AI models, a move that introduces new layers of complexity and significant costs.
This shift creates issues about trust, accountability and cost-efficiency in AI operations. Who's watching the machines when the machines are watching each other?
Table of Contents
- From AI Oversight to AI Orchestration
- The Importance of AI Accountability
- Where AI Regulations Come In
- The Echo Chamber Effect
- Building a Hybrid AI Governance Model
- The High Cost of Undetected AI Errors
From AI Oversight to AI Orchestration
Roles are changing oversight to supervision, according to Paul McDonagh-Smith of MIT Sloan. "We're seeing the rise of a 'human on the loop' paradigm where people still define intent, context and accountability, whilst co-ordinating the machines' management of scale and speed," he explained.
Humans are moving up the value chain, designing ethical frameworks and escalation paths rather than checking individual outputs. The question has changed: it's no longer whether machines should supervise machines, but how humans design the principles that make that process trustworthy.
Roles are shifting from tedious checker to strategic supervisor. "AI agents handle assignments like monitoring activity, detecting anomalies and accelerating investigations, alerting human supervisors to high-stakes and actionable decisions,” said Marla Hay, senior vice president of product management at Salesforce. Her team's data tells the story: more than 3,000 hours reclaimed just by automating incident triage and response.
The pattern is confirmed by Shriram Natarajan from ISG. "We are seeing humans at the helm as the next step on the evolution," he noted. Governance must become as intelligent and adaptive as the systems it oversees, McDonagh-Smith said. We need "humanity in every system" rather than humans in every loop.
The Importance of AI Accountability
Beneath this evolution lies the principle that accountability requires a human bearer. Machines might track responsibility, but they cannot own it. This distinction separates sustainable AI governance from merely efficient operations.
The case is definitional, according to Ann Skeet, senior director of leadership ethics at Santa Clara University's Markkula Center for Applied Ethics: "By its definition, accountability speaks to who will bear the consequences for failing to perform as expected,” she said. “Ultimately, some one person and the company that person works for will bear responsibility for the failures of AI systems."
When AI systems fail, and they will, courts won't accept "the algorithm decided" as a defense. The stakes extend beyond legal exposure, Skeet warned. "The legal battles that will be launched, if we settle for having AIs overseeing AI, will be legion, but the ethical considerations are even greater."
"Accountability can't be automated,” agreed Patrizia Bertini, managing partner at Euler Associates. Machines can produce outputs, but only humans can own outcomes, a distinction grounded in regulation and ethics, not just philosophy.
Where AI Regulations Come In
The regulatory landscape reinforces these requirements. The EU AI Act mandates real human oversight by August 2026, with fines reaching €35 million ($40.5 million) or 7% of global turnover for non-compliance, Bertini said. The Act demands AI explainability: systems that show their reasoning clearly enough for a person to challenge them, with humans able to override machine decisions.
Yet most organizations remain unprepared. When Bertini talks with product and design teams, she said she finds that "almost none have actually built it into their systems or workflows yet," treating human oversight as nice-to-have rather than foundational.
"In the foreseeable future, it is not possible to have accountability without humans,” Natarajan agreed. The gap between regulatory requirement and organizational readiness reflects a fundamental misunderstanding of what sustainable AI governance demands.
But even if organizations wanted to delegate oversight entirely to machines, the approach carries risks that could prove catastrophic.
The Echo Chamber Effect
The appeal of AI checking AI seems obvious: scale meeting scale, with machines monitoring machines at speeds humans cannot match. But this approach introduces structural risks that threaten the very trust it promises to protect.
When two models share similar data foundations or training biases, one may simply validate the other's errors faster and more convincingly. The result is what McDonagh-Smith describes as "an echo chamber, machines confidently agreeing on the same mistake." This is fundamentally epistemic rather than technical, he said, undermining our ability to know whether oversight mechanisms work at all.
"How do we know the checker is right when both are potentially learning from the same flawed datasets, algorithms or world views?" McDonagh-Smith said.
Without clear audit trails and deliberately structured human calibration, companies lose oversight and accountability.
AI checking AI inherits vulnerabilities, Hays warned. "Transparency gaps, prompt injection vulnerabilities and a decision-making chain becomes harder to trace with each layer you add." Her research at Salesforce revealed that 55% of IT security leaders lack confidence that they have appropriate guardrails to deploy agents safely.
Conference
Nov
17
KMWorld Washington 2025
Register
Webinar
On demand
AI in Customer Service: Faster Resolutions, Happier Customers
Don’t let rising demand burn out your team. See how to build a smarter, more resilient support org.
Watch Now
Webinar
On demand
Accelerating Healthcare Ops with AI
Sign up to see how your peers are putting AI to work — and making it stick.
Watch Now
Webinar
On demand
From Nice-to-Have to Non-Negotiable: Prove the Value of Your EX Stack
Transform your tech stack: prove its value, secure your future.
Watch Now
Webinar
On demand
The Great Workforce Breakdown (And How to Stop It)
Disconnected employees. Failing communication. It’s time to turn things around.
Watch Now
Webinar
On demand
Maximize Your Intranet’s Value: Enhance the Employee Experience for Greater Adoption
Learn practical strategies to enhance usability, boost adoption and ensure your intranet becomes a true productivity hub.
Watch Now
"That's not a minor gap," Hays said. "That's a majority of organizations racing toward AI deployment without the safety infrastructure to support it." What's needed is governance, security and compliance rules baked into architecture from day one.
The fragility extends to the models themselves. Anthropic research found that just 250 poisoned files compromise models even with billions of parameters. This finding exposes how vulnerable even massive systems remain, according to Bertini: "If your idea of oversight is 'let another model check it,’ we are not building safety,” she said. “We are building fragility."
Natarajan confirms this core vulnerability. "The non-determinism of AI (especially generative AI) used to provide the check and balance for underlying AI is the key risk." Properly designed systems reduce exposure, he noted, "however, we cannot guarantee that it is eliminated."
"We're trusting systems that we don't truly understand to police other systems that no one can fully explain, all while removing the very humans who could have asked the right questions,” Bertini agreed.
These risks point toward a different model, one that uses both human and machine capabilities without over-relying on either.
Building a Hybrid AI Governance Model
The path forward requires designing resilient ecosystems where human judgment and machine scale collaborate effectively.
Governance should evolve "from static policies to living systems where accountability, transparency and explainability are continuously tested and improved," McDonagh-Smith said. Machines handle repetitive verification, while humans provide contextual arbitration and moral reasoning.
It's a conclusion Gartner also reached. The analyst firm recommends businesses create an "accuracy survival kit," consisting of three parts:
- Formal metrics to compare AI outputs against a norm.
- Two-factor error checking, where an employee oversees one AI model reviewing the work of another.
- The "good enough" ratio, where teams decide in advance what level of accuracy is acceptable for a given project.
Adopting this multi-faceted approach provides a higher level of confidence than a single approach alone.
Natarajan adds another layer to this model: transparency requires auditable records of AI-to-AI interactions: what was checked, by which model, under which criteria. This requires ongoing work, he emphasized: "constant testing, checking for drift, updating golden records and pathways." He advocates adopting "a 'chaos monkey' mindset to uncover corner cases that normally wouldn't occur."
The High Cost of Undetected AI Errors
The economic case supports hybrid models. One undetected error bankrupts trust in a product, service or brand. Counterbalancing data comes from Hays: her team has moved "from reactive firefighting to strategic, cost-effective defense," with 83% of organizations still paying a premium by not automating compliance processes. Yet apparent efficiency might mask the costs of cascading failures, opaque decision chains and reputational damage.
Like financial systems using layered audits, AI ecosystems need multi-tiered governance combining self-regulation, independent audits and public accountability, said McDonagh-Smith. Natarajan disagrees, saying: "Regulators should not jump the gun and put constraints on automated checks. This is an area of innovation and the required regulations will become clearer in the future."
Yet Bertini's warning about the EU AI Act suggests that regulatory clarity is arriving faster than Natarajan anticipates, and most organizations remain unprepared.
When AI systems decide who gets a loan, a job or a diagnosis, the cost of cutting humans out isn't efficiency but risk, Bertini said. By rapidly forging ahead with agentic AI, Skeet warns, "companies are creating conditions where they have more than they can manage, exposing the company to AI safety and reputational risks."
"Tools, processes and human capital need to work in conjunction to govern the AI landscape,” Natarajan said.
So who's watching the machines when the machines are watching each other? The solution isn't choosing between human or machine oversight, but positioning humans as architects of systems that remain accountable, transparent and trustworthy.
Editor's Note: Read more thoughts on how to establish AI accountability:
- Inside the AI Accountability Gap: Why Enterprises Are Building Their Own Rules — The AI accountability gap is widening. Here's how enterprises are stepping in as lawmakers struggle to keep pace.
- The Teammate With No Manager: Who's Accountable for AI? — Why cross-functional leadership, accountability and employee experience design are central to AI’s success at work.
- Why AI Hiring Discrimination Lawsuits Are About to Explode — AI is reshaping hiring — and the courtroom. Job seekers are suing over biased screening tools, and experts say a wave of lawsuits is just beginning.