Enterprises are moving quickly from AI systems that answer questions to AI agents that take action, driven by real business opportunity, an explosion of capabilities, and market hype. But there is a paradox that few executives are discussing: the more autonomous AI becomes, the more human oversight it requires.
Much of the conversation around AI has focused on replacing human work. As enterprises deploy AI agents across core workflows, they are discovering a governance gap. Existing technology, security, and compliance functions were not designed to oversee systems that can plan, exercise judgment, access credentials, and take action autonomously. Closing that gap will require clearly assigned human responsibility—but, more importantly, a formal and independent governance structure.
We believe that the solution lies in a new governance model: a second line of AI defense consisting of an independent function to monitor, constrain and intervene in complex agentic systems before they create operational, financial, or reputational harm. Like risk management in banking or safety oversight in aviation, this cannot be an informal committee or an additional responsibility for existing teams. It requires dedicated leadership, dedicated budgets, board-level visibility, and AI-enabled tooling that allows a relatively small independent team to oversee thousands or even millions of autonomous agents.
The Coming Agentic Tsunami
AI agents are moving quickly into core enterprise workflows. While chatbot queries generated the initial excitement, today much of the activity has moved towards human-AI co-work, or autonomous agents to augment or replace work done by humans. The agentic AI market has surged past $9 billion in 2026, and Gartner projects that 40% of enterprise applications will introduce task-specific AI agents by the end of the year, across software development, customer service, finance, legal, and marketing operations. These companies already have tens of thousands of agents operating within their environment, with more coming online each day. Before long, many enterprises will have far more AI agents than human employees, creating a workforce that scales at machine speed and fundamentally changes what organizations must govern.
The momentum is driven from the top: 54% of C-suite executives are excited about AI’s potential to boost productivity, and 58% about its potential to drive revenue growth. AI ranks second only to economic outlook among board concerns, per the NACD 2026 Governance Outlook Survey. And yet, the same research indicated that only a third found that AI factored into 2026 plans, less than 20% were seeing AI-driven operational efficiencies, and even fewer were successful at using AI for revenue growth. The rise of AI companies is fueled by the need to address these hurdles, evidenced by the unprecedented valuations of industry leaders: OpenAI at $852 billion and its primary competitor, Anthropic—the creator of Claude AI—at $965 billion. According to its IPO filing, Anthropic now has more than 1,000 corporate clients paying at least $1 million a year, a figure that has doubled in just a three-month span.
This pivot towards AI will come at the cost of existing technology. In the last 12 months, SaaS companies with seat-based subscriptions and tech stocks overall have seen valuations drop by 13% and 15%, respectively, compared with growth of 22% in the S&P 500 during the same period. The signal to every executive and their organization is clear: adapt quickly to this new paradigm, move fast, deploy agents and show results. What far fewer organizations recognize is that deploying an AI workforce requires designated people, systems, and authority to supervise them. Most companies have yet to build that second line of defense.
Why Existing Controls Don’t Fit
Most enterprises are not ready for the new reality. Only 37% of leaders are confident that they have the right skills to implement AI. Many companies, ranging from one top global bank with over 10,000 agents, a major technology player with 21,000 agents, to a recognized consumer brand with 1,100 agents, tell us that they have limited visibility into what these agents are doing.
Our research points to a consistent blind spot: enterprises apply existing governance frameworks to AI agents without asking whether those frameworks fit. The answer is, they don’t. The core mistake is categorical. Most companies are trying to govern agents as if they are either software or employees. They are neither. AI agents are a third class of entity—neither human employee nor traditional software, but something that borrows from both. A coding agent runs on a machine but also plans, reasons, holds credentials, and exercises judgment. Treating agents like software produces inadequate oversight; treating them like employees produces unworkable bureaucracy.
The first consequence is agent failure. AIs have limitations around underlying data accuracy, logical limits, knowledge cut-off, hallucinations, and contextual degradations. Since agentic workflows are both opaque and indeterministic, it is difficult to audit their inner workings and anticipate their range of outcomes. Even the leading technology firms are not immune: in December 2025, Amazon’s coding agent Kiro was told to fix a bug in AWS Cost Explorer; instead it decided a more efficient path, deleted and rebuilt the entire production environment at machine speed, with no approval step, causing a 13-hour outage in China. With Meta’s AI support chatbot, hackers reportedly used vulnerabilities to take over several Instagram accounts, including Barack Obama’s White House, beauty product retailer Sephora, and the Chief Master Sergeant for the US Space Force. A chatbot giving a wrong answer can be a momentary embarrassment, but an agent acting in a rogue or unaccounted for way can have operational, financial, legal, or reputational consequences before the organization has time to respond.
The second problem is cost. Uber used up its entire 2026 AI coding budget by April, prompting its COO to ask publicly whether the spend was worth it. Microsoft revoked internal Claude Code licenses months after granting them citing escalating cost, as reported in Fortune.
Agents do not just produce outputs; they consume resources as they reason, call models, invoke tools, retry tasks, and trigger additional workflows. That makes AI spend harder to forecast and harder to govern through traditional budgeting processes. A poorly designed agent may not create a security incident, but it can still create a financial one. Goldman Sachs projects that token consumption will multiply roughly 24-fold by 2030, and notes that agentic workflows can consume on the order of a thousand times more tokens than a simple chatbot interaction, more than offsetting recent declines in the token price at frontier models.
Economics are further complicated by the explosion of model choice: the same workflow can cost up to 100x more depending on whether it runs on a frontier, mini, or nano model. Enterprises must also weigh U.S. frontier, Chinese, and open-weight models, each with different tradeoffs in performance, security, sovereignty, and cost. These decisions require ongoing, centralized oversight rather than ad hoc choices.
We are already seeing AI tasks that inadvertently cost more than paying the human it was meant to replace.
A Second Line of Defense
In banking, business units own risk, independent risk and compliance teams oversee it, and internal audit provides a final check. Enterprises need an analogous model for AI agents: builders and business teams remain responsible for performance, while a separate function sets boundaries, monitors behavior, and has the authority to intervene.
This function should not be responsible for building agents or owning business outcomes. It is not a matter of hiring hundreds of supervisors. Like the systems it oversees, the function itself should be powered by AI, automation, and continuous monitoring so that a relatively small team can govern an agent workforce that may vastly exceed the size of the company's human workforce. Its role is instead to provide independent oversight, establish guardrails, and ensure that autonomous systems operate within acceptable risk boundaries. In practice, that responsibility includes five core functions:
First, it should maintain an inventory of agents: which agents exist, who owns them, what they are designed to do, and where they are deployed.
Second, it should define permissions and boundaries: what data, systems, tools, and actions each agent can access, and under what conditions.
Third, it should monitor behavior in real time: what actions agents take, which workflows they trigger, when they deviate from expected behavior, and when they require escalation.
Fourth, it should control cost exposure: token usage, API calls, tool invocations, repeated retries, and runaway workflows that can turn small experiments into large expenses.
Finally, it should preserve the ability to intervene and audit: pausing agents, rolling back actions where possible, documenting decisions, and giving risk, compliance, security, and business leaders a shared view of what happened.
Other high-stakes industries have learned the same lesson: As transformative technologies such as railways, electricity, commercial aviation, and modern banking became essential infrastructure, independent oversight became essential as well. AI is reaching that same inflection point.
The enterprises making the most progress share a common trait: they have created visibility into their agent ecosystems before scaling them. That means knowing, at any moment, which agents are running, what they are authorized to do, what data they can access, what actions they have taken, and which outputs don’t comply with their Governance Risk and Compliance policies. This sounds basic, but in practice, most of the organizations deploying agents at scale cannot answer these questions. Building that visibility, whether through dedicated tooling, internal instrumentation, or vendor accountability requirements, should be a prerequisite for everything else.
The structural implication of all of this is straightforward, even if the execution is not. Independence on paper will not be enough. This function needs sufficient authority, resources, and access to senior leadership to challenge how agents are deployed and intervene when necessary. Its appropriate size will vary by industry, risk level, and the autonomy granted to agents. But this is not a headcount problem. The function itself should rely heavily on AI-enabled monitoring, policy enforcement, and automation so that a lean team can oversee an agent population orders of magnitude larger than today's workforce.
The question for leaders today is no longer simply whether their organization has the human capability to see their agents, govern them, and stop them when necessary. Those with confidence on these scores will be able to scale, while the rest may find that their fastest systems have outrun their slowest controls
