AI Agent Governance for Boards

AI agent governance for boards is becoming urgent as autonomous systems move from demonstrations into real business workflows. Yet most enterprises are still unwilling to let them operate without human oversight.

That is not evidence that the technology has failed. It is evidence that autonomy creates a leadership problem as well as a technical one.

The central question for boards is no longer, “Can an AI agent perform this task?” It is:

Can the organisation prove what the agent did, why it acted, what it was permitted to access and who remained accountable?

The enterprises that answer those questions well will be able to expand autonomy faster. The rest may remain trapped in pilots that work technically but cannot be defended operationally.

 

Executive summary

• A Gartner survey found that 75% of respondents were piloting, deploying or had deployed some form of AI agent.
• Only 15% were considering, piloting or deploying fully autonomous agents that required no human oversight.
• Only 13% strongly agreed that their organisation had the governance structures required to manage agents.
• Agent autonomy should be widened using risk-adjusted evidence, not enthusiasm or a single performance metric.
• For boards, the competitive advantage is the ability to govern agents safely at scale.

What the enterprise data actually says

In May and June 2025, Gartner surveyed 360 IT application leaders from organisations with at least 250 employees across North America, Europe and Asia-Pacific.

The research found that 75% were piloting, deploying or had already deployed some form of AI agent. However, only 15% were considering, piloting or deploying fully autonomous agents—defined in the research as goal-driven tools that do not require human oversight.

Those figures should not be treated as a simple 60-point “trust gap.” They measure related but different stages and forms of adoption. Their practical message is still important: enterprises are interested in agents, but few are ready to remove human oversight completely.

The governance findings explain some of that caution:

• Only 19% reported high or complete trust in vendors’ ability to provide adequate hallucination protection.
• Seventy-four percent believed AI agents represented a new attack vector into their organisation.
• Only 13% strongly agreed that they had the governance structures needed to manage agents.

In separate 2026 analysis, Gartner reported that 17% of organisations had deployed AI agents and more than 60% expected to do so within two years. It also noted that most current deployments remain narrowly scoped and that fully autonomous agents are not ready for the majority of enterprise use cases.

Adoption is accelerating. Governance maturity is not yet keeping pace.

Autonomy is a governance decision

Consider how an organisation appoints a senior executive. It does not normally provide unlimited access to company funds, customer data and production systems on the first day. The executive receives a defined mandate, delegated authority, approval limits and oversight.

AI agents need the same management discipline.

Before an agent can take consequential action, leadership should define four controls:

1. Scope and access: Which systems, data and actions can the agent use?
2. Approval and escalation: Which decisions require a person, and when must the agent stop?
3. Traceability: Can the organisation reconstruct the agent’s actions using reliable, auditable records?
4. Containment and recovery: Can the process be paused, permissions revoked and changes safely reversed?

The level of control should reflect the potential impact. Drafting an internal meeting summary does not present the same risk as issuing a payment, changing customer data, approving a loan or making an employment decision.

This is why a single organisation may allow substantial autonomy in one workflow while requiring human approval at every stage of another.

The alignment problem is a board problem

The most important Gartner finding may be the lack of agreement about what agents are meant to achieve.

Only 14% of respondents strongly agreed that IT, business users and executive leadership were aligned on the problems AI should solve. Organisations reporting that alignment were 1.6 times more likely to expect a transformative productivity impact from agents and more than three times more likely to report significant value from their generative AI tools.

That makes alignment a commercial control, not a workshop exercise.

Technical teams can configure the agent. They cannot decide which customer outcome, operating constraint or strategic priority matters most. Leadership must define:

• The business outcome
• The acceptable level of risk
• The accountable owner
• The evidence required before autonomy expands

Without those decisions, organisations tend to automate visible tasks rather than valuable ones.

Where the defensible advantage sits

Most organisations can buy access to the same agent platforms. The software alone is unlikely to create a lasting advantage.

Drawing on more than 1,000 AI projects reviewed and 300+ keynotes, I repeatedly see the same pattern: projects scale when leadership can answer three questions clearly.

1. What measurable outcome is this workflow expected to deliver?
2. Who remains legally and operationally accountable?
3. What evidence will show that the process remains safe and effective?

The model matters. The surrounding operating system—ownership, controls, data, skills and measurement—usually matters more.

What the EU AI Act changes—and what it does not

The EU AI Act does not classify every AI agent as high-risk simply because it can act autonomously. Obligations depend on the system’s intended purpose, use and legal classification.

Where an AI system falls within a high-risk category, the Act includes requirements concerning areas such as risk management, documentation and traceability, human oversight, accuracy, cybersecurity and robustness. The regulation also contains record-keeping requirements for high-risk systems.

For boards, the practical lesson is not to apply one compliance checklist to every agent. It is to maintain an inventory of agent use cases and assess each one according to what it does, what decisions it influences, which data it uses and who may be affected.

As host of the independent AI Ireland Podcast, I have interviewed European policymakers and people involved in shaping the EU AI Act. Those conversations reinforce a recurring leadership lesson: accountability cannot be delegated to the technology supplier.

This article provides strategic business guidance, not legal advice. Organisations should obtain appropriate legal and regulatory advice for their specific systems and use cases.

A practical 30-day controlled-agent test

Boards do not need to choose between permanent manual oversight and unrestricted autonomy. A controlled pilot can generate the evidence needed for a better decision.

Step 1: Create the agent register

Ask the CIO, AI lead or relevant business owner to document every agent in pilot or production. For each agent, record:

• Its purpose and business owner
• The systems and data it can access
• The actions it can take
• The human approval rule
• The escalation and shutdown procedure
• The available audit record
• The expected benefit and risk classification

If the organisation cannot produce a concise register, it has found a governance gap before it has found an autonomy opportunity.

Step 2: Choose a bounded workflow

Select a frequent, reversible and relatively low-risk process. Avoid beginning with a workflow that can move money, alter legal rights, expose sensitive data or create an irreversible customer outcome.

Keep human approval at the final execution point during the pilot.

Step 3: Measure more than time saved

Track:

• Completion time and cost
• Human intervention frequency
• Error frequency and severity
• Policy or permission violations
• Successful escalation and rollback
• Audit-record completeness
• Impact on the customer or employee receiving the outcome

Human intervention frequency is useful, but it is not an AI-readiness score by itself. A low intervention rate can reflect strong performance, weak review or a task that is too simple to justify automation.

Step 4: Expand autonomy by risk tier

At the end of the pilot, decide whether to retain, widen or reduce autonomy. The decision should be based on control evidence and outcome quality—not solely on whether the agent completed most tasks without assistance.

Questions boards should ask management

1. Which AI agents are currently operating in pilot or production?
2. Who owns the business outcome and operational risk for each agent?
3. What information and systems can each agent access?
4. Which actions always require human approval?
5. Can the organisation halt an agent and safely reverse its actions?
6. What evidence would justify widening or reducing autonomy?
7. Has each use case been assessed for applicable legal, regulatory and contractual obligations?

These questions move the conversation away from demonstrations and towards accountable deployment.

Build the trust layer before expanding autonomy

The objective is not to keep a person approving every agent action forever. It is to earn wider autonomy through evidence.

Organisations that establish ownership, risk thresholds, traceability and recovery controls early will be able to move faster as agent capabilities improve. Organisations that postpone those decisions may discover that they have built pilots their boards cannot approve and their operating teams cannot safely scale.

Competitive advantage will not come from having the highest number of agents. It will come from knowing where autonomy creates value—and having the controls to use it with confidence.

Bring this discussion to your leadership team

Mark Kelly delivers AI keynote presentations and executive AI leadership workshops for boards, leadership teams and business audiences. His sessions translate AI strategy, adoption and governance into practical decisions leaders can act on.

You can also explore Mark’s interviews with European policymakers and people involved in shaping the EU AI Act on the independent AI Ireland Podcast hub.

To discuss an event or leadership session, email [email protected].

Frequently asked questions

What is a fully autonomous AI agent?

In the Gartner research referenced here, a fully autonomous AI agent is a goal-driven AI tool that does not require human oversight. In practice, autonomy exists on a spectrum: an agent may plan independently while still requiring approval before it changes data, contacts a customer or completes a transaction.

Why are so few enterprises considering fully autonomous agents?

The research identifies concerns about vendor safeguards, cybersecurity, governance and organisational readiness. Only 13% of respondents strongly agreed that they had the governance structures needed to manage agents, while 74% believed agents represented a new attack vector.

Does human oversight eliminate the benefit of AI agents?

No. Human oversight can be concentrated at high-impact decision points rather than applied to every minor step. The appropriate level depends on the workflow’s risk, reversibility and potential effect on people or the organisation.

Where are organisations expecting AI agents to have the most impact?

In the Gartner survey, analytics and business intelligence ranked first, with 64% placing it among their top three domains. Customer service followed at 55%, with office productivity at 39%. These are expectations, not guarantees of value.

When should an organisation expand an agent’s autonomy?

Autonomy should expand when the organisation has evidence that the agent delivers the intended outcome within agreed risk limits, that exceptions are detected and escalated, that actions are traceable, and that failures can be contained and reversed. Intervention rate alone is insufficient.

Sources

• Gartner: Survey Finds Just 15% of IT Application Leaders Are Considering, Piloting, or Deploying Fully Autonomous AI Agents, 30 September 2025.
• Gartner: What the 2026 Hype Cycle for Agentic AI Reveals, 15 April 2026.
• European Commission: Navigating the AI Act.
• European Union: Regulation (EU) 2024/1689, including Articles 12 and 14 on record-keeping and human oversight for high-risk AI systems.

About Mark Kelly

Mark Kelly is an AI keynote speaker, Founder of AI Ireland and Co-Founder of Alldus. Drawing on more than 1,000 AI projects reviewed and 300+ keynotes, he helps boards, executives and business leaders turn AI strategy into practical business action. He hosts the independent AI Ireland Podcast, including interviews with policymakers, European Commission officials and people involved in shaping the EU AI Act.

---