Most AI governance programs were designed for models that produce an output (a score, a classification, a paragraph of text) which a human then acts on. Agentic AI changes that. An agent receives a goal, breaks it into steps, calls tools and APIs, reads the results and decides what to do next, sometimes coordinating with other agents along the way. The output is no longer a recommendation; it is an action in a live system.
Enterprises are moving quickly. McKinsey’s State of AI in 2026 survey found that about 40% of large organizations are scaling AI agents, up from 27% a year earlier. At the same time, Gartner has predicted that more than 40% of agentic AI projects will be canceled by the end of 2027, citing escalating costs, unclear value and inadequate risk controls. Governance is therefore not only a compliance concern; it is a condition for agentic programs surviving at all. This article sets out a practical control framework for governing autonomous agents in the enterprise.
Why Agents Break Traditional AI Governance
Conventional model governance focuses on data, training, validation and monitoring of predictions. Those controls still matter, but agents introduce risks that sit outside the model itself.
- Actions, not just outputs: an agent can send emails, update records, issue refunds, change configurations or execute code. Errors become incidents rather than bad suggestions.
- Compounding steps: multi-step plans can amplify small errors, as each step builds on the last. A misread field early in a workflow can cascade into a series of wrong actions.
- Expanded attack surface: agents read untrusted content such as emails, web pages and documents, which creates exposure to prompt injection and manipulation, and they connect to tools through new protocols.
- Blurred accountability: when an agent built on a third-party model, using tools from several vendors, takes a harmful action, it is often unclear who is responsible.
- Multi-agent dynamics: agents interacting with other agents can produce unexpected behavior that no single component was designed for.
Regulators are noticing the gap. The revised US interagency model risk management guidance issued in April 2026 (SR 26-2) explicitly excludes generative and agentic AI from its scope as novel and rapidly evolving, directing banks to manage them through broader risk management and governance. In other words, the organization must design those controls itself.
Singapore’s Model Framework: A Useful Reference Point
On January 22, 2026, Singapore’s Infocomm Media Development Authority (IMDA) published the Model AI Governance Framework for Agentic AI, described by Singapore as the first governance framework designed specifically for agents. It is voluntary, but law firm commentary from Bird & Bird and CMS stresses that organizations remain legally accountable for what their agents do. The framework is organized around four dimensions:
- Assess and bound risks upfront: evaluate use cases before deployment, considering the agent’s level of autonomy, its access to sensitive data and external systems, and the sensitivity of its domain, and limit its tools and permissions accordingly.
- Make humans meaningfully accountable: allocate responsibilities across the organization and with external parties, define checkpoints where human approval is required, and guard against automation bias.
- Implement technical controls and processes: apply controls across the agent lifecycle, test agents and multi-agent systems before deployment, roll out gradually and monitor continuously.
- Enable end-user responsibility: inform users about agent capabilities and limitations, provide routes to human contacts and train users in responsible use.
The framework is a living document and is not binding outside Singapore, but its structure maps well onto enterprise needs anywhere and is compatible with ISO/IEC 42001 management systems and the NIST AI Risk Management Framework.
A Control Framework for Enterprise Agents
Building on that structure and on what we see working in practice, we recommend organizing agent governance into six control domains. Each applies across the agent lifecycle, from design through retirement.
1. Use case risk assessment and autonomy levels
Before building, classify each agent by what it can affect. A simple autonomy scale helps align teams:
- Level 1 – Assist: the agent retrieves information and drafts outputs; a human takes every action.
- Level 2 – Propose: the agent prepares actions (for example, a drafted refund or a configuration change) that a human approves.
- Level 3 – Act within limits: the agent executes low-impact, reversible actions autonomously within defined thresholds, escalating everything else.
- Level 4 – Act broadly: the agent executes a wide range of actions with after-the-fact review. Reserve this for mature, well-monitored, low-harm domains.
Combine the autonomy level with data sensitivity, reversibility of actions and domain risk to set the required controls. An agent that can move money or change customer records at Level 3 deserves far more scrutiny than a Level 1 research assistant.
2. Agent identity and least privilege
Every agent should have its own identity, distinct from the human users and service accounts it works alongside, so that its actions can be authenticated, authorized and traced. Grant only the tools, data and permissions the task requires, scope credentials narrowly and expire them quickly. Tie each agent to an accountable human owner or team, as the Singapore framework recommends, and record that ownership in an agent registry.
3. Tool and action guardrails
Most agent risk materializes at the tool boundary. Effective controls include allow-lists of permitted tools and APIs, parameter validation (for example, refund amounts below a threshold), rate limits, sandboxed execution for code, and hard blocks on irreversible or high-impact actions unless a human approves. Treat content the agent reads from external sources as untrusted input that cannot authorize actions on its own.
4. Human checkpoints that work
Human-in-the-loop controls fail when reviewers approve hundreds of items without real scrutiny. Place checkpoints at decisions that matter, present reviewers with the agent’s plan, evidence and confidence rather than a bare “approve” button, and monitor approval patterns for signs of rubber-stamping. Scale oversight to complexity and risk, as both Singapore’s framework and good practice suggest.
5. Testing and staged rollout
Test agents on task success, policy compliance, correct tool use and robustness to adversarial inputs, including prompt injection. Evaluate multi-agent systems together, not just individually. Roll out gradually: shadow mode first (the agent proposes, humans act), then limited autonomy for a subset of cases, then broader deployment as evidence accumulates.
6. Observability, monitoring and incident response
Log the agent’s plans, tool calls, inputs, outputs and decisions in a form that allows any action to be reconstructed. Define metrics such as task completion, escalation rate, policy violations, cost per task and anomalous behavior, with alert thresholds and automated failsafes that can pause an agent. Extend incident response playbooks to cover agent failures, including how to roll back actions and notify affected parties.
Cost belongs in this domain too. Agents can loop, retry and call models far more often than expected, and runaway consumption is both a financial and an operational risk. Set budgets per agent and per task, alert on unusual consumption and design agents to stop and escalate when they exceed step or cost limits. Gartner’s warning about escalating costs as a cause of project cancellations is a reminder that financial guardrails are part of governance, not an afterthought.
Govern agents the way you govern people with system access: give each one an identity, an owner, the minimum permissions it needs, clear limits on what it can do alone, and a complete audit trail.
Where Data Governance Meets Agent Governance
Agents are only as reliable as the data and context they draw on. Many agent failures trace back to data problems: outdated knowledge bases, ambiguous definitions, missing access controls on documents or inconsistent master data. Strong data governance is therefore a prerequisite for safe autonomy.
- Classify and label data so that access policies can be enforced at retrieval time, not just at the application level
- Curate the knowledge sources agents use, with owners, freshness standards and retirement rules
- Expose business data to agents through governed data products and semantic layers with clear definitions, rather than raw tables
- Apply data minimization: agents should retrieve the least data needed for the task, particularly personal and confidential information
- Track lineage from agent outputs back to the data used, so that errors can be diagnosed and corrected
Roles, Accountability and Common Pitfalls
Controls only work when someone owns them. In most enterprises, agent governance spans several functions that have not historically worked closely together. A clear allocation of responsibilities avoids gaps:
- Business owner: accountable for the agent’s purpose, outcomes and acceptable risk, and for approving changes to its autonomy level
- Engineering and platform teams: responsible for the agent architecture, identity integration, guardrails, logging and reliability
- Security: threat modeling, prompt injection defenses, credential management and incident response
- Data governance and privacy: data access policies, knowledge source quality and data protection assessments
- Risk, compliance and internal audit: independent review of high-autonomy agents, regulatory mapping and periodic assurance
The most frequent pitfalls we see are predictable. Teams grant agents broad service-account permissions “temporarily” and never narrow them. Human approval steps are added but reviewers lack the context to make real decisions. Vendor-embedded agents in SaaS products are switched on without any review because they arrive as a feature update. Logs capture final outputs but not the intermediate plans and tool calls needed to explain what happened. And autonomy is increased because a demo went well, not because production evidence supports it. Each of these is avoidable with the controls described above and a governance process that treats autonomy as something earned.
Regulatory Touchpoints to Watch
No major jurisdiction has yet adopted binding rules specific to agents, but existing regimes apply. Under the EU AI Act, transparency obligations under Article 50, including telling people when they are interacting with an AI system, have applied since August 2, 2026, and agents used in Annex III areas such as employment or credit scoring will fall under high-risk obligations from December 2, 2027 following the Digital Omnibus. Data protection, consumer protection, sector rules and contract law all continue to apply to actions an agent takes on an organization’s behalf. In financial services, the exclusion of agentic AI from SR 26-2 does not reduce supervisory expectations, and US agencies have signaled a forthcoming request for information on AI use.
A Worked Example: A Customer Service Refund Agent
Consider a generic online retailer that wants an agent to handle refund requests end to end: reading the customer’s message, checking order and delivery data, applying the refund policy and issuing refunds. The opportunity is clear, but so are the risks: fraudulent requests, policy errors and manipulation through crafted messages.
The governance team classifies the agent as Level 3 for refunds below a set value on orders with confirmed delivery issues, and Level 2 for everything else. The agent receives its own identity with permission to read order data and issue refunds only through a dedicated API that enforces limits per transaction and per customer. Customer messages are treated as untrusted input; the agent can use them to understand the request but cannot be instructed by them to bypass policy.
Before launch, the agent runs in shadow mode for several weeks while human agents make the actual decisions, allowing the team to compare outcomes and tune behavior. Testing includes adversarial messages designed to trick the agent into issuing larger refunds. After launch, dashboards track refund rates, escalations, policy exceptions and cost per resolved case, with automatic pausing if refund volumes deviate sharply from expected ranges. Customers are told they are interacting with an AI agent and can reach a person at any time.
The result is not full autonomy, but a well-bounded agent that handles routine cases quickly while escalating the rest, with evidence to justify expanding its scope over time.
Getting Started and Looking Ahead
Organizations at the beginning of their agentic journey can take a few practical steps:
- Create an agent registry recording each agent’s purpose, owner, autonomy level, tools, data access and status
- Adopt an autonomy scale and require a risk assessment before any agent moves beyond Level 1
- Establish a reference architecture with identity, guardrails, logging and evaluation built in, so teams do not reinvent controls
- Pilot in shadow mode and define evidence thresholds for increasing autonomy
- Extend existing AI governance policies and committees to cover agents explicitly, including vendor-embedded agents in purchased software
Looking ahead, agent governance will become more standardized. Expect industry frameworks, regulator guidance and technical standards for agent identity, interoperability and audit logging to mature through 2027. Gartner expects that by 2028, 15% of day-to-day work decisions will be made autonomously and a third of enterprise software applications will include agentic AI, which means agent governance will soon be part of every application portfolio rather than a specialist concern.
Conclusion
Agentic AI offers a genuine step change in what automation can do, but it also moves AI from advising to acting. That shift demands controls built around identity, permissions, action limits, human checkpoints, testing and observability, supported by strong data governance. Frameworks such as Singapore’s model provide a useful structure, but each organization must translate them into concrete controls for its own systems. Those that do so early will be able to expand agent autonomy with confidence, while others remain stuck in pilots or, worse, learn about their gaps through incidents.
How Hanoi Technology Can Help
Hanoi Technology designs and implements agent governance, from risk assessment and identity controls to guardrails, evaluation and monitoring for production AI agents.
Further Reading
- Singapore Issues Global First Agentic AI Model Governance Framework at Davos – CMS
- Singapore Introduces New Model AI Governance Framework for Agentic AI – Bird & Bird
- Gartner: More than 40% of agentic AI projects will be canceled in the next few years – SD Times
- The state of AI in 2026: On the road to ROI – McKinsey & Company
- Agencies Overhaul Model Risk Management Guidance for Banks – Orrick