By late 2026 almost every large organization has an AI policy, and many have mapped themselves against the EU AI Act, ISO/IEC 42001 or the NIST AI Risk Management Framework. Yet the same symptoms keep surfacing: a business unit launches an AI assistant nobody in risk has heard of; a vendor quietly adds generative features to a platform already in production; an agent gains write access to a system because no one owned the decision; and when the board asks how many AI systems the company runs, the answer takes three weeks and is still incomplete.
These are operating-model failures, not policy failures. Frameworks describe what good AI risk management looks like. An operating model defines who decides, when, with what evidence, and how that decision is recorded and revisited. This article focuses on that practical layer: the roles, forums, decision rights, processes and metrics that make AI governance run inside day-to-day delivery. (For regulation-specific detail, see our separate articles on the EU AI Act after the Digital Omnibus, ISO/IEC 42001 certification and governing agentic AI.)
Why Policy Alone Does Not Govern AI
AI is unusually hard to govern with policy documents for three reasons. First, it is everywhere at once: it arrives through internal builds, SaaS features, employee tools, data science notebooks and agent platforms, often without a procurement event. Second, it changes after approval: models are upgraded, prompts edited, data sources added and autonomy extended, each of which can change the risk profile. Third, accountability is split across functions that rarely share a process: legal, privacy, security, data, risk, procurement, HR and the business.
The demand for structure is clear. The IAPP’s AI Governance Profession Report 2025 found that 77% of organizations are already working on AI governance, and that a shortage of qualified AI governance talent is the leading challenge. Scarce expertise is the strongest argument for an operating model: it lets a small central team multiply its effect through clear rules, reusable assets and embedded champions.
Borrow a Proven Template: Lessons From Public-Sector AI Governance
One of the clearest published operating models comes from the US federal government. OMB Memorandum M-25-21, issued in April 2025, requires covered agencies to appoint a Chief AI Officer, requires larger agencies to convene an AI Governance Board, and defines “high-impact AI” as AI whose output serves as a principal basis for decisions or actions with a legal, material, binding or significant effect on rights or safety. For high-impact uses it sets minimum practices that include pre-deployment testing, AI impact assessments before and during deployment, human oversight with training and intervention safeguards, remedies or appeals for affected people, and pausing or discontinuing AI that does not meet the requirements. Agencies must also inventory their AI use cases at least annually and publicly justify any waivers.
Private companies are not bound by the memo, but its structure translates well: a single accountable executive, a cross-functional decision forum, a risk-based definition that concentrates effort where harm is possible, a short list of mandatory practices for that tier, a living inventory, and a documented exception process. The rest of this article builds on that pattern.
The Core Roles and Forums
A workable AI governance operating model usually follows a hub-and-spoke design: a small central team sets standards and handles high-risk decisions, while business and delivery teams own their systems and run most controls themselves.
Accountable executive
One named leader, often a Chief AI Officer, Chief Data and AI Officer or a senior technology executive, is accountable for AI risk across the enterprise. This person chairs the governance council, owns the AI policy, approves the risk appetite and reports to the board or its risk committee.
AI governance council
A cross-functional forum with decision rights, not just advisory status. Typical members: legal, privacy, information security, enterprise risk or compliance, data governance, HR (for workforce impacts), procurement, and senior business representatives. It approves high-risk use cases, sets standards, rules on escalations and exceptions, and reviews portfolio metrics. It should meet on a fixed cadence with a published agenda and record its decisions.
AI governance office (the hub)
A small team, sometimes three to six people in a mid-sized enterprise, that runs the process: maintains the inventory, operates intake and tiering, curates templates and control libraries, coordinates reviews, tracks remediation and produces reporting. It is the engine room; without it, the council becomes a meeting with no pipeline.
Use-case owners and system owners (the spokes)
Every AI system has a business owner accountable for its purpose and outcomes, and a technical owner accountable for how it is built and run. They complete assessments, implement controls, monitor performance and trigger reassessment when something material changes. Most governance work should happen here.
Second and third lines
Risk and compliance functions challenge first-line assessments for higher tiers and test controls; internal audit provides independent assurance on whether the whole system works. Mapping AI governance onto the existing three-lines model avoids building a parallel bureaucracy.
AI champions
Trained practitioners embedded in business units and engineering teams who help colleagues register use cases, apply templates and spot issues early. They are the most cost-effective answer to the talent shortage.
Decision Rights: Who Approves What
Ambiguous decision rights are the most common reason governance stalls. Write them down. A simple matrix might look like this:
- Register a use case: anyone; mandatory before build or purchase.
- Approve low-risk use cases (internal productivity, no personal data beyond business contact details, no decisions about people): use-case owner, with automated checks and champion review.
- Approve medium-risk use cases (customer-facing content, personal data, moderate autonomy): governance office plus relevant specialists (privacy, security).
- Approve high-risk use cases (decisions with legal or similarly significant effects on people, regulated categories, safety, agents with write access to critical systems): governance council, with second-line challenge.
- Prohibited uses: defined by policy; no one can approve them, and exceptions go to the accountable executive and legal.
- Material change to an approved system (new model, new data, new user group, more autonomy): triggers reassessment at the original tier or higher.
- Pause or retire a system: system owner, governance office or council can order it when monitoring shows unacceptable risk.
Good AI governance is mostly a routing problem: send low-risk work through fast, automated lanes so experts have time for the few decisions that really matter.
The Governance Lifecycle: From Intake to Retirement
Governance should follow the AI system lifecycle and plug into existing delivery and procurement processes rather than sit beside them.
- Intake and registration: a short form (ten to fifteen questions) captures purpose, users, affected people, data used, model or vendor, level of autonomy and the business owner. Integrate it with project portfolio, procurement and architecture review so AI cannot bypass it.
- Risk tiering: rules applied to intake answers assign a tier automatically, with human review for edge cases. Criteria typically include impact on individuals, regulatory classification, data sensitivity, autonomy and scale.
- Assessment: tier-proportionate. Low risk may need only a checklist; high risk needs a full impact assessment covering intended purpose, foreseeable misuse, data provenance, fairness, security threats, human oversight design and fallback.
- Design controls: select required controls from a library mapped to your frameworks: evaluation thresholds, red-teaming, logging, transparency notices, access limits, human review points, kill switches.
- Pre-deployment approval: evidence that controls are implemented and evaluation results meet thresholds, signed off at the right level.
- Operation and monitoring: performance, drift, incidents, user complaints, cost and autonomy boundaries, with alerts routed to system owners.
- Change and reassessment: defined triggers plus periodic review (for example annually for high risk).
- Retirement: decommissioning, data retention or deletion, and inventory update.
Make the Inventory the Backbone
Every governance question, from regulatory reporting to incident response, starts with “which systems are affected?” The AI inventory must therefore be a maintained system of record, not a spreadsheet compiled for an audit. Practical requirements:
- One record per AI system, linked to owners, vendors, models and versions, data sources, tier, assessments, approvals and open issues.
- Discovery beyond self-reporting: scan SaaS contracts and release notes for embedded AI features, review cloud and AI gateway logs for model API usage, and ask procurement to flag AI terms in new contracts.
- Agents recorded with their tools, permissions and the identities they act under.
- Status visible to the business, so teams can reuse approved patterns instead of starting from scratch.
Embed Controls in Delivery: Governance as Code
The fastest way to make governance stick is to turn controls into engineering defaults. Examples that work well:
- Release pipelines that block deployment of an AI system without an inventory ID and a current approval for its tier.
- Evaluation suites with pass thresholds that run on every model, prompt or retrieval change.
- An AI gateway that enforces approved models, data-loss prevention, logging and cost limits for every application.
- Standard agent permission profiles with least-privilege tool access and mandatory human approval for defined actions.
- Templates for transparency notices, model cards and incident reports, prefilled from the inventory.
This is also where security control frameworks are heading. NIST has been developing SP 800-53 control overlays for securing AI systems, published first as a concept paper in 2025, and NIST states that the AI RMF itself is being revised under the White House AI Action Plan. An operating model that already maps controls to a library can absorb such updates by changing the library rather than re-educating every team.
Metrics That Show Governance Is Working
Report a small set of measures to the council monthly and to the board quarterly:
- Inventory coverage: registered systems versus systems discovered through scanning.
- Throughput: median days from intake to decision, by tier. Rising times signal a bottleneck that will push teams to bypass the process.
- Control health: share of high-risk systems with current assessments, passing evals and tested human oversight.
- Incidents and near misses: count, severity, time to contain, and lessons applied.
- Exceptions: number open, age and expiry.
- Value enablement: approved use cases in production and reuse of approved patterns.
Worked Example: Standing Up an Operating Model in 90 Days
Consider a generic 5,000-person services company with dozens of AI experiments, two customer-facing assistants and an HR screening tool under evaluation. It has a policy but no process.
- Days 1–30: the CIO is named accountable executive; a council charter with decision rights is approved; a three-person governance office is staffed from risk, data and engineering; a ten-question intake form and tiering rules are published; discovery scans of SaaS contracts and API logs begin.
- Days 31–60: all known systems are registered and tiered; the HR screening tool is classified high risk and goes to the council with a full impact assessment; the two assistants receive medium-risk reviews; a control library mapped to the NIST AI RMF and the EU AI Act is drafted; 20 champions are trained.
- Days 61–90: the release pipeline requires an inventory ID; the AI gateway becomes the only route to external models; the first monthly metrics pack goes to the council; low-risk approvals now take days rather than weeks, which persuades teams to use the process voluntarily.
None of this requires new headcount beyond the small office; it requires decisions about who owns what.
Choosing a Structure: Centralized, Federated or Hub-and-Spoke
There is no single correct shape for AI governance; the right structure depends on how concentrated AI work is and how mature business units are. Three patterns are common.
Centralized
A central team reviews and approves every AI use case. It suits organizations early in adoption, with a handful of systems and a strong need for consistency, such as a regulated firm making its first customer-facing deployments. It breaks down as volume grows: queues lengthen, reviewers become generalists stretched across too many domains, and teams start routing around the process.
Federated
Each business unit or region runs its own governance under a common policy. It suits large, diversified groups with mature local risk functions. The weakness is inconsistency: the same vendor tool can be approved in one division and refused in another, and enterprise-level reporting becomes a reconciliation exercise.
Hub-and-spoke
The hybrid described above: a central hub owns standards, tooling, the inventory and high-risk decisions, while spokes in each unit run day-to-day reviews with champions and local owners. Most enterprises converge here because it balances consistency with speed. A useful rule of thumb is to start more centralized while the inventory and control library are being built, then push lower-tier decisions out to the spokes once templates, automated checks and trained champions make them reliable.
Governing Third-Party and Embedded AI
A large share of real AI exposure arrives through vendors rather than internal builds: copilots switched on in productivity suites, generative features added to CRM or HR platforms, and agent capabilities offered by service providers. An operating model needs a specific lane for this.
- Procurement hooks: add AI questions to vendor due diligence (which models are used, where data is processed, whether customer data trains models, how changes are notified) and route positive answers into intake automatically.
- Contract clauses: require notice of material model or feature changes, audit and evaluation rights proportionate to the tier, incident notification timelines and clear data-use limits.
- Feature-flag governance: treat the activation of a new AI feature in an existing platform as a change that requires registration, even when no new contract is signed.
- Shared responsibility: document which controls the vendor provides and which remain yours, such as user training, human review of outputs and transparency notices to customers.
- Periodic re-validation: when a vendor upgrades its underlying model, re-run your own evaluation set for high-risk uses rather than relying on release notes.
Handled this way, vendor AI stops being a blind spot and becomes another stream in the same pipeline, with the same inventory record, tiering and monitoring as systems built in-house.
Pitfalls to Avoid
- A council without a pipeline: meetings that discuss principles but never see a queue of decisions.
- One-size-fits-all reviews: treating a meeting summarizer like a credit model creates delays and drives shadow AI.
- Approval as the finish line: most AI risk emerges after launch through change and drift.
- Ignoring vendor AI: embedded SaaS features and model upgrades are a large share of real exposure.
- Governance owned by one function: legal-only or IT-only models miss half the risks and lose the business.
Conclusion
AI governance works when it behaves like an operating system rather than a rulebook: one accountable executive, a council with real decision rights, a small office that runs the pipeline, owners who carry controls day to day, a live inventory, tier-proportionate reviews and controls built into delivery tools. Start with decision rights and the inventory, measure throughput as carefully as risk, and let fast lanes for low-risk work earn the credibility you need for the hard decisions.
How Hanoi Technology Can Help
Hanoi Technology helps organizations design and run AI governance operating models, from intake and inventory to automated controls in delivery pipelines, through projects, dedicated teams or managed services.
Further Reading
- AI Governance Profession Report 2025 at a glance – IAPP
- OMB directs agencies to accelerate AI adoption and devise governance strategy – Jones Day
- OMB issues revised policies on AI use and procurement – National Law Review
- AI Risk Management Framework – NIST
- Risk Management projects (COSAIS) – NIST CSRC