AI governance is the set of policies, roles, processes and technical controls an organisation uses to decide which AI systems it builds or buys, and to keep them safe, lawful and accountable while they run. It answers practical questions: who approves a new AI use case, what data it may touch, how its quality is checked, who is responsible when it is wrong, and how it is switched off. Good governance is mostly visible in the systems themselves, as logs, permissions, approvals and tests, not only in a policy document.
What AI Governance Covers
- Inventory. A list of every AI system and model in use, including those bought inside SaaS tools, with an owner for each.
- Risk classification. Which uses are low risk and which affect people’s money, jobs, health or rights. The EU AI Act makes this a legal question for many companies.
- Data rules. What data each system may read, where it is processed and stored (data residency), and how long it is kept.
- Quality and safety. Evaluation before launch, monitoring after, and guardrails against misuse and prompt injection.
- Accountability. Named people who approve, operate and answer for each system, and a route for users to challenge an outcome.
- Change control. How model, prompt and data changes are reviewed, tested and rolled back.
- Third parties. Due diligence on model vendors and AI features in purchased software.
Frameworks Companies Use
- NIST AI Risk Management Framework. Released by the US National Institute of Standards and Technology on 26 January 2023, it organises the work into four functions: Govern, Map, Measure and Manage. NIST added a Generative AI Profile (NIST AI 600-1) on 26 July 2024. It is voluntary and widely used as a checklist.
- ISO/IEC 42001. An international standard for an AI management system, built like other ISO management standards, so it can sit next to existing security and quality systems and be audited.
- The EU AI Act. Law, not a framework, but it sets the minimum for organisations that provide or use AI in the EU, including risk management, logging, human oversight and AI literacy duties.
- Sector rules. Banking model-risk guidance, medical device rules and data protection law such as the GDPR all apply to AI systems in their fields.
Governance in the System, Not Only on Paper
Policies only matter if the software enforces them. For GenAI systems the controls that do most of the work are:
| Control | What it prevents |
|---|---|
| Least-privilege access, read-only by default | An agent changing data it should only read |
| Human in the loop approval for writes | Unreviewed changes to prices, records or settings |
| Source citations on every answer | Unverifiable claims reaching users |
| AI observability and logs | Problems nobody can trace after the fact |
| Versioned prompts and models with LLM evaluation | Silent quality drops after a change |
| Calculations in tested code | A language model inventing a number in a report |
Common Failure Modes
- Policy without inventory. Rules exist, but nobody knows which teams use which AI tools.
- A committee as a bottleneck. Every use case waits months for review, so teams route around governance. Lighter review for low-risk uses keeps people inside the process.
- One-time approval. A system is reviewed at launch and never again, while its model, prompts and data keep changing.
- Model grading model. Quality is signed off by another model’s scores without people reading real outputs.
- Agents with their own keys. An AI agent runs with a service account that has more rights than the user it acts for.
Related Terms
AI governance overlaps with data governance, security and compliance. It relies on interpretability where decisions need to be explained, and on LLMOps practices for keeping systems measured in production.
In Practice
Governance starts with a register: every AI use case in the organization, its owner, the data it touches, the decisions it influences and its risk level. The register sets the controls. An assistant that drafts internal notes needs light review, while a system that changes customer records needs approval steps, logged decisions and a named person accountable for it. The controls are then built into the system itself rather than left in a policy document, and the register is reopened whenever a model, a vendor or a use case changes.
We help organisations set this up through AI strategy consulting and AI security solutions development, and keep it running with LLMOps services.