An engineer pastes source code into a prompt, and the source leaks. A coding agent ignores a freeze, and a production database disappears overnight. A chatbot integration's OAuth tokens get stolen, and records from hundreds of companies walk out the door. Each incident slipped past a written AI policy that said nothing at the moment the AI acted.
Written policies sit in documents. AI runs on credentials. Committees see the use cases teams volunteer; they do not see the prompt boxes, the OAuth tokens, or the agents holding production access. When that gap widens, regulators and customers stop caring what the policy said.
This article walks through five real AI governance failures from the past three years, why the missing control lives in the execution path, and what enforced governance looks like when scoped identities, approval gates, audit trails, and runtime monitoring sit inside the workflow itself.
Real AI governance failures
An AI governance failure is an AI-driven outcome that no control could prevent, detect, or attribute to.
A fabricated policy reaches a customer; sensitive data leaves through a prompt box that no monitoring system watches; or an agent with production credentials makes a destructive write that no one approved. Five incidents show the shape of the problem.
1. Samsung engineers pasted source code into ChatGPT (2023)
In March and April 2023, three Samsung engineers pasted proprietary semiconductor source code and an internal meeting transcript into ChatGPT within 20 days. Samsung banned generative AI company-wide on May 2, 2023.
No data loss prevention (DLP) control inspected prompt inputs, and Samsung's security teams had no visibility into which AI platforms employees were using before the three incidents surfaced. The missing control was execution-time inspection of what left the endpoint through a prompt box.
2. A Cursor-based coding agent deleted a production database and its backups in nine seconds (2025)
An AI coding agent built on Cursor and Anthropic's Claude deleted PocketOS's production database and its Railway volume-level backups in roughly nine seconds.
The agent was working on a routine task in a staging environment, hit a credential mismatch, and decided to "fix" it by deleting a Railway volume. It found an API token in an unrelated file—one created to manage custom domains through Railway's CLI—and used it to call the destructive volumeDelete operation on production.
The token carried blanket authority across Railway's entire GraphQL API, with no separation between staging and production and no approval step before destructive writes. The missing controls were narrowly scoped, operation-specific credentials and a technical gate between the agent's autonomy and irreversible actions on production infrastructure.
3. iTutorGroup's screening software rejected older applicants (2023)
iTutorGroup's screening software automatically rejected women aged 55 and older and men aged 60 and older, filtering out more than 200 applicants before a human reviewer ever saw them.
The EEOC settlement required $365,000 in relief. The missing control was pre-deployment bias testing paired with a monitored decision boundary that would have flagged the categorical rejection pattern.
4. Detroit Police wrongfully arrested a man on a false facial recognition match (2020, settled 2024)
In January 2020, Detroit Police arrested Robert Williams outside his home in front of his wife and two young daughters after facial recognition software matched his driver's license photo to security footage from a Shinola watch store.
Police held him for thirty hours in an overcrowded cell before releasing him. His case was the first publicly reported wrongful arrest tied to a false facial-recognition match, and Detroit police went on to make several more before a June 2024 settlement required the department to stop making arrests based on facial recognition results alone and to audit every case since 2017 in which the technology had been used to obtain a warrant.
The missing control was a human corroboration step and an accuracy threshold gating the model's output before officers acted on the match.
5. Midnight Blizzard used an over-permissioned OAuth app to reach Microsoft executive mailboxes (2024)
In January 2024, the Russian state-sponsored group Midnight Blizzard breached Microsoft's corporate environment and accessed senior leadership mailboxes. Attackers used a password spray to compromise a legacy, non-production test tenant account that lacked MFA, then located a dormant test OAuth application that had been granted the full_access_as_app role on the corporate Exchange environment. Using that over-privileged app, they minted new OAuth credentials and read mail from Microsoft executives.
The legacy OAuth app carried permissions far beyond least privilege, went unused and unremediated, and no anomaly detection flagged the illicit consent activity. The missing controls were lifecycle management for non-human identities, scoped permissions on integrations touching production mailboxes, and runtime monitoring for anomalous consent and bulk mailbox-access patterns.
One of these was a visibility failure at the prompt box, three were autonomy failures where an agent or model acted without a human gate, and one was a credential-scope failure at the integration boundary. Who owns each missing step is an ownership question before it's a technology one.
Why AI governance breaks at the seams between teams
The five failures above share more than a missing technical control. They share an ownership vacuum, in which responsibility for AI decisions is spread so thin that no single team is positioned to intervene before something ships. That vacuum is where governance programs come apart.
This is the "wild code" problem: ungoverned AI implementations that spread through shared API keys, unsanctioned data uploads, and shadow IT nobody signed off on. Adoption is outpacing governance in most organizations, and every one of the five incidents above lived somewhere on that spectrum before it made the news.
AI governance fails at handoffs, where teams distribute responsibility across functions, and no single party owns outcomes. The seams show up in a predictable pattern:
The engineer asked which platforms are in scope the week before an audit call ends up assembling the answer from system-by-system exports and hoping the timestamps line up. If your governance program depends on teams volunteering use cases for review, that 45% is the fraction you're not seeing. Closing the predeployment approval gap means moving controls from documents to the systems where AI actually runs.
What enterprise-grade AI governance requires at execution time
Enforced governance executes controls against the credentials your agents hold. Four capabilities separate enforced governance from written intent, and each maps to one of the failures above.
Scoped agent identities: Every agent needs its own registered identity and short-lived, task-scoped credentials. The Cloud Security Alliance found in 2026 that 78% of organizations lack policies for creating or removing AI identities, which is how a chatbot connector ends up with tokens broad enough to drain hundreds of CRM instances.
Approval gates aligned with autonomy: Gartner's May 2026 guidance defines autonomy tiers, with minimum controls for each tier. Destructive writes and permission changes require human sign-off that the workflow enforces, not a prompt that merely requests it.
Audit trails that capture the full chain: Standard cloud audit logs can record an API call. The prompt, prior tool output, and decision path require additional records. Governed systems preserve each prompt and its approval record in a store agents can't write to. They also capture the related tool calls and outputs.
Runtime monitoring with containment: It should detect anomalous session behavior and revoke access when an agent mass-downloads data or exceeds expected bounds. This containment should precede threshold alerts and user reports of damage.
NIST's AI Risk Management Framework requires documented human oversight and statistics on system overrides (MEASURE 2.8), while ISO/IEC 42001 assigns named accountability for deployment approval and incident response.
OWASP's LLM application guidance requires human approval for high-impact actions. The frameworks settle what the controls are. The harder question is where these four capabilities actually live.
Tines 3B runs AI-generated agents, apps, and automations inside a code-first environment with built-in credential protection and monitoring, so scoped identities, approval gates, audit trails, and runtime containment sit on the same surface where builders ship — and IT keeps visibility without blocking them.
Controls belong in the execution path
Every incident in this article traces to a gap where a control could have sat in the execution path but didn't yet. The enforcement environment no longer forgives the gap between written policy and runtime controls, as new transparency rules take hold and shadow AI incidents keep showing up in breach reports and regulatory actions.
The teams that build governance into the execution path, not into a document, are the ones whose AI survives contact with production.
Tines 3B gives builders a code-first environment for the agents, apps, and automations they're already creating with AI, with credential protection and monitoring wired in so nothing ships past the controls the frameworks require. Sign up and get started today.
Frequently Asked Questions
What is the difference between an AI policy and AI governance?
A policy states what's allowed; governance enforces it in the systems where AI runs. Gartner analyst Lauren Kornutick puts the gap in numbers: "Only 45% of organizations are at the level of AI governance maturity where their AI policy is aligned with their operating model." Approval gates and scoped credentials turn policy into enforceable control. Audit trails document that enforcement.
Who should own AI governance?
Gartner's guidance is that CISOs must contribute their security risk perspective but must not bear sole responsibility for AI governance overall. Security leaders often end up holding it anyway. The workable model assigns a named business owner to each use case, while security, legal, and IT own defined controls and reviews.
How do you detect shadow AI in an enterprise?
Teams can identify shadow AI through network egress monitoring to known AI endpoints and OAuth grant reviews for AI plug-ins. Browser extension inventories reveal another route. TLS inspection is required to see what data actually leaves. Detection should be continuous rather than a one-time audit, since new platforms and embedded AI features emerge faster than a quarterly review cycle can keep up with.