Policy on Paper

There is a file on your company's intranet that says your AI agents cannot access customer data without authorization, cannot execute financial transactions above a defined threshold, and cannot egress sensitive information to external systems.

Your AI agents can, in fact, do all three of those things.

Both of those statements are true. That is the problem.

I have read through a large number of enterprise AI governance policies over the last year. They are thoughtful. They are earnest. They have been shaped by lawyers, compliance officers, and security leaders who take their jobs seriously and know exactly how much a serious AI incident is going to cost when it lands. They reference the NIST AI Risk Management Framework. They map to ISO 42001. Many of them run thirty pages.

None of them do anything.

This is not a rhetorical flourish. NIST's own guidance on the AI RMF is explicit that policy must be implemented as dynamic controls — controls that act on real model behavior at runtime, not as static configuration filed and forgotten. Governance specialists have put it more bluntly: a governance mechanism is worth more than a governance framework, because frameworks without mechanisms do not govern anything. The industry is starting to say this out loud. Most of the industry is not yet acting on it.

A clause that describes what an agent must not do, without specifying the mechanism that stops the agent from doing it, is a description of a preference. Your enterprise does not have an AI governance program. It has an AI governance wish list, typeset nicely. The industry is currently confusing the two, and a lot of people are about to find out the hard way that they are not the same.

The shape of the incidents

Look at the publicly reported AI incidents of the last six months and pay attention to what is conspicuously absent from each of them.

Late last year, Deloitte's Australian arm had to issue a partial refund to the Australian government after a welfare-systems review it produced turned out to be padded with AI-generated hallucinations — fabricated academic citations, an invented court quote, references to research papers that do not exist. Deloitte has governance. Deloitte has more governance than almost any organization on the planet. What Deloitte did not have was a mechanism that stopped a model from inventing citations and a human from shipping them.

Earlier this year, Meta disclosed a serious internal incident in which an AI agent exposed sensitive user and company data to engineers who did not have permission to access it, for roughly a two-hour window before it was contained. Meta's policies on internal data access are not aspirational. They are detailed, well-funded, and owned by teams who deal with regulators on a full-time basis. What Meta did not have was a mechanism that stopped an agent — an internal agent, deployed by Meta's own engineers — from routing data to the wrong set of humans.

This month, Salesforce and Microsoft both patched prompt injection vulnerabilities in Agentforce and Copilot that would have allowed external attackers to exfiltrate sensitive data through crafted inputs. In response, Salesforce characterized the data exfiltration path not as a platform vulnerability but as a customer configuration matter — pointing out that customers could activate a human-in-the-loop setting if they wanted one.

Read that sentence again. The mechanism that could have stopped a prompt injection from exfiltrating customer data was a setting. Off by default.

These are three illustrations of the same shape. Every month adds more. A setting that defaults to off is not a control. A guideline in a Confluence page is not a control. A prompt instruction that asks the model to behave is not a control. A policy with executive signatures and a tasteful header font is not a control. The word control has a technical meaning, and most of what the industry is currently calling AI governance does not meet it.

The broader evidence

If individual incidents feel anecdotal, the aggregate numbers are not.

IBM's 2025 Cost of a Data Breach Report was the first year in which researchers specifically measured AI-related security characteristics. Among organizations that experienced an AI-related breach, the overwhelming majority — 97% by IBM's count — reported that they had no technical access controls on their AI systems at the time of the incident. Not "insufficient" controls. Not "immature" controls. No controls.

Ninety-seven percent.

The defining characteristic of organizations breached through their AI was not that their policies were weak. In many cases the policies were thorough. The defining characteristic was that their policies were not connected to anything that executed. They had committees. They had documented procedures. They did not have enforcement.

Adjacent surveys find the same pattern. Governance vendors and analysts repeatedly note that enterprises are strong on frameworks, councils, and training materials, and weak on policy engines, runtime guards, and automated audit trails. The documentary side of governance is almost universally over-invested. The mechanism side is almost universally under-invested. This is not a rumor. It is what the industry keeps telling itself in its own reports.

If you are running AI agents in production right now and your governance program lives in documents rather than in the runtime, you are not an exception to this pattern. You are the pattern. The only remaining question is which quarter your incident report gets filed in.

We have done this to ourselves before

This is not the first time an industry has confused documentation for control, and it is worth being honest about that, because the rerun is happening in real time.

When Sarbanes-Oxley passed in 2002, the initial corporate response was a wave of paperwork. Committee charters. Internal control memoranda. Certification letters. Audit committees that met, produced minutes, and forwarded them to other committees. None of it prevented financial misstatement. What eventually prevented financial misstatement — grudgingly, slowly, only after some CFOs went to prison and some firms collapsed — was the long, unglamorous work of moving the controls into the systems. Segregation of duties became a technical property enforced by IAM. Change management on financial-reporting code became a gate enforced by release tooling. Journal entries started flowing through approval workflows that could not be verbally overridden.

PCI-DSS went through the same arc faster, pushed by the direct financial pain of card-data breaches. HIPAA has gone through it slowly and incompletely. GDPR is still halfway through it and will be for another decade. The pattern is always the same. Governance begins as description. It becomes real when it is moved into mechanism. The organizations that make that shift early do well. The organizations that try to skip it by writing better policies eventually get breached, fined, and forced into mechanism anyway, at many multiples of the original cost.

The regulators are already here

What is different this time is that the regulators are not going to wait for the industry to figure this out on its own. They are already writing "mechanism, not memo" into law.

The EU AI Act, in its governance and enforcement provisions, requires automatic logging of events across high-risk AI systems and assigns live supervisory authority to designated market surveillance bodies. The Act is not structured to reward organizations that have written policies. It is structured to reward organizations that can demonstrate, through logs and technical evidence, that their AI systems behaved as required. If you cannot produce a log that shows what your agent did, when, why, and what your system's response was, you are not compliant. There is no partial credit for a well-drafted responsible AI statement.

In the United States, proposed updates to the HIPAA Security Rule move in the same direction: explicit emphasis on real-time monitoring for unauthorized activity and demonstrable technical controls around systems that touch protected health information. Financial services regulators are landing in similar places. The NIST AI Risk Management Framework, which many enterprise policies cite as their governing authority, treats the framework itself as scaffolding for continuous, dynamic control — not as a compliance checklist to be signed off once and filed.

If you are surprised to learn that your regulator has gotten here before you did, that is a signal about your governance function, not about your regulator.

AI governance is at the beginning of the enforcement arc. The documents are being drafted. The mechanisms, in most places, are not. What is different about this round is the clock. Agents move faster than the systems they are replacing. The time between "we wrote a policy" and "we were breached because the policy was not real" is collapsing from years to months.

If you are reading this and your plan is to write a better policy, the blog post that uses your company as the cautionary tale is already being outlined by someone.

The question you are not asking

Here is the exercise I would like every CISO, general counsel, and AI leader to run on their own organization, today, before the end of the day if possible.

Pull up whatever AI governance policies your company has published. Find every clause that uses the words may not, cannot, shall not, must not, or is prohibited. For each clause, ask one question, out loud:

What stops it?

That is the whole test. Three words. If you want to be formal about it you can call it the Mechanism Test. I call it "what stops it," because that is what I actually say in the room.

This is not a heuristic I invented. It is an operationalization of where serious AI governance guidance is already going. Contemporary policy-enforcement vendors and analysts describe modern AI governance as evaluating every request — prompt, retrieval, tool call — against a policy set, and deciding, in real time, whether to allow, warn, block, or escalate. NIST says much the same thing in less direct language. The EU AI Act effectively encodes it. All of it converges on one point: policy lives in the runtime, or policy does not live at all.

If the answer is the policy says so — nothing stops it. The policy is an opinion. The agent cannot read the policy, and if it could, it would not care.

If the answer is the developer knows the rule — nothing stops it. The next developer doesn't. The developer who modifies the agent six months from now doesn't. The contractor who builds the adjacent agent doesn't. Human memory is not a security control.

If the answer is we tell the model not to do that in the prompt — nothing stops it. Adversaries can inject. Context windows drift. Models update. Prompts are, and have always been, advisory. The Salesforce and Microsoft incidents this month are not exotic. They are what the prompt-as-control posture looks like when an adversary shows up.

If the answer is we have a content filter on the input or the output — something might stop it, sometimes, for certain classes of violation. Most violations do not happen at the input or output. They happen in the middle, where the filter is not looking. Your agent's tool calls, data reads, and inter-service actions all live inside that blind spot.

If the answer is the runtime refuses to execute the action — something stops it. The agent requests; the execution environment evaluates the request against the policy set; the action either happens or it does not. The agent's good behavior is not the load-bearing element. The mechanism is the load-bearing element. The policy is the specification for the mechanism, which is exactly what a policy is supposed to be.

Now go through your clauses and count. For how many of them is the answer in that last category?

In my experience the number is almost never above ten percent. In most organizations it is zero. Ten percent is not governance. Zero is not governance. Those are decorations. You are, at best, in the business of telling a story about governance. The story may be compelling. It may even be true in the narrow sense that the policies exist, the committees meet, the certifications are signed. The agents don't care about any of it.

What real governance actually feels like

Enforcement does not look like a policy document. It looks like a boring afternoon.

A developer tries to wire a new external API into an agent. They cannot, because the organization's policy says agents in this department are not permitted to call tools that egress to external payment providers, and the runtime refuses to register the binding. The developer opens a ticket. Compliance looks at it. Compliance decides whether the policy should be adjusted. If yes, the policy changes, and the binding is granted. If no, the developer finds a different way to accomplish the business goal.

Nobody raised a voice. Nobody wrote a memo. Nobody forgot a rule and then remembered it three months later in a postmortem. The policy worked because the policy was the system, not a document about the system.

Or: a prompt injection lands in the input stream of a customer-service agent, telling it to look up and return a different customer's records. The agent, following the injected instruction, constructs a request to the CRM. The runtime evaluates the request against the policy set, notices that the tool invocation would cross a data classification boundary the agent is not authorized to cross, denies the call, and logs the denial with a content hash and a decrement to the agent's trust score. The agent returns an error and moves on to other work.

No data left the environment. No incident report was filed. A security engineer sees the denial the next morning in the audit summary, notices the pattern, traces it to the injection source, and addresses it. The governance worked not because someone happened to be watching in real time but because the mechanism was there when no one was watching.

This is boring. This is what governance is supposed to look like. The companies that have taken AI governance seriously at scale — Microsoft, Google, and IBM among them — have spent the last several years embedding controls into their product lifecycles: model cards, AI registries, bias audits, and monitoring wired into the systems themselves rather than described on top of them. The shared feature of these programs is that the governance continues to function when the responsible AI council is not in session. Which is to say: always.

The organizations that treat AI governance as a performance — the press releases, the responsible AI charters, the steering committees that meet quarterly — are going to be outrun by the organizations that treat it as an engineering problem. Not because the latter are cooler or more responsible, but because their agents will actually behave, and their competitors' agents will not.

The honest conversation

I want to say one last thing, directly to the executives who are currently deciding what shape their AI governance investment should take.

The version of this conversation that gets you into trouble is the one where your CISO tells you the governance is under control, and they genuinely believe it, and it isn't. They mean the policy is written. They mean there is a review process. They mean the AI council has met five times this quarter. All of that is probably true. None of it is what an auditor is going to ask you about in 2027.

The auditor — or the EU market surveillance authority, or the HHS investigator, or the plaintiff's attorney in the data exposure suit — is going to ask: Show me the log of every time an agent tried to access a data class it was not authorized to access, and show me what the runtime did in response. If the answer is "let me get back to you," you do not have governance. You have a document.

The gap between these two things — between having a document and having a mechanism — is the only gap that matters. Everything else is prologue. The frameworks, the councils, the training, the certifications: all prologue. The mechanism is the play.

You should write the policy. Of course you should. But when you have written it, walk across the hall to the engineering team, and for every clause, ask what stops it. If nothing stops it, you have a wish. Wishes have a poor record against adversaries, against incidents, against regulators, and eventually against boards and plaintiffs' attorneys.

This is a solvable problem. Most organizations are not yet trying to solve it. They are still writing.

Do not be one of them.