When Rules Meet Capabilities
AI Governance's Five Failures

After the Pentagon placed Anthropic on a "supply chain risk" blacklist, all government agencies should theoretically have stopped using its products. But an Axios exclusive revealed: the NSA is widely using Anthropic Mythos Preview internally—and instead of scaling back, its usage is expanding.

This incident itself isn't surprising—government agencies bypassing procurement rules to use effective tools has plenty of historical precedent. What's truly worth noting is the underlying logic: when a tool's capabilities are powerful enough, internal demand automatically generates the motivation to bypass restrictions. Mythos can autonomously discover 27-year-old unfound security vulnerabilities in the Linux kernel; faced with such a tool, any security officer would find it hard to truly say "no."

This logic isn't limited to government. Every enterprise IT compliance team faces the same tension: the more capable the tool, the greater its compliance controversy tends to be. Copilot, Claude Code, Cursor—each has faced internal "prohibited use" policies, and each has survived through engineers' quiet, persistent use.

MIT Technology Review's piece on the illusion of "human-in-the-loop" in AI warfare raises a deeper question: when AI systems make decisions far faster than human approval processes, "human oversight" itself becomes a ceremonial existence. Compliance frameworks are designed assuming humans have enough time to intervene—but AI's speed has already invalidated that assumption.

In the same week, two entirely different data privacy incidents pointed to the same larger problem.

Atlassian quietly enabled data collection in Jira and Confluence to train its AI models, with the opt-out deliberately hidden. This isn't the first time—Zoom, Slack, and LinkedIn have all followed similar paths. "Default consent" has become standard operating procedure for SaaS platforms, because every case where users genuinely pushed back proved one thing: most users will take no action at all.

Meanwhile, Notion's public page vulnerability—allowing anyone to obtain editors' full names and emails without authentication—was reported in 2022 and remains unpatched four years later. This isn't technical incompetence; it's more likely a priority judgment: fixing this vulnerability incurs costs, while most affected users neither know nor care.

Taken together, these two events reveal the core dilemma facing enterprise data sovereignty: you cannot control what you do not understand. Enterprises store their most sensitive internal information—product roadmaps, customer data, engineering documents—in SaaS tools, yet most haven't even carefully read these tools' data usage terms.

The urgency of AI training data needs will only worsen this problem. SaaS platforms now have stronger incentives than ever to expand their data rights, because the competitiveness of their internal AI products depends on it directly. Users have fewer and fewer choices—either accept, or migrate to a competitor (who most likely has the same terms).

Claude Desktop silently registers a browser Native Messaging host on installation, granting itself permissions to read browser state and manipulate the DOM—this is to enable browser integration features, but there is no explicit user notification throughout the process. In the same week, Vercel confirmed a security incident, where the attack was launched through a weak link in a third-party AI OAuth integration, affecting some customer data.

These two events point to a new risk surface in AI platform expansion: AI tools are acquiring permissions faster than enterprise security teams can understand and audit them.

Claude Desktop's situation is particularly typical. Anthropic's intentions are benign—browser integration allows Claude to serve users better. But the combination of "benign intent + unnotified permission expansion" is precisely the problem by privacy tool community standards. More importantly, when enterprise IT departments deploy Claude Desktop, chances are no one checked what it does to the Chromium browser on every employee's computer.

Vercel's incident represents another category of risk: the high-value target nature of AI development platforms. Developers store not just code on AI development platforms, but also API keys, database connections, and production data samples used for AI testing. Any third-party OAuth integration with lax security auditing becomes a weak entry point for the entire platform.

The fractures discussed above are all at the security and governance level, but this week another set of data pointed to questions about AI's commercial value itself.

A survey of 6,000 executives showed nearly 90% believe AI has had no substantial impact on productivity over three years, with executives using AI only 1.5 hours per week; Uber's $3.4 billion AI partnership saw its Eats division's AI copy unanimously rated as having "no practical effect"; the launch of Claude Design put Figma in an absurd position: paying API costs that fuel a competitor's capabilities, while that competitor's marginal cost is nearly zero.

These three cases appear independent, but together they point to one issue: there is a chasm between AI's technical demonstrations and its commercial value that has yet to be systematically crossed. Mythos can find 27-year-old undiscovered security vulnerabilities; Claude Design can let non-designers "explore a dozen design directions in minutes"—these are genuine technical achievements. Yet most business leaders cannot perceive this capability in their daily work, for three reasons:

First, there are integration costs between capability and workflow. No matter how powerful AI is, without appropriate use-case packaging and process embedding, executives won't see it.

Second, multi-tool fatigue obscures the value of any single tool. When employees are simultaneously required to use five AI tools, each one seems "not very useful."

Third, the measurability of commercial ROI is itself an engineering problem. Most organizations haven't built the data infrastructure to measure AI's contribution at all.

Looking back at all of this: the NSA's unauthorized use of Mythos, Atlassian silently enabling data collection, Notion's vulnerability unpatched for four years, Claude Desktop silently registering browser permissions, Tesla concealing self-driving accidents—behind each event, there are no bad actors, but rather rule-makers and rule-enforcers collectively losing their rhythm amid a rapidly advancing technological wave.

Behind these five cases are three counter-consensus insights worth repeating—more important than any single news item:

What does this mean for three different groups of people?