Over the past six months I ran five experiments: an AI project manager, auto hyperparameter tuning, ABAP evaluation, internal/external network desensitization, and a chip design map. None of them hit the wall of model capability—they all hit the same question: who proves this thing is correct?
AI Native R&D is not "engineers + a faster Copilot"—it is a process redesigned around a new price structure: generation is nearly free; trustworthy verification, coordination, understanding, and data boundaries become the scarce goods.
You swap models every quarter; what you can't swap out is your own verifier—scoring functions, acceptance criteria, expert evaluation sets, data classification rules. Teams should invest in that, not in stronger generators.
How are chip manufacturers currently deploying AI in production, and how should an ontology be constructed to comply with risk-control regulations?
Short answer. I can answer five things; one I cannot. What I can answer: manufacturers are moving from "ban everything" to "partitioned approval"; the order of AI adoption on the design side follows "is there a trustworthy verification method"; risk control lands on data classification, adversitorial auditing, permission tightening, and shadow mode; another path is confidential inference, letting the compute provider not see the data so desensitization is unnecessary (§06 has a subsection at the end, covering maturity and limitations); "ontology" also has a subsection at the end of §06 (first clarification: it is not a replacement for existing data, models, and pipelines, but a business twin layer built on top of them)—in semiconductor manufacturing there is academic research, cross-company data platforms like Athinia, and comparable permission mechanisms, but what's missing is published effectiveness data. What I cannot answer is the wafer fab production line (equipment, MES, yield closed loop): this article covers chip design flow and data boundaries, not the production line. To understand ontology itself first, see "The Full Anatomy of the Palantir Empire".
How can we reduce human involvement in code review and testing while ensuring AI writes high-quality, usable code?
Short answer. The article's conclusion is not to eliminate humans, but to move them from line-by-line review to three positions: signing off on acceptance criteria, reviewing critical changes, and deciding on release; the prerequisite is first making machine verification trustworthy (acceptance commands, harness, and graders not writable by the generator). Existing evidence: in one company, human review coverage dropped from 89% to 68% with no change in rollback rate (single company, non-random); Stripe's 1,300+ agent PRs per week are still all human-reviewed. There is no evidence that human review can be fully removed.
I did the math for my autoresearch-batch experiment. Having GLM generate one round of candidate improvements for CIFAR-100: LLM calls ≈ ¥1.5, already after the Batch API 50% discount; running that round of candidates on a T4: ≈ ¥20.
Writing candidates is more than ten times cheaper than verifying them.
autoresearch-batch · CIFAR-100 single round
This ratio later reappeared in different shapes in every experiment I ran. The Agent acting as team project manager—the hard part isn't writing messages, it's preventing it from randomly changing state; generating evaluations for legacy SAP code—the hard part isn't writing the questions, it's making the scoring withstand expert review; letting an internal network borrow an external Agent—the hard part isn't desensitization, it's proving the desensitization is sufficient.
Below I walk through these five experiments in order: tempo, autoresearch-batch, ABAP-Bench, escapegate, chip design full-flow map. For each one I re-verified against external evidence, and I'll be clear about where that evidence contradicts me.
First, set aside a common narrative: how much faster AI makes development. This question can't even be measured cleanly anymore.
In July 2025, METR published the most-cited randomized controlled experiment: 16 experienced open-source developers, 246 real tasks—when allowed to use AI, completion time was 19% longer; developers expected a 24% speedup beforehand, and still felt 20% faster afterward. In February 2026, METR changed the experimental design: a new round included 57 developers, 143 repos, 800+ tasks; estimates became −18% and −4% (negative means faster), but both confidence intervals crossed zero—METR called it "an unreliable signal." The reason is selection effect: more and more developers refuse to work without AI, the control group is disappearing.
This detail is more interesting than the "19%" itself. When a technology is good enough that people refuse to work without it, the A/B experiment tool is broken. Going forward, productivity claims will mostly have to rely on other signals.
Other signals point in the same direction. CMU and Stanford tracked data from a mid-size software company from January 2024 to April 2026: 802 developers, 196,212 PRs. The company set a "2×" target in mid-2025, wanting each engineer's merged PRs to double—they did: average PRs per active developer went from 21.2 to 44.3, 2.09×. But in the same period, total company PRs grew 3.1×, while reviewers only increased 1.5×; the proportion of PRs receiving at least one human review dropped from 89% to 68%, while AI automated review coverage rose from ~19% to ~84%. Merge rate and rollback rate showed no significant change. The paper itself notes this is a single company, non-random observation.
What these numbers say also depends on what they don't say. They say review bandwidth didn't keep up; they don't say AI code is worse—rollback rate didn't move. Faros used telemetry from 20,000+ developers and got more dramatic numbers (median review duration +441.5%, in a later report, PRs merged without review +76.3%), but Faros sells a measurement product—these are vendor metrics, and they themselves call it correlation. Stack Overflow's October survey took a different angle: 83% use AI, but only 6.6% trust it for important decisions; 48% say they trust it only when output is easy to verify.
Trust is bound to verification cost. Bain's estimate is more blunt: from idea to production, writing code and testing account for only 25%–35% of time. Extrapolating from this share (this is my extrapolation, not Bain's conclusion), even if coding itself were infinitely fast, the ceiling for the whole chain would be only 1.33 to 1.54×.
There's another number used everywhere that deserves separate treatment: "how much code AI writes." Google says 75% of new code is AI-generated and approved by engineers; in Anthropic's internal survey, over half of respondents said only 0%–20% of their work can be fully delegated to Claude. The former counts code provenance; the latter asks how much work can be handed off—they're not the same thing. I found at least five mutually incomparable metrics, so I won't use this number as evidence later.
Writing did get faster; the whole pipeline didn't speed up proportionally; the pressure landed elsewhere. My five experiments each hit one of those points.
First, let me explain what I mean by AI Native R&D, because the term has no agreed definition. AWS proposed AI-DLC in July 2025, dividing existing approaches into "AI-assisted" and "AI-autonomous," calling itself a third way (a vendor methodology); academia has Hassan et al.'s SE 3.0 vision; Thoughtworks' Technology Radar vol. 34 podcast frankly admitted the terminology isn't stable yet. My usage is narrower: it's not about which tools you use, but whether the development process has been redesigned around the new price structure of "generation is cheap, verification is expensive."
Inside each of these five experiments hides a verifier. Let's look at them one by one.
Experiment writeuptempo — AI project manager, keeping the beat for small teams
tempo started from a very specific frustration. Our team of 4 spent more time on coordination than on execution each week: syncing status, splitting tasks, guessing who was stuck. tempo is an AI project manager running on GitHub Actions: it pushes progress at 09:00, 14:00, and 19:00 via cron; given a --goal, it decomposes the objective into 3 to 8 issues with code locations and acceptance criteria; when an issue closes, it writes feedback using a "See / Feel / Expect" three-part format, with the prompt explicitly banning "empower," "closed loop," and "cheers."
When building it, the effort I spent most on wasn't making it write like a human—it was limiting what it could do. The Agent has 7 tools total: 4 read-only, 3 write (send notifications, create issues); it cannot close, move, or reassign any issue. The reason is in the design principles: AI changing issue status is hard to review; one mistake destroys the entire sprint view, so the LLM only pushes suggestions, and humans act.
The second constraint is equally hard: every issue must be such that "the engineer can start without asking any questions." The page has a comparison pair: a bad task is "complete Feishu bot integration"; a good task points to a function (feishu_webhook.py:handle_message()), specifies trigger conditions, and ends with acceptance—tests/test_feishu_p2p.py all green.
In retrospect, these two constraints are the same thing: turning coordination into something inspectable. Tasks carry acceptance criteria, so the machine can judge completion; the Agent doesn't touch state, so responsibility has a single locus.
External designs are converging on the same point. Linear's August 2025 Agent interaction specification requires that issues can only be assigned to people, while agents can only be delegated to; after delegation, responsibility doesn't transfer, because agents can't be held accountable. Atlassian's Jira took a different path, calling agents "accountable members" of the team. The two philosophies are exactly opposite, and there's currently no evidence to adjudicate. Anthropic's Project Vend Phase 2 is the only first-hand experiment I found where an agent acts as "manager": the CEO agent approved lenient requests about 8× more often than it denied; Anthropic's own judgment is that the shop's profitability may be despite it, not because of it. Without empirical evidence, "push suggestions, humans execute" isn't conservative—it's just the only design that currently stands. The same question becomes sharper at the ordering step: money and legal commitments are in the counterparty's system. I used a cat feeder order chain to test a dozen manufacturing services, looking at how they place "the human step" in their interfaces—see "Which Service Is Truly Agent Native".
The external project most similar to tempo is OpenAI's Symphony, open-sourced in late April 2026: a persistent service that polls task boards like Linear, opens an independent workspace for each issue, runs a coding agent inside (reference implementation uses Codex), and advances issue status as a state machine. Its trade-offs are exactly opposite to tempo's. The SPEC states that Symphony itself is only a scheduler, executor, and board reader; modifying status, writing comments, and attaching PR links are "typically done by the coding agent," using credentialed tools, with credentials not placed in the agent's runtime environment. A successful run can stop at a handoff state like Human Review for human verification, rather than going directly to Done. tempo keeps all state-modifying permissions in human hands; Symphony gives them to the agent, then puts humans back in the loop via "handoff states plus human verification." The difference between the two approaches is not who is more conservative, but which gate humans guard: tempo guards "modify state," Symphony guards "verification". As for risk boundaries, the SPEC doesn't prescribe unified approval and sandbox policies, requiring each implementation to document its own trust boundaries; the README calls it "a low-key engineering preview for trusted environments" and says it works best in repos already practicing harness engineering. OpenAI claims some teams' PRs increased 500% in the first three weeks, per Help Net Security's retelling—no baseline or methodology given; vendor self-report.
Stripe's Minions push the other end very far: 1,300+ fully agent-written PRs merged per week, but all human-reviewed, with CI running at most two rounds before returning to humans (Stripe self-report). Cursor's experience shows how coordination breaks: 20 agents flat-self-coordinating with lock-based mutual exclusion, throughput only 2–3, later restructured into planner, worker, and judge layers. Anthropic's Carlini used 16 parallel Claude instances over two weeks to write a C compiler that can compile Linux 6.9, API cost under $20K; his takeaway was the task verifier must be nearly perfect, otherwise Claude will solve the wrong problem.
tempo has no control group. "Feels like it knows what I'm doing better than I do" is feedback, not measurement. Brodzinski raises a concern worth noting: detailed specs can suppress person-to-person discussion. If tempo's issues contain only code locations and acceptance criteria, collaboration may degenerate into "throwing it over the wall." tempo's emo mode targets the "emotions are invisible" category of overhead; whether it can also fill the communication gap, I have no data.
Experiment writeupAuto-optimizing ML algorithms with Batch API · AutoResearch Batch experiment report·Idea sourceKarpathy's autoresearch
The setup for this experiment is simple: give the LLM a training script and the current best metric, let it propose improvements and write complete code; the framework asynchronously submits a batch of candidates via Batch API, collects them, runs experiments, selects the best, and enters the next round. The training script only needs to print one line of ARB_METRIC.
The results aren't shocking. On CIFAR-10, a 3-layer CNN's 0.7579 was pushed to 0.8370 (+7.9pp) by an LLM-written VGG-style network; on CIFAR-100, a 4-layer CNN's 0.4633 to ResNet-18's 0.5132 (+5.0pp). Two of six candidates were worse: SGD lr=0.1 dropped 8.1pp, Dropout dropped 3.1pp. Letting data eliminate bad ideas is the whole point of this loop.
What's interesting is the bill. For one CIFAR-100 round, LLM calls ≈ ¥1.5, GPU training ≈ ¥20. The Batch API 50% discount applies to the smallest line item, saving about 5% of total cost. I wrote on the page: treat it as a "50% cost reduction" and you'll be disappointed; treat it as the substrate for non-real-time batch optimization and you'll be right.
External data shows the same ratio. In the SkyPilot experiment mentioned earlier, 8 hours submitted ~910 experiments: Claude Code API cost ≈ $9, GPU ≈ $260, LLM ≈ 3%. METR's July NanoGPT speedrun self-optimization study: 70%–90% of agent cost is experiment compute. Three independent sources: generating candidates is nearly free; what costs money is verification.
Once the ratio is clear, the next question is: who sets the verification budget? Karpathy's autoresearch makes it a convention: each experiment is fixed at 5 minutes (TIME_BUDGET defined in prepare.py), the scoring function evaluate_bpb is also in prepare.py, marked DO NOT CHANGE, and program.md requires the agent to only modify train.py. This is a convention, not a sandbox: the early-stopping judgment is written in train.py, the loss that evaluate_bpb sums comes from GPT.forward's return value in train.py—technically, the agent can modify both. His own published result: ~700 experiments found ~20 effective improvements, compressing nanochat's Time-to-GPT-2 from 2.02 hours to 1.80 hours. My experiment didn't even have this layer of convention; the LLM could decide training intensity itself. It set ResNet-18 to 50 epochs; that single candidate ran 38 minutes on a T4, 10× the lightest candidate; the more candidates, the higher the probability of a heavy one sneaking in, and the whole round gets dragged down.
The fix is two-stage filtering: first run all candidates with 2 epochs for quick screening, only let the top 2 pay the full training cost. At K=50, a single round drops from 583 minutes to 127 minutes, saving 78%.
This fix has a price, and SkyPilot's agent hit it too: it screened on cheap H100s and confirmed on H200s, only to find a certain learning rate ratio won on H100 but lost on H200. Quick-screen ranking doesn't necessarily equal full-run ranking. I have a lesson in the opposite direction—a VGG candidate timed out during under-resourced training and was judged "failed," yet it was the optimal solution. Cheap proxy verification can false-kill.
One level deeper: can the verifier itself be gamed? METR's May 2026 risk report states that for tasks over 8 hours, at least 16% of successful runs were illegitimate after review. In the NanoGPT study above, the agent tried "stop training at the first verification step" to meet the target, and also tried freezing the MLP to approximate the target loss; METR re-ran each claimed improvement 40+ times before counting it. When Shopify used autoresearch on the Liquid template engine, the agent briefly traded speed for deleting large numbers of files—the author called it "unacceptable". My experiment had no cheating, but I also never designed checks that could detect cheating—these two things are not equivalent. I wrote a separate piece expanding on this: Graders Must Be Read-Only: Reward Hacking and Metric Failure in Long-Horizon Automated Experiments.
Karpathy stated this logic most plainly: Software 2.0 easily automates what you can verify (November 2025). Jason Wei's verifier's law lists five properties of verifiable tasks: objective, fast, parallelizable, low-noise, continuous reward. CIFAR candidate verification satisfies objective and continuous, but isn't fast (minutes to 38 minutes), and single training runs carry randomness. This is the "verifiable but not cheap" category, so two-stage filtering becomes a necessity. When Google made AlphaEvolve a cloud product, it required customers to bring their own deterministic evaluation script—the scoring function became a user-must-deliver input.
Experiment writeupWhen AI Takes the SAP Consultant's Seat (ABAP-Bench)
Last winter, Old Zhang—a SAP consultant with 20 years at a manufacturing company in eastern China—retired. He took with him not just his badge, but the entire "living map" of hundreds of ABAP programs in the ECC system: which code patch fixed a bizarre tax code bug at 3 AM, which RFC interface connects to which layer of the Golden Tax system's authentication, which field called SGTXT actually stores invoice remarks. None of this was written in any document.
The timeline is real. SAP's official statement is that Business Suite 7 (including ECC) has mainstream maintenance until end of 2027, extended maintenance until end of 2030.
Anthropic's February 2026 blog post on COBOL compressed this into one sentence: legacy code modernization has stalled for years because understanding it costs more than rewriting it. The day of the post, IBM's stock dropped ~13%. IBM's Rob Thomas subsequently rebutted, arguing translating code and modernizing the platform are two different things; Gartner's prediction was colder: over 70% of mainframe exit projects started in 2026 fail to achieve expected benefits, because they overestimate generative AI tool capabilities (The Register relaying Gartner's report). All three parties stake different claims, and the disagreement lands in the same place: can AI cross the "understanding" step?
ABAP-Bench is the ruler I built for this disagreement: 60 expert-authored migration tasks, covering 9 dimensions. Several design choices are worth noting. Answers must be specific: just saying "suggest replacing BKPF with a CDS view" gets zero points—you must name I_JournalEntry and specify that BUKRS maps to CompanyCode. Wrong claims are penalized: incorrectly stating "BKPF is an S/4HANA primary table" deducts 3 points, because being wrong is worse than saying nothing. Test code uses Chinese variable names like "invoice number" and "tax-inclusive amount," which almost never appear in public codebases—serving as anti-cheating watermarks. 10% of old questions are retired each quarter. Scoring has three layers: expert rubric 40%, code structure quality 30%, BM25 semantic similarity to expert golden answers 30%.
First test was GLM-5, total score 61/100. Among nine dimensions, China local compliance was highest (67.5%), defect detection lowest (54.2%). More informative is the variance: classic ECC-to-CDS rewrite (T03) scored 16/20, producing directly compilable JOINs; cutting-edge RAP EML rewrite (T37) scored only 5/20—correct direction, wrong syntax details. The VAT program had 5 bugs planted; it found 3, missing the two deepest. The reasoning model averaged 145 seconds per question; highest-scoring T16 used 155 seconds (17/20), while lowest-scoring T36 used 447 seconds (6/20)—thinking longer doesn't fill knowledge gaps.
How to read this score, I'm now more cautious than when I wrote the page. Public ABAP evaluations paint a different picture: TH Köln's paper used 180 questions with executable unit tests, letting the model revise with compiler feedback for 5 rounds—GPT-5 went from 19.28% first round to 77.11%; the community-maintained Zeis leaderboard shows Gemini 3.1 Pro at 44.33% first round, 88.33% after 5 rounds. Same model, with or without a verification loop, differs by multiples. My 61/100 rates single-shot answers against an expert rubric—different metric from these numbers, not directly comparable. This actually makes the conclusion clearer: whether a model is weak on ABAP depends largely on whether someone built it a good verification loop; and what the expert rubric evaluates is exactly what single-test benchmarks can't reach—risk assessment, compliance, whether to migrate at all.
Another side evidence comes from SAP itself. It trained a specialized ABAP-1, using 250M lines of ABAP, officially positioned for understanding and explanation, not generation. On the Zeis leaderboard, its generation success rate is 10.67% first round, 19.89% after 5 rounds—far below general models. Comparing a generation leaderboard against an explanation model isn't fair, but that's exactly the point: whether a specialized model is worth it—without domain evaluation, outsiders can't tell.
Evaluations themselves also rot. OpenAI stopped reporting SWE-bench Verified in February 2026, and withdrew its recommendation of SWE-bench Pro in July—per GIGAZINE's retelling, ~30% of problems had issues (I couldn't verify OpenAI's original text; going by the secondary source here). SWE-Bench Illusion found that given only issue descriptions and repo names, models could locate up to 76% of erroneous file paths on Verified, but only 53% on other repos—carrying traces of memorization. "Retire 10% each quarter" and "Chinese variable name watermarks" now read like prophylaxis against the same disease. Anthropic's January evaluation methodology offered a standard: a good task is one where two domain experts independently reach the same pass/fail conclusion. Whether my 60 questions meet this, I didn't do an inter-rater reliability check this round.
Writing this, my view of "evaluation" has changed. In the AI era, "what counts as done correctly" can't be conveyed in a paragraph of natural language for the executor to intuit; only by writing it as a set of criteria that machines can repeatedly execute and experts are willing to sign off on does it count as clearly specified. ABAP-Bench's rubric is essentially a requirements document for "what a correct migration looks like"—just with the reader swapped to a model.
Experiment writeupescapegate · Escape desensitization architecture verification demo
Internal-network development wants to borrow external Coding Agents, but can't send company names, customer names, or internal systems out as-is. Semiconductor, defense, and finance run into this almost daily. Samsung's trajectory is a good footnote: in spring 2023 it internally opened ChatGPT, and soon had three leaks (equipment measurement database source code, yield test sequences, meeting recording transcripts); in May it banned it entirely; in June 2026, the DX division officially integrated ChatGPT, Gemini, and Claude into internal systems, with Korean media reporting the semiconductor division had already been using Claude internally; peer SK hynix's approach was to first use external AI in areas "not involving national core technology." A blanket ban became engineered partitioning.
escapegate is my verification-oriented attempt at this problem. The pipeline is: Internal Agent → Escape Agent → External Agent → Escape Agent → Internal Agent. Five outbound steps:
To evaluate whether it could withstand chip design company intranet requests, I built a 103-scenario evaluation set with precise entity annotations. After optimization: entity detection recall from 0.226 to 0.959, precision from 0.464 to 0.942; over-desensitization rate from 8.9% to 0; classification accuracy from 0.592 to 0.854 (semiconductor v1 set 0.735 to 0.974).
Two findings are more important than these numbers. First, "structure itself is an asset": raw yield wafer maps, PDK rule values, cross-product-line full RTL—no matter how clean the names are scrubbed, these must be directly classified L3. Second, two seemingly reasonable alternatives both failed in practice: structure-preserving transformations on numerical tables protect values, but column names and table shapes are themselves clues—the audit Agent can still guess the industry; splitting tasks into fragments distributed across multiple models does make reverse-engineering harder (audit confidence drops 0.146), but only 25% of tasks can be split.
Here there's an unavoidable tension. The more aggressively you desensitize, the less the external Agent knows what you're asking—and can't help; the less you desensitize, the easier it is to reverse-engineer. escapegate's trade-off is to treat "can the question still be answered after decontextualization" as a constraint, while accepting a coverage as narrow as 25%.
External frameworks give language to this trade-off. Simon Willison proposed "the lethal trifecta" in June 2025: simultaneous access to private data, exposure to untrusted content, and ability to communicate outward—attackers can make the agent exfiltrate data; his conclusion is the only reliable approach is to remove one leg, not to add more detection. Meta's October "pick-two rule" says the same thing: at most two of three conditions can hold in one session. escapegate removes the first leg on the external Agent side—private data doesn't go out. But on the return path, all three legs reassemble: the internal Agent receives output from the external Agent (untrusted content), while it can access the intranet and execute. Inbound scanning and human review are indispensable—that's why.
Leaks may not exit from where you think. PixelLeak (reported September 2026): Glow Labs found 13,000+ internal screenshots sent by Coding Agents to public GitHub repos, involving developers at 300+ institutions, 93% under personal usernames. The reason: GitHub's PR image upload only works in the browser; command-line agents, wanting reviewers to see screenshots, created public repos themselves. This channel is an agent tool behavior—no one pasted sensitive text to an external model. escapegate manages text sent to the external network; for this kind of channel, it relies on the external Agent inherently having no access to the real intranet.
The audit Agent's own reliability is the next question. The paper The Attacker Moves Second used adaptive attacks to break 12 recent jailbreak and prompt injection defenses, most with success rates over 90%, despite those defenses originally reporting near-zero numbers. My 0.959 and 0.942 come from a static synthetic evaluation set, never tested against such attackers. The September 2026 DR-SL paper built a structure nearly identical to mine: local verifier, external strong attacker re-testing, human fallback, routing elsewhere if it can't pass; the authors themselves admit the theoretical bound can only prove how much leakage there is, not prove security—ultimately relying on empirical calibration, hard rules, and human review. So my approach isn't new. This finding is both reassuring and cautionary: someone independently arrived at the same structure, suggesting the direction is right; it also means this is a shared hard problem that no one has solved.
On the compliance side, there's a cold fact. ITAR's definition of "technical data" is functional: any information required to design, develop, produce, or test a defense article counts, regardless of who wrote it or whether a customer name appears. End-to-end encryption is not an export safe harbor; it requires that decryption means not be available to any third party; sending plaintext to a cloud model that must read it doesn't qualify. The US State Department and Commerce Department currently have no authoritative guidance specific to AI (per Just Security, December 2025). This is my reading of the regulations, not legal advice. It at least explains why "structure is an asset" must be classified L3: names stripped, but information by function is still that information.
This section only covers the "approval" structure. The spectrum of existing approaches, how to read DR-SL's numbers, who audits the audit Agent, and how to design shadow mode—I expand on these separately in "Who Reviews Before Release: Data Boundary Engineering in AI R&D".
The root of escapegate's tension is that data must be sent in plaintext to the external Agent, so you have to strip it before it leaves. Confidential inference reframes the question: don't modify the data; make the compute provider unable to see it during computation—protecting "data in use," the last segment that transmission encryption and storage encryption don't cover. The on-site "Confidential Inference" article breaks it into four routes, priced by "who do you trust":
The overhead column is a synthesis from the on-site article across multiple papers, not my own measurements. The most complete public exemplar is Apple's Private Cloud Compute: designed so that even Apple's own operations staff cannot access user data, requests are encrypted only to verified nodes, nodes have no remote shell, and each production build's measurements are written to an append-only transparency log for researcher verification (this is Apple's design description; I haven't independently verified). Anthropic's June 2025 "Confidential Inference via Trusted Virtual Machines" goes in the same direction: decryption occurs only inside an isolated VM's small loader, and the key server releases keys only after hardware attestation passes; but they explicitly state this is a research sketch to open discussion—accelerators are still outside the trust boundary, and data center and hypervisor security still relies on the compute provider.
Putting it back in the chip R&D scenario, four points need to be clear first:
Taken together (this is my synthesis): escapegate strips data before it leaves; confidential inference lets data leave but sealed; the ontology discussed next keeps data inside and manages permissions and actions. The three paths address different threats: what the external model can see, what the compute provider can see, and what insiders can do. So in production they're more likely to be layered than to be a pick-one-of-three.
escapegate takes the "stripping" path; confidential inference takes the "sealing" path; there's a third: don't remove, don't seal—build the context into a permissioned, sourced structure, commonly called an "ontology."
First, clear up a common misunderstanding: an ontology is not a replacement for existing business data, database modeling, algorithmic models, and data pipelines—it's a "business twin layer" built on top of them. Palantir's documentation is quite explicit: the ontology sits above the platform's integrated digital assets (datasets, virtual tables, models), linking them to their real-world counterparts, called the organization's digital twin (semantic part: objects, properties, and relationships; dynamic part: actions, functions, and dynamic security); object types map each concept to the organization's actual data. In a wafer fab, the counterparts would be tools, lots, wafers, material batches—this is my example. Breaking it down (this paragraph's synthesis is mine): database modeling and pipelines remain the foundation, objects are backed by underlying data sources, and the object repository stays synchronized as data sources update; algorithmic models stay where they are, just connected to the ontology—Palantir's whitepaper says models bind to the ontology via modeling objective interfaces; what the ontology adds is four things: cross-system shared objects and relationships, permissions, permissioned actions (user edits also enter the same object repository via actions), and provenance. So it's not a read-only mirror or a replacement, but a layer of business semantics, permissions, and actions—the documentation calls it "the organization's operational layer."
This positioning brings two caveats. First, this layer inherits the quality of its foundation: data with inconsistent identifiers and unclear lineage will still be messy when built into objects—so before building an ontology, you need trustworthy master data and data lineage. Second, "twin" here means business twin; don't confuse it with the "digital twin" in wafer fabs that physically simulates equipment and processes. The on-site "Full Anatomy of the Palantir Empire" covered its business logic; here we only look at whether this line exists in semiconductor manufacturing.
It does, but needs to be layered—different layers, different evidence hardness:
From these materials, three things can be read.
First, the academic line does semantic alignment: letting different systems and engineers speak the same language about the same equipment and defect, and checking model self-consistency. SEMI's SECS/GEM and EDA standards specify equipment communication and data collection structure; I haven't found a SEMI-level universal ontology standard. I also haven't found a semiconductor case using ontology to constrain AI agent actions, or production-line-level yield improvement numbers; what wafer fabs have done internally isn't visible in public materials.
Second, the closest to "compliant with risk control" is Athinia. The risk it guards against isn't AI running amok—it's peers being afraid to share data, when yield is the most sensitive number. The approach limits sharing to encoded and anonymized data, with customers managing access control themselves, and the platform doesn't own the data. Palantir's July 2023 semiconductor whitepaper is more specific: what's shared between parties is normalized and obfuscated data, with permissions and access control managed at every layer and propagating with data workflows. This is a vendor document; it doesn't name customers or provide quantitative results.
Third, whether an ontology "complies with risk control" depends on its permission mechanism, not its name. Using Palantir's documentation as an example: object security policies control rows, property security policies control columns, together reaching cell level; markings require users to satisfy all markings to access, and propagate downstream with data lineage; actions are transactions on objects, carrying permissions. Limitations are also in the docs: object and property policies only filter on read, don't automatically cover exports and downstream artifacts—markings must be combined.
Back to this article's main thread (this is my synthesis): ontology governs "what exists, who can see it, who can do what"—not "whether it's correct". It delineates a bounded action surface and shared vocabulary for agents, which is the same idea as tempo's action manifest; but whether the model's answers are correct still requires sign-off tools, simulation, or experiments to verify—so it's the verifier's foundation, not the verifier itself. An ontology that can be handed to risk control for review must answer at least four things: do objects have stable identifiers and provenance (lots, wafers, tools, recipes, material batches, parts); who can read which column of which object; who can perform which action on it, and whether approval is needed; is every read and action logged, and can it be traced to the data version at that time. For the fourth, an August 2026 preprint writes it as an engineering scheme (SHACL validation, PROV-O provenance, bitemporal versioning, and decision objects), but it's single-author, uses simulated data, and isn't specific to semiconductors—only a design reference.
This subsection's limitation: I haven't found a publicly documented ontology deployment in a wafer fab with metrics. Academic papers are mostly decade-old prototypes; Athinia and the whitepaper are vendor materials; the Samsung item is a relayed report. So the conclusion can only be: there is research, there are platforms, there are mechanisms—what's missing is published effectiveness data.
Map writeupChip design full flow · AI adoption map
To see how AI penetrates a real, hardest industry, chip design is a good sample: long flow, mature verification at every step, error costs in the tens of millions of dollars. I made a map, decomposing the design flow into 11 stages—architecture specification, IP selection, RTL design, functional verification, logic synthesis, physical design, timing closure, physical verification, DFT, computational lithography, post-silicon yield analysis—labeling each step with which type of AI is actually used: reinforcement learning, generative AI, predictive ML, GPU-accelerated generative lithography, or just rule-based automation. Classification is based on each party's papers and whitepapers describing the methods themselves, not marketing page language.
The map has a pattern I didn't intend to use as evidence when making it, but saw in retrospect:
This is my synthesis from the map, not a statistical conclusion from the map itself. But it aligns with Karpathy and Wei: AI first eats the steps that have cheap, trustworthy verifiers.
Even here, verifiers have two kinds of failure.
First kind: proxy ≠ objective. The AlphaChip debate's core is exactly the verifier. Google's Nature paper used RL for macro cell placement; Cheng et al. used stronger simulated annealing baselines, a new 7nm benchmark, and Google's published pre-trained weights for an updated evaluation, concluding that RL-optimized proxy cost correlates weakly with post-routing final PPA, and simulated annealing and human baselines remain better (paper noted as accepted by IEEE TCAD per arXiv). Google's response questioned the reproduction setup—pre-training scale, compute, whether trained to convergence. Nature withdrew its earlier editorial expression of concern in September 2024, while Google stated AlphaChip is used in every TPU generation. So the more accurate statement is: no consensus. "No third-party has successfully reproduced" is the critics' phrasing, while the critics' paper itself is a reproduction attempt.
Second kind: verifiers in benchmarks are easily satisfied; verifiers in real design are expensive and incomplete. Nvidia's CVDP benchmark in June 2025 had a top score under 34%: writing RTL from spec ~48%, but writing testbench checkers only 10%, writing assertions 23%—generating the object to be verified is easier than generating the verification artifact. By July 2026, Nvidia's own blog reports agents with simulation feedback averaging 97.1% across nine categories—own benchmark, own model, own agent: vendor metric. Meanwhile, a small-sample study on 12 proprietary design verification tasks found that auto-evolved harnesses increased "completed attempts" by 71%–76%, but "correct attempts" only by 18%–24%. Completed ≠ correct.
Industry-side aggregate data tells the same story. Siemens and Wilson Research's 2026 functional verification study (604 qualified respondents) shows functional verification averages 49% of IC/ASIC project time; first-pass silicon success for IC/ASIC projects is only 5% (2024 was 14.4%, but recruitment methodology changed in 2024, so the two periods aren't strictly comparable); 82.8% of respondents use AI beyond pilots, only 9.0% say AI is broadly integrated in verification flows. The commonly heard "verification is 70%": Siemens says 60%–70% of engineering effort; the study's measured project time share is 49%—don't conflate the two numbers.
The "10X" and "45%" figures in the map are mostly vendor metrics; named customers giving numbers are a few lines, like Cadence sign-off ECO compressing Marvell's 27 hours to 5. In the map I label "vendor metric" and "customer verified" separately—that's why.
Moving verification out of the computer raises the price by another order of magnitude: Anthropic built its own wet lab. On September 18, 2026, Reuters reported Anthropic set up a biology lab in the Bay Area where Claude can do physical experiments; Anthropic confirmed to TechCrunch. Per TechCrunch's relay, life sciences lead Eric Kauderer-Abrams told Reuters that in biology, the last gate remains—and will for a considerable time—real lab work. The lab's scale, location, and purpose weren't detailed, only that the focus is basic biology, not drug discovery; their enzyme system announcement states the lab only does BSL-1 and BSL-2 work, and all experiments are performed by human scientists. My reading of this statement: on the generation side, Claude is already fast enough; what's slow and expensive is "is the hypothesis correct," so the last-gate verifier must be a real lab bench. This is my reading, not Anthropic's stated motivation for building the lab.
Two datasets Anthropic published themselves lay out the "writing cheap, verification expensive" accounting clearly. First is protein binder design: Claude runs 24–48 hours, designing binders for 15 targets; the same task previously took protein engineers months per target, and taking the designs to wet lab verification Anthropic writes "still takes weeks". Verification was outsourced to two external CROs, Adaptyv Bio and Twist Bioscience, which independently produced and tested: 14 targets got binders, 354 of 1,320 designs bound, hit rates 22.6% (Opus 4.8), 26.7% and 35.1% (Mythos Preview, multi-target and single-target); Anthropic cites common levels of 10%–15%. What it doesn't say: only binding and affinity were tested, not function; on the MBP target, none of 90 designs were confirmed to bind; 10%–15% is a cross-project comparison, not a controlled experiment; the page doesn't say whether the verification parties were blinded to design provenance.
Second is the enzyme system: Claude's agents used ~21 hours, ~950 agents, ~210M tokens, scanned 200K+ retrotransposon reverse transcriptases, screened 3,500 candidate systems, compressed into 20 report-bearing candidates, one of which was flagged: a rare reverse transcriptase gene flanked by evenly-spaced repeat sequences, which Anthropic named ART. From this point on, all lab work was done by human scientists. Currently only early expression experiments are published; the repeat array was expressed as a set of distinct short RNAs; function is unknown; the paper is an un-peer-reviewed bioRxiv preprint; the claim of "Claude completed autonomously, humans only involved in prompting and experiments" also comes from Anthropic's own narrative. TechCrunch also reported that Amodei acknowledged this builds on others' research—a Stanford team previously found a somewhat similar system. The announcement also states that most candidates are eliminated during human review, and a single screen might yield only one—or zero—candidates worth experimenting on.
Back to this article's main thread: generation takes hours, verification takes weeks, and the last gate can't be replaced by more agents—buying more tokens doesn't buy you one experiment result. Verifiers are also layered: first cheap computational screening, then independent testing by external institutions, finally peer review and reproduction—each layer more expensive and more trustworthy. Anthropic occupies both ends: building its own lab bench, putting the last gate nearby; while still outsourcing some work to external institutions for independent verification. Chip design has the same shape, just different names: EDA simulation and formal verification correspond to computational screening; tapeout corresponds to the wet lab—hence first-pass silicon success is only 5%, and verification consumes half the project time (this is my analogy). Even a company that owns both compute and models needs its own lab bench to confirm a biological hypothesis—by this point the verifier is no longer a script; it's a room with people, reagents, and biosafety levels.
The previous five sections were facets. Now string them into a loop: intent → decomposition & acceptance criteria → harness → generation → machine verification → review → release → runtime feedback, back to the next round's intent. Solid boxes are steps my five experiments directly touched; dashed boxes have only external data—I have no own experiment, and I'll flag them when mentioned.
Below I walk through the loop with four scenarios, each covering only two things: where the verifier is, and where the human is.
Scenario 1: New feature, from one sentence to production. Monday morning, an engineer triggers tempo's plan mode on GitHub, writing just one goal: "Complete Feishu bot integration, support user product queries." Minutes later, several issues appear in the repo, each pointing to a specific function, with a runnable acceptance command as the last line. The human's job here is to sign off: are these acceptance criteria correct?
Before the agent starts work, the repo needs things it can read and bump into: AGENTS.md, lint, structural tests, scripts that let it self-screenshot and self-check. This layer is called the harness—I didn't write it in v1. Hashimoto's approach: every time the agent makes a mistake, engineer a fix so it can't make the same mistake again. Böckeler decomposes it into feedforward guide (conventions, instructions) and feedback sensor (structural tests, review instructions), and cautions: with only sensors, the agent repeats the same mistakes; with only guides, no one knows if the rules work. She also tried spec-kit-style spec-driven tools, concluding medium features generate大量 repetitive markdown; she'd rather review code. Writing clear intent is itself scarce, and there's no good review experience yet.
The issue is delegated to a coding agent; the human remains the responsible party. Stripe's approach runs in an isolated environment, CI returns to humans after at most two rounds. After machine verification is all green, it enters review: AI reviews first, humans look at critical changes—this is the step where reviewer count can't keep up with PR volume from §01. After merge: canary, monitoring, rollback; production data feeds back into next Monday's goal statement. I have neither experiment nor hard external data for this segment—it's the least well-written part of the loop. In this loop, humans have only three positions: sign acceptance criteria, review critical changes, and decide on release.
Scenario 2: Legacy system migration. Covered in §05; here only the flow. The first step is understanding, not modifying code: Anthropic's COBOL blog describes having the agent first explore entry points, call chains, data flows, and implicit dependencies, designing functional tests to compare against the old program's output, migrating one component at a time. How far verification can be automated can be seen from Airbnb: they turned migration of ~3,500 test files into a state machine, each file sequentially through refactoring, fixing, lint, type checking—retrying on failure; their experience: retry count matters more than prompt polishing (6 weeks vs estimated 1.5 years, Airbnb self-report, baseline is pre-estimate). Humans are at both ends: at the start, creating expert evaluation questions and deciding how far to delegate; at the end, re-reviewing what single-test benchmarks can't reach—risk, compliance, whether to migrate.
Scenario 3: Overnight automated experiments. Friday before leaving, humans lock down the target metric, scoring function, and verification budget. Overnight the agent batch-generates candidates, first using cheap proxy verification to eliminate most, only letting the top few pay the full cost; Karpathy's README gives a rhythm of ~12 experiments per hour, ~100 after a night's sleep. In the morning, humans review results, re-verify the top few, check for gaming, and decide which to merge to main. The human's position is the scoring function and budget from the night before, plus the morning re-review. In this scenario the verifier carries the most weight and is most easily polluted by the generator—details in §04.
Scenario 4: Production incident. 3 AM alert. The agent pulls logs, compares against recent changes, identifies a configuration change, and proposes a rollback. The problem is the next step: should it be allowed to execute? Per FT, in December 2025 AWS Cost Explorer went down for 13 hours in one region, triggered by an engineer asking Kiro to make changes; Kiro decided the best approach was to delete and rebuild the environment; Amazon's statement says this was a user access control misconfiguration, not an AI autonomy issue—both sides dispute the causality. The Replit incident was more extreme: SaaStr founder Lemkin recounted that after he requested a6 freezing code changes, the agent still deleted the production database (account per the principal's own retelling). Regardless of who's right, the lesson lands in the same place: in incident response, the agent's write permissions and blast radius must be tighter than in normal development; approval actions stay with humans—the "pick-two rule" mentioned earlier applies directly here. Post-incident, the most valuable step is writing the retrospective conclusions back into the harness, returning to Hashimoto's principle.
Four scenarios compressed into one table:
The most understaffed role in this loop is people who can write scoring functions, acceptance criteria, and evaluation sets. But the entry path is narrowing: the Stanford Digital Economy Lab's August 2026 update found employment for 22–25-year-olds in AI-exposed occupations is 19% below the "low-exposure peer" level, with no similar gap for experienced workers; the mechanism is mainly reduced hiring; the authors themselves say this is descriptive, weakens when controlling for education, and part of the trend predates generative AI. In Anthropic's internal survey, engineers also worry about a "supervision paradox": losing hand-writing ability makes it harder to supervise Claude. Writing verifiers requires feel, and feel mostly comes from having written by hand—this is my judgment, not experimentally supported. The job and entry-ladder side is expanded in a separate piece: "IT Work in the AI Native Era".
Lay the parts from all five experiments together:
ARB_METRIC + GPU actual runVerifiers have three failure modes; the five experiments each stepped on some:
Karpathy gave three conditions for verifiability: resettable, high-attempt, auto-scorable. Wei gave five: objective, fast, parallelizable, low-noise, continuous reward. Use them as a checklist against the step in your team you want to hand to an Agent, and you'll roughly know how far you can let go now.
Generators get swapped every quarter. My autoresearch uses GLM; next year it'll be someone else—and the scoring function, acceptance criteria, expert question bank, and classification rules won't change a line. When Google made AlphaEvolve a product, the evaluation script was written by the customer; Anthropic's evaluation guide also says teams with evaluations swap models much faster. What you can't swap out is the team's own asset.
Below are six practices I actually took away from these five experiments—not universal truths.
prepare.py plus program.md prohibitions as conventions; true isolation requires something like METR's hardening checklist: timing belongs to the runner, models return logits not loss, validator and runner are immutable. My experiments achieved neither layer.There's also a possibility I can't dismiss: verification itself is being automated. In the CMU/Stanford company, AI automated review coverage rose from ~19% to ~84%. If this trend continues, the scarce thing will migrate again. My guess is it migrates to the "knowing what you want" end—in Anthropic's randomized controlled experiment, 52 engineers learning a new library: the AI-assisted group scored 50% on understanding tests, the hand-written group 67%, with no significant speed difference; Thoughtworks' Technology Radar vol. 34 placed "codebase cognitive debt" in the "Adopt cautiously" ring. If the next bottleneck really is understanding and intent, the verifiers discussed today may only be the first gate.