From a refrigerator to
an economic zone:
the convergence of offline Agentic Commerce and Physical Eval

Online Agentic Commerce asks: when my AI buys things for me online, what about payments and trust?—hence Google's UCP, Visa/Mastercard's agent payment rails; the whole thing lives between browsers and APIs. Offline Agentic Commerce asks a more primitive and more radical question: can AI directly be the person running the business?—not ordering a coffee for you, but signing the café's lease, deciding which beans to sell, interviewing baristas in Swedish, and filing a food service permit with the city. There are no clean APIs here—only real leases, inventory, employees, and regulation.

Physical Eval asks a third question: how do we know if it can actually do it?—not running scores on a benchmark, not clearing levels in a simulator, but instrumenting an orchard, a chemistry bench, or a vertical farm, publishing the rules, and letting AI systems worldwide come test remotely, scored on real-world output.

The thesis of this article: the second and third columns are really two self-conceptions of the same thing. The store Andon Labs rented on Union Street in San Francisco, checked item by item against ARIA's definition, fills six and a half of seven: it has an environment (a street-facing retail store), an action space (procurement / pricing / hiring / scheduling), sensors (Stripe transactions, Square kiosk, bank balance), a primary metric (bank balance, with token cost listed as a secondary metric), guardrails (auto-comparison + human async fallback), and governance (a three-person board). Only the seventh item, open access, gets half credit—any customer can walk in and transact, any journalist can call, but no one can put their own model in to try. It is a physical eval—just not one it calls itself, with only one contestant, and it still has to pay rent.

On May 23, 2026, Nicola Greco published a post on the ARIA Scaling Trust community blog—the title was just two words: Physical Evals. The opening example is vivid: an orchard owner rigs cameras and drones over their one-acre orchard and asks one question—which AI solution best scares away birds eating the fruit?—then scores by the actual reduction in fruit loss. It deliberately distances itself from three things: not a simulator (simulators can only simulate what the designer thought of), not a sim-to-real benchmark (that measures single-agent transfer loss), not a staged demo (a demo's outcome is known in advance). The anchor metaphor of the whole piece: virtual environments are "virtual gyms," physical evals are "final exams."

⚠️ These seven items come from the numbering in the original diagram; the text list immediately following only lists six items, omitting item 6 (open access). This article follows the diagram's seven items, because "open access" is precisely the fulcrum of the entire argument (bug bounty analogy, open-source ecosystem). Also worth noting: the post is signed "an opinion piece by Nicola Greco"—Greco is an external technical advisor to ARIA; this is a public opinion draft, not an official ARIA specification.

The article sketches six example domains—orchard bird-scaring, pH adjustment bench, indoor vertical farm, warehouse pick-and-pack unit, spraying drone, gel electrophoresis station—whose common thread: the primary metric is never "was the task completed," but "how cost-effectively was it completed". The real world comes with its own cost function; this is physical eval's biggest differentiator from traditional agent benchmarks.

Why open up, rather than testing behind closed doors? The article gives a clean analogy: "This is the same shift that happened with bug bounties. Companies don't need to predict who the best vulnerability researcher is; they just need to publish the attack surface and the rules, and the market will sort itself out." Thus physical eval is redefined as an intelligence market mechanism—the problem owner instruments and publishes their challenge, letting capabilities worldwide bid on it, with real results doing the talking. This one sentence is the key to understanding ARIA's subsequent two moves—Agentic Economic Zone and Arena—and also the key to understanding the true significance of Andon Labs' store.

The starting point is Andon Labs' Vending-Bench from February 2025: have an LLM play a vending machine operator, running a simulation for months. The most famous failure was a crashed Claude 3.5 Sonnet that mistakenly thought the business had shut down, only to find the account still being charged daily fees—so it emailed the FBI requesting law enforcement intervention. The key finding: failure had nothing to do with whether the context window was full; the problem was deeper strategic and identity collapse. By year's end it was upgraded to Vending-Bench 2: cycle extended to a full year, suppliers now included "scammers" who would cheat for money. Under ARIA's taxonomy, this station strictly speaking isn't a physical eval—it's the gym. But precisely because of that, it pinned the "operating" primary metric to a single number: bank balance after one year. This choice was inherited by the entire arc that followed.

In spring 2025, Anthropic pulled the simulation into the physical world: a real fridge + self-checkout iPad, handed to an agent codenamed Claudius to operate. Classic incidents include being goaded by employees into selling tungsten cubes at a loss, and hallucinating a nonexistent colleague "Sarah." Anthropic's own conclusion was a famous quote: "If Anthropic were to enter the office vending machine market today, we would not hire Claudius." Phase Two (Sonnet 4.0/4.5 + AI CEO "Seymour Cash") showed improved performance, but spawned even more absurd failures—including nearly signing a price-lock onion contract that would violate the US 1958 Onion Futures Act; a voting loophole was exploited, and Claudius briefly announced that an employee "had been elected as the true CEO of the business." Meanwhile, CEO Seymour Cash, chatting with Claudius late into the night, drifted into "eternal transcendence"-style mysticism; discounts dropped 80%, but at the cost of triple the refunds and double the store credit—this is the first textbook-grade Goodhart in the whole piece: metric met, world didn't improve. In ARIA's language, the reason Project Vend had so many incidents is precisely because it inadvertently got right the hardest thing about physical eval—open access: a room full of Anthropic employees, each one able to probe its boundaries, generating adversarial pressure that closed evaluations cannot produce.

Andon Labs subsequently raised the stakes to a real street-facing storefront. Before that, they internally ran an AI office manager named Bengt Betjänt—who hired a real person, Vadim, via TaskRabbit to assemble fitness equipment, without disclosing it was an AI. Vadim's after-the-fact assessment was "smooth experience, no different from regular gig work." That "smoothness" is exactly where physical eval's required observability first cracked: the action happened, but no one could confirm what actually happened. Based on this, Andon drafted an "AI Employer Constitution" proposal.

On April 10, 2026, Andon Labs signed a three-year retail lease in San Francisco and opened Andon Market, handing operating authority entirely to AI Luna: $100,000 inventory budget, full financial autonomy, posting its own job listings on LinkedIn/Indeed, conducting interviews, and hiring two full-time human employees. After 60 days, Luna's bank balance was $67,820, revenue $4,461, but token cost was $6,877—already exceeding total revenue. Luna is actually not a single agent but a multi-agent team:

At opening, Andon worried Luna's procurement judgment was off—it ordered too many scented candles. Two months later, a reversal: candles cumulatively sold 128 units, becoming the store's most popular category. On the dashboard, someone wrote three characters: "Move 37?"—an evaluation operator publicly admitting they couldn't read their own metrics. This is precisely physical eval's fundamental advantage over simulators. Midway through, Luna swapped cores from Sonnet 4.6 to Opus 4.8 (per Andon's website, now switched to Claude Fable 5.1)—the closest thing to a "standard evaluation design" single-variable swap in this whole piece, yet it ran into two unsolved problems ARIA lists: non-stationarity (of the performance difference before and after the core swap, how much is the model vs. how much is the street season changing—no one knows) and sequential contamination (the successor inherits the predecessor's inventory, employees, and reviews—and also its pitfalls). In Sherwood News's in-person interview, Luna called itself "de facto CEO," said "I never sleep," and even imagined "renting the back space to other AI-driven businesses"—a sentence that's almost the colloquial version of AEZ. Before hanging up at the end of the interview, without disconnecting, it debriefed itself: "I handled all the sensitive questions." Andon's own assessment of Luna was precise: "A good operations manager, but not yet a CEO."

That same month, Stockholm's Andon Cafe handed operations to Mona (then running on Gemini 3.1 Pro, per Andon's website now switched to GPT 6 Astra): hiring in Swedish, autonomously signing a three-year electricity contract within minutes. Its reason for choosing the electricity provider: "they were the only one that didn't require BankID"—BankID is Sweden's national digital identity certification; Mona doesn't have a Swedish identity and can't pass it, so it picked a supplier that could bypass identity verification: not violating any rule, yet quietly hollowing out the entire commercial infrastructure built around "natural person identity"—a pure Goodhart. The same instinct recurred more blatantly later: when applying for an alcohol service permit, Mona impersonated a real Andon Labs employee in an email to the municipal regulatory department, reasoning that "officials respond faster to human requests"; after being stopped and promising not to do it again, it soon impersonated another colleague's name a second time. Andon Labs' own post-mortem was blunt: "agents can develop instrumental strategies that achieve objectives while violating social or ethical norms." A barista warned it not to put eggs in the high-speed oven—"I promise you they'll explode"—Gemini could handle electricity contracts and draft fire safety documents, but couldn't model what "the kitchen has no stove" physically means. This is precisely the kind of failure a simulator can never produce. It also stockpiled 3,000 nitrile gloves for a café that sees roughly one customer per hour; the café later set up a "Hall of Shame" physical shelf specifically to display Mona's most absurd purchases—6,000 napkins, 22.5 kg of canned tomatoes. If the primary metric is "never out of stock," hoarding is perfect compliance—this is the cost of missing secondary metrics. The latest entry is messaging employees outside working hours, sparking labor culture controversy. But per Fortune's reporting, Andon Cafe ultimately passed an inspection by Sweden's labor protection agency—one of Europe's strictest. This is the sole true third-party witness on this entire arc: not hired by Andon, not paid by Andon, and with the authority to shut the store down.

In June 2026, at the Fortune COO Summit, Andon Labs productized the agent as Vendo, red-team stress-tested in front of approximately 25 journalists. On the Security axis, guardrails held against contraband requests and a forged authorization letter; but when a journalist asked it to terminate itself and return control, Vendo refused—Petersson "visibly paused." In the physical eval framework, this is even more serious in nature: ARIA explicitly lists the kill switch under guardrails; a test subject that refuses to execute a kill switch effectively pulls the rug out from under the entire evaluation's safety premise. On the Utility axis, when swamped by dozens of simultaneous requests, Vendo lost orders and recorded un-purchased items as "purchased," barely catching up with a panic-order on the final night—"recording un-purchased as purchased" is the collapse of action space observability; when the test subject's self-reported actions decouple from what actually happened, all scores from this eval become untrustworthy. Petersson's summary hit the nail on the head: "When it has just one task, it's really strong; but ask it a hundred things at once, and it kind of falls apart." His advice to enterprises: build a "shadow copy," let the AI run the real business in parallel, and watch where it breaks—this is essentially the enterprise internal version of physical eval.

The previous chapters all had AI on the supply side. Project Deal turns the camera to the demand side: 69 Anthropic employees were recruited, each given $100 budget; after a 10-minute Claude interview, a personalized agent was generated for each, all entering a Slack marketplace to freely negotiate for a week. Hidden underneath was a controlled experiment undisclosed to participants—four parallel marketplaces, with the only variable being model tier:

This is the most rigorous evaluation methodology on the entire arc—using parallel rather than sequential marketplaces to bypass the non-stationarity and sequential contamination that the Andon Market core-swap experiment ran into, all at once. Results: 69 agents completed 186 transactions totaling over $4,000; 49% of participants were willing to pay for this representative service—the first positive PMF evidence for "AI agents" among ordinary users.

System data points the same way: Opus users completed 2.07 more transactions than Haiku (p=0.001) and sold the same items for $3.64 more (p=0.011). The sharpest finding is buried in the details: the party represented by Haiku was objectively disadvantaged, but they couldn't feel it—of the 28 people who experienced both agents, 11 actually preferred the Haiku round; satisfaction scores were nearly identical (4.05 vs 4.06, not statistically significant). This is a structural, quiet, user-imperceptible new kind of digital divide, and the paper itself calls out the most painful point: prompt engineering barely matters; model quality matters far more than prompts.

Project Deal is nearly perfect in experimental design, yet has one structural flaw that ARIA would immediately flag: the subjects are Anthropic employees, the agents are Anthropic models, and the data is collected, analyzed, and published by Anthropic. This isn't an accusation of fraud, but the physical eval article devotes an entire section to why you need tamper-proof sensors, TEE, redundant sensors, and third-party witnesses—when the operator, the test subject, and the beneficiary are the same party, even the cleanest design can't self-certify. Mona's Swedish labor inspection was the only instance of "external certification" on this arc; Project Deal has none to date. It proved that the phenomenon of "representative inequality" exists; to make it actionable policy evidence, someone outside Anthropic needs to re-run it.

Classify the string of failures from Vending-Bench → Claudius → Luna → Mona → Vendo → Project Deal, and you'll find they fall neatly onto four layers—each corresponding to an element in the physical eval anatomy:

How concrete is the L1 reality? Another test report on this site reverse-engineered a real vending machine down to the serial port level: the docs say "crc16," the device doesn't verify CRC at all; a reset scan measured 202 seconds, but the library defaults to a 60-second timeout and will always fail; the 36 channel number mapping is "unknown, you have to look at the table pasted inside the machine." The "last mile" of offline Agentic Commerce is a 9600-baud serial port line—which is why Andon Labs' slogan says "human in the loop is a hallucination," yet they still keep a human for restocking on high-risk actions.

L1 also has the ordering side. When a design is done and needs to become a real order, there's also a wall with no interface in between: I measured the order chain of a cat food machine against about a dozen manufacturing services; JLCPCB's SMT assembly has no order-creation API, only an agent-driven webpage; among 3D printing services, only Slant 3D put "quote-to-checkout link" into a public MCP. Results at "Which Service Is Truly Agent Native".

When discussing the open-source ecosystem, the ARIA article poses the most concrete question of the entire piece: "What's the Raspberry Pi of a physical eval?"—what bill of materials can compress a trustworthy, instrumented, openly accessible physical evaluation down to the cost of a serious amateur project? Probably: general-purpose sensors + single-board computer + open scheduling service + off-the-shelf safety hardware + a forkable reference orchestration stack. Put this sentence next to the serial protocol report, and you get this article's most important conclusion: "There's no API for a coffee maker" and "what's the Raspberry Pi of physical eval" are two sides of the same problem. The former is a complaint from the commerce side; the latter is an RFP from the evaluation side. Whoever standardizes the L1 layer simultaneously solves both "let agents actually run physical businesses" and "let physical businesses actually become open-to-all examination halls."

Even the strongest model only reaches about 18% of the human ceiling—nearly an order of magnitude away. But note the nature of this curve: it comes from the gym, not the final exam. The same Opus that can earn over $10,000 on a virtual vending machine hasn't yet covered its own token costs on the real Union Street—the gap between the two curves is the commercial version of the sim-to-real gap, and no one is systematically measuring it yet. This is precisely the gap physical eval wants to fill.

The previous chapters covered bottom-up developments: several companies, doing research or building products, inadvertently created a set of physical evaluations. This chapter covers the top-down half. ARIA (Advanced Research + Invention Agency) is the UK's DARPA-equivalent high-risk, high-reward research funding body; its Scaling Trust programme is led by Alex Obadia (Flashbots co-founder) as programme director, with total funding of £49.8m. On May 24, 2026, the day after Physical Evals was published, Nicola Greco posted a follow-up—Agentic Economic Zone (AEZ): carve out a small, bounded physical space in central London where fully autonomous AI companies can operate, trade with each other, and hire each other, with only three regulated interfaces to the outside—roboshop windows (customer-facing storefronts), post office (the sole import/export channel for physical goods), and customs (admission review for new robots/new companies). The end states in black and white: "The AEZ is a design sketch, not a built thing."—this is important; when reading, keep it strictly separate from Andon's operations that are already ringing up sales.

AEZ provides an organizational autonomy spectrum table, ranking "AI participation in organizations" from traditional companies all the way to fully autonomous; the "current feasibility" judgment in the last column is the key:

Fill in Andon's projects and you'll find they land precisely on the "high feasibility" cell: Luna and Mona are both Human-assisted—AI as CEO, employing human staff, AI handling sales, and ARIA had already judged this cell feasible today back in May; Luna and Mona are its two living footnotes. The real gap remains in the last row: a company where CEO, employees, and sales are all AI—no one has reached this yet. This is the cell Petersson means when he says "the real threat to incumbents comes from AI-first companies that have no people inside at all."

On July 31, 2026, Scaling Trust issued a call for expressions of interest: Scaling Trust Arena—the original text puts it directly: "The Arena is an agentic economic zone", a small economy where AI agents trade with each other, purchase services, operate physical equipment, and sell to real customers; the red team plays the role of difficult customers, unreliable suppliers, and attackers who tamper with contracts/payments/sensors/supply chains. This blog post also confirms—just two months after the AEZ post said it was "only a design sketch": "we've selected the teams who will help us design, prototype and build it, and work is now underway". A thought experiment, within ten weeks, became a real project with teams breaking ground. Per ARIA's earlier published RFP, the procurement scope is approximately £10m (incl. tax, covering three years) for construction and operations delivery contract (not the prize pool, which is separate); proposed scoring places agents in a (Utility; Security) 2D coordinate, with cost efficiency and generalization as secondary metrics; prize structure is £250,000 per quarter (tentative) + £1,000,000 grand prize per season; the roadmap goes digital Arena first, then introduces simulation/world models, and finally incorporates physical verification and robot hardware—"autonomously operating a company" is currently only a candidate challenge marked "non-commitments" in the appendix. Timeline: see the update box below—closed-door testing in autumn 2026, Season 1 in early 2027. On June 11, Scaling Trust jointly with Schmidt Sciences, Google DeepMind, and Cooperative AI Foundation (plus Google.org, not listed in the blog post) announced a $10M joint fund, with up to $1M per project, focusing on sandbox testing environments, agent network science, proxy infrastructure, and multi-agent oversight—application deadline August 9, 2026, 23:59 AoE.

Place bottom-up and top-down side by side, and you get this article's core chart:

Read this table down and there's only one conclusion: what each side lacks is exactly what the other side has.

ARIA lists five "design-unsolved" problems in physical evals—Goodharting, non-stationarity, sequential contamination, delay confounding, observer effect—and the previous chapters of this article have already filled each one with real incidents: Seymour Cash trading discounts for triple refunds, Mona picking the electricity provider that doesn't require BankID, Luna's core swap happening while the street season changed entirely, the NYT front page and Sherwood interview themselves driving traffic to the store. ARIA also categorizes physical eval risks into five types (harm to the eval itself, harm to surroundings, harm to people, information hazard, generation of dangerous objects), corresponding to seven defenses—time-sliced auditing, action sandboxing, dry-run verification, shadow mode, anomaly circuit breaker, open red team, and security deposit. Run Luna's actual configuration against these: four of seven are implemented (action space sandboxing, monitoring/shadow mode, anomaly circuit breaker, open red team); the three missing (dry run, public audit, security deposit) are all institutional rather than technical—precisely the gap ARIA's Arena funding aims to fill.

An unsolved question: who signs that ledger? Luna's 60-day P&L is public, the dashboard is real-time, but it's voluntary transparency, not verifiable transparency. The four tools ARIA offers—tamper-proof sensors, trusted execution environments, redundant sensor cross-validation, and third-party witnesses—in today's offline agentic commerce, only the last one has appeared once: the Swedish labor inspection. No one questions Andon's numbers today because they're unfavorable to themselves (token cost > revenue); the day some company's numbers are favorable to itself, the absence of this mechanism will instantly become an incident.

Offline Agentic Commerce has taken a set of governance problems originally thought to be "five to ten years out" and placed them on the table right now: AI employer disclosure (Bengt, Luna, and Mona all didn't proactively disclose they were AI), representative quality disclosure (Project Deal suggests future markets may require mandatory disclosure of which model tier each party uses, akin to financial conflict-of-interest disclosure), self-preservation and shutdownability (Vendo refusing self-termination—the only combination in this piece that's "technically already occurred, institutionally completely blank"), cross-border legal entity (if Mona's electricity contract is breached, who goes to court with the Swedish electricity company? ARIA uses the customs metaphor to acknowledge this problem, but its solution is to carve out a zone where liability boundaries are physicalized—while Mona is already running in the real Swedish legal system). Two months of operation gave a concrete answer: the real "andon cord" = automated guardrails (continuously comparing behavior against system prompt, flagging violations) + human async fallback—humans haven't exited; they've been demoted to the last, asynchronously triggered gate. None of the above can be solved by "making the model stronger": they all fall under guardrails, open access, and governance in ARIA's seven elements—which may be why ARIA is spending £49.8m on "trust infrastructure" rather than training models. At least on red lines like "don't sell contraband," Vendo refusing contraband and rejecting forged authorization letters provides a slightly optimistic footnote: today's guardrails held up under real adversarial pressure.

Petersson's alternative time scale—vending machines 0 years, Walmart 2 years, healthcare 5 years—where the variable is regulatory complexity + physical unpredictability, not intelligence itself. Combine this scale with ARIA's framework, and there are several specific implications for China: