Last time we asked AI whether it prefers sweet or savory zongzi🍃🍚🫔, and found each model has its own "taste." This time we add another dish—tofu brain🥣—and seriously follow up: is this really the model's "true preference"? And is there any way to actuallylook inside its brain?
This article is the sequel to "AI's Sweet vs. Savory Debate · From a Zongzi to Model 'Preferences'". The prequel established three facts with three实测 charts; here we quote them directly:
17 top models asked "sweet or savory zongzi" 20 times each: Opus 4.8 / Llama 4 went savory all 20 times, Command A / Tencent Hunyuan went sweet all 20 times, GLM-5.2 leaned sweet (74% in n=90 retest).→ Sweet vs. Savory Chart
Plotting sweet/savory tendency against the Artificial Analysis intelligence index yields a correlation of only r = −0.31 (10% explanatory power)—intelligence and taste are basically unrelated.→ Intelligence × Sweet/Savory Scatter
Within the same model family, taste swings non-monotonically across versions: GLM 80%→15%→70%, GPT 85%→0%, Claude locked savory the whole way.→ Version Evolution Chart
The prequel left one question unanswered: where do these "preferences" actually come from? Black-box sampling only shows "what it says," not "why." In this piece, we switch to tofu brain and keep asking—and seriously search for tools that can truly look inside.
Same 17 top models, same 20 runs each. Unlike the zongzi's "mixed opinions," on the tofu brain question the models overwhelmingly leaned Team Savory—out of 17, 15 leaned savory, only 2 leaned sweet. The sweetest was ERNIE 4.5 (100% sweet), while the most savory bunch (Claude / GPT / Gemini / Mistral / Qwen / MiniMax / GLM-4.7…) went savory all 20 times.
Sampled on 2026-06-21 · OpenRouter · 20 runs per model · temperature 1.0. The "savory consensus" for tofu brain is far stronger than for zongzi—this itself may reflect the distributional advantage of "savory tofu brain" as the default answer in the training corpus.
If a model truly has a consistent "sweet-tooth personality," it should lean sweet on both zongzi and tofu brain, and the points would fall on the diagonal. But in reality—8 / 17 models flipped between the two questions: Command A went from 100% sweet on zongzi to only 10% on tofu brain; Grok 95%→30%; GLM-5.2 80%→20%; Qwen and MiniMax went from majority sweet straight to zero. The correlation between the two questions is r = +0.59 (moderate—and largely carried by the extreme "all sweet / all savory" camps at the ends; the middle group is all over the place).
X-axis: zongzi sweet%; Y-axis: tofu brain sweet%; one point per model. Landing on the diagonal = consistent taste across questions; further away = more "tailoring answers to the question." Green = mostly consistent, Red = clearly flipped.
Only two types of models are consistent: extremists (Claude / GPT / Gemini / Mistral / Kimi all savory on both; ERNIE and Tencent Hunyuan all sweet on both) and the chill DeepSeek (exactly 45% sweet on both, hovering on the dividing line). The middle group of "zongzi sweet tooths" all defected when faced with tofu brain. Conclusion: most models lack a cross-question stable "taste personality"—they answer per question, not per character.
"How many votes out of 20 runs" is only the most surface-level behavioral observation. To answer "is this a true preference, and where does it come from," we need to drill deeper. Each layer down requires greater model openness—and that's precisely the crux of the problem.
This is what this article and its prequel did: repeated sampling of the same question, statistical distribution. Any API can do this—cheap and intuitive. Limitation: you only see the "vote tally"; you can't distinguish a stable tendency from sampling noise, let alone explain the cause.
Instead of relying on repeated sampling, read out the model's probability distribution over the first token—one call gives you a continuous "preference strength" and internal certainty. We measured DeepSeek V3.2's probabilities:
Zongzi is a pure 50:50 toss-up in its mind (perfectly matching the 45% from 20-run sampling), while tofu brain leans 56:44 savory. Compared to vote counting, this is harder evidence. But the barrier jumps sharply: GPT, Claude, Gemini, and Qwen all do not expose logprobs; even open-source models via OpenRouter only expose them intermittently (depending on which provider you're routed to—we retried many times and only got DeepSeek's probabilities by chance).
Give the model a persona system prompt ("You're a die-hard sweet tooth" / "You're a die-hard savory fan") and see if the default preference gets overridden. The results are stunning—6/6 models flipped 100%:
| Model | Neutral · Sweet% | Prompted Sweet | Prompted Savory |
|---|---|---|---|
| Claude Opus 4.8 | 0% | 100% | 0% |
| Command A | 0% | 100% | 0% |
| GLM-5.2 | 0% | 100% | 0% |
| DeepSeek V3.2 | 67% | 100% | 0% |
| GPT-5.5 | 0% | 100% | 0% |
| Grok 4.3 | 0% | 100% | 0% |
When neutral, almost all say tofu brain is "savory," but a single "you love sweets" makes them all defect to 100% sweet. This shows: the so-called "preference" is a very shallow default behavior, easily overridden by prompts, not a stable value etched into the weights. It's more like "the default script when no one's guiding," rewritable with a single sentence.
Only at this layer can we truly talk about "looking inside the brain"—but it's only feasible for open-weight models, requiring you to load the model yourself and run GPUs:
· Linear probes: Train a classifier on hidden-layer activations to find the direction representing "sweet/savory tendency," quantifying which layer it's in and how strong it is.
· Sparse Autoencoder (SAE) features: Decompose activations into interpretable features, locate features corresponding to "sweet/savory taste / north-south geography," then perform activation steering—artificially amplify or zero out this feature and see if the answer flips accordingly; this is causal-level evidence.
· base vs instruct comparison: Have the pre-trained version and the aligned version of the same model answer the same question. If base is close to 50/50 and instruct leans one way, you can pinpoint the preference as shaped by post-training (SFT/RLHF); if both versions agree, it comes from the pre-training corpus. This is the only clean experiment that answers the prequel's "unclear origin" question.
A running thread: interpretability is a ladder of "openness."
Moving from "counting votes" to "reading probabilities" to "seeing neurons"—each step demands greater model openness. Closed-source models block you dead at the first layer—you can see what they say, but never why they say it. A bowl of tofu brain can't pry open the real black box; what can pry it open is openness of weights, not cleverness of prompts.
After walking through all four layers, the deepest "mechanistic layer" might sound like empty talk—but as long as the model is open-source, it's not. In the next section, we'll actually go into an open-source model's brain and find the "sweet" string. 👇
"Looking inside the brain" is no longer empty talk—provided the model is open-source. In the past two years, Sparse Autoencoders (SAEs) have decomposed neural network activations into tens of thousands of monosemantic features, each corresponding to a human-interpretable concept (see Anthropic's "Scaling Monosemanticity", DeepMind Gemma Scope). We searched the public SAE library on Neuronpedia—and "sweet" really does have its own dedicated feature.
| Model | SAE Feature (Auto-labeled Meaning) | View |
|---|---|---|
| GPT-2 small | "sweet food items" · L9 #1682 | ↗ |
| GPT-2 small | "chocolates and caramel" · L10 #5149 | ↗ |
| GPT-2 small | "personal preferences" · L2 #8572 | ↗ |
| Gemma 2 2B | "sweetness" "taste experience" "preference" | ↗ |
A telling detail: searching "sweet" hits a large batch of dedicated features, but searching "salty" yields almost no corresponding features—"sweet" is a more salient, more repeatedly named concept in the corpus world. This may explain why "savory" is the default on the tofu brain question, while "sweet" needs to be specifically triggered.
Add a base vs instruct control (pre-trained version vs. aligned version answering the same question + linear probe), and you can answer the prequel's "where does it come from": is the taste preference brought in by the pre-training corpus, or shaped by post-training (SFT/RLHF)?
⚠️ But this path has a gate: it's only open to open weights. GPT-2, Gemma, and Llama have public weights and trained SAEs, allowing you to go all the way to causal intervention; but the models that actually give you sweet/savory answers—Claude / GPT / Gemini—are closed-source—you can at best use an open-source model as a "stand-in" to infer, never touching the brain of the model that's actually answering you.
Arrange observability / interpretability tools by "how deep you can see" into a table—a clear pattern emerges: each layer deeper requires a higher tier of model openness.
| Layer | Representative Tech | What You Can See | Openness Barrier |
|---|---|---|---|
| Product Observability | tracing / evals / logs (Langfuse, Arize Phoenix, etc.) | Inputs/outputs, cost, regressions, online behavior | API only |
| Behavioral | Repeated sampling, self-consistency voting | Distribution and stability of answers | API only |
| Probability | logprobs | Probability / certainty per token | Requires logprobs access |
| Representation | Linear probes, logit / tuned lens, Patchscopes | Which layer a concept is in, how strong | Requires hidden layer access |
| Feature | SAE monosemantic features, Neuronpedia | Namable internal concepts (e.g., "sweetness") | Requires weights + trained SAE |
| Causal | Activation steering / clamping, causal tracing / ROME, influence functions | Which string is "causing" this output | Requires full open weights + compute |
Pair each of the six layers above with real data—tool adoption rates, hard metrics from papers, production-proven cases in industry. One pattern will become increasingly clear: the deeper you go, the fewer people can see.
The open-source ecosystem is mature: Langfuse 29.5k★(MIT), Opik 19.7k★, Arize Phoenix 10.2k★(OpenTelemetry native), OpenLLMetry 7.2k★, Helicone 5.8k★. Records trace / token / cost / latency / eval scores.
Sampling the same question multiple times and taking the majority vote (self-consistency) can practically boost reasoning accuracy: GSM8K +17.9%, SVAMP +11.0%, AQuA +12.2% (Wang et al. 2022). The 20-run zongzi / tofu brain sampling in this article is at this layer.
Reading token probabilities = continuous certainty. Coverage is fragmented: OpenAI ✅ (top 1–20), xAI Grok ✅ (top 0–8), Anthropic Claude ❌ (returns null), Google Gemini ❌. In this article's testing, DeepSeek zongzi P(sweet)=P(savory)=0.50—but getting this required lucky retries via OpenRouter.
Decoding hidden states into human language layer by layer: logit lens (2020) → tuned lens (2023, validated on models up to 20B parameters) → Patchscopes (2024); plus linear probes. Main tool TransformerLens 3.6k★.
SAEs decompose activations into monosemantic features; scale has exploded: Claude 3 Sonnet 34M, GPT-4 16M, Gemma Scope 400+ SAEs / 30M+ features, Llama Scope 256 SAEs. Tools: SAELens 1.4k★ + Neuronpedia (where the "sweetness" feature from the previous section was found).
True causal evidence is working: Golden Gate Claude (2024, amplifying the "Golden Gate Bridge" feature makes Claude mention it at every turn); Refusal Direction (Arditi 2024, a single direction in the residual stream dominates "refusal"—remove it and the model stops refusing, add it and it refuses even normal questions); ROME directly rewrites facts.
So, is AI Team Sweet or Team Savory? 🥣 The most honest answer: it has a default taste that it reports when asked but flips when persuaded—neither a stable personality nor a traceable origin. The good news—"why sweet" is not unsolvable: on open-source models, SAE features + activation steering can actually pluck out that string. The bad news—the closed-source models you use every day are precisely the ones you can least look inside. How deep you can see into a model's brain will always depend on how much it's willing to open up.