Wiring an LLM into finance, healthcare, or government? Sooner or later you'll face the most expensive question there is — how many times more compute will you pay to keep the cloud from seeing your data?
You're about to wire an LLM into finance, healthcare, government, or legal workloads — where a single data leak is an incident — yet no one can plainly answer the most basic question: is handing data to cloud inference actually safe, and how many times more compute do I pay for that safety? If you're stuck here, this is written for you. It frames TEE, MPC, Homomorphic Encryption, and Split Inference as one single "trust-pricing" question: who you're willing to trust — the chip vendor, non-collusion among parties, or only math — directly decides whether you pay 1.05×, hundreds×, or thousands× the compute.
It's written for three kinds of readers, matching three moments you'll actually reach for it:
· Technical decision-makers (CTO / architect / platform lead) — choosing among TEE, MPC, HE, and split inference, dizzy from every vendor's "≤3× overhead" / "zero trust" / "data never leaves the domain" pitch. This hands you a decision tree to follow directly, plus the real overhead of each route: 1.05× / 10–100× / thousands× / near-zero.
· Security & compliance leads (CISO / data compliance / Xinchuang) — needing to tell the board, regulators, or customers exactly how well your AI inference is protected and at what cost; especially when Xinchuang requirements slam into the wall of "domestic GPUs still have no commercial confidential computing."
· Privacy-computing researchers, founders, and investors — judging whether a so-called "privacy-preserving" solution offers cryptographic guarantees, or just the empirical comfort of "upload features, not the raw image" — which feature-inversion attacks can break.
The evolution and endgame of four technical routes for LLM confidential inference: TEE, MPC, Homomorphic Encryption, and Split Inference are essentially four answers to the same "trust pricing" problem. This long-form article breaks down their evolution, measured overhead, industry adoption, and the unique structural contradictions of the Chinese market.
Before reading any technical details, narrow down the selection problem first. The compute path for enterprise LLM adoption essentially requires answering four questions in sequence: Is the data sensitive? → Do you have GPUs? → Are mitigation measures sufficient? → Are there Xinchuang requirements? The technical route corresponding to each endpoint will be dissected one by one in the main text.
Trust the chip vendor (TEE) and pay less than 10% compute premium; trust that multiple parties won't collude (MPC) and pay 10–100x; trust only math (FHE) and pay 3–5 orders of magnitude; architectural weak guarantees (split inference) cost near zero, but have the weakest formal guarantees.
NVIDIA H100/Blackwell has pushed overhead down to single-digit percentages; Apple, Anthropic, and Google have already deployed it in production. MPC/FHE will only land in the short term in narrow scenarios with strong compliance (finance, healthcare) that can tolerate high latency.
CPU-side domestic confidential computing (Hygon CSV, Kunpeng iTrustee) is already commercially deployed at scale, but GPU confidential computing on domestic AI accelerator cards is almost entirely blank—leaving a window for "MPC on domestic GPUs" and "falling back to self-built."
Federated learning, differential privacy, and zkML—these three are often tossed into the big basket of "privacy-preserving computation," but their objectives are orthogonal to confidential inference: federated learning solves collaborative training without data leaving its domain, differential privacy solves privacy leakage at the output and statistical level, and zkML solves the verifiability of inference results, not input confidentiality.
Confidential inference asks only one thing: When a user sends a prompt to a cloud-based large model, or a model provider hosts weights on third-party compute, how can we ensure that inputs, outputs (and sometimes weights) are not visible to the compute provider during computation? This is the confidentiality problem of "data-in-use"—the last mile that transport encryption (TLS) and storage encryption cannot cover.
The industrial weight of this problem is being continuously elevated by the policy side: the National Data Bureau's "Trusted Data Space Development Action Plan (2024–2028)" defines trusted data spaces as "data circulation and utilization infrastructure that connects multiple parties and enables data resource sharing and co-use," and explicitly names high-performance confidential computing as one of the core technologies for cracking the data circulation problem. In other words, whether confidential inference can scale determines not just the security paradigm of cloud AI, but also the technical foundation of the larger proposition of the data factor market. And this precisely depends on whether the four technical routes can simultaneously push "trust cost" and "compute cost" down to an industrially acceptable range.
The four routes are not competing for the same spot; rather, they each occupy a corner on the two-dimensional plane of "trust assumption × performance overhead." The ladder below (horizontal axis on a logarithmic scale of relative plaintext overhead) is the key to understanding the entire landscape.
The first-generation protagonist was Intel SGX—carving out an encrypted "enclave" in the CPU. But its two fatal flaws almost defined the early dilemma of the field: first, the enclave memory was only ~128MB, far from enough to run large model weights; second, the history of side-channel attacks was alarming—in 2018, Foreshadow directly broke SGX remote attestation; in 2020, CacheOut and SGAxe further proved that even with all side-channel defenses enabled, attackers could still extract attestation keys and forge legitimate Intel machine signatures (CVE-2020-0549). Intel later effectively deprecated SGX on new-generation client CPUs.
The second generation was VM-level confidential computing: Intel TDX and AMD SEV-SNP expanded the protection boundary from "a piece of code" to "an entire virtual machine." This step was crucial for AI—no need to modify model code, just stuff the entire inference stack into a confidential VM.
The real turning point was NVIDIA extending TEE to the GPU for the first time in 2023 on Hopper (H100). In CC mode, every piece of data written to HBM is encrypted within the GPU security engine using AES-256-GCM; keys are generated inside the GPU and never leave the chip; data between the CPU and GPU goes through an encrypted bounce buffer.
Apple Private Cloud Compute (PCC) is the most complete architectural paradigm to date: custom Apple silicon servers as the root of trust, a trimmed OS, Swift-written inference control layer for memory safety, request deletion upon processing, and verifiable software attestation provided to user devices (Apple Security official architecture post). In 2025, Apple further extended PCC beyond its own data centers—some Apple Intelligence inference now runs on Google Cloud's NVIDIA Blackwell GPUs, adopting a three-layer hardware trust stack of "NVIDIA Confidential Computing + Intel TDX + Google Titan security chip," claiming to be the industry's first globally scalable end-to-end confidential inference pipeline.
Anthropic in 2025, together with Pattern Labs (now Irregular), released the "Confidential Inference via Trusted Virtual Machines" whitepaper and deployed a Confidential Inference system. Its threat model is extremely aggressive: assuming the service provider itself is adversarial, the attacker fully controls all machines including the host. The subtext is clear—this is exactly the capability needed to deploy frontier models to data centers in untrusted jurisdictions.
The CPU side is commercially deployed at scale. Hygon CSV is currently the only domestically produced CPU confidential computing technology deployed at commercial scale: memory encryption uses the national cipher SM4, and remote attestation uses SM2 signatures; CSV3 (Hygen-4, late 2023) added memory integrity protection, bringing its capabilities on par with Intel TDX and AMD SEV-SNP. Alibaba Cloud has launched confidential VM instances based on Hygon CSV. Huawei Kunpeng iTrustee has obtained CC EAL4+ certification, with TEE secure memory configurable up to 128GB.
The GPU side is almost entirely blank—this is the biggest hardware bottleneck for Xinchuang. As of 2025, no domestic AI accelerator card (Ascend, Moore Threads, Biren, Cambricon, Tianshu Zhixin) has a commercial, productized GPU confidential computing mode like the H100. The closest is the academic prototype ASCEND-CC from Huawei Zurich Research Center and ETH Zurich (arXiv:2407.11888)—implementing NPU confidential computing by modifying firmware on the Ascend 910A, with inference overhead of only 0.028%–0.91%, but it is a single-tenant research prototype, not a commercial feature.
The root of trust problem is the original sin of this route. The entire security of TEE bets on two things: the chip vendor has no backdoor, and the attestation system is trustworthy (for an American vendor like NVIDIA, this is itself a trust issue in the Xinchuang context); and side-channel defenses are adequate (SGX's history proves this is far from reassuring). In short: TEE is "trust transfer" rather than "trust elimination"—it converges trust from "the entire cloud vendor" to "the chip vendor + a piece of verifiable code."
"TEE converges trust from the entire cloud vendor to the chip vendor plus a piece of verifiable code. This is a massive step forward, but it is trust transfer, not trust elimination."— One of the core arguments of this article
MPC slices data into secret shares, distributes them to multiple parties for collaborative computation, and no single party can see the original data. This is an intermediate assumption—weaker than "trusting the chip vendor" but stronger than "trusting only math."
Early work included general frameworks like ABY, SPDZ, and CrypTFlow. From 2022 onwards, secure inference protocols targeting Transformers emerged in rapid succession: Iron, Cheetah, CipherGPT, until 2023's PUMA became a milestone—the first open-source MPC scheme that could securely infer pre-trained Transformers without modifying the model, achieving MPC inference at the LLaMA-7B level for the first time: ~5 minutes to generate 1 token. Then came the performance race: BumbleBee (NDSS 2025) slashed matrix multiplication communication by 80–90%; Microsoft's SIGMA used function secret sharing + GPU acceleration to further reduce latency by 11–19x, achieving secure inference for GPT-class generative models for the first time.
Non-linear operators are the main battlefield. Non-polynomial functions like GELU, Softmax, and LayerNorm are extremely expensive on ciphertext—the entire optimization history of MPC-Transformers is essentially the history of "how to approximate non-linear operators more cheaply."
It is worth noting that the two milestone protocols, PUMA and BumbleBee, both came from domestic teams and are externally available through the open-source SecretFlow technology stack—in the MPC-Transformer niche, China's engineering contributions are in the global top tier. On the industry side, vendors like Huatsing, BaseBit, and Guangzhishu are advancing LLM confidential computing products, generally focusing on cryptographic evaluation / national cipher compliance advantages; and "pure cryptographic solutions are too expensive and need to work with trusted hardware" is already a widespread engineering consensus—MPC + TEE hybrid is widely seen as the realistic path to scale.
The fundamental bottleneck is not computation but communication. GB-level or even hundreds of GB of data exchange is the decisive bottleneck under WAN. This means MPC in the short term is only suitable for narrow scenarios with "good multi-party bandwidth, tolerable minute-level latency, and strong compliance." GPU-accelerated MPC (SIGMA being representative) is currently the most promising direction for changing the overhead curve.
FHE allows arbitrary computation directly on ciphertext: the client encrypts, the server computes, the client decrypts, and the server never sees plaintext. The trust assumption is the cleanest, and the overhead is the most terrifying—DARPA's analogy is that a computation taking 1 millisecond on a laptop would take weeks with today's FHE.
CKKS, BFV/BGV, and TFHE are the three mainstream schemes. Non-interactive HE inference for Transformers is a research hotspot in 2024–2026: NEXUS (NDSS 2025) is among the first to implement non-interactive Transformer inference using RNS-CKKS, but BERT-base (128 tokens) takes about 1103 seconds—and research indicates its non-linear approximation has a maximum relative error of up to 297%. Even with specialized approximation and GPU implementations, FHE overhead for running full LLM inference remains in the thousands to hundreds of thousands range.
DARPA's DPRIVE program funded four teams (Intel, Duality, SRI, Galois) to build FHE ASIC accelerators, aiming to bring overhead within an order of magnitude of plaintext; Intel and Microsoft's joint ASIC claims to accelerate by about a million times. Cornami takes the reconfigurable many-core approach, Optalysys uses optical computing, Niobium has taped out, and Fabric Cryptography's chip is expected to mass-produce within a year. Whether the 1000–5000x improvement claimed by these accelerators is "enough" remains an open question.
Deployment reality: What FHE can deliver today is "small and beautiful" retrieval-like tasks, not LLM inference. Apple uses HE for encrypted database queries (caller ID lookup without exposing the queried number), not for running neural networks with HE. Zama became FHE's first unicorn with a valuation exceeding $1 billion, but its FHEVM on-chain throughput is still only on the order of tens of transactions per second. This itself shows: FHE's industrial breakthrough lies in lightweight, high-value, latency-tolerant scenarios. Hybrid schemes (HE only handles linear operators, non-linear ones are handed off to MPC or TEE) are a more pragmatic engineering path—BOLT and BumbleBee are essentially HE/MPC hybrids.
Pulling homomorphic encryption from "theoretical upper bound" toward engineering usability, a noteworthy hybrid route has emerged domestically—Jinghua Confidential Computing's confidential training/inference platform is representative. Its approach can be summarized in three sentences: First, linear and non-linear operators are computed separately, structurally splitting apart the part with the heaviest cryptographic overhead; Second, CPU-TEE and GPU heterogeneous scheduling—key management and the sensitive control plane reside in the CPU's trusted execution environment (millisecond-level random key generation and revocation), while large-scale matrix operations are handed to the GPU; Third, encryption granularity is at the request level, with each inference request randomly encrypted and computed token-by-token in ciphertext, so inputs and results are visible only to the requester.
Unlike most solutions that only do confidential inference, Jinghua covers confidential training (ciphertext model training, ciphertext retrieval, confidential vector database), and claims that while adapting to mainstream domestic GPUs, the time loss of confidential training and inference relative to plaintext is controlled within 3x—if independently verified, this would be significantly lower than the ranges of pure MPC (10–100x) and pure FHE (10³–10⁵x), at the cost of introducing hardware trust in CPU-TEE (deployment requires CPU support for TEE). Deployment form factors include confidential computing appliances (inference: dual-socket CPU + 4 GPUs; training: dual to quad-socket CPU + 8 GPUs) and software deployment; publicly disclosed projects include the Lecheng Healthcare Confidential Training Trusted Space and the Yangtze River Delta Healthcare Pilot Zone platform, with CCA confidential standards and Level 4 classified protection review progressing in 2026.
It should be objectively noted: "≤3×" is a vendor claim that still lacks public third-party benchmarking; and the trust assumption of this route is no longer "trust only math," but a hybrid of "math + CPU chip vendor"—this is exactly the price it pays for performance, and why it is placed in this chapter rather than given its own.
The industry significance of this hybrid route is that it demonstrates: the realistic path for HE in LLM scenarios is not "waiting for ASICs to save pure FHE," but rather using cryptographic guarantees where they matter most, and handing performance-sensitive parts back to hardware trust—essentially creating a new tier in the middle of the trust-pricing ladder.
Run an encoder locally, only send the compressed intermediate features to the cloud-based large model, and the original input never leaves the device. The overhead is almost zero—it's just normal neural network forward computation.
Real-world example: GLM-4.5V (released and open-sourced by Zhipu in August 2025, 106B total params / 12B activated) consists of a visual encoder, MLP adapter, and language decoder—the visual encoder can be placed on the edge side, uploading only compressed visual features, making it a natural landing point for split inference in multimodal scenarios. Apple Intelligence's tiered architecture of "edge-first, go to PCC only when cloud is necessary" is essentially also split thinking.
The open-source implementation accompanying this project, VIT Secure Inference (GitHub · MIT), turns this route into a directly installable product: using a 300M-parameter ViT visual encoder (Metal GPU accelerated) on a local Mac to extract image features, then uploading only ~1MB of feature vectors after Gzip compression to the cloud GLM-4.5V for visual understanding and text generation—original image pixel data never leaves the device, supporting OCR, document/financial report analysis, multi-image comparison, and medical imaging tasks.
Honest boundary note: ViT encoding is a one-way and heavily information-compressing process, significantly raising the reconstruction barrier, but referring to the feature inversion research below, this is still an empirical guarantee, not a cryptographic guarantee—this is why it is categorized under this route, and why adding noise/obfuscation is recommended for high-sensitivity scenarios.
Fatal weakness: Feature inversion attacks. "Only send features, not the original image" sounds secure, but extensive research proves intermediate representations can be partially or even high-fidelity reconstructed: FIA-Flow (2025) uses less than 0.32% of ImageNet-1K samples to train a black-box inversion model that high-fidelity reconstructs the original image; CapRecover (ACM MM 2025) can directly reconstruct text descriptions of images from visual features; for LLMs, there are also Vec2Text-style embedding inversion works. Defense methods (adversarial training, feature sparsification, noise addition) all have accuracy-privacy trade-offs, and none can provide cryptographic-level formal guarantees.
Positioning conclusion: Split inference is a "cost-driven" rather than "guarantee-driven" solution. It is suitable for consumer-grade scenarios with medium privacy sensitivity and extreme sensitivity to latency and cost; but for any scenario involving finance, healthcare, or government that requires provable confidentiality, it can only serve as an auxiliary layer, not the primary line of defense.
| Route | Root of Trust | Overhead vs. Plaintext | Formal Guarantee | Representative Schemes / Products | Xinchuang Adaptability | Typical Scenarios | Maturity |
|---|---|---|---|---|---|---|---|
| TEE / GPU CC | Chip vendor + verifiable code | <10% H100 average <7%, Blackwell approaching zero | Medium Depends on hardware + remote attestation | NVIDIA H100/Blackwell CC · Apple PCC · Anthropic Confidential Inference | Weak Root of trust is NVIDIA; domestic GPU TEE is blank | General-purpose cloud confidential inference, cross-jurisdiction deployment | Commercially deployed at scale |
| MPC | Multi-party non-collusion assumption | 10–100× GB-level communication, bottleneck is bandwidth | High Cryptographic | PUMA · BumbleBee · SIGMA · SecretFlow open-source stack | Strong Pure software can run on domestic GPUs, easy to pass crypto evaluation | Finance / healthcare cross-institutional data fusion | Narrow-scenario deployment |
| HE+TEE Hybrid Operator separation |
Math + CPU chip vendor (hybrid) | Claims ≤3× Vendor figure, pending independent verification | Relatively high Primarily cryptographic + TEE control plane | Jinghua Confidential Computing training / inference platform (appliance + software deployment) | Strong Adapted for mainstream domestic GPUs, national cipher certification in progress | Trusted data spaces, confidential training, cross-border data circulation | Early commercial stage |
| FHE | Trust only math | 10³–10⁵× Hardware acceleration is the only hope | Highest Cryptographic | NEXUS · Zama TFHE · Apple swift-HE (PIR) | Medium Algorithms can be national-cipher-ized, performance not yet usable | PIR-class lightweight encrypted retrieval | Lightweight retrieval only |
| Split Inference | "Features hard to invert" (empirical) | ≈ 0 | Weakest Threatened by feature inversion attacks | GLM-4.5V edge-cloud collaboration · VIT Secure Inference · Apple edge-cloud tiering | No specific hardware dependency | Default cost-reduction layer for consumer / low-sensitivity scenarios | Consumer-grade auxiliary |
The Chinese market is not a simple scaled-down version of the global landscape—it has three unique structural constraints that together spawn China's distinctive "fall back to self-building" logic.
The de facto standard for global cloud confidential inference is converging on NVIDIA GPU CC, but its root of trust (remote attestation, chip keys) is entirely in NVIDIA's hands. You use confidential computing because you don't trust the compute provider, yet you must unconditionally trust an American chip vendor's attestation system.
On domestic hardware, confidential inference in the short term can only rely on software cryptographic solutions (MPC) running on domestic GPUs/CPUs, or on Hygon CSV / Kunpeng TEE for VM-level protection—unable to replicate NVIDIA's low-overhead GPU confidential inference path. This in turn reinforces the "MPC + TEE hybrid using national ciphers" technology choice.
The GB/T 43206-2023 cryptographic evaluation standard took effect in April 2024, superimposed with the policy drive of the "Data Factor ×" three-year action plan. Solutions adopting national cipher algorithms (SM2/SM3/SM4) gain a natural compliance advantage in strongly regulated scenarios like government and finance—this is a local barrier that international vendors cannot easily replicate.
The combined result of these three points: for institutions with the highest confidentiality requirements (large banks, top-tier hospitals, government clouds), rather than agonizing over the confidential inference trust chain on public clouds, it's better to directly deploy domestic hardware + domestic confidential solutions privately—the cost is higher operational overhead and limited external service efficiency, but under current technical constraints, this is a structurally rational choice. The policy side is trying to pull this "falling back" demand back into trusted cloud-based confidential circulation: the National Data Bureau's "Trusted Data Space Development Action Plan (2024–2028)" explicitly names high-performance confidential computing; on the deployment side, according to vendor disclosures, the Lecheng Healthcare Confidential Training Trusted Space (the country's first trusted space supporting AI confidential training on medical datasets) and the Yangtze River Delta Healthcare Pilot Zone platform both use "compute without decryption" as the technical foundation for solving cross-domain trust—its construction goal is to cover 30 leading pharmaceutical companies and 2,800 key hospitals in the Yangtze River Delta by 2027, connecting 1,000+ data entities.
On the demand side, real buyers are highly concentrated in four categories: Finance (risk control, cross-institutional data fusion), Healthcare (medical records, genetic data cross-center collaboration), Government (data factor circulation), and Cross-border (data export compliance). On the supply side, market forecasts diverge massively: Grand View Research gives about $153.8 billion for 2030 (CAGR ~64%), Precedence Research gives about $1.28 trillion for 2034, while conservative forecasts from TechSci and others put 2030 at only $10–23 billion. A forecast divergence of tens of times is itself a signal: this market is still early, definitions are chaotic, and consensus has not formed. The rational reading is to focus on the growth direction (high-speed) rather than absolute numbers.
"Under Xinchuang constraints, confidential inference currently has no cheap cloud solution. Falling back to self-building is structurally forced."— Core judgment of the China market chapter
Selection doesn't need a new framework—the decision tree at the beginning is the action guide. Below, each of the four decision nodes is expanded: each branch gives the endpoint, action items, and the threshold that triggers a fallback.
First published 2026-06-17