Skip to content
← DeepDive Compute & Business · v3 Updated 2026-10-14 EN
DEEPDIVE · CONFIDENTIAL LLM INFERENCE · 2026.06

Who you're willing to trust,
determines how many times more compute you pay

When you connect large models into sensitive scenarios like finance, healthcare, or government, you'll eventually have to answer the most expensive question: for "don't let the cloud see my data," how many times more compute are you willing to pay?

AUTHOR / Feng Xiaoping SERIES / AI DEEPDIVE FILED / DD · 0018 · 2026.06.12 READING / ~25 MIN
SCROLL
Encrypt Decrypt Prove Trust Ciphertext Plaintext Enclave Certificate Key Multi-party Shard
Reading Guide · Who this is for, and when you'll suddenly need it

When you're first forced to choose between "performance" and "don't let anyone see my data" — you don't need a vendor's PPT, you need the real overhead, trust assumptions, and deployment boundaries of all four routes.

You're about to connect a large model into finance, healthcare, government, or legal — businesses where "data leakage = incident" — yet no one can give you a straight answer to the most basic question: Is it safe to send data to cloud inference? And for that safety, how many times more compute do I need to pay? If you're stuck here, this piece is for you. It lays out TEE, MPC, homomorphic encryption, and split inference as answers to the same "trust pricing" problem: who you're willing to trust (the chip vendor, that multiple parties won't collude, or only mathematics) directly determines whether you pay 1.05×, hundreds of times, or thousands of times more compute.

It's written for three kinds of people, corresponding to three moments when you'll actually need it:

· Technical decision-makers (CTO / architect / platform lead) — choosing between TEE, MPC, homomorphic encryption, and split inference, getting dizzy from every vendor's "≤3× overhead," "zero-trust," and "data never leaves the domain" pitches. This piece gives you a decision tree you can follow directly, and the real overhead of all four routes: 1.05× / 10–100× / thousands× / near-zero.

· Security & compliance leads (CISO / data compliance / Xinchuang) — needing to explain to the board, regulators, or clients "what level of protection our AI inference has, and at what cost"; especially when Xinchuang requirements hit the hard wall of "no domestic GPU has commercial confidential computing yet."

· Privacy-computing researchers, founders, and investors — judging whether a "privacy-preserving" scheme offers cryptographic guarantees, or just the empirical comfort of "send features, not the original image" that can be shattered by inversion attacks.

START

Answer Four Questions First: Compute & Confidential Route Decision Tree

COMPUTE & CONFIDENTIALITY DECISION TREE

The evolution and endgame of four technical routes for LLM confidential inference: TEE, MPC, homomorphic encryption, and split inference are essentially four answers to the same "trust pricing" problem. This long-form article dissects their evolution, measured overhead, industry deployment, and the structural contradictions unique to the China market.

Before reading any technical details, narrow down the selection problem first. The compute path for enterprise LLM adoption essentially requires answering four questions in sequence: is the data sensitive → do you have GPUs → are mitigation measures sufficient → are there Xinchuang requirements. Each endpoint corresponds to a technical route that the main text will dissect one by one.

Is this use case's data sensitive? Not sensitive Sensitive Vendor Cloud Platform Most daily use cases · Call API directly Have own GPU compute? Have GPUs No GPUs or not enough Internal Private Deployment Few need internal isolation (confidential compute/TEE) Are trust mitigations sufficient? Sufficient Insufficient Desensitization + Dedicated Instance Contractual no-training promise · Most common tier Xinchuang requirements? Yes No Domestic Confidential Compute Options ① MPC on domestic GPUs · 10–100× ② HE+TEE hybrid (operator separation) claims ≤3× ③ CPU-TEE (Hygon CSV) + domestic accelerator GPU TEE First <10% overhead Upgrade to MPC/HE as needed When confidential compute is immature, fall back to self-build Split Inference / Edge-Cloud Synergy: a cost-reduction layer that can be stacked before any path — raw data never leaves the edge, overhead ≈ 0, guarantee is empirical Example: VIT Secure Inference (local ViT encode + cloud GLM-4.5V) · See Chapter 04
Green = low-friction path · Gold = Xinchuang confidential compute menu · Red = confidential computing path · Dashed = structural fallback · Click nodes to jump to corresponding sections
TL;DR

Read the Whole Article in Three Sentences

THREE TAKEAWAYS
01 / Essence

Confidential inference is a "trust pricing" problem

Trust the chip vendor (TEE) and pay less than 10% compute premium; trust that multiple parties won't collude (MPC) and pay 10–100×; trust only mathematics (FHE) and pay 3–5 orders of magnitude; architectural weak guarantees (split inference) cost near zero, but have the weakest formal guarantees.

02 / Landscape

GPU TEE will become the de facto cloud standard

NVIDIA H100/Blackwell has pushed overhead to single-digit percentages, Apple has deployed it in production (some inference runs on Google Cloud), and Anthropic has published a research proposal. MPC/FHE will only land in the near term in narrow scenarios with strong compliance and tolerance for high latency, such as finance and healthcare.

03 / China

Xinchuang × confidential inference has an unsolvable hardware gap

Domestic CPU-side confidential computing (Hygon CSV, Kunpeng iTrustee) is commercially deployed at scale, but GPU confidential computing on domestic AI accelerators is nearly blank — leaving a window for "MPC on domestic GPUs" and "fallback to self-build."

00

Opening Cut: First, Show the "Close Relatives" Out

SCOPE & DEFINITION

Federated learning, differential privacy, and zkML — these three are often lumped into the "privacy computing" basket, but their goals are orthogonal to confidential inference: federated learning solves collaborative training without data leaving the domain, differential privacy addresses privacy leakage at the output and statistical level, and zkML addresses the verifiability of inference results, not input confidentiality.

Confidential inference asks only one thing: When a user sends a prompt to a cloud-based large model, or when the model provider hosts weights on third-party compute, how do you ensure that inputs, outputs (and sometimes weights) are not visible to the compute provider during computation? This is the confidentiality problem for "data-in-use" — the last mile that transport encryption (TLS) and storage encryption cannot cover.

The industrial weight of this problem is being continuously elevated by policy: the National Data Bureau's "Trusted Data Space Development Action Plan (2024–2028)" defines trusted data spaces as "data circulation and utilization infrastructure that connects multiple parties and enables data resource sharing," and explicitly names high-performance confidential computing as one of the core technologies for solving data circulation challenges. In other words, whether confidential inference can scale determines not just the security posture of cloud AI, but the technical foundation of the larger data-factor market proposition. And that depends precisely on whether the four technical routes can simultaneously compress "trust cost" and "compute cost" into an interval the industry can accept.

FRAME

Trust Assumption × Performance Overhead: One Diagram to Understand the Landscape

THE TRUST-PRICING LADDER

The four routes aren't competing for the same position — they each occupy a corner of the "trust assumption × performance overhead" plane. The ladder below (horizontal axis on a log scale of relative plaintext overhead) is the key to understanding the entire landscape.

Trust Pricing Table / TRUST-PRICING LADDERRelative Plaintext Inference Overhead (LOG SCALE)
Split InferenceSPLIT INFERENCE · Trust "features are hard to invert"
≈ 0Weakest formal guarantees
GPU TEECONFIDENTIAL COMPUTING · Trust chip vendor
< 10%H100 average <7%, approaching zero for large models
MPCSECURE MULTI-PARTY · Trust parties won't collude
10–100×Communication in GB range, bottleneck is bandwidth
FHEFULLY HOMOMORPHIC · Trust only mathematics
10³–10⁵×Hardware acceleration is the only hope
1×10×10²×10³×10⁴×10⁵×
Core insight: There is no free lunch in confidential inference — the trust you save, you pay for in compute. The less willing you are to trust anyone (FHE), the more compute you pay; the more willing you are to stake trust on the chip vendor (TEE), the cheaper the compute. So-called technology selection is essentially answering: "In your business, who are you least willing to trust, and how much cost can you tolerate?"
01

TEE: Pushing Overhead to Single Digits,
But the Trust Root Is in Someone Else's Hands

TRUSTED EXECUTION ENVIRONMENT
Trust assumption: chip vendor + a piece of remotely verifiable code
Edge · CLIENT Cloud · HOST (Untrusted) User Holds plaintext input Host / Cloud vendor — cannot see inside enclave Confidential Execution Environment · TEE Enclave GPU-TEE / Confidential VM — decrypt and compute only inside enclave HBM written with AES-256-GCM · Key never leaves chip Plaintext exists only in this box ① Encrypted channel ② Remote attestation Trust root = chip vendor + verifiable code · Overhead <7% (H100 CC) Residual risk: side channels / vendor backdoor — trust transfer, not trust elimination
TEE · Stuff the entire inference stack into a hardware enclave: plaintext is decrypted only inside the enclave, the host and cloud vendor are locked out, but security is staked on "the chip vendor has no backdoor + remote attestation is trustworthy."

Evolution: From Enclave to Confidential VM, Then to GPU

The first-generation protagonist was Intel SGX — carving out an encrypted "enclave" in the CPU. But its two fatal flaws nearly defined the early困境 of the entire track: first, enclave memory was only ~128MB, far from enough to hold large model weights; second, the side-channel attack history was alarming — in 2018, Foreshadow directly broke SGX remote attestation; in 2020, CacheOut and SGAxe further proved that even with all side-channel defenses enabled, attackers could still extract attestation keys and forge legitimate Intel machine signatures (CVE-2020-0549). Intel later effectively deprecated SGX on new-generation client CPUs.

The second generation was VM-level confidential computing: Intel TDX and AMD SEV-SNP expanded the protection boundary from "a piece of code" to "an entire virtual machine." This step was crucial for AI — no need to modify model code; just stuff the entire inference stack into a confidential VM.

The real turning point came in 2023, when NVIDIA first extended TEE to the GPU on Hopper (H100). In CC mode, every write to HBM is encrypted with AES-256-GCM inside the GPU security engine; the key is generated inside the GPU and never leaves the chip; data between CPU and GPU travels through an encrypted bounce buffer.

<7%
Average overhead of LLM inference under H100 CC mode; the larger the model and longer the sequence, the closer overhead approaches zero
arXiv:2409.03992
4–8%
Throughput penalty range from independent academic benchmarks, decreasing with larger batch and input sizes
arXiv:2509.18886
≈0
Throughput overhead for compute-heavy loads like Llama-3.1-70B under H100 CC — bottleneck is only PCIe data transfer
PHALA BENCHMARK
1/2/4/8
Blackwell launch TEE-I/O: NVLink inline encryption, multi-card confidential VM with "near-zero performance loss"
NVIDIA CONFIDENTIAL COMPUTING

Industry Deployment: The Only Route Already at Scale

Apple Private Cloud Compute (PCC) is the most complete architectural exemplar to date: custom Apple silicon servers as trust root, trimmed OS, Swift-written inference control layer for memory safety, request data deleted after processing, and verifiable software proofs provided to user devices (Apple Security official architecture post). In 2025, Apple further extended PCC beyond its own data centers — some Apple Intelligence inference runs on Google Cloud's NVIDIA Blackwell GPUs, using a three-layer hardware trust stack of "NVIDIA Confidential Computing + Intel TDX + Google Titan security chip," claimed as the industry's first globally scalable end-to-end confidential inference pipeline.

Anthropic published "Confidential Inference via Trusted Virtual Machines" with Pattern Labs (now Irregular) in June 2025, but this is a research sketch, not a description of a deployed system — the paper itself states it is "just a research sketch meant to start a discussion," and that it's too early to predict what specific design it will evolve into. The idea: most of the inference server stays on the untrusted side; only a small trusted "model loader and invoker" in an isolated VM handles decrypted data; a key server releases keys only after hardware attestation passes; model weights are stored encrypted and decrypted only inside the loader. Its motivation is primarily preventing model weight theft, and it claims this design is intended to resist a malicious hypervisor and some forms of physical attack. The boundaries are stated clearly: accelerators are not yet inside the trust boundary (the required accelerator memory protection features are not mature); data center physical security and hypervisor-layer security still depend on the compute provider; API server security is out of scope. In other words, it neither claims to "assume the service provider is adversarial" nor states it has been deployed; the publicly available material that explicitly designs for "even operators cannot touch the data" is Apple PCC above.

Verifiable Productization: TrustedRouter Makes "Trust Transfer" Transparent for Client Verification

Apple proved GPU TEE can scale in production; per its design specification, devices only send data to nodes whose measurements are written into a transparency log, and it plans to open every production version to security researchers. Anthropic's proposal is still a research sketch — external clients have nothing they can run themselves. For enterprise clients who want to verify things themselves, neither is a ready-made entry point. AI gateway startup TrustedRouter (parent company Lore Hex, codebase formerly Quill) provides the opposite example: it's an OpenAI-compatible multi-model routing API whose core selling point isn't the routing itself, but making "prompts and outputs are never persisted to disk" something externally verifiable item by item, rather than relying on promises alone — the complete method is published on its trust.trustedrouter.com trust boundary page.

CASE / TRUSTEDROUTER · Three-cloud attestation chain + supply chain transparency log

Three different attestation chains, one common hard constraint

The gateway is deployed across GCP Confidential Space, AWS Nitro Enclaves, and Azure SEV-SNP, corresponding to three measurement methods: on GCP, comparing the image digest from the proof JWT issued by confidentialcomputing.googleapis.com with audience quill-cloud; on AWS, measuring the enclave image into PCR0 (SHA-384), where the official script verify-pcr0.sh lets clients rebuild the image from source and compare; on Azure, measuring the SHA-256 (hostdata) from the decoded CCE policy, reflected in the Microsoft Azure Attestation (MAA) token. The common prerequisite across all three chains is api.trustedrouter.com must use DNS-only or TCP passthrough — it cannot be TLS-terminated by any CDN — once terminated, the certificate private key leaves the measured confidential environment, and the trust chain breaks on the spot.

Release artifacts (image digests, PCR0, hostdata comparison table) are signed with cosign keyless and recorded into Sigstore's Rekor transparency log; signatures are bound to the specific GitHub Actions workflow identity that produced the file, not a broad repository regex — verifiers can independently verify offline using a command like cosign verify-blob --certificate-identity https://github.com/Lore-Hex/quill-cloud-proxy/.github/workflows/publish-trust-aws.yml@refs/heads/main. TrustedRouter also clearly documents what this mechanism does not cover: Confidential Space allows overriding an environment variable allowlist under the same image digest — the image digest locks down code, not every parameter at code startup; a signature only proves "who produced this record," not "whether this is the latest record" — to exclude replayed old measurements, you must check Rekor's append-only log, not just trust the website page itself.

GCP Confidential SpaceAWS Nitro Enclaves · PCR0Azure SEV-SNP · MAAcosign keyless + Sigstore RekorFail-closedEd25519 signed inference receipts (optional)

This practice of proactively disclosing boundaries perfectly illustrates this article's core judgment on TEE: remote attestation solves trust transfer, not trust elimination. TrustedRouter's increment isn't inventing new confidential computing primitives, but making every link of this transfer — who signed, what they signed, what step it covers — transparent for clients to verify item by item.

When the gateway forwards requests to downstream providers, it applies "attest first, encrypt second" a second time: when routing to providers like Chutes that offer TEE inference, the enclave first separately verifies fresh GPU attestations from NVIDIA (nras.attestation.nvidia.com) and Intel TDX evidence from Intel (api.trustedservices.intel.com), compares them against preset workload measurements, and only after confirmation encrypts the request with ML-KEM-768 (post-quantum key encapsulation) plus ChaCha20-Poly1305; if attestation fails at any link, the gateway only retries with a different verified instance — it never silently falls back to plaintext transmission — this is the concrete implementation of "fail-closed" on a multi-hop chain. It also offers an optional capability orthogonal to confidentiality: Signed Inference Receipts — the gateway uses an Ed25519 key generated at enclave startup and committed into the hardware attestation to sign a JWS over "the exact bytes of this request and response + the actual model and provider routed to," which can be verified offline without reconnecting to TrustedRouter. TrustedRouter draws this boundary equally clearly: receipts prove integrity and provenance, not confidentiality — a middleman who forwarded the request and read the plaintext can equally hold a valid receipt, and the receipt contains no requester identity. This distinction provides a sufficiently concrete product-level counterexample for two things — "verifiability" and "confidentiality" — that are often conflated.

Domestic TEE: CPU-Side Mature, GPU-Side Blank

CPU-side is commercially deployed at scale. Hygon CSV is currently the only domestically produced CPU confidential computing technology with commercial-scale deployment: memory encryption uses the national SM4 cipher, remote attestation uses SM2 signatures; CSV3 (Hygon C86-4, late 2023) added memory integrity protection, bringing capabilities on par with Intel TDX and AMD SEV-SNP. Alibaba Cloud has launched confidential VM instances based on Hygon CSV. Huawei Kunpeng iTrustee has obtained CC EAL4+ certification, with TEE secure memory configurable up to 128GB.

GPU-side is nearly blank — this is Xinchuang's biggest hardware bottleneck. As of 2025, no domestic AI accelerator (Ascend, Moore Threads, Biren, Cambricon, Tiansu Zhixin) has a commercial, productized GPU confidential computing mode like H100. The closest is the academic prototype ASCEND-CC from Huawei's Zurich Research Center and ETH Zurich (arXiv:2407.11888) — implementing NPU confidential computing on Ascend 910A by modifying firmware, with inference overhead of only 0.028%–0.91%, but it's a single-tenant research prototype, not a commercial feature.

The trust root problem is this route's original sin. TEE's entire security is staked on two things: the chip vendor has no backdoor, and the attestation system is trustworthy (for an American vendor like NVIDIA, this is itself a trust issue under Xinchuang); and side-channel defenses are adequate (SGX's history proves this is far from assured). In one sentence: TEE is "trust transfer," not "trust elimination" — it narrows trust from "the entire cloud vendor" down to "the chip vendor + a piece of verifiable code."

"TEE narrows trust from the entire cloud vendor to the chip vendor plus a piece of verifiable code. This is a huge advance, but it is trust transfer, not trust elimination."
— One of this article's core arguments
02

MPC: Distributing Trust Across Multiple Parties,
At a Cost of 10–100×

SECURE MULTI-PARTY COMPUTATION
Trust assumption: participating parties do not collude (between "trust chips" and "trust only math")

MPC splits data into secret shares, distributes them to multiple parties for collaborative computation, and no single party can see the original data. This is a weaker assumption than "trust the chip vendor" but stronger than "trust only mathematics."

Edge · CLIENT Multi-party · NON-COLLUDING PARTIES User Secret-shares input Reassembles result at end Party A Holds share #1 Cannot see original data Party B Holds share #2 Cannot see original data Share #1 Share #2 Every nonlinear operator requires multiple rounds of communication Trust = parties do not collude · Overhead 10–100× · Decisive bottleneck is communication (GB~hundreds of GB), not compute
MPC · Data is split into shares distributed to non-colluding parties; no one sees the plaintext; the cost is repeated communication between parties for every nonlinear operator — the bottleneck is bandwidth, not compute.

From General Frameworks to Transformer-Specific Protocols

Early work included general frameworks like ABY, SPDZ, and CrypTFlow. From 2022 onward, secure inference protocols targeting Transformers emerged densely: Iron, Cheetah, CipherGPT, until 2023's PUMA became a milestone — the first open-source MPC scheme that could securely infer pre-trained Transformers without modifying the model, first achieving LLaMA-7B-level MPC inference: ~5 minutes to generate 1 token. Then came the performance race: BumbleBee (NDSS 2025) cut matrix multiplication communication by 80–90%; Microsoft's SIGMA used function secret sharing + GPU acceleration to improve latency by another 11–19×, first achieving secure inference for GPT-class generative models.

Nonlinear operators are the main battlefield. GELU, Softmax, LayerNorm — these non-polynomial functions are extremely expensive on ciphertext. The entire optimization history of MPC-Transformer is essentially the history of "how to approximate nonlinear operators more cheaply."

5 min
PUMA: time to generate 1 token for LLaMA-7B under MPC (8-token input)
arXiv:2307.12533
5.64 GB
BumbleBee: communication for LLaMA-7B generating 8 tokens (~13.87 min, LAN)
IACR 2023/1678 · NDSS 2025
44 s
SIGMA: Llama2-13B secure inference time — at the cost of hundreds of GB of FSS preprocessing keys
IACR 2023/1269 · PETS 2024
6.7×
SHAFT LAN speedup over BOLT, saving 82% communication and integrating Hugging Face
NDSS 2025

Domestic Industry: An Area Where China Is Relatively Ahead

Notably, both milestone protocols — PUMA and BumbleBee — came from domestic teams and were made available externally through the open-source SecretFlow tech stack — China's engineering contributions sit in the global first tier on the MPC-Transformer sub-track. On the industry side, vendors like Huakong Qingjiao, Yifang Jianshu, and Guangzhishu are advancing LLM confidential compute productization, generally emphasizing crypto-evaluation / national-cipher compliance advantages; and "pure cryptographic schemes are too costly and need to work with trusted hardware" is already a widespread engineering consensus — MPC + TEE hybrid is widely seen as the realistic path to scale.

The fundamental bottleneck is communication, not computation. GB-level or even hundreds-of-GB data exchange is a decisive bottleneck under WAN. This means MPC in the near term is only suitable for narrow scenarios with "good multi-party bandwidth, tolerable minute-level latency, and strong compliance." GPU-accelerated MPC (SIGMA being representative) is currently the most promising direction for changing the overhead curve.

03

FHE: Trust Only Mathematics,
At a Cost of 3–5 Orders of Magnitude

FULLY HOMOMORPHIC ENCRYPTION
Trust assumption: only "the math problem holds" — no trust in chips, no trust in multiple parties

FHE allows arbitrary computation directly on ciphertext: the client encrypts, the server computes, the client decrypts — the server never sees plaintext. The trust assumption is cleanest, the overhead most terrifying — DARPA's phrasing: a computation that takes 1 millisecond on a laptop would take weeks with today's FHE.

Edge · CLIENT Cloud · SERVER (sees only ciphertext) User Plaintext → Encrypt → Upload Download → Decrypt → Plaintext Private key stays only on edge Server Computes directly on ciphertext CKKS / BFV / TFHE Never sees plaintext Ciphertext Enc(x) Ciphertext Enc(f(x)) Trust = only the math problem (LWE/RLWE) — no trust in chips, no trust in multiple parties Cost: full LLM inference overhead 10³–10⁵× · What can land today is PIR / encrypted retrieval, not neural networks
FHE · The server computes blindly on ciphertext, never decrypting; the trust assumption is cleanest (only mathematics); the cost is 3–5 orders of magnitude overhead, and real-world deployment still stops at lightweight tasks like encrypted retrieval.

Research Frontier and Real-World Overhead

CKKS, BFV/BGV, and TFHE are the three mainstream schemes. Non-interactive HE inference for Transformers is a research hotspot in 2024–2026: NEXUS (NDSS 2025) is among the first to implement non-interactive Transformer inference using RNS-CKKS, but BERT-base (128 tokens) takes ~1103 seconds — and research indicates its nonlinear approximation has a maximum relative error as high as 297%. Even with specialized approximation and GPU implementations, FHE overhead for full LLM inference remains in the thousands-to-hundreds-of-thousands range.

Hardware Acceleration Is FHE's Only Hope

DARPA's DPRIVE program funded four teams — Intel, Duality, SRI, and Galois — to build FHE ASIC accelerators, aiming to pull overhead within one order of magnitude of plaintext; Intel's ASIC with Microsoft claims ~million× acceleration. Cornami takes reconfigurable many-core, Optalysys uses optical computing, Niobium has taped out, and Fabric Cryptography's chip is expected to tape out within a year. Whether the 1000–5000× improvement claimed by these accelerators is "enough" remains an open question.

1103 s
NEXUS time for one BERT-base (128 token) non-interactive HE inference
NDSS 2025
10³–10⁵×
FHE overhead range for full LLM inference relative to plaintext — still orders of magnitude away from real-time
Synthesized from multiple papers
$1B+
Zama 2025 Series B valuation — FHE's first unicorn, but main battlefield is confidential blockchain, not LLM
2025.06 SERIES B
PIR
Apple iOS 18 Live Caller ID: BFV + Private Information Retrieval — the best footnote for FHE's current real capability boundary
SWIFT-HE, APACHE 2.0

Deployment reality: What FHE can land today is "small and beautiful" retrieval, not LLM inference. Apple uses HE for encrypted database queries (caller ID lookup without exposing the queried number), not for running neural networks with HE. Zama became FHE's first unicorn at a $1B+ valuation, but its FHEVM on-chain throughput is still only dozens of transactions per second. This itself shows: FHE's industrialization breakthrough is in lightweight, high-value, latency-tolerant scenarios. Hybrid schemes (HE for linear operators only, nonlinear handed to MPC or TEE) are a more pragmatic engineering path — BOLT and BumbleBee are essentially HE/MPC hybrids.

Engineering Hybrid Route: Operator Separation + CPU-TEE Heterogeneity

Pulling homomorphic encryption from "theoretical upper bound" toward engineering usability, a noteworthy hybrid route has emerged domestically — Jinghua Confidential Compute's confidential training/inference platform is representative. Its approach can be summarized in three sentences: First, linear and nonlinear operators are computed separately, structurally splitting apart the most cryptographically expensive parts; Second, CPU-TEE and GPU heterogeneous scheduling — key management and the sensitive control plane land in the CPU TEE (millisecond-level random key generation and destruction), while large-scale matrix operations go to the GPU; Third, encryption granularity is at the request level, with each inference request randomly encrypted and computed token-by-token in ciphertext, so inputs and results are visible only to the requestor.

CASE / Jinghua Confidential Compute · HE + TEE Hybrid (Operator Separation)

An Engineering Exemplar of Homomorphic Encryption: Full-Pipeline Confidential Training and Inference

Unlike most schemes that only do confidential inference, Jinghua covers confidential training (ciphertext model training, ciphertext retrieval, confidential vector DB), and claims that while adapting to mainstream domestic GPUs, both confidential training and inference time overhead relative to plaintext are controlled within 3× — if independently verified, this would be significantly below the pure MPC (10–100×) and pure FHE (10³–10⁵×) ranges, at the cost of introducing hardware trust in CPU-TEE (deployment requires CPU TEE support). Deployment form factors include confidential compute appliances (inference: dual-socket CPU + 4 GPUs; training: dual-to-quad-socket CPU + 8 GPUs) and software deployment; publicly disclosed projects include the Lecheng Healthcare Confidential Training Trusted Space and the Yangtze River Delta Healthcare Pilot Zone platform, with CAICT confidential standards and Level 4 classified protection evaluation progressing in 2026.

Operator separation: linear / nonlinearCPU-TEE + GPU heterogeneousRequest-level random encryptionClaims ≤3× (pending independent verification)Adapts to domestic GPUsCovers confidential training

To be objective: "≤3×" is a vendor claim lacking published third-party benchmarks; and this route's trust assumption is no longer "trust only math," but a hybrid of "math + CPU chip vendor" — this is the price it pays for performance, and why it's covered in this chapter rather than given its own.

The industry significance of this hybrid route is: it demonstrates that HE's realistic path in LLM scenarios isn't "wait for ASIC to save pure FHE," but use cryptographic guarantees where they matter most, and hand performance-sensitive parts back to hardware trust — essentially creating a new gear in the middle of the trust-pricing ladder.

04

Split Inference: Near-Zero Overhead,
But Weakest Guarantees

SPLIT INFERENCE / EDGE-CLOUD
Trust assumption: "intermediate features are hard to invert" — empirical, not cryptographic

Run an encoder locally, send only the compressed intermediate features to the cloud large model — raw input never leaves the device. Overhead is nearly zero — it's just normal neural network forward computation.

Edge · CLIENT (original image stays on device) Cloud · BIG MODEL Local Encoder ViT · 300M · Metal GPU Raw pixels Never leave device Cloud Large Model GLM-4.5V · Visual understanding + generation Receives only feature vectors Feature vector ~1MB (3× Gzip) Feature inversion: FIA-Flow / CapRecover can partially reconstruct original image from features Overhead ≈ 0 (just one forward pass) Guarantee is empirical, not cryptographic — cost-driven approach; can only be auxiliary layer for high-sensitivity scenarios
Split Inference · Original image stays on-device, only compressed features are uploaded; overhead is near zero, but features can be partially reconstructed via inversion attacks — this is an empirical, not cryptographic, guarantee.

Real example: GLM-4.5V (released and open-sourced by Zhipu in August 2025, 106B total / 12B active params) consists of a visual encoder, MLP adapter, and language decoder — the visual encoder can be placed on the edge side, uploading only compressed visual features, making it a natural fit for split inference in multimodal scenarios. Apple Intelligence's "edge-first, go to cloud only when necessary via PCC" tiered architecture is essentially also a split-inference design.

CASE / VIT SECURE INFERENCE · Edge-Cloud Synergistic Privacy Inference

Images Locally, Intelligence in the Cloud: The Open-Source Implementation of This Approach

The companion open-source implementation of this project, VIT Secure Inference (GitHub · MIT), turns this route into an installable product: on a local Mac, a 300M-parameter ViT visual encoder (Metal GPU accelerated) extracts image features, which are Gzip-compressed and only ~1MB of feature vectors are uploaded to the cloud GLM-4.5V for visual understanding and text generation — raw image pixel data never leaves the device, supporting OCR, document/financial report analysis, multi-image comparison, and medical imaging tasks.

Local ViT · 300M paramsUploads only ~1MB feature vector3× Gzip compression0 raw pixel uploadsCloud GLM-4.5VMIT open-source

Honest boundary note: ViT encoding is a one-way, heavily information-compressing process that significantly raises the reconstruction barrier, but per the feature inversion research below, this is still an empirical guarantee, not a cryptographic one — which is why it's classified under this route, and why we recommend adding noise/obfuscation for high-sensitivity scenarios.

Fatal weakness: feature inversion attacks. "Send features, not the original image" sounds safe, but extensive research proves intermediate representations can be partially or even high-fidelity reconstructed: FIA-Flow (2025) trains a black-box inversion model using less than 0.32% of ImageNet-1K samples, achieving high-fidelity reconstruction of original images; CapRecover (ACM MM 2025) can directly reconstruct text descriptions of images from visual features; for LLMs, Vec2Text-style embedding inversion work also exists. Defenses (adversarial training, feature sparsification, noise addition) all involve accuracy-privacy tradeoffs, and none provides cryptographic-level formal guarantees.

Positioning conclusion: Split inference is a "cost-driven" rather than "guarantee-driven" approach. It suits consumer-grade scenarios with moderate privacy sensitivity and extreme sensitivity to latency and cost; for any scenario involving finance, healthcare, or government that requires provable confidentiality, it can only serve as an auxiliary layer, not the primary defense line.

⊞

Four-Route Comparison Table

SIDE-BY-SIDE COMPARISON
RouteTrust RootRelative Plaintext OverheadFormal GuaranteeRepresentative Scheme / ProductXinchuang FitTypical ScenarioMaturity
TEE / GPU CC Chip vendor + verifiable code <10% H100 average <7%, Blackwell approaching zero Medium Depends on hardware + remote attestation NVIDIA H100/Blackwell CC · Apple PCC · Anthropic Confidential Inference (research sketch) · TrustedRouter (three-cloud attestation gateway) Weak Trust root is NVIDIA; domestic GPU TEE is blank General cloud confidential inference, cross-jurisdiction deployment Commercially deployed at scale
MPC Multi-party non-collusion assumption 10–100× Communication in GB range, bottleneck is bandwidth High Cryptographic PUMA · BumbleBee · SIGMA · SecretFlow open-source stack Strong Pure software runs on domestic GPUs, easy crypto-evaluation pass Finance / healthcare cross-institution data fusion Narrow-scenario deployment
HE+TEE Hybrid
Operator Separation
Math + CPU chip vendor (hybrid) Claims ≤3× Vendor figure, pending independent verification Fairly High Primarily cryptographic + TEE control plane Jinghua Confidential Compute confidential training / inference platform (appliance + software deployment) Strong Adapts to mainstream domestic GPUs, national-cipher certification in progress Trusted data spaces, confidential training, cross-border data circulation Early commercial stage
FHE Trust only mathematics 10³–10⁵× Hardware acceleration is the only hope Highest Cryptographic NEXUS · Zama TFHE · Apple swift-HE (PIR) Medium Algorithms can be national-cipher adapted, performance not yet usable PIR-class lightweight encrypted retrieval Lightweight retrieval only
Split Inference "Features hard to invert" (empirical) ≈ 0 Weakest Vulnerable to feature inversion attacks GLM-4.5V edge-cloud synergy · VIT Secure Inference · Apple edge-cloud tiering No specific hardware dependency Consumer-grade / low-sensitivity default cost-reduction layer Consumer-grade auxiliary
05

The China Market's Particularity:
Xinchuang, Crypto-Evaluation, and Fallback to Self-Build

THE CHINA STRUCTURE

The China market is not a simple rescaling of the global landscape — it has three unique structural constraints that together give rise to China's distinctive "fallback to self-build" logic.

The "American Vendor" Paradox of GPU TEE Trust Roots

The de facto standard for global cloud confidential inference is converging on NVIDIA GPU CC, but its trust root (remote attestation, chip keys) is entirely in NVIDIA's hands. You use confidential computing to avoid trusting the compute provider, yet you must unconditionally trust an American chip vendor's attestation system.

GPU Confidential Computing on Domestic Cards Is Blank

On domestic hardware, confidential inference in the near term can only rely on software cryptographic schemes (MPC) running on domestic GPUs/CPUs, or on Hygon CSV / Kunpeng TEE for VM-level protection — unable to replicate NVIDIA's low-overhead GPU confidential inference path. This in turn reinforces the "MPC + TEE hybrid with national ciphers" technology choice.

Crypto-Evaluation and National-Cipher Compliance Dividend

GB/T 43206-2023 crypto-evaluation standard took effect in April 2024,叠加 the "Data Factor ×" three-year action plan's policy drive — schemes using national ciphers (SM2/SM3/SM4) gain natural compliance advantages in strongly regulated scenarios like government and finance, a local moat that international vendors cannot replicate.

The combined result of these three forces: for institutions with the highest confidentiality requirements (major banks, top-tier hospitals, government clouds), rather than wrestling with the confidential inference trust chain on public clouds, it's better to directly deploy domestic hardware + domestic confidential schemes privately — the cost is higher ops overhead and limited external service efficiency, but under current technical constraints this is a structurally rational choice. The policy side is trying to pull this "fallback" demand back into trusted cloud-based confidential circulation: the National Data Bureau's "Trusted Data Space Development Action Plan (2024–2028)" explicitly names high-performance confidential computing; on the deployment side, per vendor disclosures, the Lecheng Healthcare Confidential Training Trusted Space (the country's first trusted space supporting AI confidential training on medical datasets) and the Yangtze River Delta Healthcare Pilot Zone platform both use "compute-without-decrypting" as the technical foundation for solving cross-domain trust — with a build-out target of covering 30 leading pharma companies, 2,800 key hospitals, and 1,000+ data entities in the Yangtze River Delta by 2027.

Market Size: Forecast Divergence Is Itself a Signal

On the demand side, real buyers are highly concentrated in four categories: finance (risk control, cross-institution data fusion), healthcare (medical records, genetic data cross-center collaboration), government (data factor circulation), and cross-border (data export compliance). On the supply side, market forecasts diverge enormously: Grand View Research gives ~$153.8B by 2030 (CAGR ~64%), Precedence Research gives ~$1.28T by 2034, while conservative forecasts from TechSci et al. put 2030 at only $10–23B. A forecast divergence of tens of times is itself a signal: this market is still early, metrics are chaotic, and consensus hasn't formed. The rational reading is to focus on the growth direction (high-speed) rather than absolute numbers.

"Under Xinchuang constraints, confidential inference currently has no cheap cloud solution. Falling back to self-build is structurally forced."
— Core judgment of the China Market chapter
06

Selection Guide: Follow the Decision Tree

DECISION-TREE PLAYBOOK

Selection doesn't need a new framework — the decision tree at the opening is the action guide. Below, each of the four decision nodes is expanded: every branch gives the endpoint, action items, and the threshold that triggers fallback.

Q1Is this use case's data sensitive?
No →
Vendor Cloud Platform▸Call API directly; don't pay for security you don't need. Action item: implement data classification at the gateway layer, making the "sensitive / non-sensitive" judgment automated and auditable — classification is the entry point for the entire tree, worth investing in heavily.
Yes →
Proceed to Q2. Note the decision object is the use case, not the client: for the same client, 80% of traffic can usually take the cheaper path above.
Q2Do you have your own GPU compute?
Yes →
Internal Private Deployment▸In most cases, stop here. Only add CPU-TEE or a confidential compute layer when there are internal isolation needs (group multi-entity data isolation, board-level confidentiality from IT admins, classified requirements).
No / Not enough →
Proceed to Q3, consider external cloud services.
Q3Are trust mitigation measures sufficient? (Desensitization gateway / dedicated instance / contractual no-training promise)
Sufficient →
Desensitization + Dedicated Instance▸The highest-share tier in real transactions, costing a fraction of confidential computing. Action item: productize the desensitization rules and dedicated instance contract terms as the default quote item for "concerned but budget-limited" clients.
Insufficient →
The client needs provable confidentiality, proceed to Q4.
Q4Are there Xinchuang requirements?
No →
GPU TEE First▸Overhead <10%, evidence is ample, can directly benchmark against the Apple PCC narrative (Anthropic's Confidential Inference is currently a research sketch, usable as a design reference). Action item: include GPU CC mode as an option in enterprise / financial editions, and make "remote attestation" a product feature clients can self-verify. For narrow scenarios with strong compliance and tolerable minute-level latency, upgrade to MPC / HE as needed.
TRIGGER / Fallback ThresholdClient threat model explicitly includes "do not trust chip vendors / do not trust American hardware" → process per the "Yes" branch, even if the client has no nominal Xinchuang requirement.
Yes →
Domestic Confidential Compute Menu, Pick One of Three▸Trade off by latency tolerance and compliance depth: ① MPC on domestic GPUs — pure software, easy crypto-evaluation pass, accept 10–100× overhead and minute-level latency, suitable for cross-institution data fusion; ② HE+TEE hybrid (operator separation) — represented by Jinghua Confidential Compute's appliance / software deployment route, claims ≤3× and covers confidential training, recommend verifying with POC before scaling; ③ CPU-TEE (Hygon CSV) + domestic accelerator — VM-level protection, minimal engineering changes, but no hardware confidentiality on the GPU side.
TRIGGER / Fallback and Inflection PointSingle-token latency can't fit within business bounds (conversational needs second-level) → fall back to desensitization tier or self-build; once a domestic card ships commercial GPU TEE (track Ascend CC productization timeline) → decisive inflection point for Xinchuang confidential inference, deploy immediately.
+Overlay Layer and Tracking Items (applies to all branches)
Overlay →
Split Inference / Edge-Cloud Synergy▸A cost-reduction layer prependable to any path (local encoding, features to cloud; implementation reference: VIT Secure Inference). Must clearly disclose to internal stakeholders and clients that its guarantee is empirical, and add noise / obfuscation to intermediate features — never use it as the primary defense line in finance, healthcare, or government scenarios.
Track →
Don't Bet on Pure FHE Yet▸Keep tracking the technology; PIR-class lightweight retrieval (privacy-preserving knowledge base recall) can be trialed at small scale.
TRIGGER / Re-evaluation ThresholdFHE hardware acceleration (DPRIVE-series ASICs, Zama ecosystem) can pull LLM inference overhead below 100× → re-evaluate for investment.
⚠

Caveats: Keep These in Mind While Reading

EPISTEMIC STATUS

Update Log

3 versions total
  1. v3 Corrected Anthropic's description: its "Confidential Inference via Trusted Virtual Machines" (2025-06) is a research sketch, not a deployed system, and the original text did not claim to assume the service provider is adversarial; synchronized corrections to TL;DR, industry deployment, comparison table, and decision tree, and rewrote the "verification process is closed internally" phrasing per Apple's design specification (synced in both Chinese and English) View v3
  2. v2 Added TrustedRouter case to TEE section: three-cloud attestation chain (GCP/AWS/Azure), cosign+Sigstore supply chain signatures, Chutes TEE-to-TEE cross-verification, signed inference receipts View v2
  3. v1 Initial release: evolution, measured overhead, industry deployment, and China market structural contradictions of four LLM confidential inference technical routes (TEE/MPC/FHE/Split Inference) View v1