IOSG：AI的十字路口，為什麼華爾街們正在向ChatGPT與Claude們說「不」？

Original Author: Jeff, IOSG Ventures

Why We Need Private AI

On July 1st, Palantir CEO Alex Karp delivered a 20-minute interview on CNBC that some media outlets described as a "meltdown." According to Karp, enterprises are paying premium token prices to frontier labs while watching their own IP flow to model providers. He called this leakage a transfer of alpha, happening at the architectural layer: every request sent to a closed-source model arrives at the provider's server in plaintext. Just days before the broadcast, Palantir had announced a partnership with NVIDIA to run the open-source Nemotron model in customer-controlled environments, accompanied by a nine-point AI sovereignty manifesto. Following the CNBC appearance, PLTR's stock jumped 8%.

For the past two decades, enterprises adopted cloud software based on trust at the protocol level, and it worked. Each SaaS vendor only saw a slice of enterprise data, and most had little incentive to use customer data to fuel their core products. Salesforce saw sales pipelines, Workday saw HR data, Jira saw development iterations, and AWS provided the storage and compute foundation. However, today's AI workflows demand uploading all assets at once, along with the structured context connecting various departments, to maximize productivity. Putting goodwill aside, upstream service providers can now use this data to develop new features rather than letting it sit idle on servers.

No one is slowing down. Anthropic's annualized revenue reached $470 billion in May, a massive jump from $90 billion at the end of 2025, while OpenAI surpassed 900 million weekly active users in February. Both companies completed new funding rounds this spring, nearing a $1 trillion valuation, and are expected to IPO at even higher market caps. Years of privacy and IP allegations haven't dented their momentum.

Some enterprises have already taken action. In February 2023, within three months of ChatGPT's release, major Wall Street banks restricted its use. In May 2023, after Samsung engineers leaked chip source code into ChatGPT, the company banned generative AI across its network. In response, OpenAI launched ChatGPT Enterprise in August 2023, promising not to train on business data and offering a zero-data-retention (ZDR) protocol, which has since become a standard enterprise procurement requirement.

But contracts only lock down company accounts. IBM found that by 2025, shadow AI (employees feeding company data into unauthorized AI tools via personal accounts) was involved in one-fifth of data breaches, with heavy shadow AI use adding an average of $670,000 to breach costs. In a 2025 survey by security training firm Anagram, 40% of employees said they would violate AI use policies to complete tasks faster.

Enterprises can at least buy their way out with ZDR contracts and non-training service tiers, and sovereign deployment is available if you're a government or Palantir client. But for ordinary users like you and me, the importance of private AI is still debated—until a court subpoena arrives.

A court order in May 2025 forced OpenAI to retain even deleted consumer chats. In November, a judge ordered the transfer of 20 million of these chats to the New York Times lawyers as discovery material. Then came criminal cases: ChatGPT records of a defendant in the Palisades fire arson case entered evidence, and an affidavit in a Florida double homicide case cited the suspect's questions about how to dispose of a body. Sam Altman also admitted in a July 2025 interview that ChatGPT conversations are not protected by legal privilege and that OpenAI "could be required to hand over" user chat logs in litigation.

The point is not that only criminals need private conversations. The fact that people's conversations with AI are archived and subject to subpoena represents a surveillance blind spot most users are unaware of. A Kolmogorov Law survey of 1,000 US AI users in October 2025 found that 50% were unaware these conversations could be subpoenaed, while two-thirds believed these chats should receive the same protection as consultations with a lawyer or doctor.

Self-hosted or open-source models running in verifiable environments are catching up quickly, but the strongest ones still lag behind frontier closed-source models by about 4 months in general capabilities. This puts token-maximizing enterprises and individuals at a crossroads: either sacrifice a few months of model quality for privacy, or continue sending sensitive materials to Anthropic's servers because competitors are using them to gain a productivity advantage.

There is no perfect solution on the market today. This report outlines various attempts to narrow the gap and examines how far frontier intelligence with provable privacy is from reaching enterprises and ordinary users.

How Privacy is Achieved Today

Private AI is not a single engineering feat, but each mechanism on the market handles the same event: a prompt leaves your device, traverses the network, lands on the machine running the model, and returns a response. The difference between mechanisms lies in where plaintext exists on this path, who can read it there, and what verifies the privacy of the response.

Protocol-Level Privacy

At this level, someone besides you can still read your plaintext prompt. What happens next relies entirely on a promise.

• Contractual zero-retention is the enterprise-grade solution. The provider knows who you are, processes your prompt, and promises not to retain it, enforced by contract and reputation.
• Anonymous proxies erase who you are but not what you said. Downstream providers still handle plaintext according to their own policies. Terms vary: proxies like Duck.ai (DuckDuckGo's chatbot product) negotiate deletion agreements with model providers, while Venice directly lets users assume providers will store everything—but neither can be verified.

Every segment of machine-to-machine communication runs over TLS, which only encrypts the pipe; the receiving party can read all information. Relays typically use Oblivious HTTP (RFC 9458) to decouple this knowledge, similar to asking a friend to pass a note. The friend knows who sent it but can't read the content, and the recipient can read the content but doesn't know who wrote it. OHTTP has been an IETF standard since January 2024, and many companies now run production traffic through OHTTP relays rented from Cloudflare and Fastly.

This represents the maximum privacy attainable when accessing closed-source models, and the reason is a simple arithmetic problem. A single flagship training run now costs in the billions of dollars, and these labs' near-trillion-dollar valuations are staked on the exclusivity of their model weights. The premium persists as long as the capability gap exists, so labs guard their weight files like state secrets.

Meta has already passively experienced this experiment. LLaMA, released in February 2023, was initially available only to researchers, but within a week, its weights leaked via a torrent to 4chan. A week later, llama.cpp allowed the smallest 7B model to run locally on a MacBook. Three days after that, Stanford fine-tuned the same model into the Alpaca chatbot assistant for under $600. This leakage reduced LLaMA's operating cost to just electricity—anyone with the file could run it at home. In July 2023, Meta officially open-sourced LLaMA 2 under a commercial license with a 700 million monthly active user exclusion clause. The weights ran, and so did the premium.

Frontier labs could theoretically provide attestation for closed-source model inference, but attestation can only prove which code read the prompt, not what that code did with it. To determine whether a server retained data, one would need to audit the serving code and reconstruct it to match the hash reported by the hardware. But once the serving code is surrendered, the lab also gives up the batching and caching techniques that underpin its profit margins—techniques that will transfer to every future generation of models. Apple and Meta can provide remote attestation for the service stacks behind iPhone and WhatsApp because their profits lie in devices and advertising, making public serving code nearly costless.

This is why the weights and serving code of flagship models cannot reach external operators. Without external operators, there is no third-party attestation. Without attestation, verifiable privacy exists only on open-source models.

Structural-Level Privacy

Each mechanism in this category replaces trust promises with proofs based on hardware, cryptography, or physics. However, each requires a different cost for the privacy upgrade, the primary one being that they can only run open-source models.

TEE (Trusted Execution Environment) confidential computing runs inference inside a hardware enclave (a sealed compartment on the chip that even the machine operator cannot open). The chip signs an attestation stating exactly which model and code were run.

The prompt is only sealed at the endpoint. Along the path relayed via the platform proxy, there remains a role capable of reading plaintext. Only the protocol prevents the proxy from recording or leaking the relayed content.

E2EE (End-to-End Encryption) seals the readable relay. The user's device encrypts the prompt using the enclave's key. Each hop in between carries a sealed envelope that only the enclave can open.

• Trust falls on the client side. The code responsible for encrypting the prompt and verifying the attestation also has the power to revoke this guarantee. Therefore, verifiable E2EE requires not only a proven enclave but also open, reproducible client code.
• Compared to the simplicity of TEE, E2EE's cost is engineering burden, which slows down feature integration. E2EE turns the proxy into a blind messenger, so all functions that rely on reading plaintext must be rebuilt around the client-side key or reconstructed only within the enclave.

FHE (Fully Homomorphic Encryption, and MPC variants) eliminates the trusted party altogether. The server performs computations on ciphertext within a locked box it can never open; the key remains with you. MPC (Multi-Party Computation) splits the prompt into secret shares distributed among multiple parties, creating an equivalent effect unless all parties collude.

• The cost is speed. FHE natively only performs addition and multiplication, so the non-linear steps required for transformer operation must be reconstructed at a high cost. Inference on ciphertext costs 10,000 to 100,000 times more than on plaintext, taking seconds to minutes per token on small models compared to milliseconds for unencrypted computation.
• Chips customized for encrypted computation could close the gap, but the first prototype only completed demos in early 2026, with commercial versions still years away.

Local inference eliminates the path entirely. The model runs on your own hardware: no relays, no servers, no providers, and no verification requirements.

• The obvious costs are capability and expense. gpt-oss-120b scores about half of GLM-5.2 on the Artificial Analysis index but takes up 65GB, exceeding the combined VRAM of two top-tier consumer graphics cards. A full-precision GLM-5.2 can only run on an 8-GPU data center node, costing over $300,000 just for the GPUs.

However, beyond these structural constraints, the cost of putting inference into an enclave is decreasing. For single-GPU inference, benchmarks from enclave cloud provider Phala show that H100 throughput loss in enclave mode averages less than 7%, and approaches zero for large models because the primary cost is moving data into the chip, not computing inside it. For multi-GPU inference, NVIDIA's next-generation GPU Blackwell supports direct encryption of inter-chip traffic, while the older H100 could only achieve the same effect by routing through the CPU host at one-seventh the bandwidth. NVIDIA's own benchmarks on Blackwell show less than 8% throughput loss for a 397B model in enclave mode. With these advances, the performance overhead of private inference itself is no longer a decisive constraint.

In fact, the enclave itself adds almost no extra operating cost for the operator. Every H100 produced after 2023 comes with an enclave mode natively; the extra cost is the throughput loss from encryption, not an additional chip. Currently, the confidential H100 SKU rental price on Azure is still $8.90 per hour, compared to $6.98 without enclave—a 27% premium over traditional cloud infrastructure. On the other hand, with specialized enclave providers like Phala, confidential-mode H100s are renting from $3.80 per hour, lower than Lambda's standard SXM card price range of $3.99 to $4.29. For managed API solutions, NEAR AI offers gpt-oss-120b with attestation endpoints at $0.15 per million input tokens and $0.55 per million output tokens, on par with plaintext routes like Amazon Bedrock, Together, and Groq. Even for models requiring multi-chip parallelism, NEAR AI's pricing on GLM 5.2 matches Fireworks exactly, and on the larger Kimi K2.6, its input is 15% cheaper and output 4% cheaper.

While these new private inference providers might be burning profits to grab market share (a statement true for any company seeking growth in this market), the structural trend is that the cost of privacy is decreasing for both consumers and operators.

How Open-Source Models Can Win

Despite the shrinking performance overhead, there remains a visible gap between frontier models and state-of-the-art open-source models. An entity pursuing maximum productivity still needs to trust that frontier labs won't steal their IP to stay at the forefront.

The gap exists, but on June 30th, Bridgewater's AIA Labs and Thinking Machines provided a case study: an open-source model fine-tuned with expert annotations outperformed frontier models in both accuracy and cost simultaneously.

In the study, the team fine-tuned Qwen3-235B on Tinker (Thinking Machines' managed fine-tuning API service). They first sourced annotations from vendors, trained an initial round, then handed divergence samples back to the company's investment professionals for re-annotation. The training used reinforcement learning (GRPO) with three modifications: round-robin batching (one batch per task in rotation), CISPO loss (limiting how far a single answer could pull the model), and on-policy distillation (anchoring to the current best checkpoint to ensure the model doesn't learn from weaker copies).

The tasks were all taken from investment professionals' daily workflows: determining if a news article is important for a C-suite investment professional, whether a central bank document hints at future interest rate direction, and where boilerplate template language starts in a document or email. Scoring came from an independent test set. Frontier models averaged around 50% with simple prompts and only reached 78.2% with expert prompts, still below the investment professionals' 80% threshold. The fine-tuned Qwen scored 84.7%, which according to the report translates to 29.8% fewer errors than the best frontier model with 13.8x lower inference cost.

https://thinkingmachines.ai/news/learning-to-replicate-expert-judgment-in-financial-tasks/

This case proves that open-source models can win on accuracy and cost, but the training process itself was not private. The expert annotations used were Bridgewater's proprietary data, passing through Tinker's third-party service, landing at the same trust level as a ZDR agreement. The fund also rented compute power, with the entire training run happening on machines it never controlled. Buyers wanting this formula without the trust assumptions have few options today. Renting a bare GPU cluster leaves the training process readable to the cloud operator. Buying the cluster solves the data hosting problem, but costs skyrocket.

The attestation-based route has just arrived. In March, Workshop Labs and Tinfoil released Silo, a post-training stack running inside a Tinfoil enclave on a single 8-GPU node, with keys controlled solely by the customer. The article reported enclave overhead as just 11 minutes for a two-hour training run. Additionally, by freezing the base model weights and only training a small adapter on top, the stack can fit a trillion-parameter model (Kimi K2 Thinking). The challenge is that reinforcement learning requires moving data back and forth between components, and moving data is precisely where enclave costs lie.

Less than a month after Silo's release, Workshop Labs was acquired by Thinking Machines. The components needed to run a Bridgewater-style RL loop inside an enclave are now all under the same roof.

Privacy at the Harness Layer

There's another problem that lies outside all private inference mechanisms. Each mechanism manages the path from the prompt to the model. However, every external tool call initiated by an agent opens a path the inference layer cannot touch. The recent trend of harness engineering amplifies this issue exponentially: every tool, memory bank, and data source connected around the model represents another destination reading its slice of the workflow in plaintext. A calendar server reads appointments, a database server reads queries. A fully local agent, if it needs anything beyond its training set, still must send search terms in plaintext to the search engine. If the server cannot read plaintext, it cannot answer the question.

Mainstream solutions still default to the protocol layer. Companies like Runlayer and MintMCP use a central gateway to manage all tool traffic, masking personally identifiable information (PII) before the request leaves. The gateway also decides which servers can receive traffic, blocking unvetted ones, and logs every invocation's destination and content for forensics. Even if these controls carry independent audits (SOC 2), the tool server still needs to read the plaintext query to answer.