IOSG: AI at a Crossroads — Why Wall Street is Saying "No" to ChatGPT and Claude?

Original author: Jeff, IOSG Ventures

Why Private AI is Necessary

On July 1st, Palantir CEO Alex Karp delivered a 20-minute interview on CNBC that some media described as a "meltdown." According to Karp, companies are paying a token premium to frontier labs while watching their own IP flow to model providers. He called this leakage an alpha transfer, occurring at the architectural layer: every request sent to a closed-source model arrives at the service provider's server in plain text. Just days before the broadcast, Palantir announced a partnership with NVIDIA to run the open Nemotron model in customer-controlled environments, accompanied by a nine-point AI sovereignty manifesto. Following the CNBC segment, PLTR jumped 8%.

Over the past two decades, enterprises adopted cloud software based on trust at the protocol level, and it worked. Each SaaS vendor saw only a slice of enterprise data, and most had little incentive to use customer data to feed back into their core products. Salesforce saw sales pipelines, Workday saw HR, Jira saw development iterations, and AWS provided the foundation for storage and computing. However, today's AI workflows demand uploading all assets at once, along with the structured context connecting different departments, to maximize productivity. Setting aside goodwill, upstream providers can now use this data to build new features rather than letting it sit idle on servers.

No one is slowing down. Anthropic's annualized revenue reached $47 billion in May, a massive jump from $9 billion at the end of 2025, while OpenAI surpassed 900 million weekly active users in February. Both companies completed new funding rounds this spring, with valuations approaching $1 trillion, and are expected to IPO at even higher market caps. Years of privacy and IP allegations have not dented their momentum.

Some enterprises have already taken action. In February 2023, less than three months after ChatGPT's launch, major Wall Street banks restricted its use. In May 2023, after Samsung engineers leaked chip source code into ChatGPT, the company banned generative AI across its entire network. In response, OpenAI launched ChatGPT Enterprise in August that year, promising not to train on business data, along with a zero-data-retention (ZDR) protocol, which has since become a standard requirement for enterprise procurement.

But contracts only lock down company accounts. IBM found that by 2025, shadow AI (employees feeding company data into unauthorized AI tools via personal accounts) was involved in one-fifth of data breaches, and heavy shadow AI usage added an average of $670,000 to breach costs. In a 2025 survey by security training company Anagram, 40% of employees said they would violate AI usage policies to complete tasks faster.

Enterprises can at least buy their way out with ZDR contracts and no-training service tiers, or if they are government or Palantir clients, sovereign deployment. But for ordinary users like you and me, whether privacy AI matters is still debated—until a court subpoena comes knocking.

A court order in May 2025 forced OpenAI to retain even deleted consumer-level chats. In November, a judge ordered the transfer of 20 million of these chats to *The New York Times* lawyers as discovery material. Then came criminal cases: the ChatGPT records of the defendant in the Palisades arson case became evidence, and an affidavit in a Florida double homicide case cited the suspect's questions about disposing of bodies. Sam Altman also admitted in a July 2025 interview that ChatGPT conversations are not protected by legal privilege, and in litigation, OpenAI "may be required to hand over" user chat records.

The point is not that only criminals need private conversations. People's conversations with AI are archived and subject to subpoena—a surveillance surface most users don't know exists. A Kolmogorov Law survey of 1,000 U.S. AI users in October 2025 found that 50% were unaware these conversations could be subpoenaed, while two-thirds believed these chats should receive protections equivalent to consulting a lawyer or doctor.

Open-source models that are self-hosted or run in verifiable environments are rapidly catching up, but the strongest ones still lag behind frontier closed-source models by about four months in general capabilities. This puts token-maxxing enterprises and individuals at a crossroads: either sacrifice a few months of model quality for privacy, or continue sending sensitive materials to Anthropic's servers, because competitors are precisely stealing a productivity advantage this way.

Currently, there is no perfect solution on the market. This report examines various attempts to close the gap, observing how far provably private frontier intelligence is from being delivered to enterprises and ordinary users.

How Privacy is Currently Achieved

Private AI is not a single engineering problem, but each mechanism in the current market addresses the same event: a prompt leaves your device, traverses the network, lands on the machine running the model, and returns a response. The difference between mechanisms lies in where plaintext exists along this path, who can read it there, and what verifies the privacy of the response.

Protocol-Level Privacy

At this level, someone other than you reads your plaintext prompt, and what happens next relies entirely on a promise.

• Contractual zero-retention is the enterprise-grade solution. The provider knows who you are, processes your prompt, and promises not to retain it, enforced by contract and reputation.
• Anonymous proxies erase who you are but not what you said. The downstream provider still processes plaintext according to its own policies. Terms vary; for example, proxies like Duck.ai (DuckDuckGo's chatbot product) arrange deletion agreements with model providers, while Venice simply lets users assume the provider retains everything, but neither side can verify.

Every segment of communication between machines runs on TLS, which only encrypts the pipe; the receiving end can read all information. Relays typically use Oblivious HTTP (RFC 9458) to split this knowledge, similar to passing a note through a friend. The friend knows who handed it but cannot read the content; the recipient reads the content but doesn't know who wrote it. OHTTP has been an IETF standard since January 2024, and many companies now route production traffic through OHTTP relays rented from Cloudflare and Fastly.

This is the privacy ceiling achievable when accessing closed-source models, due to a simple math problem. A single flagship training run now costs in the billions of dollars, and these labs' valuations near a trillion dollars are predicated on the exclusivity of model weights. As long as the capability gap persists, the premium persists, so labs guard weight files like state secrets.

Meta has already passively conducted this experiment. LLaMA, released in February 2023, was initially open only to researchers, but within a week, the weights leaked as a torrent to 4chan. A week later, llama.cpp allowed the smallest 7B model to run locally on a MacBook. Three days after that, Stanford fine-tuned the same model into the chat assistant Alpaca for under $600. This leak reduced Llama's operating cost to electricity; anyone with the files could run it at home. In July 2023, Meta officially open-sourced Llama 2 under a commercial license with a 700 million MAU exclusion clause. The weights ran, and so did the premium.

Frontier labs could theoretically provide attestation (remote attestation) for closed-source model inference, but attestation only proves which code read the prompt, not what that code did with it. To determine if the server retained data, we need to audit the serving code and reconstruct it to match the hash reported by the hardware. However, once the serving code is handed over, the lab also gives away the batching and caching techniques that underpin its profit margins—techniques that transfer to every future generation of models. Apple and Meta can provide remote attestation for the service stacks behind iPhone and WhatsApp because their profits lie in devices and advertising; publishing the service code costs them almost nothing.

This is why the weights and serving code of flagship models cannot reach external operators. Without an external operator, there is no third-party attestation; without attestation, verifiable privacy exists only for open-source models.

Structural-Level Privacy

Each mechanism in this category replaces trust promises with proofs based on hardware, cryptography, or physics, though they all come at different costs for privacy upgrades, the primary one being that they can only run open-source models.

TEE (Trusted Execution Environment) confidential computing runs inference inside a hardware enclave (a sealed compartment on the chip that even the machine operator cannot open). The chip signs an attestation detailing which model and code were executed.

The prompt is only sealed at the endpoint. Along the path routed through a platform proxy, a role capable of reading plaintext remains, and only the protocol prevents the proxy from recording or leaking the relayed content.

E2EE (End-to-End Encryption) eliminates readable relays. The user's device encrypts the prompt with the enclave's key, and every hop in between carries a sealed envelope that only the enclave can open.

• Trust falls on the client side. The code responsible for encrypting the prompt and verifying the attestation also has the power to revoke this guarantee. Therefore, verifiable E2EE requires both a proven enclave and open, reproducible client-side code.
• Compared to the simplicity of TEE, the cost of E2EE is engineering overhead, which slows feature integration. E2EE turns the proxy into a blind messenger, so all functions relying on reading plaintext must be rebuilt around the client key or entirely within the enclave.

FHE (Fully Homomorphic Encryption, and MPC variants) eliminates the trusted party entirely. The server computes on ciphertext in a locked box it can never open; the key is only in your hands. MPC (Multi-Party Computation) splits the prompt into secret shares distributed among multiple parties, achieving the same effect unless all parties collude.

• The cost is speed. FHE natively only supports addition and multiplication, so the non-linear steps required for transformer operation must be rebuilt at high cost. Inference on ciphertext costs 10,000 to 100,000 times more than on plaintext, taking seconds to minutes per token even on small models, compared to milliseconds without encryption.
• Chips custom-built for encrypted computation could narrow the gap, but the first prototype was only demonstrated in early 2026, with commercial versions still years away.

Local inference simply removes this path. The model runs on your own hardware: no relays, no servers, no provider, no verification needed.

• The obvious costs are performance and model capability. gpt-oss-120b scores roughly half of GLM-5.2 on the Artificial Analysis index but takes up 65GB, exceeding the combined VRAM of two flagship consumer graphics cards. Full-precision GLM-5.2 can only run on an 8-GPU data center node, costing over $300,000 just for the GPUs.

However, beyond these structural constraints, the cost of placing inference inside an enclave is decreasing. For single-GPU inference, benchmarks from enclave cloud provider Phala show H100 throughput loss in enclave mode averages less than 7%, approaching zero for large models since the main cost is moving data into the chip, not computing inside it. For multi-GPU inference, NVIDIA's next-generation GPU, Blackwell, supports direct encryption of inter-chip traffic, whereas the older H100 achieves the same effect only by routing through the CPU host at one-seventh the bandwidth. NVIDIA's own benchmarks on Blackwell show less than 8% throughput loss for a 397B model in enclave mode. With these advancements, the performance penalty of private inference itself is no longer the decisive constraint.

In fact, an enclave adds almost zero extra operating cost for the operator. Every H100 manufactured after 2023 has enclave mode built-in; the additional cost is the throughput loss from encryption, not an extra chip. Currently, Azure's confidential H100 SKU is rented at $8.90 per hour, compared to $6.98 without enclave mode, a 27% premium over traditional cloud infrastructure. Conversely, on specialized enclave providers like Phala, confidential-mode H100s are rented starting at $3.80 per hour, below Lambda's standard SXM card price range of $3.99 to $4.29. For hosted API solutions, NEAR AI provides gpt-oss-120b with attested endpoints at $0.15 per million input tokens and $0.55 per million output tokens, on par with plaintext routes like Amazon Bedrock, Together, and Groq. Even for models requiring multi-chip parallelism, NEAR AI's pricing for GLM 5.2 matches Fireworks exactly, and for the larger Kimi K2.6, it is 15% cheaper for inputs and 4% cheaper for outputs.

While these new private inference providers may be burning profits to gain market share (a statement true for any company seeking growth), the structural trend is that the cost of privacy is decreasing for both consumers and operators.

How Open-Source Models Can Win

Despite the shrinking performance overhead, a visible gap remains between frontier models and SOTA open-source models. An entity seeking to maximize productivity must still trust frontier labs not to steal its IP to stay at the forefront.

The gap persists, but a case study from Bridgewater's AIA Labs and Thinking Machines on June 30th demonstrated that an open model fine-tuned with expert annotations can outperform frontier models in both accuracy and cost.

In the study, the team fine-tuned Qwen3-235B on Tinker (Thinking Machines' hosted fine-tuning API service). They first purchased annotations from vendors, used this data for an initial training round, and then had company investment professionals re-annotate divergent samples. Training used reinforcement learning (GRPO) with three modifications: round-robin batching (each task takes turns producing a batch), CISPO loss (limits how far a single answer can pull the model), and on-policy distillation (anchoring to the current best checkpoint to prevent learning from weaker copies).

All tasks were drawn from the daily workflow of investment professionals: whether a news article is important for C-suite investment professionals, whether a central bank document signals future interest rate direction, and where boilerplate language begins in a document or email. Scoring was based on an independent test set. Frontier models scored an average of ~50% with simple prompts, reaching only 78.2% with expert prompts, below the 80% threshold set by investment professionals. The fine-tuned Qwen achieved 84.7%, translating to 29.8% fewer errors than the best frontier model and costing 13.8 times less in inference cost.

https://thinkingmachines.ai/news/learning-to-replicate-expert-judgment-in-financial-tasks/

This case proves open-source models can win on accuracy and cost, but the training process was not private. The expert annotations used were Bridgewater's proprietary data, passing through Tinker's third-party service, residing at the same trust level as ZDR agreements. The fund also rented compute, running the entire training on machines it never controlled. Buyers wanting this formula without trust assumptions have few options today. Renting a bare GPU cluster makes the training process readable to the cloud operator. Buying the cluster solves data custody but costs a fortune.

The attested route has just arrived. In March, Workshop Labs and Tinfoil released Silo, a post-training stack running inside a Tinfoil enclave on a single 8-GPU node, with keys controlled solely by the client. The stated enclave cost is an extra 11 minutes for a two-hour training run. Moreover, by freezing base model weights and training only small adapters on top, the stack can accommodate a trillion-parameter model (Kimi K2 Thinking). The challenge is that reinforcement learning requires moving data back and forth between components, and moving data is precisely where enclave costs lie.

Less than a month after Silo's release, Workshop Labs was acquired by Thinking Machines, bringing the components needed to run a Bridgewater-style RL loop inside an enclave under one roof.

Privacy at the Harness Layer

There remains a problem spanning all private inference mechanisms. Each mechanism addresses the path from prompt to model, but every external tool invocation initiated by an agent opens a new path that the inference layer cannot touch. The recent wave of harness engineering amplifies this problem; every tool, memory bank, and data source connected to the model is a destination that reads its own slice of the workflow in plaintext. A calendar server reads schedules, a database server reads queries. A fully local agent, if it needs anything beyond its training set, must still send search terms in plaintext to a search engine. If the server cannot read the plaintext, it cannot answer the question.

The mainstream solution still defaults to the protocol layer. Companies like Runlayer and MintMCP use a central gateway to manage all tool traffic, stripping Personally Identifiable Information (PII) before requests leave. The gateway also decides which servers can receive traffic, blocking unvetted ones, and logs every call's destination and content for forensics. Even with independent audits (SOC 2), tool servers still need to read plaintext queries to respond. Whether they keep a copy depends on their retention policy, multiplied by every tool in the harness. Furthermore, the gateway itself represents an additional trust-dependent reader on the path, not a verification.

Structural-level solutions target the middle layer. For instance, Phala hosts MCP servers directly inside TEEs, covering wallets, code execution, and data sources. Users can verify privacy