IOSG: At the Crossroads of AI, Why Are Wall Street Investors Saying "No" to ChatGPT and Claude?

Original text by Jeff, IOSG Ventures

Why Private AI Is Necessary

On July 1st, Palantir CEO Alex Karp delivered a 20-minute interview on CNBC that some media described as a "meltdown." According to Karp, enterprises are paying frontier labs a token premium while watching their own IP flow to model providers. He called this leakage an alpha transfer occurring at the architectural layer: every request sent to a closed-source model arrives at the service provider's server in plaintext. Just days before the broadcast, Palantir had announced a partnership with NVIDIA to run the open Nemotron model in customer-controlled environments, accompanied by a nine-point AI sovereignty manifesto. After the CNBC segment aired, PLTR jumped 8%.

For the past two decades, enterprises adopted cloud software based on trust at the protocol level, and it worked. Each SaaS vendor saw only a slice of enterprise data, and most had little incentive to use customer data to feed their core products. Salesforce saw sales pipelines, Workday saw HR data, Jira saw development iterations, and AWS provided the foundation for storage and computation. However, today's AI workflows advocate uploading everything at once, along with the structured context linking various departments, to maximize productivity. Putting goodwill aside, upstream service providers can now use this data to develop new features rather than letting it sit idle on servers.

No one is slowing down. Anthropic reached an annualized revenue of $47 billion in May, a significant jump from $9 billion at the end of 2025, while OpenAI surpassed 900 million weekly active users in February. Both companies completed new funding rounds this spring, with valuations approaching $1 trillion, and are expected to go public at even higher valuations. Years of privacy and IP allegations have not cost either company any momentum.

Some enterprises had already taken action. In February 2023, less than three months after ChatGPT's launch, major Wall Street banks restricted its use. In May 2023, after Samsung engineers leaked chip source code into ChatGPT, the company banned generative AI across its network. In response, OpenAI launched ChatGPT Enterprise in August of that year, promising not to train on business data and offering a zero-data-retention (ZDR) agreement, which has since become a standard requirement for enterprise procurement.

But contracts only lock down company accounts. IBM found that by 2025, shadow AI (employees feeding company data into unauthorized AI tools via personal accounts) was involved in one-fifth of data breaches, and heavy shadow AI use added an average of $670,000 to breach costs. In a 2025 survey by security training company Anagram, 40% of employees said they were willing to violate AI usage policies to get tasks done faster.

Enterprises can at least buy their way out with ZDR contracts, non-training service tiers, and, for government or Palantir customers, sovereign deployment. For ordinary users like you and me, the importance of private AI is still debated—until a court subpoena comes knocking.

A court order in May 2025 forced OpenAI to retain even deleted consumer-level chats. In November, a judge ordered 20 million of those chats to be handed over to The New York Times' lawyers as discovery material. Then came criminal cases: ChatGPT records of the defendant in the Palisades fire arson case became evidence, and an affidavit in a Florida double homicide case cited the suspect's questions about how to dispose of a body. Sam Altman also admitted in a July 2025 interview that ChatGPT conversations are not protected by legal privilege, and in litigation, OpenAI "may be required to hand over" user chat records.

The point is not that only criminals need private conversations. The fact that people's conversations with AI are archived and subject to subpoena is a surveillance surface most users don't know exists. A Kolmogorov Law survey of 1,000 US AI users in October 2025 found that 50% were unaware these conversations could be subpoenaed, while two-thirds believed these chats should receive the same protection as consulting a lawyer or doctor.

Self-hosted or verifiable environment open-source models are catching up quickly, but the strongest ones still lag behind frontier closed-source models by about 4 months in general capability. This puts token-maxxing enterprises and individuals at a crossroads: either give up a few months of model quality for privacy, or continue sending sensitive materials to Anthropic's servers because competitors are grabbing productivity advantages that way.

There is currently no perfect solution in the market. This report sorts through the various attempts by different parties to narrow the gap, observing how far frontier intelligence under provable privacy is from being delivered to enterprises and ordinary users.

How Privacy Is Currently Achieved

Private AI is not a single engineering challenge, but each mechanism in the market currently addresses the same event: a prompt leaves your device, travels across the network, lands on the machine running the model, and returns a response. The difference between mechanisms lies in where the plaintext exists on this path, who can read it there, and what verifies the privacy of the response.

Protocol-Level Privacy

At this layer, someone other than you can still read your plaintext prompt. What happens next depends entirely on a promise.

• Contractual zero retention is the enterprise solution. The service provider knows who you are, processes your prompt, and promises not to retain it, enforced by contract and reputation.
• Anonymous proxies strip away who you are but not what you said. The downstream service provider still handles plaintext according to its own policies. Terms vary. For example, Duck.ai (DuckDuckGo's chatbot product) and similar proxies negotiate deletion agreements with model providers. Venice, on the other hand, directly lets users assume the service provider will retain everything, but neither side can verify.

Every segment of machine-to-machine communication runs on TLS. It only encrypts the pipe; the receiving party can read all information. Relays typically use Oblivious HTTP (RFC 9458) to split this knowledge, like asking a friend to pass a note. The friend knows who sent it but can't read the content; the recipient reads the content but doesn't know who wrote it. OHTTP has been an IETF standard since January 2024, and many companies now run production traffic on OHTTP relays rented from Cloudflare and Fastly.

This is the privacy ceiling achievable when accessing closed-source models, and the reason is a math problem. A flagship training run now costs on the order of billions of dollars, and these labs' nearly trillion-dollar valuations are based on the exclusivity of their model weights. The longer the model capability gap lasts, the longer the premium lasts. So labs guard their weight files like state secrets.

Meta has already been passively subjected to this experiment. LLaMA, released in February 2023, was initially only open to researchers, but within a week, the weights were leaked to 4chan as a torrent. A week later, llama.cpp allowed the smallest 7B model to run locally on a MacBook. Three days later, Stanford fine-tuned a chatbot assistant, Alpaca, on the same model for less than $600. This leak brought Llama's running cost down to the price of electricity—anyone with the files could run it at home. In July 2023, Meta officially open-sourced Llama 2 with a commercial license containing a 700 million monthly active user exclusion clause. The weights were gone, and so was the premium.

Frontier labs could theoretically perform attestation for inference on closed-source models. But attestation can only prove which code read the prompt, not what that code did with it. To determine if the server retained data, we need to audit the serving code and reconstruct it to match the hash reported by the hardware. However, once the serving code is handed over, the labs also hand over the batching and caching techniques that support their profit margins, techniques that will transfer to every future generation of models. Apple and Meta can provide remote attestation for the service stacks behind iPhone and WhatsApp because their profits lie in devices and advertising; making the serving code public costs them almost nothing.

This is why the weights and serving code of flagship models cannot reach external operators. Without an external operator, there is no third-party attestation. Without attestation, verifiable privacy only exists on top of open-source models.

Structural-Level Privacy

Each mechanism in this category replaces trust promises with hardware-based, cryptographic, or physical proofs. However, each comes with a different cost for the privacy upgrade, primarily that they can only run open-source models.

TEE (Trusted Execution Environment) confidential computing places inference inside a hardware enclave (a sealed compartment on the chip, inaccessible even to the machine operator). The chip signs an attestation specifying which model and which code were executed.

The prompt is only sealed at the destination. On the path relayed through the platform proxy, there remains a role that can read plaintext. What prevents the proxy from logging or leaking the relayed content is only the protocol.

E2EE (End-to-End Encryption) seals the readable relay. The user's device encrypts the prompt using the enclave's key. Each hop in between carries a sealed envelope that only the enclave can open.

• Trust resides on the client side. The code responsible for encrypting the prompt and verifying the attestation also has the power to revoke this guarantee. Therefore, verifiable E2EE requires not only a proven enclave but also open, reproducible client code.
• Compared to the simplicity of TEE, the cost of E2EE is the engineering burden, which slows down feature integration. E2EE turns the proxy into a blind messenger, so all functions that rely on reading plaintext must be rebuilt around the client key or only within the enclave.

FHE (Fully Homomorphic Encryption, and MPC variants) completely removes the trusted party. The server performs computations on ciphertext within a locked box it can never open; only you hold the key. MPC (Multi-Party Computation) splits the prompt into secret shares distributed among multiple parties. The effect is equivalent unless all parties collude.

• The cost is speed. FHE natively only performs addition and multiplication, so the nonlinear steps required for transformer operation must be rebuilt at a high cost. Inference on ciphertext costs 10,000 to 100,000 times more than on plaintext. For small models, each token takes seconds to minutes, compared to milliseconds without encryption.
• Custom chips designed for encrypted operations could narrow the gap, but the first prototype was only demonstrated in early 2026, and commercial versions are still a few years away.

Local inference simply eliminates this path. The model runs on your own hardware: no relay, no server, no service provider, and no verification required.

• The obvious cost is capability and hardware expense. The gpt-oss-120b scores about half of GLM-5.2 on the Artificial Analysis index, but its 65GB size exceeds the combined VRAM of two flagship consumer graphics cards. Running a full-precision GLM-5.2 requires an 8-GPU data center node, costing over $300,000 for the GPUs alone.

However, beyond these structural constraints, the cost of putting inference in an enclave is decreasing. For single-GPU inference, benchmarks from enclave cloud provider Phala show an average throughput loss of less than 7% for H100s in enclave mode, and near zero for large models because the main cost is moving data into the chip, not computing inside it. For multi-GPU inference, NVIDIA's next-generation GPU, Blackwell, supports direct encryption of inter-chip traffic, whereas the older H100 requires routing around to the CPU host at one-seventh the bandwidth to achieve the same effect. NVIDIA's own benchmarks on Blackwell show a throughput loss of less than 8% for a 397B model in enclave mode. With these advancements, the performance overhead of private inference itself is no longer the decisive constraint.

In fact, the enclave itself adds almost no extra operating cost for the operator. Every H100 manufactured after 2023 comes with enclave mode built-in. The extra cost is the throughput loss from encryption, not an additional chip. Currently, Azure's confidential H100 SKU rents for $8.90 per hour, while the non-enclave version is $6.98 per hour, representing a 27% premium over traditional cloud infrastructure. On the other hand, specialized enclave operators like Phala rent confidential-mode H100s starting at $3.80 per hour, lower than Lambda's standard SXM card price range of $3.99 to $4.29. For managed API solutions, NEAR AI offers gpt-oss-120b with attested endpoints at $0.15 per million input tokens and $0.55 per million output tokens, on par with plaintext routes like Amazon Bedrock, Together, and Groq. Even for models requiring multi-chip parallelism, NEAR AI's pricing for GLM 5.2 matches Fireworks exactly, and for the larger Kimi K2.6, its input is 15% cheaper and output 4% cheaper.

While these new private inference providers might be burning profits for market share (a statement applicable to any company seeking growth in the market), the structural trend is that the cost of privacy is decreasing for both consumers and operators.

How Can Open-Source Models Win?

Despite shrinking performance overhead, a noticeable gap remains between frontier models and SOTA open-source models. An entity seeking maximum productivity must still trust frontier labs not to steal its IP to stay at the forefront.

The gap persists, but a case study from Bridgewater's AIA Labs and Thinking Machines on June 30th showed an open model fine-tuned with expert annotations beating a frontier model in both accuracy and cost.

In the study, the team fine-tuned Qwen3-235B using Tinker (Thinking Machines' managed fine-tuning API service). They first procured annotations from vendors, used this data to train an initial round, then had the company's investment personnel re-annotate the divergent samples. The training used reinforcement learning (GRPO) with three modifications: round-robin batching (processing one batch per task sequentially), CISPO loss (limiting how far a single answer can pull the model), and on-policy distillation (anchoring to the current best checkpoint to prevent the model from learning from weaker copies).

The tasks were all taken from the daily workflow of investment personnel: determining if a news article is important for C-suite investment professionals, if a central bank document hints at future interest rate direction, or where boilerplate language begins in a document or email. Scoring was based on an independent test set. Frontier models averaged about 50% with simple prompts and reached only 78.2% with expert prompts, below the 80% threshold set by the investment personnel. The fine-tuned Qwen scored 84.7%. According to the original report, this translates to 29.8% fewer errors than the best frontier model, with 13.8x lower inference costs.

https://thinkingmachines.ai/news/learning-to-replicate-expert-judgment-in-financial-tasks/

This case proves that open-source models can win on accuracy and cost, but the training process itself was not private. The expert annotations used were Bridgewater's proprietary data, passing through Tinker's third-party service, landing on the same trust level as a ZDR agreement. The fund also rented compute power, running the entire training on machines it never controlled. Buyers wanting this recipe without trust assumptions have few options today. Renting bare GPU clusters leaves the training process readable to the cloud operator. Buying the cluster solves the data hosting problem but makes costs skyrocket.

The attested route has just arrived. In March, Workshop Labs and Tinfoil released Silo, a post-training stack running within a Tinfoil enclave on a single 8-GPU node, with keys controlled solely by the customer. The reported enclave cost was an extra 11 minutes for a two-hour training run. By freezing the base model weights and training only small adapters on top, this stack can accommodate a trillion-parameter model (Kimi K2 Thinking). The challenge is that reinforcement learning requires moving data back and forth between components, and moving data is precisely where enclave costs lie.

Less than a month after Silo's release, Workshop Labs was acquired by Thinking Machines. The components needed to run a Bridgewater-style RL loop within an enclave are now all under one roof.

Privacy at the Harness Layer

Another issue lies outside all private inference mechanisms. These mechanisms each govern the path from the prompt to the model. However, every external tool call initiated by an agent opens a path that the inference layer cannot touch. The recent wave of harness engineering has amplified this problem exponentially. Every tool, memory bank, and data source connected around the model is another destination that reads its slice of the workflow in plaintext. The calendar server reads the schedule, the database server reads the query. A fully local agent still needs to send search terms in plaintext to a search engine if it wants anything outside its training set. If the server can't read the plaintext, it can't answer the question.

Mainstream solutions still default to the protocol layer. Companies like Runlayer and MintMCP use a central gateway to manage all tool traffic, redacting Personally Identifiable Information (PII) before requests leave. The gateway also decides which servers can receive traffic, blocking unauthorized ones, and logs the destination and content of each call for forensic purposes. Even with independent audits (SOC 2), the tool server must still read