AI大模型中轉站不僅需要跑分，也需要審計：GatewayBench 評測框架上線，官網 Check4U.ai 同步開放

Are the GPT-5 or Claude models that companies spend巨额 amounts on each month really the official original models?

To navigate the complex geographical restrictions, enterprise compliance processes, and payment barriers of AI large language models, an increasing number of R&D teams are choosing a shortcut: the third-party AI relay market (Shadow API). This alternative appears extremely attractive, allowing developers to use mainstream models like GPT and Claude with lower access barriers, offering an experience close to the official API while filling gaps that official channels cannot cover.

However, behind this seemingly convenient "gateway" lies an unfathomable black box.

In March 2026, an audit report released by the CISPA Helmholtz Center for Information Security in Germany revealed a chilling truth: at least 187 top academic papers globally used AI large model relay APIs (Shadow API) for their research (62% of which have been accepted by top-tier academic conferences like CVPR and ICLR). However, because these relay gateways secretly replaced, quantized, or downgraded the underlying models, a large number of research results face the risk of being irreproducible.

What is a "disaster" in academia is a "time bomb" ready to explode at any moment in a company's production environment.

Today's large models are no longer just test products in labs; these core infrastructures are deeply integrated into customer service centers, code generation pipelines, Agent workflows, and risk control business chains. When dealing with systems supporting critical business operations, relay service providers often pitch selling points like "original models," "low prices," "high speed," and "cache support" to enterprises.

But the problem is that traditional evaluation tools simply cannot detect the tricks inside this black box. Existing benchmarking software mainly focuses on the speed and price outside the interface. These tools presuppose that the data returned by the gateway is entirely authentic and reliable. Conventional evaluations simply cannot answer the following core business security questions:

● During business traffic spikes, has the expensive model you purchased been secretly "swapped" or "downgraded"?

● Behind the so-called high speed and low price, have inflated Token counts been sneaked into the bill, and have failed requests been charged forcibly?

● Has the cache discount promised by the gateway actually been returned to the enterprise, and has cross-account private data been strictly isolated?

The AI Relay API does solve the accessibility problem of large models but also creates a new crisis of trust. Without understanding these invisible backend paths, purchasing decisions based solely on surface-level unit prices and speed is equivalent to running blindfolded. Breaking the black box era and making honest delivery a market advantage again is the top priority for the current AI supply chain.

Unveiling the Black Box: The "Three Unspoken Rules" of the AI Large Model Relay Market

Why can't existing conventional benchmarking software detect the problems? Because traditional evaluation tools only stay outside the interface, focusing on comparing response speed and listed prices. But behind the extremely opaque gateways, providers are exploiting this information asymmetry, turning complex technical means into hidden arbitrage tools.

An in-depth analysis of the current AI large model relay market reveals three highly concealed "unspoken rules" that are eroding enterprise business quality and budgets:

Unspoken Rule 1: Dynamic Model Adulteration

In the gray relay market, the hardest arbitrage tactic to detect is dynamic model replacement.

Many service providers, when facing conventional evaluation tools or low system traffic, will obediently call the official original models. However, once they encounter high-concurrency business peaks or are in blind spots where monitoring is difficult, these gateways will quietly switch the backend to a lower-cost quantized model, a weaker distilled version, or even an identically named open-source model with much lower costs.

Although the interface still appears to be outputting text on time, because the underlying probability distribution has been tampered with, the actual output quality received by the enterprise no longer matches the metrics originally promised by the provider. This model-swapping behavior exposes the accuracy of enterprise customer service replies and the quality of code generation to the uncontrollable risk of downgrade at any time.

Unspoken Rule 2: The Invisible Bill (Hidden Billing & Privacy Exposure)

Since the popularization of reasoning models, a cost item that is harder to verify has appeared in large model bills: thinking tokens. As this reasoning process is invisible by default, it's difficult for the buyer to judge whether the reasoning consumption claimed by the platform is real, leaving room for malicious gateways to inflate costs.

More frightening than inflated bills is uncontrolled cache fraud. Some service providers display a cache hit indicator on the bill but do not actually pass the corresponding discount back to the enterprise, turning cache optimization into a purely book-entry numbers game. Even worse, to forcibly increase the cache hit rate and reduce their own costs, certain gateways forcibly stuff prompts from different enterprises into the same shared cache pool. This directly impacts the data isolation boundaries of multi-tenant systems, exposing core business data and commercial privacy to cross-account mixing risks.

Unspoken Rule 3: The Deceptive "Lowest Price on the Market" (Carefully Designed Financial Trap)

When purchasing large model APIs, the "lowest price on the market" often catches the most attention. However, in actual business, the nominal unit price does not equal the true cost. Especially in the relay gateway market, input, output, cache hit, and cache write prices are often bundled into a single composite price, making comparison look easy. But once subjected to real business workloads, this price easily becomes misleading.

This distortion arises from the fact that there is no fixed input-output ratio when enterprises call large models. RAG, long document analysis, and complex Agent workflows are typically high-input, low-output; code generation and content creation might be low-input, high-output. If a platform only shows a blended unit price, the buyer struggles to pinpoint where the costs are actually incurred. A platform with a seemingly low blended price might optimize its headline offer by lowering the output unit price while raising the prices for input, cache writing, or other less conspicuous line items. Consequently, the actual bill for certain high-input scenarios could be far higher than expected.

Furthermore, hidden costs are buried in various obscure terms and conditions. For instance, when the system experiences disconnections, timeouts, or 5xx errors, failed requests may still be charged forcibly; low unit prices are often bundled with high minimum top-up amounts, non-refundable balance clauses, and opaque foreign exchange and payment channel fees. When all these financial frictions are added together, the actual cost debited from the enterprise's account per million Tokens can be several times higher than the nominal price advertised on the webpage.

GatewayBench: A Specialized Audit Framework for Large Model Gateways

Faced with highly opaque gateway backends, traditional speed testing and benchmarking tools have clear limitations. These metrics can compare response time, model coverage, and nominal price, but they struggle to answer more critical questions: Does the model actually called by the gateway match the promise? Is the billing transparent? Is the cache and data isolation trustworthy?

Against this backdrop, the GatewayBench audit and evaluation framework has been officially launched, with the evaluation entry open on the official website Check4U.ai. As an open-source audit framework for large model gateways, GatewayBench does not just look at speed and surface prices. Instead, it breaks down gateway evaluation into three dimensions: Trustworthiness, Economy, and Performance, using a weighting system of 40% Trustworthiness + 40% Economy + 20% Performance.

This weighting reflects GatewayBench's fundamental judgment: in the AI large model relay API scenario, trustworthiness and true cost take precedence over speed. A gateway must first prove its model authenticity, billing transparency, and cost explainability before it can qualify for performance competition.

To achieve this goal, GatewayBench provides three core audit capabilities:

L1 Trustworthiness Audit: From Platform Claims to Verifiable Trust

In GatewayBench's scoring system, L1 Trustworthiness accounts for 40% of the weight. The logic behind this design is: in the AI large model relay API scenario, speed and price are certainly important, but if the model isn't real and the billing isn't transparent, other metrics become moot.

The core risk of third-party large model relay gateways stems from the invisible processes behind a successful API call. A normal response at the interface level only indicates that the request was processed. It does not prove that the model source, billing process, and cache handling align with the platform's promises. In the past, these aspects lacked externally verifiable evidence and were difficult to incorporate into systematic audits.

The L1 dimension of GatewayBench is precisely designed to turn these vague suspicions into verifiable engineering signals. By breaking down trustworthiness into three questions—Is the model real? Is the billing transparent? Is the cache trustworthy?—and employing statistical tests, cryptographic structures, and latency fingerprints, it observes what actually happens inside the gateway black box from the outside.

For model authenticity, GatewayBench introduces the RUT (Rank-based Uniformity Test) algorithm, which checks the position of output tokens within the probability ranking of a reference model. Different models might generate similar text, but the probability distribution of tokens is much harder to fake. If the backend undergoes quantization, downgrade, or substitution, the distribution drift leaves traces. Simultaneously, GatewayBench can combine Logprob Tracking, requesting only one output token for a fixed prompt and tracking whether its log probability shows a stable shift over different periods. This provides a lower-cost signal for continuously monitoring model updates, fine-tuning, quantization adjustments, or route switching.

For billing transparency, GatewayBench uses PALACE to estimate a reasonable range for thinking tokens, helping to identify abnormally high reporting in reasoning models. Simultaneously, by leveraging verifiable structures like CoIn, it makes billing records more traceable and resistant to tampering.

For cache trustworthiness, GatewayBench uses latency fingerprints to determine if cache hits are real and cross-account isolation tests to identify potential tenant boundary issues. If the bill shows a cache hit but the TTFT hasn't decreased correspondingly, the discount might only exist on paper. If abnormal cache reuse occurs between two independent accounts, it may indicate potential risks in cache isolation.

Through these methods, GatewayBench transforms what was previously only assessable by intuition and suspicion into a set of measurable, verifiable, and comparable signals, achieving truly "verifiable trust."

L2 Performance: Stress Testing the Limits of Stable Delivery Under Extreme Pressure

Only gateways that pass the L1 Trustworthiness audit are eligible for performance and cost-effectiveness comparison.

In the large model infrastructure ecosystem, performance has always been the metric that relay stations and aggregation API providers emphasize most. Terms like "Fastest on the web" or "150 tokens/s per concurrent request" are common. However, GatewayBench designed its indicator system with a very restrained weight allocation for performance: L2 Performance accounts for only 20% of the comprehensive score.

The reason is simple. Speed is important; it determines basic system usability and can weed out services with frequent freezes or uncontrollable long-tail latency. But speed should not outweigh trustworthiness and economy. A gateway might be very fast, but if it suffers from model substitution, inflated billing, or opaque caching, it cannot be considered a trustworthy enterprise-grade infrastructure.

Therefore, in the L2 Performance dimension, GatewayBench does not chase single-point peak speed. Instead, it breaks down performance into questions more relevant to production environments: Latency determines how long users wait; Goodput measures how much effective capacity can be delivered within latency redlines; and long-context tests observe how the system degrades under heavy load.

The premise behind this design is: performance is a threshold, but not the ultimate goal. What enterprises are truly buying is the ability to deliver stably, on time, and predictably under the premise of trustworthiness.

In commercial-grade applications, the peak speed an idle system achieves has extremely limited reference value. GatewayBench rejects simply competing on throughput (Tokens/s) and instead introduces SLO (Service Level Objectives) as the business red line. Enterprises can stipulate, for example, that P95 TTFA must be less than 1.5 seconds, P95 E2E must be less than 8 seconds, and streaming output must not show significant jitter. Goodput is built upon this red line: only throughput delivered while meeting the SLO counts as effective capacity (Goodput).

To test the true scheduling capability of relay gateways, GatewayBench launches stress tests with ultra-long contexts at the 100k token level. This test aims to observe whether the system maintains stable, graceful degradation under heavy load, or suffers from long-tail issues and even silent downgrades. What enterprises pay for is precisely this stable delivery capability that can still deliver results on time under extreme business pressure.

L3 Economy: Penetrating the Billing Fog to Reveal the "True Cost per 1M Tokens"

Price is the most sensitive variable for enterprises procuring large model relay APIs, and also the easiest to repackage. Unit prices on web pages look clear, but once actual production calls begin, the cost is impacted by input-output ratios, cache rules, failed requests, top-up terms, exchange rates, and payment channels.

Therefore, GatewayBench assigns L3 Economy a 40% weight, putting it on par with L1 Trustworthiness as a core dimension. The focus here is not on the nominal unit price on the rate card, but on the True Cost per 1M Tokens – the cost an enterprise ultimately pays under real business workloads.

For listed pricing, GatewayBench breaks down the input price, output price, cache hit price, and cache write price, preventing a single blended price from obscuring the cost structure. The input-output ratio varies significantly across different business scenarios; RAG, long document analysis, and Agent workflows are typically high-input, low-output, while code generation and content creation may incur higher output costs. Looking only at a composite price can easily lead to misjudging actual expenditure.

To assess value relative to official pricing, GatewayBench introduces a "Platform Price / Official Price" ratio, combined with the gateway's role to determine if the premium is reasonable. Aggregated routing, multi-channel failover, and unified billing can justify a certain service premium; simple forwarding proxies should be closer to the official price. A low price doesn't necessarily represent an advantage, and a high price needs to correspond to real engineering value.

The GatewayBench framework also delves into the hidden friction behind bills: Are failed requests charged forcibly? Is the advertised cache discount genuinely returned? Are there strict usage limits on the fund account? Through layer-by-layer decomposition, GatewayBench ultimately reconstructs for the enterprise the true debited cost stripped of all marketing packaging.

Join GatewayBench: Let Honest Delivery Gain Market Reward

The API relay market will not disappear due to controversy. As long as geographical, payment, risk control, and compliance differences exist in model access, third-party gateways will continue to serve real demand. Since this is an unavoidable reality, GatewayBench's goal is to make this relay station more transparent, more trustworthy, and more sustainable.

The biggest contradiction in the current API relay market is that information asymmetry is amplifying the "bad money drives out good" effect. Some service providers can gain short-term traffic through model swapping, billing packaging, cache arbitrage, or low-price marketing. Meanwhile, providers who insist on original model passthrough, transparent billing, and stable service find it harder to be noticed in the noisy market because they cannot break through their real cost constraints.

This is not a structure that an infrastructure market can rely on long-term. Any mature supply chain requires a market-based trust mechanism built on metrics: good services should be seen, stable performance should be recorded, and honest delivery should be rewarded with more traffic, more trust, and higher-quality procurement budgets. Service providers who have long profited from information asymmetry should bear higher trust costs.

Based on the vision of reshaping industry trust consensus, the AI large model relay station evaluation platform Check4U.ai has announced