NVIDIA Unveils NeMo Switchyard, Using AI Model Routing Technology to Reduce Agent Operating Costs

Odaily News: NVIDIA has launched the AI model routing system NeMo Switchyard, designed to help developers dynamically allocate AI Agent tasks across multiple large models, reducing costs and latency while maintaining performance.

NVIDIA stated that building an AI agent does not mean relying on a single large model. Different models have their own advantages in reasoning capability, response speed, and operating costs. For instance, classification tasks may be suitable for lightweight models, while complex reasoning requires more powerful frontier models. If all requests are sent to the largest-scale model, costs and latency will increase; conversely, using only small models for all tasks may degrade the quality of complex task completion. NeMo Switchyard uses an intelligent routing mechanism to automatically select the most appropriate execution model among multiple specialized models and frontier models based on factors such as task requirements, model capabilities, cost, latency, and system status. The system supports developers in switching between different model providers and model versions without changing the application architecture.

NVIDIA explained that NeMo Switchyard offers various routing strategies, including training-free LLM classifier routing, Stage Router, Escalation Router, and an adjustable routing model trained on real workload data. Among these, the Escalation Router prioritizes assigning tasks to low-cost models and escalates requests to more powerful models when it detects increased task complexity, persistent errors, or stalled execution, thereby achieving a balance between performance and cost.

NVIDIA stated that in related tests, NeMo Switchyard significantly reduced AI Agent operating costs while maintaining a high task completion rate by distributing tasks across different models. For example, compared with using only frontier models, the Escalation Router-based approach reduced costs by approximately 74% in LangChain multi-turn agent tests, with only 7% of requests requiring calls to frontier models.

Additionally, NVIDIA has partnered with companies such as Cognition, Nous Research, Ramp, LangChain, LiteLLM, and Kong to integrate NeMo Switchyard into AI agent development workflows and enterprise application infrastructure.