NVIDIA Launches NeMo Switchyard, Using AI Model Routing Technology to Reduce Agent Operating Costs

Odaily News NVIDIA has launched an AI model routing system called NeMo Switchyard, designed to help developers dynamically allocate AI Agent tasks across multiple large models, reducing costs and latency while maintaining performance.

NVIDIA stated that building AI agents does not necessarily mean relying on a single large model. Different models have their own advantages in reasoning capability, response speed, and operating costs. For example, classification tasks may be suited for lightweight models, while complex reasoning requires more powerful frontier models. If all requests are sent to the largest model, costs and latency will increase; conversely, using small models for everything may degrade the quality of complex task completion. NeMo Switchyard uses an intelligent routing mechanism to automatically select the most appropriate execution model among multiple specialized and frontier models based on factors such as task requirements, model capabilities, cost, latency, and system status. The system allows developers to switch between different model providers and model versions without changing the application architecture.

NVIDIA explained that NeMo Switchyard offers multiple routing strategies, including a training-free LLM classifier router, Stage Router, Escalation Router, and an adjustable routing model trained on real workload data. Among these, the Escalation Router prioritizes assigning tasks to low-cost models and escalates requests to stronger models when it detects increased task complexity, persistent errors, or stalled execution, thereby achieving a balance between performance and cost.

NVIDIA stated that in related tests, NeMo Switchyard significantly reduced AI Agent operating costs while maintaining a high task completion rate by distributing tasks across different models. For example, compared to using only frontier models, the escalation-based routing approach reduced costs by approximately 74% in LangChain multi-turn agent testing, with only 7% of requests requiring calls to frontier models.

Additionally, NVIDIA has partnered with companies such as Cognition, Nous Research, Ramp, LangChain, LiteLLM, and Kong to integrate NeMo Switchyard into AI Agent development workflows and enterprise application infrastructure.