What Is NVIDIA SwitchYard? The Open-Source Local AI Model Router
NVIDIA SwitchYard routes agent tasks between local and frontier models automatically. Here's what it does, how routing works, and why it matters.
Drafted with Claude from source material, checked by automated verification, and reviewed before release. How we make these.
What is NVIDIA SwitchYard?
SwitchYard is an open-source routing library from NVIDIA that sits between an AI agent and the models it calls, deciding per request which model should handle the job. It is not a model itself. It is infrastructure: a classifier, a proxy for translating between different API formats, and a set of routing strategies that let a team mix local models with frontier APIs based on the difficulty of each task. It builds on an earlier project called RouteLLM and adds session state tracking, action traces, and multiple routing algorithms out of the box.
TL;DR
- SwitchYard is a routing layer, not a model, built on top of RouteLLM, that decides which model handles each step of an agent’s work.
- It ships with four routing strategies: random (for A/B testing), an LLM classifier that picks a tier before the call runs, a stage router that reads signals already in the conversation, and an escalation router that starts cheap and upgrades when a judge model spots trouble.
- The library includes an SDK that translates between API formats (OpenAI-style, Anthropic-style, and the newer responses API), so switching providers doesn’t mean rewriting your integration code.
- NVIDIA claims combining open and proprietary models through this kind of routing can deliver 50% faster responses and 25% better token efficiency, according to the company’s own framing.
- Routing exists for four practical reasons: cost, speed, specialization, and privacy, each with different tradeoffs and different ideal strategies.
- Nemotron 3.5 Lightning, a 30-billion-parameter mixture-of-experts model NVIDIA released alongside SwitchYard, is one example of the kind of fast local model meant to pair with a router.
- Most of the routing decision logic runs on CPU, not GPU, meaning the router itself doesn’t compete for GPU memory with the models it’s coordinating.
Plans first. Then code.
Remy writes the spec, manages the build, and ships the app.
Why do agent systems need a router at all?
Most people building with AI today default to using the most capable model available for every task, especially if they’re on a flat-rate subscription rather than paying per token. That habit breaks down fast once you’re running an agent that takes hundreds of steps to complete a job: planning, tool calls, retrieval, summarization, error handling, and final output generation. Treating every one of those steps as if it needs frontier-level reasoning is expensive and often unnecessary.
There are four distinct reasons teams reach for a routing layer, and they don’t all point to the same solution:
Cost. Enterprises running agents at scale burn through tokens quickly if every call goes to the most expensive model. Routing lets simpler tasks, like classification or entity extraction, get handled by cheaper models while complex planning steps still go to a stronger one.
Speed. Agentic workflows involving many sequential steps can’t afford to wait on a slow model for each one. A faster model, even if less capable on paper, keeps the whole pipeline moving. Some frontier providers are already optimizing for this: OpenAI’s planned deployment on Cerebras hardware is one example of the industry chasing latency, not just intelligence.
Specialization. No single model is the best choice for every kind of task. Enterprises increasingly use fine-tuned or purpose-built models for narrow jobs, and a router needs to know which specialist to call.
Privacy. Sensitive data, like hostnames, IP addresses, or account numbers, often shouldn’t leave a company’s own infrastructure. A hybrid approach keeps local models in the loop for anything containing private information and only sends sanitized, aggregated data to external APIs.
How does SwitchYard actually route requests?
SwitchYard defines a few core building blocks. A “profile” sets which router mode a given route uses, whether that’s a stage router or an escalation router. “Targets” define the available models, for example labeling one model “capable” (a frontier API) and another “efficient” (a local model), along with which provider serves each one. The application then just points at the profile it wants to use.
From there, SwitchYard offers four routing strategies:
Random routing splits traffic across models at a fixed ratio. It sounds pointless on its own, but it’s useful for running A/B tests or establishing a cost and quality baseline before switching to something smarter.
LLM classifier routing uses a secondary model to read the incoming request and decide which tier of model should handle it, before the actual task runs. This is well suited to domain-based routing: coding requests go one way, general writing or lookup tasks go another.
Stage routing skips the extra classification call entirely. Instead, it reads signals already present in the conversation, like whether tool calls are failing or whether code edits are landing successfully, and uses those signals to decide whether to escalate or stay put. Because it doesn’t need a separate model call, it’s close to free to run.
Other agents start typing. Remy starts asking.
Scoping, trade-offs, edge cases — the real work. Before a line of code.
Escalation routing starts every session on the cheaper model and has a judge model watch the run. When it detects real trouble, it bumps the session up to a stronger model and keeps it there for the rest of the task, rather than re-evaluating every single turn.
That last detail matters operationally: re-routing on every conversational turn is expensive and breaks prompt caching. The general pattern is to make a routing decision once, near the start of a task, and stick with it.
What does routing actually cost?
Each strategy carries a different overhead. Random routing costs nothing extra. The LLM classifier adds one additional model call per query, since something has to read the request and decide. Stage routing is close to free because it reuses signals the system already has. Escalation routing costs one judge call per session, until that session locks onto a stronger model. For anyone deploying a router, that cost column is the one to scrutinize most carefully, since an inefficient routing strategy can eat into the savings routing is supposed to deliver.
Beyond the four ready-made strategies, SwitchYard also supports a more advanced “tunable” approach: a prefill router, a trained model that reads early-stage signals in a request and predicts the likelihood of success on a given model, then blends that prediction with cost and latency targets to pick a route. This is closer to a learned policy than a fixed rule.
Does SwitchYard actually reduce cost without hurting quality?
NVIDIA’s own claim, cited when SwitchYard was released, is that combining open and proprietary models through this kind of routing can produce roughly 50% faster responses and 25% better token efficiency. That’s the company’s framing rather than an independently verified benchmark, but the underlying logic is sound and matches what other routing systems (like OpenRouter’s model fusion or Sakana’s Fugu) have already demonstrated: most agent steps don’t need frontier-level reasoning, and paying frontier prices for simple extraction or summarization tasks is wasteful.
The practical case for this pairs a strong general model for hard reasoning and planning with a fast, efficient local model for routine steps. In one demonstrated setup, network alert data with sensitive fields like hostnames and IP addresses was processed locally by a small model that produced sanitized summaries, with only the cleaned, combined data forwarded to a larger API-based model for root-cause analysis. That’s routing solving a privacy problem and a cost problem simultaneously.
Is SwitchYard worth setting up for your own agents?
If you’re running agent workflows with more than a handful of steps, or you’re paying API pricing rather than a flat subscription, a routing layer is likely to pay for itself quickly. SwitchYard’s value is less about any single model and more about giving teams infrastructure they’d otherwise have to build themselves: format translation across OpenAI-style, Anthropic-style, and responses-API endpoints, observability into which model handled which call and why, and a starting set of routing strategies rather than a blank page.
One coffee. One working app.
You bring the idea. Remy manages the project.
The tradeoff is that SwitchYard gives you the plumbing, not the policy. Deciding which tasks are simple enough for a local model, which signals indicate an agent is struggling, and where the cost/quality line sits for your specific workload is still work you have to do. For teams already running local models like Nemotron 3.5 Lightning or similar fast open-weight models, that setup cost is likely worth it. For a single hobbyist project calling one API, it’s probably overkill.
Frequently Asked Questions
What is the difference between SwitchYard and RouteLLM?
RouteLLM is a trained classifier for selecting between models. SwitchYard is built on top of it, adding a proxy layer for API format translation, multiple routing strategies (random, LLM classifier, stage router, escalation), and state tracking across sessions and conversation turns.
Does SwitchYard require a GPU to run?
The routing decision logic itself runs on CPU in most configurations, since it’s coordinating calls rather than generating text. GPU resources are only needed by the actual models being routed to, not by the router itself.
What is Nemotron 3.5 Lightning and how does it relate to SwitchYard?
Nemotron 3.5 Lightning is a 30-billion-parameter mixture-of-experts model NVIDIA released around the same time as SwitchYard, designed for high-throughput, low-latency workloads. It’s an example of the kind of fast local model a router like SwitchYard would pair with a larger frontier model.
Can SwitchYard route based on data privacy needs?
Yes. One common pattern is using a local model to process sensitive data (like hostnames or account numbers), strip or summarize it, and then send only sanitized, combined output to an external API model for tasks requiring more capability.
How is SwitchYard different from services like OpenRouter?
OpenRouter provides a single API endpoint that gives access to many hosted models, which is a form of routing but runs as a hosted service. SwitchYard is an open-source library you run yourself, giving you control over which models are included, how routing decisions are made, and where your data goes.