NVIDIA-NeMo /
Switchyard
Routing library that picks which LLM handles each agent request
- 3.4k stars
- 170 stars since 3 Oct
- Rust
- Updated today
- Advanced
- Token & cost savings
AI agents often send every request to one expensive model. Switchyard lets you set up a pool of models, such as a cheaper one and a stronger one, and picks which should answer each request. Your gateway or application still makes the actual model call and owns credentials and retries.
You choose a routing algorithm. Some look at the request or the agent's recent tool activity. Others ask a model to judge whether the efficient model can handle the task. Routing can change as the agent keeps working. The default, Auto, is a fixed preset that tries the efficient model first with a confidence threshold of 0.5.
It keeps OpenAI and Anthropic API compatibility, so it can sit behind existing clients. The README says a cheaper call does not guarantee a cheaper finished task, so test the whole setup against a single-model baseline. It is pre-1.0 software and APIs may change.
What it does
Several routing algorithms
Auto, Task (a model judges the task), Execution (uses recent tool activity) and Composite (combines both). The wider catalogue adds Plan/Execute, escalation, advisor, sub-agent and random routing.
Gateway integrations
Works with LiteLLM (experimental, checkout-only) and NeMo Relay through a native plugin. Which algorithms work varies by gateway.
Local proxy for trying it out
A standalone server speaks the OpenAI and Anthropic APIs, including a local Codex service. It is labelled demo-only, not for production.
Embeddable in your own harness
The libsy library runs routing inside a Rust application, and there is a Python embedding example. You own the model calls.
Benchmark and soak-test guides
Docs explain how to compare routing against single-model baselines and how to test latency and routing overhead.
What you could build
- Route an coding agent's easy steps to a cheaper model and hard steps to a stronger one.
- Compare routing setups against a single-model baseline on your own workload.
- Add model routing to a LiteLLM or NeMo Relay gateway through a plugin.
- Try routing locally by pointing an OpenAI- or Anthropic-compatible client at the demo proxy.
- Embed routing decisions inside your own Rust or Python agent harness.
Get started
The README gives no install commands inline. Start with the local proxy path in docs/getting_started.md to try routing, or follow the LiteLLM or NeMo Relay guide if you already use one. Pin the version you use.
Commands are copied from the project's README. Read it on GitHub for the full setup.
More in Token & cost savings
caveman
Makes coding agents answer tersely and shrinks the logs and output they read
rtk
A command-line proxy that shrinks shell output before your AI coding agent reads it
headroom
Shrinks tool outputs, logs, and files before your AI agent sends them to the model
codegraph
A local code knowledge graph that lets AI coding agents look up code in one call
freellmapi
One OpenAI-style API that routes across dozens of free LLM provider tiers
context-mode
Keeps bulky tool output out of your coding agent's context window
9router
Local proxy that routes AI coding tools to subscription, cheap, or free models
pxpipe
Local proxy that renders Claude Code's bulky context as images to cut input tokens
billion-context
A proxy that compresses long coding-agent chats so one session can run for weeks
token-optimizer-mcp
MCP server and Claude Code plugin that cuts agent token use and keeps a project graph