# Switchyard

> Routing library that picks which LLM handles each agent request

- Repository: https://github.com/NVIDIA-NeMo/Switchyard
- Category: Token & cost savings
- Stars: 3,435 · Forks: 341 · Language: Rust · License: Apache-2.0
- Experience: Advanced
- Page: https://ir.adyog.com/repos/nvidia-nemo/switchyard/

AI agents often send every request to one expensive model. Switchyard lets you set up a pool of models, such as a cheaper one and a stronger one, and picks which should answer each request. Your gateway or application still makes the actual model call and owns credentials and retries.

You choose a routing algorithm. Some look at the request or the agent's recent tool activity. Others ask a model to judge whether the efficient model can handle the task. Routing can change as the agent keeps working. The default, Auto, is a fixed preset that tries the efficient model first with a confidence threshold of 0.5.

It keeps OpenAI and Anthropic API compatibility, so it can sit behind existing clients. The README says a cheaper call does not guarantee a cheaper finished task, so test the whole setup against a single-model baseline. It is pre-1.0 software and APIs may change.

## What it does

- **Several routing algorithms**: Auto, Task (a model judges the task), Execution (uses recent tool activity) and Composite (combines both). The wider catalogue adds Plan/Execute, escalation, advisor, sub-agent and random routing.
- **Gateway integrations**: Works with LiteLLM (experimental, checkout-only) and NeMo Relay through a native plugin. Which algorithms work varies by gateway.
- **Local proxy for trying it out**: A standalone server speaks the OpenAI and Anthropic APIs, including a local Codex service. It is labelled demo-only, not for production.
- **Embeddable in your own harness**: The libsy library runs routing inside a Rust application, and there is a Python embedding example. You own the model calls.
- **Benchmark and soak-test guides**: Docs explain how to compare routing against single-model baselines and how to test latency and routing overhead.

## What you could build

- Route an coding agent's easy steps to a cheaper model and hard steps to a stronger one.
- Compare routing setups against a single-model baseline on your own workload.
- Add model routing to a LiteLLM or NeMo Relay gateway through a plugin.
- Try routing locally by pointing an OpenAI- or Anthropic-compatible client at the demo proxy.
- Embed routing decisions inside your own Rust or Python agent harness.

## Who it's for

Developers running LLM agents who care about cost and latency; you should already know how model gateways and API keys work, and be comfortable with pre-1.0 software.

## Get started

The README gives no install commands inline. Start with the local proxy path in docs/getting_started.md to try routing, or follow the LiteLLM or NeMo Relay guide if you already use one. Pin the version you use.



Tags: llm-routing, cost-optimization, agents, proxy, rust