BerriAI /
litellm
One OpenAI-style API and proxy server for calling 100+ LLM providers
- 61.9k stars
- 1.8k stars since 3 Oct
- Python
- Updated today
- Some experience
- Infrastructure & security
Every LLM provider has its own SDK, login method, request shape and error types. If your app talks to several, that mess spreads through your code. LiteLLM puts one interface in front of all of them, using the OpenAI format you may already know.
You can use it two ways. The Python SDK is a library: you call completion() with a model name like openai/gpt-4o or anthropic/claude-sonnet-4-20250514, and it handles the provider details. The AI Gateway is a proxy server you run for a team. Apps point any OpenAI client at its URL, and the gateway forwards each call to the right provider.
Beyond plain chat calls, the gateway adds virtual keys, spend tracking, guardrails, load balancing and an admin dashboard. It also fronts MCP tool servers and A2A agents. The README reports 8ms P95 latency at 1k requests per second, which you should check against your own setup.
What it does
One format for 100+ providers
Call OpenAI, Anthropic, Gemini, Bedrock, Azure and others with the same OpenAI-style request. Endpoints include chat completions, responses, embeddings, images, audio, batches and rerank.
Python SDK or proxy server
Import it as a library, or run the AI Gateway so a team shares one endpoint. Existing OpenAI client code works by changing the base URL.
Team controls in the gateway
The proxy offers virtual keys, spend tracking, guardrails, load balancing and an admin dashboard.
MCP and A2A support
Connect MCP tool servers to any model through the SDK or the gateway. You can also call A2A agents such as LangGraph or Bedrock AgentCore.
Agent harnesses on any model
The SDK can run Claude Code, Codex, OpenCode or Deep Agents, sending their model calls through your gateway so they are tracked.
What you could build
- Swap between OpenAI, Anthropic and Bedrock models in an app by changing one model string.
- Run a shared gateway that gives each teammate a virtual key and tracks what they spend.
- Point existing OpenAI client code at a self-hosted proxy to route calls across providers.
- Expose MCP tool servers to any model, or to Cursor, through one authenticated endpoint.
- Route a coding agent such as Claude Code through your own gateway to log its cost.
Get started
For the library, install it, set your provider API keys as environment variables and call completion() with a model name. For the gateway, install the proxy extra and start it with a model; it listens on port 4000, and any OpenAI client can use that address as its base URL.
uv add litellmuv tool install 'litellm[proxy]'litellm --model gpt-4oCommands are copied from the project's README. Read it on GitHub for the full setup.
Anything better than litellm?
LiteLLM is the strongest pick for a production-grade, multi-provider LLM gateway with spend tracking, keys and team limits. Look elsewhere only if you're in TypeScript app code (vercel/ai) or just want free-tier pooling for experiments.
-
aiAlternative
Pick it if your app is TypeScript and you want an in-process SDK with streaming, structured output, agents and chat UI hooks, not a separate gateway server.
Trade-off: It's a library only, with no hosted proxy, central key management or spend and budget controls across teams.
-
freellmapiAlternative
Pick it for hobby or experimental use where you want to pool free-tier keys from many providers behind one /v1 endpoint with automatic failover.
Trade-off: Far less mature for production, with no team controls or spend tracking, and it relies on free-tier limits that can change.
More in Infrastructure & security
strix
Autonomous AI agents that pentest your app and prove each bug with a working exploit
NemoClaw
Reference stack for running AI agents inside NVIDIA OpenShell sandboxes
SkillSpector
Scans AI agent skills for security risks before you install them
CubeSandbox
Self-hosted MicroVM sandboxes for running AI agent code, compatible with the E2B SDK