# litellm

> One OpenAI-style API and proxy server for calling 100+ LLM providers

- Repository: https://github.com/BerriAI/litellm
- Category: Infrastructure & security
- Stars: 61,851 · Forks: 12,585 · Language: Python · License: not stated
- Experience: Some experience
- Page: https://ir.adyog.com/repos/berriai/litellm/

Every LLM provider has its own SDK, login method, request shape and error types. If your app talks to several, that mess spreads through your code. LiteLLM puts one interface in front of all of them, using the OpenAI format you may already know.

You can use it two ways. The Python SDK is a library: you call completion() with a model name like openai/gpt-4o or anthropic/claude-sonnet-4-20250514, and it handles the provider details. The AI Gateway is a proxy server you run for a team. Apps point any OpenAI client at its URL, and the gateway forwards each call to the right provider.

Beyond plain chat calls, the gateway adds virtual keys, spend tracking, guardrails, load balancing and an admin dashboard. It also fronts MCP tool servers and A2A agents. The README reports 8ms P95 latency at 1k requests per second, which you should check against your own setup.

## What it does

- **One format for 100+ providers**: Call OpenAI, Anthropic, Gemini, Bedrock, Azure and others with the same OpenAI-style request. Endpoints include chat completions, responses, embeddings, images, audio, batches and rerank.
- **Python SDK or proxy server**: Import it as a library, or run the AI Gateway so a team shares one endpoint. Existing OpenAI client code works by changing the base URL.
- **Team controls in the gateway**: The proxy offers virtual keys, spend tracking, guardrails, load balancing and an admin dashboard.
- **MCP and A2A support**: Connect MCP tool servers to any model through the SDK or the gateway. You can also call A2A agents such as LangGraph or Bedrock AgentCore.
- **Agent harnesses on any model**: The SDK can run Claude Code, Codex, OpenCode or Deep Agents, sending their model calls through your gateway so they are tracked.

## What you could build

- Swap between OpenAI, Anthropic and Bedrock models in an app by changing one model string.
- Run a shared gateway that gives each teammate a virtual key and tracks what they spend.
- Point existing OpenAI client code at a self-hosted proxy to route calls across providers.
- Expose MCP tool servers to any model, or to Cursor, through one authenticated endpoint.
- Route a coding agent such as Claude Code through your own gateway to log its cost.

## Who it's for

Developers and teams who use more than one LLM provider and want one API and cost tracking; you should know basic Python and what an OpenAI-style API call looks like.

## Get started

For the library, install it, set your provider API keys as environment variables and call completion() with a model name. For the gateway, install the proxy extra and start it with a model; it listens on port 4000, and any OpenAI client can use that address as its base URL.

```
uv add litellm
```
```
uv tool install 'litellm[proxy]'
```
```
litellm --model gpt-4o
```

Tags: llm-gateway, openai-compatible, proxy, mcp, cost-tracking