LiteLLM, OpenRouter, and Portkey all promise one API in front of every model provider. Here is how their routing, caching, and failover architectures actually differ once you look past the pitch.
Every team that calls more than one model provider eventually builds the same piece of infrastructure, usually by accident. First it's a try/except around an OpenAI call that falls back to Anthropic. Then it's a cost dashboard someone hacks together after a surprise bill. Then it's a shared Redis cache because three services are sending the identical system prompt a thousand times a day. At some point a team member says the quiet thing out loud: "we should probably put a gateway in front of this."
That's the right instinct, but "LLM gateway" has quietly become a category with real architectural disagreements inside it, not just a commodity "add a base URL" decision. LiteLLM, OpenRouter, and Portkey all describe themselves as solving the same problem — one API, many providers — and they genuinely overlap on paper. But they make different bets about where the gateway should live, who operates it, and what you're trading away to get it. Those bets matter more than the feature checklist once you're past the prototype.
Strip away the marketing and a gateway earns its place in the request path for four reasons:
None of this requires a gateway to be hosted by a third party, self-hosted, open source, or proprietary. Those are deployment choices layered on top of the same core job.
Whichever product you pick, the request lifecycle looks roughly like this:
The interesting engineering is in steps 2 through 4: what counts as a cache hit when two prompts are semantically but not textually identical, what counts as "equivalent enough" to fail over to, and what happens to a response that was already streaming when the primary provider drops the connection mid-token. That last one has no clean answer — you either buffer the whole response before forwarding it (killing time-to-first-token) or you accept that a failover mid-stream means the client sees a truncated, restarted answer.
LiteLLM is an open-source (MIT core), Python-based proxy that translates OpenAI-shaped requests into whatever each of 100+ providers actually expects, then translates the response back. You either import it as an SDK inside your own service or run the LiteLLM Proxy Server as a standalone process — Docker, Helm chart, or bare Python — sitting inside your own network boundary.
That self-hosting is the whole value proposition: no per-token markup, no third party in the request path, and nothing stopping you from pointing it at an internal vLLM cluster alongside OpenAI and Anthropic. Routing strategies (latency-based, weighted, least-busy), configurable fallback chains, per-key and per-team spend budgets, and — more recently — an MCP gateway for routing tool calls sit in the same config file as your model list. The cost is operational: you own the uptime, the Redis instance backing the cache and rate limiter, and the Postgres database backing usage logs. A minimal production deployment realistically runs in the neighborhood of $50–200 a month in cloud infrastructure before you've sent a single token — and SSO is only free up to a handful of users before enterprise licensing kicks in.
OpenRouter takes the opposite bet: it's a hosted router with no self-hosting option at all. You get one API key, a catalog of several hundred models from dozens of providers, and automatic routing to whichever provider instance of a model is cheapest or fastest at that moment — useful when the same open-weight model is served by three different inference providers at three different prices. Its economics are also notably simple: there is no per-token markup on top of what the underlying provider charges. OpenRouter instead takes a fee on how you fund the account — around 5.5% when you buy credits by card, with a minimum charge that makes small top-ups disproportionately expensive, or nothing at all if you bring your own provider keys (free up to a monthly request volume in the low millions).
The trade is control: you're trusting OpenRouter's infrastructure and uptime for every request, your fallback logic is whatever OpenRouter's routing policy decides "equivalent" means, and you have no self-hosted escape hatch if you later need the gateway to live inside a VPC for compliance reasons.
Portkey sits architecturally between the two: an MIT-licensed gateway core that can be self-hosted, plus a managed control plane most teams actually run instead. Where it differentiates is everything around routing — dozens of built-in guardrails (PII detection, content filtering, jailbreak checks), virtual keys for per-team access control, semantic caching, prompt versioning, and A/B testing between models, all surfaced through a dashboard rather than a config file. Pricing is a flat subscription rather than a percentage fee — a free tier capped by log volume, a fixed monthly rate once you outgrow it, with log overages billed separately. Worth knowing if governance is the reason you're adopting a gateway at all: Portkey is now part of Palo Alto Networks, which changes who you're trusting with request-level visibility into every prompt your organization sends, even if the gateway's open-source core keeps shipping independently.
A handful of adjacent options are worth knowing exist even if they don't fit this head-to-head: Cloudflare AI Gateway and Vercel AI Gateway both give you a managed router for free if you're already running on that platform's edge or hosting, at the cost of being SaaS-only with no self-host path; Kong AI Gateway extends an API gateway many enterprises have already standardized on for everything else.
| LiteLLM | OpenRouter | Portkey | |
|---|---|---|---|
| Deployment | Self-hosted (OSS) | Hosted only | Self-hosted gateway or managed |
| License | MIT (core) | Proprietary | MIT (gateway core) |
| Provider breadth | 100+ | Several hundred models | 1,000+ |
| Token markup | None | None (fee on credit purchase) | None (subscription instead) |
| Caching | Exact + semantic | None natively | Exact + semantic |
| Guardrails | Limited | None | Extensive, built-in |
| Best fit | Teams that want infra in their own network, zero marginal cost | Teams that want the widest model catalog with no ops burden | Teams that need governance/compliance surfaces more than raw config flexibility |
A few things don't show up in any comparison table until they bite in production:
Most of what matters architecturally is visible in a few lines of a LiteLLM proxy config — provider list, a fallback chain, and a budget:
model_list:
- model_name: primary
litellm_params:
model: anthropic/claude-sonnet-5-5
api_key: os.environ/ANTHROPIC_API_KEY
- model_name: fallback
litellm_params:
model: openai/gpt-5
api_key: os.environ/OPENAI_API_KEY
- model_name: overflow
litellm_params:
model: hosted_vllm/llama-3-70b
api_base: http://vllm-cluster.internal:8000
router_settings:
fallbacks: [{ "primary": ["fallback", "overflow"] }]
routing_strategy: latency-based-routing
general_settings:
cache: true
cache_params:
type: redis
semantic_cache_embedding_model: text-embedding-3-small
budget:
max_budget: 500
budget_duration: 30d
A client calling this gateway never needs to know that primary resolved to Claude on this request and hosted_vllm/llama-3-70b on the next — it just sends an OpenAI-shaped chat completion request to http://localhost:4000 and gets an OpenAI-shaped response back.
A gateway earns its place once you've actually hit one of the four problems it solves — an outage that took your app down with it, a bill nobody can explain, or a second provider you're bolting on by hand. Don't add one preemptively; it's one more process that can fall over, and "my gateway is down" is a worse incident than "my model provider is down," because now two things are broken instead of one.
If you do need one: pick LiteLLM when the deciding factor is cost and control and you're willing to operate infrastructure; pick OpenRouter when you want the widest catalog with zero ops burden and can live with a hosted dependency; pick Portkey when the actual requirement is governance — guardrails, audit trails, spend policy — and routing is secondary. All three speak the same OpenAI-compatible protocol on the way in, which is the one decision you don't have to get right the first time: switching gateways later is a base-URL change, not a rewrite.