LLM Gateway vs Proxy vs Router: What the Three Words Actually Mean
If you've shopped for LLM infrastructure lately, you've watched the same three words get swapped in and out of the same sentence: gateway, proxy, router. A tool calls itself an "LLM gateway" on its homepage, a "proxy" in its docs, and a "router" in a launch tweet — and reviewers copy whichever word was closest. The result is that "which LLM gateway should I use?" threads compare products that aren't actually solving the same problem.
The words aren't interchangeable. They describe three different axes, and most real products sit on all three at once — which is exactly why the terms blur. Untangle the axes and the shopping question gets easy. Disclosure first, as always: this is published by TierUp, and TierUp is one of the tools in the last section. We'll tell you where it's the wrong pick.
Proxy = where it sits (the architecture)
A proxy is a positional fact. It's a server that sits between your app and the model provider and forwards requests. Your code stops calling api.openai.com directly and calls the proxy instead; the proxy relays to the upstream and relays the response back.
That's the whole definition — it's about position in the request path, not features. A proxy is where you bolt on anything you want applied to every call without touching app code: a single place to hold provider keys, centralized logging, a rate limit, a response cache, a retry policy. LiteLLM's self-hosted proxy is the canonical open-source example — you run it, point your apps at it, and it becomes the one hop everything passes through.
The trade-off is inherent to the shape: a proxy is another hop that can add latency and become a single point of failure, and if you self-host it, you own its uptime. LiteLLM's own production docs recommend recycling workers to bound memory growth — that's the on-call tax of running a proxy yourself.
Gateway = what it manages (the control plane)
A gateway is a proxy that grew a management layer. The word is borrowed straight from classic API gateways (Kong, Apigee): the thing in front of your services that handles auth, rate limiting, routing, and observability so individual services don't each reinvent it. An LLM gateway is that pattern pointed at model providers.
So a gateway is (almost always) a proxy — but not every proxy is a gateway. The gateway label signals a control plane: budgets and spend caps, API-key issuance and scoping, guardrails and PII filtering, prompt management, per-team analytics. Cloudflare's AI Gateway leads with caching, rate limiting, and analytics; Kong's AI Gateway is its existing gateway plugins retargeted at LLM traffic; Portkey packages routing inside observability and governance. The common thread isn't "forwards requests" (that's the proxy underneath) — it's "gives an ops team a place to govern LLM traffic."
If your real problem is "I can't see or control what my org spends on LLMs," you want a gateway. The routing is incidental.
Router = how it decides (the decision function)
A router is the piece that answers which backend serves this request. It's a decision function, and it's orthogonal to the other two: you can have a proxy with no routing (always hits one provider) or routing logic with no standalone proxy (baked into an SDK).
Routing comes in three flavors, and conflating them causes most of the confusion:
- Provider routing — same model, different hosts. OpenRouter's provider selection picks which upstream serves, say, Llama-70B based on price, latency, or your allow/deny list. You still choose the model; the router chooses the seller.
- Policy routing — you write the rules. LiteLLM lets you author fallbacks and model-to-key mappings. The router executes your logic; it doesn't decide anything you didn't.
- Abstraction routing — the router owns the model choice. You express intent (a quality tier, a task type) and it maps that to a concrete model, re-shopping as the market moves. This is the rarest flavor and the one people mean least often when they say "router."
When a review says "Tool X is a better router than Tool Y" without saying which flavor, it's usually comparing a provider router to a policy router — two tools that aren't substitutes at all.
Why one product is usually all three
Here's the punchline that the loose vocabulary hides: most tools are a proxy (shape) that is a gateway (scope) that does routing (decision) — all at once. LiteLLM is a proxy you self-host, a gateway's worth of features, and a policy router. OpenRouter is a hosted proxy, a light gateway, and a provider router. Calling any of them "a proxy" or "a router" isn't wrong — it's just naming one axis and dropping the other two.
That's why the useful question isn't "gateway or proxy or router?" It's: which axis is your actual problem on?
| Your real problem | The axis | What to shop for |
|---|---|---|
| "I need one hop to add keys/logging/caching to every call" | Proxy | Self-hosted proxy — LiteLLM |
| "I can't see or govern org-wide LLM spend and access" | Gateway | Gateway/control plane — Cloudflare AI Gateway, Kong, Portkey |
| "I want the cheapest/fastest host for a model I've chosen" | Router (provider) | OpenRouter |
| "I want to stop choosing and maintaining models entirely" | Router (abstraction) | Tier routing — see below |
Where TierUp sits, stated plainly
TierUp is a hosted proxy (you call our OpenAI-compatible endpoint) with a thin gateway (keys, usage, wallet) — but the axis we're actually built on is the rarest one: abstraction routing. Instead of "model": "some-id", you send tier-1 through tier-4 (Speed → Reasoning), and we map each tier to what we assess as the best value in that class, re-shopping as the market shifts — so a better model shipping means your tier silently upgrades, no code change. That's the Reason 4 trigger: you want out of the model-maintenance business. Tier details here; tier 1 is free.
The honest trade-offs, same bluntness we hold everyone else to: you give up model pinning — if you need a specific checkpoint, that's a policy-router or provider-router job (LiteLLM, OpenRouter), full stop. We're very early with roughly zero production users, and the pricing (~50% under the underlying models' retail) is transparently subsidized while we test whether tier-routing is something developers actually want. If that makes you cautious, good — it should factor in.
If none of that is your problem — if you just need a hop for logging, or org-wide spend governance, or the cheapest host for a model you've already picked — we're the wrong tool, and now you know exactly which of the other three axes to shop instead. If abstraction routing is your problem, the playground needs no signup: paste a real prompt into tier 2 and the "is this good enough" question answers itself faster than any explainer, ours included. For the wider field mapped by why you'd switch, the OpenRouter alternatives guide and the 2026 router field guide go tool by tool.
Vendor capabilities described here verified 2026-07-08 against the linked primary docs (Cloudflare, Kong, LiteLLM, OpenRouter, Portkey). The category names drift; the axes don't — re-check a specific product's docs before you commit.
