Projects · AIForge
LiteLLM the model gateway
One OpenAI-compatible front door for every model. Routing, virtual API keys, budgets and rate limits, fallbacks and retries, usage tracking, and a straight argument for why a gateway beats letting applications connect to model servers directly.
Applications should never know where a model lives.
Not the service name, not the port, not the engine, not the provider, not whether it is a KServe InferenceService or a plain Deployment or a hosted API in another country. One base URL, one key, one model name that means something to a human.
Everything else in this chapter, the routing, the keys, the budgets, the fallbacks, is downstream of that single decision. Indirection at the API boundary is the cheapest architectural insurance in the entire platform.
LiteLLM the model gateway
Two chapters in, the platform can serve models. What it cannot do is answer a single one of the questions a real organisation asks within a week of going live.
Who called the model yesterday. How much did team B spend. Can I stop this one team from consuming the whole cluster. What happens to requests when the pod is being rescheduled. How do I move a team from a small local model to a bigger one without touching their code. How do I revoke access for a service that got compromised.
None of those are model problems. All of them are gateway problems, and LiteLLM is where AIForge puts them.
1. The problem, drawn honestly
Without a gateway, every application connects to every model it needs. That is N applications times M models of configuration, credentials and failure handling, and every single one of those pairs is somebody's copy pasted client code.
Every arrow in that diagram is a place where somebody hardcoded a URL. And here is the part that actually hurts: those arrows are invisible. Nothing in the cluster tells you they exist until you try to change one of the boxes on the right and something you had forgotten about breaks.
The gateway version collapses all of it to one hop:
Because there is exactly one hop, all of the following are now configuration rather than engineering:
Swap the engine. vLLM today, llama.cpp tomorrow, a GPU build next month. No application changes.
Revoke a key. One row, one command, immediate.
Attribute spend. Every request already carries an identity, so cost per tenant is a query rather than a project.
Rate limit a noisy neighbour. Enforced before the request reaches an engine that would have queued it.
Fail over. The retry logic lives in one place written by one person, instead of five slightly different versions in five repositories.
2. What LiteLLM actually is
Two products share the name, and mixing them up leads to confusion.
The SDK is a Python library that normalises calls to a hundred plus providers behind the OpenAI function signature. Useful in application code, not infrastructure.
The proxy, sometimes called the LLM gateway, is a server. It speaks the OpenAI API on the front and everything else on the back, and it holds keys, budgets, routing and logging. That is what AIForge deploys, and everything below refers to it.
Its whole behaviour comes from one config file plus a database.
They are not a mistake. Two backends, one public name.
One is the raw vLLM Deployment from the vLLM chapter. The other is the KServe InferenceService from the lifecycle chapter. Two completely different deployment philosophies, arguing with each other about complexity, and to a tenant application they are one model called qwen2.5-1.5b-instruct.
That is the abstraction paying for itself in the most literal way possible. I can migrate traffic from one to the other by editing weights in a config file, and nobody has to be told.
3. Deploying it
Nothing exotic. A stateless Deployment, a ConfigMap for the config, a Secret for the keys, and Postgres for state.
The salt key encrypts provider credentials stored in the database. Lose it and every stored credential becomes undecryptable ciphertext. Change it on a running system and the proxy can no longer read its own secrets, which presents as authentication failures against every backend at once.
It goes in a real secret store, it gets backed up, and it never rotates casually. Treat it like a database encryption key, because that is exactly what it is.
The master_key is different: it is the admin credential for the proxy API itself. It should never be handed to an application. Applications get virtual keys, which is the next section.
4. Virtual keys, the feature that makes this a platform component
This is the part that turns LiteLLM from a routing convenience into infrastructure.
A virtual key is a credential the gateway issues, stores hashed in Postgres, and attaches policy to. It looks exactly like an OpenAI key to the client, so every SDK on earth already knows how to use it. Attached to it: which models it may call, a spend limit, a reset period, rate limits, a team, and arbitrary metadata.
Tenant A tried to reach the expensive hosted model. Rejected at the gateway, before any provider was contacted, before a single token was spent.
Compare that to the alternative universe where the application holds a provider key directly. In that world the request succeeds, the money is spent, and you find out at the end of the month.
Policy enforced at the boundary is policy that cannot be forgotten by a developer in a hurry.Budgets and rate limits, and how they differ
Three limits that people conflate constantly:
| Limit | Unit | What it protects | What the client sees |
|---|---|---|---|
rpm_limit | Requests per minute | The engine's queue. Stops one client filling every batch slot | HTTP 429, retry sensibly and it recovers |
tpm_limit | Tokens per minute | Actual compute. A single 30,000 token request is worth hundreds of small ones | HTTP 429, same shape |
max_budget with budget_duration | Currency over a window | The invoice. Cumulative, not instantaneous | An error, and no amount of retrying helps until the window resets |
Tenant A is blocked. Tenant B, on the same gateway, hitting the same model server, is unaffected. That is the isolation gate from the volume overview, demonstrated rather than described.
Here is a wrinkle that catches people running local models. LiteLLM computes spend from token counts multiplied by per token prices. For a hosted provider those prices are real. For a model running on my own hardware there is no invoice, so I set them to zero in the config above, which means the budget never trips.
Two honest options:
Keep them at zero and enforce fairness with rpm_limit and tpm_limit, which are the resources actually being contended. This is what the lab does today.
Set a synthetic internal price derived from what the hardware costs per hour divided by the tokens it can produce. That turns spend into a real capacity signal and makes chargeback possible.
The second is the more interesting engineering, and it is Volume 4 work, because the number is only meaningful once GPU hours, utilisation and idle time are being measured properly.
5. Routing, fallbacks and retries
This is the machinery that turns a proxy into something that improves reliability rather than adding a failure point.
Four behaviours worth naming precisely, because their differences are where outages come from:
Retry repeats the same logical request, possibly against a different deployment in the same model group. Good for transient failures such as a pod being rescheduled.
Fallback switches to a different model group entirely when the first is exhausted. This is a quality decision disguised as a reliability feature, because the answer now comes from a different model.
Cooldown takes a deployment out of rotation after allowed_fails failures. This is the one that prevents a hammering loop against a pod that is restarting, and it is the least appreciated of the four.
Context window fallback is specific to LLMs and genuinely clever: when a request exceeds the model's context length, route it to a model with a bigger one instead of returning an error. A 6,000 token prompt against a 4,096 token model becomes a successful answer somewhere else rather than a support ticket.
That is the whole argument in one experiment, and it is worth being precise about who did what.
The engine did nothing clever. It died. The client did nothing clever either: it sent five ordinary requests and got five ordinary answers, with no retry logic of its own.
The gateway noticed, retried, cooled down the dead deployment, and sent traffic to the other member of the model group.
Now imagine that same failure with direct connections. Every application would need its own retry logic, its own cooldown, and its own knowledge that a second backend exists. Reliability written once at the boundary beats reliability written five times in five languages.
Fallbacks have a sharp edge and it deserves a warning rather than a footnote.
If qwen2.5-1.5b-instruct falls back to a much stronger hosted model, requests keep succeeding, and the answers get better. Nobody complains. Nobody notices. The bill notices, about three weeks later.
The reverse is worse: falling back to a weaker model produces answers that are still fluent and quietly less correct, which in a RAG system is nearly undetectable without evaluation.
So two rules in AIForge. Every fallback is logged with the model that actually answered, and the response carries that in its metadata rather than pretending it came from the requested model. And an alert fires on fallback rate, not just on error rate, because a fallback is a failure that returned 200.
6. Usage tracking, and why it is the quiet killer feature
Every request through the gateway writes a spend log row: key, team, model, input tokens, output tokens, computed cost, latency, timestamp, and whatever metadata the key carries.
That table answers the questions that otherwise require a project.
Nine calls to escape-hatch from tenant A, whose key is not allowed to call it.
Those are fallbacks. Something failed nine times in the last day and was silently rescued by the hosted model, and the only reason I know is that the spend log records the model that answered rather than the model that was asked for.
This is the single most valuable row in the entire chapter. It is a hidden dependency on an external provider, invisible in every dashboard that tracks error rates, and it would have shown up first as a line item on a credit card.
The same hook feeds LLM specific observability. LiteLLM's callbacks push prompts, responses, token counts and latencies to Langfuse, which is where the request level view lives. That wiring belongs to Volume 4, alongside the Prometheus and tracing work that the APM project already prototyped with SigNoz on a k3smp cluster.
Prompt and response logging captures whatever users typed. In a RAG system it also captures chunks of internal documents, which is arguably worse.
That is a data protection decision, not a debugging preference. AIForge logs metadata always, and full payloads only for explicitly opted in namespaces with a retention window. The gateway is the correct place to enforce that, precisely because it sees everything.
Being the chokepoint for all traffic makes you the chokepoint for all sensitive traffic. That is a responsibility, not just a capability.
7. Why a gateway beats direct connections
The summary, stated as the tradeoff it is.
| Concern | Direct connections | Through the gateway |
|---|---|---|
| Changing model backends | A code change per application, coordinated | One config edit, nobody is told |
| Credentials | Provider keys in N applications, rotated never | Virtual keys per tenant, revoked in one command |
| Cost attribution | One invoice, no breakdown | Per key, per team, per model, per request |
| Rate limiting | Per application, uncoordinated, therefore useless | Global and per tenant, enforced before the engine |
| Failover | Reimplemented per application, tested never | One retry, cooldown and fallback policy |
| Observability | Whatever each team remembered to instrument | Uniform, complete, including the platform's own calls |
| Added latency | None | A few milliseconds of proxy overhead |
| Failure blast radius | One application at a time | Everything. This is the real cost, and section 8 is about it |
The latency row deserves a number rather than a shrug. Measured in the lab, the proxy adds roughly 4 to 9 milliseconds per request. Against a CPU inference call taking 30 seconds, that is noise below the level of caring. Against a 200 millisecond embedding call it is 3 percent, which is still fine. It only becomes an argument at very high request rates against very fast models, and at that point you are running a different kind of system.
8. Failure modes, named
Every request in the platform goes through this one component. Including the RAG service's calls. Including the platform's own. If LiteLLM is down, the platform is down, no matter how healthy the model servers look.
That is a genuine cost of the design, and the honest response is to treat it like the critical path it is:
More than one replica, on different nodes, which is why the manifest says replicas: 2.
A PodDisruptionBudget, so a node drain cannot take both at once.
Stateless pods. All state is in Postgres, so any replica can serve any request and a restart loses nothing.
Postgres becomes critical too. Uncomfortable but true: the keys live there. LiteLLM caches key data, so a brief database blip is survivable, but a long outage is not.
A gateway concentrates risk in exchange for concentrating control. That is a good trade, and it is only good if you actually pay for the availability side of it.
The rest of the list, from things that bit me or that I have watched bite others:
| Failure mode | What it looks like | What to do about it |
|---|---|---|
| Retry storm | A backend slows down, retries triple the load, it dies properly | Low num_retries, a real cooldown_time, and client side backoff. Retries are not free capacity |
| Streaming plus fallback | The stream fails after 40 tokens. The fallback cannot un send them, so the client gets a spliced answer | Accept that mid stream failover is not seamless. Fail the request cleanly and let the client decide |
| Token counting drift | Gateway estimated token counts do not match what the engine reports | Trust the usage object from the backend when it is present. Estimates are for pre flight limits only |
| Timeout mismatch | Gateway times out at 60s, the engine happily generates for 90s, work is thrown away | Gateway timeout above the slowest realistic backend. On CPU that means minutes, and the ingress needs telling too |
| Key sprawl | Two hundred keys, nobody knows who owns forty of them | Mandatory owner metadata and an expiry on every key. A key without an owner is an incident waiting to be attributed |
| Config drift | Someone edits models through the admin API, the ConfigMap in git no longer matches reality | Pick one source of truth. AIForge uses the ConfigMap and treats the admin API as read only for models |
| Readiness probe too strict | One unhealthy backend makes the whole proxy unready, so healthy models become unreachable | Readiness should mean the proxy can serve something, not that every backend is perfect |
What I would watch
- Fallback rate per model group. A fallback is a failure that returned 200, and nothing else will tell you.
- Cooldown events. A backend entering cooldown is the earliest honest signal that something downstream is unwell.
- 429 rate per key. Distinguishes a tenant that needs a bigger quota from a tenant with a runaway loop.
- Proxy p99 latency minus backend latency. If the gap grows, the problem is the gateway, not the model, and that is a very different investigation.
Next
There is now one front door. Authenticated, budgeted, rate limited, load balanced, failing over, and logging everything including its own internal calls. Applications hold a base URL and a key, and know nothing about Kubernetes.
The last piece is knowledge. The models being served know nothing about your documents, your policies, or anything written after their training cutoff, and asking them anyway produces fluent invention.
RAG pipeline and Qdrant builds the retrieval layer end to end: ingestion, chunking, embeddings, a Qdrant collection with tenant filters, reranking, context assembly, and an evaluation harness that tells you when retrieval is quietly failing. It is also the chapter where every component built so far finally gets used at once.