Projects · AIForge
Serving LLMs with vLLM
What an inference engine actually does. KV cache arithmetic, PagedAttention, continuous batching, the OpenAI-compatible server, a real Kubernetes deployment, and an honest account of running all of it on CPU with the GPU path written down.
There is no GPU in this lab. Every command in this chapter ran on CPU cores inside a Multipass VM, with Qwen2.5 1.5B Instruct and, when I wanted headroom, Qwen2.5 0.5B Instruct.
That changes what this chapter can prove. It cannot prove throughput. It can prove that the engine starts, loads, serves a correct OpenAI-compatible API, exposes metrics, survives restarts, refuses work it cannot do, and behaves predictably when I abuse it.
Section 8 is the GPU path, written as a diff rather than a wish. Everything above the engine is identical on both sides of that line.
Serving LLMs with vLLM
Running a language model is easy. Three lines of Python, a model name, and you have text coming out of a laptop.
Serving one is a completely different job, and the gap between those two sentences is where vLLM lives. This chapter is about what happens inside that process: why the memory runs out before the compute does, why a naive server wastes most of the hardware it is given, and what an inference engine does about it.
I am going to spend the first third of this chapter on theory before a single manifest appears. That is deliberate. Every operational decision later, how much memory to request, why the pod refuses to start, why latency collapses at eleven concurrent users, is a direct consequence of the mechanics in sections 1 to 3. Skip them and the YAML is cargo cult.
1. What an inference engine actually does
Start with the thing itself. A decoder-only language model takes a sequence of tokens and predicts the next one. That is the entire operation. Everything else is repetition and bookkeeping.
Generation happens in two phases with wildly different characteristics, and confusing them is the source of most bad intuition about LLM performance.
| Phase | What happens | Bottleneck | How long |
|---|---|---|---|
| Prefill | The whole prompt is processed in one shot. Every token attends to every earlier token, in parallel, as a few large matrix multiplications | Compute. Big dense matmuls, high arithmetic intensity, exactly what an accelerator is built for | One pass. Cost grows with prompt length, roughly quadratically in the attention part |
| Decode | One token is produced, appended, and fed back in. Repeat until a stop condition | Memory bandwidth. Every single output token requires reading the entire set of model weights from memory to do very little arithmetic | One pass per output token. A 300 token answer is 300 sequential passes |
Here is the fact that reframes the whole problem. To generate one token for one user, the hardware must stream all the model weights out of memory and through the compute units. For a 1.5B parameter model at 16 bit precision that is about 3 GB moved, to produce a couple of bytes of text.
The arithmetic performed on that data is trivial. The compute units sit mostly idle waiting for memory.
Which leads directly to the key insight: if you are reading all the weights anyway, you may as well use them for more than one user at the same time. Serving 16 requests together costs almost the same memory traffic as serving one. That is why batching is not a nice optimisation in LLM serving. It is the optimisation.
So an inference engine is not really "the thing that runs the model". It is the thing that decides, on every single forward pass, which requests get to ride along, and where their intermediate state is kept. The model is a library call. The engine is a scheduler and a memory allocator.
2. The KV cache, and why memory runs out first
To generate token number 400, the model needs to attend to tokens 1 through 399. Recomputing their keys and values every step would be catastrophically wasteful, so they are computed once and kept. That store is the KV cache, and it is the single most important object in LLM serving.
It is also the thing that fills up your memory.
2.1 The arithmetic, on the actual lab model
The size of the KV cache per token is fixed by the model architecture:
For Qwen2.5 1.5B Instruct, which has 28 layers, 2 key/value heads thanks to grouped query attention, and a head dimension of 128, in 16 bit precision:
Twenty eight kibibytes per token sounds like nothing. Now put it in context:
| Scenario | KV cache required |
|---|---|
| One conversation, 4,096 tokens of context | about 112 MiB |
| Sixteen concurrent conversations at 4,096 tokens | about 1.8 GiB |
| Sixty four concurrent conversations at 8,192 tokens | about 14 GiB, which is more than the weights by a factor of nine |
And Qwen2.5 is the friendly case, because grouped query attention collapses 12 attention heads down to 2 key/value heads. A model without GQA is brutal. The classic example from the PagedAttention paper is a 13B model where a single sequence at 2,048 tokens needs roughly 1.6 GB of KV cache. Two dozen users and the cache alone outweighs the model.
Everybody sizes for weights. "The model is 7B at bf16, so 15 GB, so a 24 GB card is fine."
Then they put it in production, twenty people use it, and it falls over. Because the weights are a fixed cost and the KV cache is a cost per concurrent token, and only the second one grows with your success.
Memory available for KV cache is what determines your maximum concurrency. Not FLOPs, not core count. Concurrency is a memory question wearing a compute costume.
2.2 Where the memory actually goes
vLLM does this calculation at startup. It profiles memory, subtracts the weights and the activation peak, and declares everything left over as the KV cache pool. That pool, divided by the per token cost, is printed in the logs as the number of tokens it can hold concurrently. It is the most useful line in the whole startup output and almost nobody reads it.
3. PagedAttention and continuous batching
Two ideas, both about waste. They are the reason vLLM exists and the reason it is worth learning rather than just installing.
3.1 PagedAttention: stop allocating for the worst case
The naive way to manage a KV cache is to give each request one contiguous block of memory, sized for the longest output it might produce. If max_model_len is 4,096, every request reserves 4,096 tokens of space the moment it arrives.
Now consider what actually happens. A user asks a short question and gets a 60 token answer. The system reserved 4,096. It used 1.5 percent of the reservation. The rest was unusable by anyone else for the whole life of that request.
The PagedAttention paper measured this in real systems and found that 60 to 80 percent of KV cache memory was wasted to internal fragmentation and over reservation. Most of the expensive memory in the machine was doing nothing.
The fix is borrowed wholesale from operating systems. Split the KV cache into fixed size blocks, typically 16 tokens each. Allocate blocks on demand as a sequence grows. Keep a block table per sequence mapping logical positions to physical blocks, which do not have to be contiguous. This is virtual memory with paging, applied to attention state.
Two consequences fall out of paging that are easy to miss:
Prefix sharing becomes free. If ten requests share the same 800 token system prompt, they can point at the same physical blocks instead of storing ten copies. In a RAG system, where every request carries a similar instruction block, this is a real saving rather than a theoretical one.
Preemption becomes possible. Because a sequence's state is a list of blocks rather than one slab, the engine can evict a request, free its blocks, and either recompute or swap them back later. That turns "out of memory, crash" into "out of memory, this request goes back in the queue and gets slower". A degraded response beats a 500.
3.2 Continuous batching: stop waiting for the slowest request
Static batching is the intuitive design. Collect requests, run them together, return results, collect the next batch.
It performs terribly for text generation, for one specific reason: outputs have wildly different lengths. Batch eight requests where seven finish in 40 tokens and one runs to 900, and seven slots in your expensive batch sit idle for 860 steps. They are not just idle, they are occupying their KV cache blocks the entire time, blocking new arrivals.
Continuous batching, also called iteration level scheduling, changes the unit of decision from "a batch" to "a single forward pass". After every step the scheduler asks again: who finished, who can be dropped, who is waiting, who fits.
The practical effect is that the engine keeps the hardware busy instead of keeping a batch tidy. Throughput improvements over static batching are measured in multiples, not percentages, and the queueing behaviour is far more graceful under bursty load.
Continuous batching means your latency depends on your neighbours. A request that would take 4 seconds alone may take 7 when the batch is full, because each decode step now processes more sequences.
That is a real tradeoff, not a free win, and it is why the metrics in section 9 matter. You are trading per request latency for total throughput, and you need to be able to see where on that curve you are sitting. A serving system with no queue depth metric is a serving system you cannot reason about.
4. The OpenAI-compatible server
Everything above is a library. What makes vLLM usable as platform infrastructure is that it ships an HTTP server speaking the OpenAI API, which has become the de facto interface that every SDK, framework and tool already knows.
The endpoints that matter:
| Endpoint | Purpose | Why the platform cares |
|---|---|---|
GET /v1/models | Lists served model IDs | Cheap liveness signal, and the thing LiteLLM introspects when registering a backend |
POST /v1/chat/completions | The main event. Messages in, completion out, optionally streamed as server sent events | The contract every tenant application writes against |
POST /v1/completions | Legacy prompt style completion | Some older tooling still speaks it. Cheap to keep |
POST /v1/embeddings | Vector embeddings, when serving an embedding model | The RAG pipeline uses this shape, whichever server implements it |
GET /health | Returns 200 when the engine is up | Kubernetes readiness and liveness probes |
GET /metrics | Prometheus exposition of engine internals | Section 9. This is where operational truth lives |
It is tempting to see the OpenAI shape as a detail. It is the most strategically important property in this entire volume.
Because the interface is standard, the engine becomes replaceable. vLLM on GPU, a llama.cpp server on CPU, a hosted provider during an outage, all speak the same dialect. The gateway can route between them and no application ever notices.
I have watched teams hand roll a bespoke inference API because it felt cleaner. Two years later they own an SDK, a client library in three languages, and a migration project. Adopting a boring standard interface is the cheapest optionality you will ever buy.
Here is the server answering, in the lab:
That usage object is not decoration. It is the raw material for every budget, quota and cost number in Volume 4. An engine that does not report token usage honestly makes the whole cost story guesswork.
5. Deploying it on Kubernetes
Now the manifests. The cluster is the k3s environment from Volume 1, built with k3smp on real Multipass VMs.
5.1 Weights need a home before anything else
My first deployment had no volume for the model cache. It worked. The pod pulled about 3 GB of weights from Hugging Face on startup, loaded them, and served requests.
Then I restarted it. Another 3 GB. Then a node drained and the pod moved. Another 3 GB, and this time the download failed halfway because my home connection had opinions, leaving a pod in CrashLoopBackOff with an error about a corrupt safetensors file.
Model weights are data, and data belongs on a volume. Treating a multi gigabyte download as part of pod startup is the AI equivalent of running apt install in an entrypoint.
5.2 The Deployment
The startupProbe. Without it the liveness probe starts counting during weight loading, fails, restarts the pod, and you get an infinite loop where the pod never lives long enough to finish loading. This is the single most common way a first LLM deployment fails, and the symptom looks nothing like the cause.
No CPU limit. A throttled decode loop does not fail, it crawls, and crawling shows up as client timeouts rather than as a resource problem. Requests are set so the scheduler still places the pod honestly.
A memory limit that is real. The KV cache is the thing most likely to grow without bound. I want that to be an OOMKilled event on one pod with a clear signal, not a node under memory pressure evicting Qdrant.
The shm mount. Empty by default in Kubernetes at 64 MiB, and multi process inference wants shared memory. On a single CPU replica it rarely bites. The moment tensor parallelism is switched on, an undersized /dev/shm produces a hang with no useful error at all.
5.3 Watching it come up
Maximum concurrency for 4096 tokens per request: 17.9xThat line is vLLM telling you exactly what it can hold. Two GiB of KV cache pool, divided by 112 MiB per full length sequence, gives roughly 18 sequences at full context.
This is the number that decides whether request nineteen queues. It is the number to compare against your --max-num-seqs. And it is the number that moves when you change --max-model-len, which is why context length is an infrastructure decision and not a product preference.
If that number is below 1, the server will refuse to start, and it is right to.
6. The CPU reality
Time to be specific rather than hand wavy.
Single request, Qwen2.5 1.5B Instruct, 4 vCPU, 8 GiB VM, short prompt:
- Time to first token: 1.2 to 3.0 seconds
- Decode rate: 4 to 9 tokens per second
- A 300 token answer: 40 to 70 seconds
- Second concurrent request: both slow down by roughly 40 percent, because there is no idle compute to absorb it
With Qwen2.5 0.5B Instruct the decode rate roughly triples and quality drops off a cliff for anything requiring reasoning. It is a useful model for testing plumbing and a poor model for answering questions.
vLLM's CPU backend exists and works, but it is not the project's centre of gravity. It requires a separate image, some kernels and optimisations are CUDA only, and the supported quantisation story is thinner.
So the lab runs a pragmatic split, and I think it is the correct engineering answer rather than a compromise:
vLLM is the designed target. The manifests, the KServe runtime, the gateway entries and the metric names in this volume are all written for it, because it is what runs the moment an accelerator exists.
A llama.cpp based server or Ollama is the pragmatic CPU path. Both are genuinely good at quantised CPU inference, both expose an OpenAI-compatible endpoint, and a 4 bit quantised 1.5B model on llama.cpp comfortably beats the same model unquantised under vLLM's CPU backend on my hardware.
The reason this is not a mess is the gateway. To a tenant application, both are the model qwen2.5-1.5b-instruct at the same URL. Swapping which one is behind that name is a config change in LiteLLM, covered in the gateway chapter.
7. Quantisation, the lever that actually helps on CPU
Since I cannot add compute, the honest lever is to make the model smaller. Quantisation stores weights at lower precision, which cuts both memory and, because decode is bandwidth bound, the time spent moving those weights.
| Precision | Weights for a 1.5B model | Practical note |
|---|---|---|
| bf16 or fp16 | about 3.1 GiB | The baseline. What the lab runs under vLLM on CPU |
| int8 | about 1.6 GiB | Quality loss is usually small. Kernel support varies by backend |
| 4 bit, for example Q4_K_M in GGUF | about 0.9 GiB | Roughly 3x less memory traffic per token. This is the CPU sweet spot, and it is llama.cpp territory |
The tradeoff is real and I will not pretend otherwise. Aggressive quantisation degrades quality, and it degrades it unevenly: chat and summarisation hold up well, precise instruction following and arithmetic degrade first. In a RAG system the degradation shows up as the model paraphrasing a retrieved passage slightly wrong, which is exactly the failure mode that is hardest to notice.
Given a fixed memory budget, a quantised larger model usually beats a full precision smaller one. A 4 bit 3B model tends to be more useful than an fp16 1.5B, for about the same bytes.
That heuristic breaks at very aggressive quantisation, below 4 bits, where models start losing coherence in ways that benchmarks catch late.
8. The GPU path, written as a diff
This section exists so that "we will add a GPU later" is a plan and not a hope.
The cluster side first. A Kubernetes node cannot schedule GPU work until something advertises the resource, and that something is the NVIDIA GPU Operator, which installs the driver, the container toolkit and the device plugin, then labels the node. After that, nvidia.com/gpu appears in kubectl describe node as an allocatable resource and the scheduler can do its job. Nothing about the Kubernetes scheduling model changes, GPUs are simply an extended resource that cannot be oversubscribed.
Then the workload side, which is the whole change:
No change to the Service. No change to the port. No change to the endpoint path. No change to the served model name, which is deliberate: the public name stays qwen2.5-1.5b-instruct right up until I decide to publish a new name, so a hardware upgrade is not an API break.
No change to LiteLLM, to the RAG service, or to any tenant application.
That is the definition of a configuration change rather than a redesign, and it is the property the whole volume is built to protect. The abstraction earns its keep on exactly this day.
Fractional GPUs do not exist. nvidia.com/gpu: 0.5 is not valid. Two pods cannot share a card through the standard device plugin. Sharing needs MIG on supported hardware, or time slicing configured explicitly, and both have caveats. Plan one model per GPU until proven otherwise.
--gpu-memory-utilization is a fraction of the whole card, not of what is free. Set it to 0.9 on a card that already has something else on it and the engine will try to allocate memory that is gone, then fail late and confusingly.
Cold start gets worse, not better. Bigger models mean bigger downloads and longer loads. The startupProbe tuned for CPU is not generous enough for a 70B model, and the resulting CrashLoopBackOff looks exactly like a broken image.
9. The metrics that tell you the truth
vLLM exposes Prometheus metrics on /metrics, and a handful of them are worth more than every dashboard panel you will be tempted to build.
| Metric | What it tells you | What to do when it moves |
|---|---|---|
vllm:num_requests_running | Sequences in the current batch | Pinned at max-num-seqs means you are at the concurrency ceiling |
vllm:num_requests_waiting | Requests queued, admitted by nobody yet | Persistently above zero is the clearest scale up signal there is |
vllm:gpu_cache_usage_perc | Fraction of the KV cache pool in use. Reported for the CPU cache too | Sustained near 1.0 means preemption is imminent and latency is about to get strange |
vllm:time_to_first_token_seconds | Histogram of prefill plus queue time | This is what a user experiences as responsiveness. Alert on the p95 |
vllm:time_per_output_token_seconds | Decode speed per token | Rises as batch size rises. That is expected, and it is the batching tradeoff made visible |
vllm:num_preemptions_total | Sequences evicted due to KV pressure | Anything other than zero on a steady workload means the memory budget is wrong |
Eight running, four waiting. --max-num-seqs=8 is doing exactly what I told it to, and the queue is absorbing the rest instead of the memory allocator finding out the hard way.
Cache usage at 0.41 with twelve requests in flight says the KV pool is not the binding constraint here. Compute is. On a GPU that ratio typically inverts, and the cache becomes the thing you run out of first.
Zero preemptions means nothing has been evicted. Running plus waiting plus cache usage plus preemptions is the minimum viable dashboard for an inference engine. Four numbers. Everything else is decoration.
10. What went wrong, and what I watch now
Recorded honestly, because these all cost me an evening.
| Symptom | Actual cause | How I found it |
|---|---|---|
Pod in CrashLoopBackOff, logs stop mid weight load with no error | OOMKilled. Weights plus KV pool plus activations exceeded the memory limit | kubectl describe pod showed the reason. The logs never mention it, which is why people blame the model |
| Pod restarts forever, never becomes ready, model download restarts each time | Liveness probe firing during a slow first load. No startupProbe | Restart count climbing on a fixed cadence, roughly probe period times failure threshold |
| Engine refuses to start, complains the model max length exceeds what the KV cache can hold | --max-model-len=8192 with only 2 GiB of KV pool. The arithmetic in section 2 did not close | The startup log states both numbers. Fix is to lower the context or raise the pool, and both are real tradeoffs |
| Streaming works with curl, arrives all at once through the ingress | A proxy buffering the server sent event stream | Compared a direct port forward against the ingress path. Fixed with proxy buffering disabled on that route |
| Latency fine at eight concurrent users, terrible at twelve, no error anywhere | Queueing. Exactly as configured, but invisible without the waiting metric | vllm:num_requests_waiting above zero. This is not a bug, it is the system working and me not looking |
| Client sees a truncated answer with no error when a node is drained | Pod terminated mid stream. Server sent events have no built in resumption | Deliberate chaos test. Led to fallback and retry policy at the gateway, not at the engine |
Five of those six failures produce a symptom that points somewhere other than the cause. An OOM kill looks like a corrupt model. A probe misconfiguration looks like a networking problem. Queueing looks like the engine being slow.
Inference engines fail quietly and blame themselves. Which is precisely why section 9 exists, and why I would rather have four correct metrics than forty panels.
What I would watch in production
Three alerts, and I would resist adding a fourth until something hurt:
vllm:num_requests_waitingabove zero for more than two minutes. Capacity, not health.vllm:num_preemptions_totalincreasing at all on a steady workload. The memory budget is wrong.- p95 of
vllm:time_to_first_token_secondsabove target. The only one a user would recognise as a complaint.
Next
There is an engine running in the cluster, it speaks a standard API, and I know what it does with its memory.
What it does not have is a lifecycle. That Deployment is a hand written manifest with a hardcoded model name. Nobody can tell me which version is serving, nothing scales it down when it is idle, and promoting a new model means editing YAML and hoping.
KServe and the model lifecycle puts a real abstraction on top of this pod, and MLflow puts a real registry behind it. It also answers the question this chapter deliberately left open: whether that abstraction is worth what it costs.