Read this before anything else: there is no GPU in this lab

I do not own a GPU. Not a small one, not a rented one running permanently. Everything in this volume runs on CPU cores inside Multipass VMs on my own machine.

That is not a disclaimer buried at the bottom of the page. It is the single constraint that shaped every decision in Volume 2, and I would rather you know it in the first ten seconds than discover it when a benchmark number looks strange.

What it means in practice: the lab proves wiring, contracts, behaviour and failure handling. It does not prove throughput. Section 4 below draws the line precisely.

Volume 2 The AI Serving Layer

Volume 1 ended with a real Kubernetes cluster. Real nodes, real kubelets, real CNI, built with k3smp inside actual Multipass virtual machines rather than containers pretending to be nodes.

And that cluster could not answer a single question.

That is the gap Volume 2 closes. By the end of it, an application holding nothing but an API key and a base URL can send an OpenAI-shaped request, have it authenticated, budgeted, routed, enriched with private documents retrieved from a vector database, generated by a model server running inside the cluster, and logged on the way out. No component in that chain knows or cares which model is behind it.

Here is the part everybody gets wrong about AI infrastructure: the model is the easy bit. You can run a model in one command on a laptop. What you cannot do in one command is answer the questions that follow. Who is allowed to call it. What happens when it is down. How much did team B spend last week. Which document did that sentence come from. How do I swap the model without redeploying nine applications. Every one of those questions is answered by a piece of infrastructure that is not the model.

1. The four moving parts

Volume 2 introduces four things. They are easy to confuse because all four sit somewhere between "an application" and "a model", so it is worth fixing the mental model before any YAML shows up.

Part	What it is	The question it answers	What it costs you to skip it
vLLM	The inference engine. The process that actually holds model weights in memory and turns tokens into tokens	How do I run this model efficiently and expose it over HTTP	You write your own batching and KV cache management, badly, and get a fraction of the hardware you paid for
KServe plus MLflow	The lifecycle layer. A Kubernetes CRD that owns model deployments, and a registry that owns model versions	Which version is in production, and how did it get there	Model versions live in someone's head and in a hand edited image tag
LiteLLM	The gateway. One OpenAI-compatible front door for every model, local or hosted	Who called what, with which key, at what cost, and where does traffic go when a backend dies	Every application hardcodes a model URL, and you find out about spend at the end of the month
RAG plus Qdrant	The knowledge layer. An ingestion pipeline and a vector database holding embeddings of private documents	How does a general model answer questions about our specific documents	Confident, fluent, completely invented answers

The one sentence version

vLLM runs the model. KServe decides what running means. MLflow remembers which version. LiteLLM decides who may ask. Qdrant supplies what the model does not know.

If you keep only one thing from this page, keep that. Every chapter in this volume is an expansion of one clause in that sentence.

2. How they fit together

This is the whole serving layer on one screen. Everything above the dashed boundary is the platform. Everything below it is a workload that the platform manages.

Preparing diagram

Two things in that diagram are deliberate and worth saying out loud.

The RAG service calls back into the gateway. It does not call vLLM directly, even though it could and it would be one hop shorter. If retrieval bypassed the gateway, then the tokens it spends would be invisible to budgets, invisible to spend logs, and invisible to rate limits. A tenant could blow through a quota simply by asking questions with long contexts. Everything that spends tokens goes through the front door, including the platform's own components.

MLflow has a dashed arrow. It is not on the request path. Not once, not for anything. It is consulted at deployment time to answer "which artifact is version 4 of this model", and then it can be down for a week without a single user request failing. Keeping registries off the hot path is one of those decisions that costs nothing on day one and saves an outage on day two hundred.

3. A single request, end to end

The diagram above shows components. This one shows time, which is where the interesting failure modes live.

Preparing diagram

Look at step 9. The reranking and assembly step is pure CPU work in my lab, and on a small machine it can take longer than the retrieval it is cleaning up. That is not a diagram detail, it is the kind of thing that turns into a latency budget argument in Volume 4. I would rather draw it now than pretend the pipeline is only network hops.

4. The CPU reality, stated without apology

Now the constraint, properly.

What CPU inference actually feels like

On a 4 vCPU Multipass VM, Qwen2.5 1.5B Instruct in a CPU inference server produces roughly 4 to 9 tokens per second for a single request, with a time to first token of one to three seconds on a short prompt. A 300 token answer therefore lands somewhere between forty seconds and a minute.

Load a second concurrent request and both get slower, because there is no spare compute to hide behind. On a GPU, that second request is nearly free until the batch fills up. On CPU there is no such thing as nearly free.

So I do not run throughput benchmarks in this lab and quote them as results. That would be dishonest and, worse, useless. A number produced on the wrong hardware is not a small version of the right number.

What that leaves is still a lot. Nearly everything a platform engineer is judged on has nothing to do with tokens per second. Here is the split I hold myself to.

Capability	Lab treatment	Why
OpenAI-compatible API contract	Real. Real request and response shapes, streaming, tool call fields, usage accounting	This is the contract every application depends on. It is hardware independent, so there is no excuse for faking it
Gateway behaviour: keys, budgets, rate limits, fallbacks	Real. Real Postgres backed keys, real spend rows, real failover when I kill a backend	Entirely a control plane concern. A GPU would change nothing about it
RAG pipeline correctness	Real. Real parsing, chunking, embedding, real HNSW index, real tenant filters, real citations	Retrieval quality is decided by chunking and filtering, not by how fast the generator runs
KServe lifecycle and autoscaling behaviour	Real. Real InferenceService objects, real scale to zero, real measured cold starts	Cold start pain is worse on CPU, which makes it easier to study, not harder
Observability and failure handling	Real. Real metrics scraped, real traces, real chaos tests where I delete pods mid stream	Prior art already exists in the APM project, which stood up SigNoz on a k3smp cluster
Token throughput and latency at scale	Deferred. Measured, recorded, never presented as a performance result	CPU numbers do not extrapolate to GPU numbers in any useful way
PagedAttention and continuous batching under real load	Deferred. Explained and configured, exercised only lightly	Both are memory management wins that only show their value when the accelerator is saturated
Tensor parallelism across GPUs	Deferred. Designed, documented, one flag away	Needs at least two GPUs to mean anything. The manifest change is four lines and it is written down
Large models, 7B and above	Deferred. The lab tops out around 1.5B parameters, plus small embedding models	A 7B model at bf16 wants roughly 15 GiB of weights before any KV cache. My VM has 8 GiB total
NVIDIA GPU Operator, device plugin, MIG	Deferred. Installation path documented, never installed	Installing a GPU operator on a cluster with no GPU produces a pod stuck in Init and zero learning

The rule that makes the deferred column safe

Every deferred item is a configuration change, not a redesign. That is the bar I set for myself, and I check it chapter by chapter.

Concretely, when a GPU arrives: install the GPU Operator, add nvidia.com/gpu: 1 to the pod resources, swap the CPU image for the CUDA image, set --gpu-memory-utilization and optionally --tensor-parallel-size, and raise --max-model-len. Nothing above the engine changes. Not the gateway config, not the RAG service, not a single application.

If a GPU arriving would force me to rewrite the platform, I built the platform wrong.

An honest word about vLLM on CPU

vLLM has a CPU backend, and it works, but it is clearly the second class citizen of that project. It needs a separate build, some optimisations are GPU only, and support for quantised formats is narrower than on CUDA.

So in the lab I keep two paths open. vLLM is the designed target, because it is what I would run the moment there is an accelerator, and it is what the manifests and the KServe runtime are written for. Alongside it I keep a pragmatic CPU path using a llama.cpp based server or Ollama, both of which are genuinely good at quantised CPU inference and both of which expose an OpenAI-compatible endpoint.

That second path is not a cop out, it is the whole argument for a gateway. Because LiteLLM sits in front, swapping the engine underneath is a config file edit. The applications never learn that it happened.

5. Where this runs

Same lab as Volume 1, same tool. The cluster comes from k3smp, which provisions K3s on real Multipass VMs, so nodes have their own kernels, their own network stacks, and their own ways of running out of memory. That last part matters more than it sounds when the thing you are scheduling wants six gigabytes of anonymous memory.

The APM project already did this once, standing up a single node k3s cluster and deploying SigNoz on it. I reuse that pattern rather than reinventing it, and the observability wiring in Volume 4 picks up where it left off.

Two namespace conventions run through every manifest in this volume:

Namespace	What lives there	Who can write to it
`aiforge-system`	LiteLLM, MLflow, Qdrant, KServe controller, the RAG service, shared model servers	Platform engineers only. No tenant ever gets a role here
`aiforge-tenant-a`, `aiforge-tenant-b`, and so on	Tenant applications, tenant specific model deployments, tenant quotas	The tenant, through the control plane, never with direct cluster credentials

The isolation that makes this more than a naming convention, network policies, resource quotas, RBAC and Keycloak identity, is Volume 4 work. Volume 2 respects the boundary so that Volume 4 has something to enforce.

6. How this volume is organised

Four chapters, in build order. Each one leaves the cluster in a state the next one depends on, so read them in sequence the first time.

#	Chapter	What you have when you finish it
2	Serving LLMs with vLLM	An inference engine you understand from the inside: KV cache, PagedAttention, continuous batching. A model server running in the cluster behind an OpenAI-compatible endpoint, and a written down GPU path
3	KServe and the model lifecycle	The same model served through an InferenceService, autoscaling and scaling to zero, plus MLflow holding versions and artifacts and a clear answer to when KServe is worth its complexity
4	LiteLLM the model gateway	One front door. Virtual keys per tenant, budgets and rate limits that actually reject, fallbacks that survive a deleted pod, and a spend table you can query
5	RAG pipeline and Qdrant	Documents in, cited answers out. Chunking you can defend, a Qdrant collection with tenant filters, reranking, an evaluation set, and a catalogue of the ways retrieval lies to you

Why vLLM comes before the gateway

It is tempting to build the gateway first, because it is the part users touch. I build it third on purpose.

A gateway with nothing behind it is a proxy to nowhere, and every interesting property of a gateway, fallbacks, cooldowns, routing, load based decisions, can only be tested against a backend that can genuinely be slow or dead. So I need a real engine first, then something that manages it, then something that fronts it.

RAG comes last because it is the only component that consumes all three.

7. The gate that ends this volume

Volume 2 is not finished when the four chapters are written. It is finished when all of these are true on my cluster.

A vLLM style model server runs in aiforge-system and answers /v1/chat/completions and /v1/models with correct OpenAI-shaped payloads
Streaming works end to end, including partial chunks and a clean terminating event, not just non streaming replies
The model server survives a pod delete and comes back without manual steps, with weights served from a cached volume rather than re downloaded
Prometheus style metrics are exposed and scraped, including queue depth, running requests, and time to first token
The GPU manifest exists in the repository, is documented line by line, and differs from the CPU manifest only in image, resources and engine flags

The last tab is the real gate

The first three are checklists, and checklists can be satisfied by copying manifests. The fourth cannot.

The specific failure I am guarding against is the one that makes AI infrastructure content mostly worthless: a person installs four tools, screenshots four dashboards, and has learned nothing about why any of them exist. If I cannot explain the tradeoff, I did not learn the tool. I learned its README.

8. What this volume deliberately does not cover

Saying no clearly is part of the design.

Not here	Where it lives
The self service API that turns a developer request into these manifests	Volume 3, the platform control plane
Keycloak identity, RBAC, network policies, tenant isolation enforcement	Volume 4
Prometheus, Grafana, Loki, OpenTelemetry and Langfuse wiring	Volume 4, building on the APM project
Cost per token, GPU hour accounting, quota economics	Volume 4
Cluster build, storage classes, ingress, the node layout itself	Volume 1
Fine tuning, training, experiment tracking as a discipline	Out of scope for AIForge. The platform serves models, it does not train them