Projects · AIForge
The lab on k3smp
Building the AIForge cluster on real Multipass VMs with k3smp: the resource budget, namespaces, quotas and default deny networking, a small instruct model answering on CPU, and the deviation register that keeps this project honest.
The lab on k3smp
Time to build the thing.
Everything in the previous two documents was reasoning. This one is the cluster that the other three volumes deploy onto, and it is where the design starts getting contradicted by reality, which is the point of building it.
A running K3s cluster on real virtual machines, created and destroyed with one command each. Namespaces with quotas, limits, and default deny networking, all created by code. A small instruct model answering a prompt on CPU inside the cluster. A measured tokens per second number, written down honestly. And a deviation register listing every gap between this lab and production.
1. The constraint, stated before the design
I am building an AI platform on the laptop I also work on, with no GPU. Pretending otherwise would make every later decision unreadable, so here is the ledger first.
| Resource | What I have | Reality check |
|---|---|---|
| CPU | 8 threads | Enough for a small instruct model, nowhere near enough for a 7B at usable speed |
| RAM | 15 GiB usable | The hard ceiling on this entire project. Everything below is arithmetic against this number |
| Disk | Model weights plus images plus volumes | Weights are the surprise cost. Small models keep it manageable |
| GPU | None | The defining constraint of this lab, and the reason for the deviation register in section 8 |
The instinct is to pick a model you want to serve and then find hardware for it. On a laptop that gets you an OOM kill and a wasted evening.
So I inverted it. RAM budget first, then model. A cluster of virtual machines needs its own memory, K3s needs memory, the platform components need memory, and whatever is left over is what the model gets. Work it out in that order and the model choice makes itself.
The output is unglamorous: a 1.5B parameter model, and I am fine with that, because everything this volume needs to prove is proven just as well by a small model as a large one.
The budget
Rough allocation, and rough is honest here because container memory usage moves around.
| Consumer | Memory | Notes |
|---|---|---|
| Host OS and my own tooling | Reserved first | Non negotiable. Starving the host to feed the lab makes both unusable |
| Control plane VM | ~2 GiB | K3s API server, scheduler, controller manager, embedded datastore |
| Worker VM for platform components | ~4 GiB | PostgreSQL, Qdrant, LiteLLM, observability |
| Worker VM for AI workloads | ~6 GiB | Model server plus KV cache plus embedding model |
The arithmetic above does not survive my browser, my editor, and a dev server all running.
So the lab is designed as a deliberate mode with a documented shutdown of everything else first. I would rather tell you that on page one than have you discover it when the OOM killer reaps the model server mid demo.
2. Why real virtual machines
k3smp is a tool I wrote that provisions K3s clusters inside real Multipass virtual machines rather than Docker containers.
The trade is obvious: it is slower to create than k3d or kind, and it costs more memory. Here is what I get back.
| Capability | Containers as nodes | Real VMs | Why AIForge cares |
|---|---|---|---|
| Node isolation | Shared host kernel | Separate kernel per node | Resource pressure on one node stays on that node, which is the whole point of a saturation drill |
| Node loss drill | Simulated | Genuine VM shutdown | Volume 4 kills a node and watches rescheduling for real |
| Memory limits | Fuzzy, shared with host | Hard VM ceiling | OOM behaviour matches production instead of taking the laptop down with it |
| Scheduling and affinity | Mostly meaningful | Fully meaningful | Placing AI workloads on a specific node is the rehearsal for GPU node selectors |
AIForge is not k3smp's first real deployment. The APM project already used it to run a full SigNoz install with OpenTelemetry collectors across the cluster.
That is not a nostalgia note, it is a risk reduction. The bootstrap path, the local-path storage behaviour, the NodePort access pattern, and the kubeconfig handling were all already debugged before AIForge existed. When something breaks here, I know it is AIForge's fault rather than the lab's.
3. Creating the cluster
k3smp is configuration driven, which matters because a lab you cannot recreate identically is not a lab, it is a pet.
The cluster configuration
Three decisions in that file are worth explaining rather than skipping past.
metrics-server is on because kubectl top and the Horizontal Pod Autoscaler both need it, and I want autoscaling working from day one rather than retrofitted.
The bundled monitoring stack is off deliberately. Observability in AIForge is a designed part of the platform, not a checkbox at cluster creation. I install Prometheus, Grafana, Loki, OpenTelemetry, and Langfuse myself in Volume 4, through Helm and Argo CD, because how they get installed is part of what the project is demonstrating.
One worker would fit the memory budget more comfortably. Two exist so that scheduling is a real decision.
With one worker, node selectors, affinity rules, disruption budgets, and node loss drills are all theatre. With two, the AI workload lands on a specific node because I told it to, and Volume 4 can shut that node down and watch what happens. That is worth the extra memory.
Creating it
Three nodes, three real kernels, three real memory ceilings.
Labelling the nodes for their job
Scheduling only means something if nodes are distinguishable. This is also the rehearsal for GPU node selectors later.
Right now aiforge.io/workload=inference just means "the node with more headroom for a model server".
When a GPU exists, that same label is where the GPU node selector goes, alongside the nvidia.com/gpu resource request. The scheduling logic in the control plane does not change, only what the label points at.
4. Namespaces, quotas, and default deny
The cluster is up. Before a single workload lands on it, the guardrails go in.
Order matters here and I want to be emphatic about it.
It is genuinely tempting to deploy the model first, see it work, and add quotas and NetworkPolicy afterwards.
Do not. Once a workload runs without constraints, adding them later means breaking a thing that currently works, and there is always a reason to do that next sprint. Constraints applied to an empty namespace cost nothing. Constraints applied to a running system cost a negotiation.
The namespace layout
The quota, including the GPU line that is currently zero
requests.nvidia.com/gpu: "0" in a cluster with no GPUs looks pointless. It is one of the more useful lines in this file.
Two reasons. First, it means the quota schema is already correct, so enabling GPU for a tenant is editing a number rather than designing a policy under pressure. Second, it makes the intent explicit and reviewable: this tenant is deliberately not entitled to GPU, which is a different statement from nobody having thought about it.
When hardware arrives, the change is one integer. That is the whole design goal.
Default deny first
Then, and only then, the explicit allows: DNS resolution, egress to the LiteLLM gateway in aiforge-system, and ingress from the observability namespace for scraping.
NetworkPolicy is only enforced if your CNI implements it. K3s ships Flannel with a network policy controller by default, so it does work, but the failure mode when it does not is silent: your policy is accepted by the API server and simply never enforced.
That is the worst kind of security control, one that reports success while doing nothing. So it gets tested rather than assumed, and Volume 3 does exactly that with a pod that actively tries to reach across the boundary.
5. Getting a model to answer, on CPU
Now the part that makes it an AI platform rather than a Kubernetes cluster.
Choosing the model
| Model | Why it is on the list | Verdict for this lab |
|---|---|---|
| Qwen2.5 0.5B Instruct | Tiny, fast to load, follows instructions surprisingly well for the size | Smoke tests and CI. This is what pipelines use |
| Qwen2.5 1.5B Instruct | Still fits comfortably, noticeably better answers, usable for RAG demos | The default. Most of the lab runs on this |
| TinyLlama 1.1B Chat | A second family, so nothing accidentally depends on one vendor's quirks | Comparison and routing tests. Two models makes the gateway meaningful |
| all-MiniLM-L6-v2 | Small, fast, 384 dimension embeddings, well understood | Embeddings for RAG. Cheap enough to run alongside everything else |
vLLM is the designed inference engine for AIForge, and its whole reason for existing is GPU efficiency: PagedAttention and continuous batching are about using expensive accelerator memory well.
On CPU, most of that advantage is unavailable. vLLM does have a CPU path, and it works, but running it here is about exercising the same API surface and the same deployment shape, not about performance.
Where CPU throughput actually matters for day to day iteration, llama.cpp based servers and Ollama are the pragmatic option, and because everything sits behind an OpenAI compatible gateway, swapping the backend changes one route entry and nothing else.
That swap being trivial is itself the architecture working. Volume 2 goes through this properly.
Deploying it
The model cache PVC. Without it, every pod restart re downloads gigabytes of weights. With node local local-path storage this cache is tied to one node, which is a real limitation and it is in the deviation register.
The 120 second readiness delay. Loading weights is slow, and a model server that is up but not loaded will fail probes and get killed in a restart loop that looks exactly like a crash. This single number is one of the most common reasons a first model deployment "does not work".
Does it answer
That is the moment the project becomes real. A model, in a pod, on a node I provisioned, answering through an OpenAI compatible API.
The number I promised to write down honestly
On this hardware, a 1.5B parameter model on CPU produces tokens at a rate best described as single digit tokens per second, with first token latency measured in seconds rather than milliseconds.
For a chat product that is unusable. I am not going to dress it up.
For what this lab is proving, it is completely sufficient, and I want to be precise about why. Every one of these is fully testable at this speed: does the gateway route correctly, does the tenant filter hold, does the quota block an oversized request, does the alert fire on saturation, does the trace show the right spans, does the fallback engage when the primary is down, does the pod reschedule when I kill a node.
None of those care how fast tokens come out. All of them are the actual platform.
6. Destroy and rebuild, or it was never reproducible
The strongest claim I can make about a lab is that I am willing to delete it.
A lab I can rebuild is a lab I am willing to experiment on.
That changes behaviour in a way that is hard to overstate. I will happily break the network policy, corrupt the database, or kill a node when the cost of being wrong is a rebuild. On a precious environment I would be cautious, and cautious is how you finish a project having learned nothing about failure.
Volume 4's failure drills only exist because this command works.
The cluster is reproducible. The workloads on top of it become reproducible in Volume 3, with Terraform, Helm, and Argo CD.
7. What actually went wrong
Every honest lab document needs this section, so here are the real ones rather than a tidy narrative.
The readiness probe restart loop
First deployment went into CrashLoopBackOff. Nothing was crashing. The probe had a 30 second initial delay, weight loading took longer than that on CPU, Kubernetes concluded the container was unhealthy and killed it, and the restart began loading weights again from scratch.
The fix is the 120 second initialDelaySeconds in the manifest above. The lesson is broader: on CPU, every timeout assumption you carry from GPU work is wrong, and the symptom looks like a crash rather than a timeout.
Memory limits are not advisory
Set the limit too low and the container is OOM killed mid generation, with a 137 exit code and no useful application log, because the kernel does not ask politely.
The lesson I actually took: the KV cache is not free, and --max-model-len is a memory decision, not just a quality decision. Cutting the context window from 4096 to 2048 was the difference between stable and dead.
local-path storage pins a pod to a node
The model cache PVC bound to aiforge-worker-2. Which means that pod can now only ever run on aiforge-worker-2, because that is where its data physically is.
That is fine in a lab and unacceptable in production, where you would want a shared volume or a proper object store for weights. It is deviation number three below, and it is the kind of thing that silently limits your scheduling long before it announces itself.
Every one of those failures was a hardware and storage reality contradicting a design assumption, which is exactly what building the lab is for.
If I had written Volumes 2 through 4 from the architecture document alone, all three assumptions would have survived into the design, and I would have discovered them later in something more expensive than a laptop.
8. The deviation register
Every gap between this lab and a production deployment, with the production equivalent next to it. This table is what keeps the project honest, and later volumes refer back to it.
| # | Lab reality | Production equivalent | What it hides from me |
|---|---|---|---|
| 1 | No GPU. Small models on CPU | GPU nodes with NVIDIA GPU Operator and nvidia.com/gpu requests | Throughput, PagedAttention efficiency, tensor parallelism, real cost per token, GPU memory exhaustion behaviour |
| 2 | Single control plane node | Three control plane nodes with etcd quorum | Control plane failover, quorum loss, and rolling upgrade behaviour |
| 3 | local-path node local storage | Networked storage or an object store for weights and data | Volume reattachment on reschedule, and true node independence |
| 4 | Two worker nodes | Node pools sized per workload class, autoscaled | Cluster autoscaling, bin packing pressure, and pool level capacity planning |
| 5 | Self signed TLS inside the cluster | Real certificates from a real issuer at the edge | Certificate rotation and expiry incidents, which are a genuine outage cause |
| 6 | One user, me | Many concurrent users across many tenants | Contention, noisy neighbour effects, and fairness under real load |
| 7 | Everything in one cluster | Separate clusters or at minimum separate node pools per environment | Blast radius containment and environment promotion realism |
The failure mode is writing a register like this and never opening it again. Then a year later the lab has quietly become "the system", and its shortcuts have become architecture by accident.
So the rule I hold myself to: any claim I make in Volumes 2 to 4 has to be checked against this table first. If a conclusion depends on something in the right hand column, I say so in the text rather than letting it pass as proven.
9. Volume 1 gate, checked
Back to the checklist from the volume overview, with honest answers.
| Gate | Status |
|---|---|
| Cluster builds from empty state with one command | Yes, k3smp create with a config file, no manual steps |
| Destroy and rebuild reproduces it | Yes, verified in section 6 |
| Namespaces, quotas, LimitRange, default deny in place by code | Yes, including the GPU quota line at zero |
| A model answers on CPU, end to end | Yes, Qwen2.5 1.5B Instruct through an OpenAI compatible API |
| Throughput measured and written down | Yes, single digit tokens per second, and I have said what that does and does not invalidate |
| Every component placeable in exactly one layer | Yes, per the reference architecture |
| At least one assumption proven wrong on real hardware | Yes, three of them, in section 7 |
There is a cluster. There are guardrails. There is a model answering questions. There is a written list of everything this environment cannot tell me.
That last one is the part I am most confident about, and it is the reason the next three volumes are worth reading.
Next
The foundation is done. Time to put a real serving layer on it.
Volume 2, the AI serving layer takes this cluster and builds the part developers actually consume: vLLM properly configured, KServe as a serving contract, MLflow tracking which weights are in production, LiteLLM as the single front door, and a working RAG pipeline on Qdrant with tenant filtered retrieval.