Read this before anything else in Volume 4: there is no GPU in this lab

Every technique in this volume is real. Not every number in this volume is measured.

The AIForge lab runs on CPU only. Small instruct models, Qwen2.5 0.5B and Qwen2.5 1.5B, TinyLlama, and all-MiniLM-L6-v2 for embeddings. That means GPU utilization panels are designed, wired and empty, and the euro figures in the cost chapter are modelled from published GPU hourly prices rather than read off an invoice.

I could have hidden that. Instead every chapter tells you exactly which line is measured and which line is modelled, because a methodology you can re run on real hardware is worth more than a screenshot you cannot reproduce.

Volume 4 Operations, Security and Cost

Volumes 1 to 3 built a platform.

Volume 1 gave us a real Kubernetes cluster on k3smp, running K3s inside real Multipass virtual machines. Volume 2 put models behind it: vLLM serving inference, KServe managing the model lifecycle, LiteLLM as the gateway, and a RAG pipeline on Qdrant for retrieval. Volume 3 wrapped all of it in a control plane, gave tenants identities and boundaries, and made the whole thing reproducible with IaC, GitOps and CI/CD.

At the end of Volume 3 a developer can post a JSON body and get a working AI application back. That is a genuine achievement and it is also, roughly, twenty percent of the work.

Because here is the part everybody gets wrong.

Shipping the platform is day one. Day two is every other day for the rest of the platform's life. Day two is the 3am page, the tenant who says "it feels slow", the CVE in a base image you built four months ago, the invoice that tripled, the model server that OOMs only under concurrency, and the deploy that looked green in CI and broke retrieval in production. None of that is solved by a better POST /deployments endpoint.

The single sentence this volume is built around

A platform is not something you build. It is something you operate, and the operating is where the engineering judgement lives.

Why AI platforms make day two harder than normal platforms

I want to be precise about this, because "AI is different" is usually a marketing sentence rather than a technical one. Here it is actually true, and for four concrete reasons.

Property	A normal web platform	An AI platform	What that does to operations
Unit of work	A request. Milliseconds. Bounded.	A generation. Seconds to minutes. Length depends on the output.	Request count is a useless load signal. You have to measure tokens.
Cost per unit	Fractions of a cent, dominated by developer time	Dominated by accelerator time, which is billed by the hour whether busy or idle	Idle capacity is a real, visible, ongoing loss. Utilization becomes a first class metric.
Correctness	Deterministic. A test asserts equality.	Probabilistic. The same input can give two different valid answers.	"Is it working" cannot be answered by a status code. You need output level telemetry.
Attack surface	Code, dependencies, network, identity	All of that, plus prompts, retrieved documents, and model weights	Untrusted text becomes an instruction channel. Your index becomes an exfiltration path.

Look at the last row again, because it is the one that surprises infrastructure people. In a normal system, data is data and code is code, and the whole industry has spent thirty years learning to keep them apart. In an LLM system, the data channel and the instruction channel are the same channel. A document your RAG pipeline retrieved is fed to the model in exactly the same way as the system prompt you wrote. That is not a bug you can patch. It is a property of the architecture, and it changes how you design controls.

The four pillars

Volume 4 has four chapters and each one is a pillar. They are not independent. Each one feeds the others, and that feedback loop is what "operating a platform" actually means in practice.

Preparing diagram

Notice that observability sits between the workload and everything else. That is deliberate and it is the reason it is chapter one.

You cannot secure, cost, or stabilise something you cannot see. Security without telemetry is a policy document. Cost control without telemetry is a guess dressed up as a spreadsheet. Reliability without telemetry is hoping a user tells you. Every other pillar in this volume reads from the pillar built in chapter one, so chapter one comes first and it is the longest.

The pillars argue with each other, and that is the interesting part

These four goals are not aligned. They pull in opposite directions and the job is to referee them.

Reliability wants more replicas. Cost wants fewer. Security wants a locked down read only root filesystem with no capabilities. The model server wants to write a compilation cache and mmap a 3 GB weights file. Cost wants scale to zero. Reliability wants to avoid a 90 second cold start on the first request after a quiet period.

Anyone who tells you there is no tension is selling you something. The engineering is in choosing where to sit on each of those axes and writing down why.

The shape of a day two incident

Before the chapters, here is the loop all four pillars serve. Every drill in the reliability chapter walks this path, and every tool in the observability chapter exists to shorten one specific arrow on it.

Preparing diagram

The two arrows that separate a mature platform from an immature one are the first and the last.

Detection. If your users detect failures before your alerts do, you do not have an observability stack, you have a dashboard you look at when someone complains. Time to detection is a number you can measure and improve.

Postmortem. If an incident produces a Slack thread and nothing else, you will have the same incident again next quarter with a different pod name. The output of an incident is not a fix. It is a fix plus a control that makes the class of failure impossible or loud.

Mitigation comes before diagnosis, always

This is the hardest habit to build and the one that pays the most.

When a tenant's chat endpoint is returning 503, your job is not to understand why. Your job is to stop the 503s: roll back, shift traffic to the fallback model, scale up, drop the bad query. Then you investigate, with the pressure off and the evidence preserved.

I have watched smart engineers spend forty minutes reading logs during an outage they could have ended in ninety seconds with a rollback. Curiosity is a wonderful engineering trait and a terrible incident response strategy.

What the CPU only constraint actually costs me

I opened with the constraint, so let me be specific about its blast radius rather than leaving it vague. This table is the honest ledger for the whole volume.

Capability	Status in this lab	Why it still teaches the real thing
Latency, throughput, error rate, saturation signals	Fully real. Measured on CPU inference	The four golden signals do not care what silicon runs underneath
Token accounting, prompt and completion telemetry, retrieval quality	Fully real. Langfuse sees every generation	This is application level telemetry, hardware independent
Security controls: scanning, admission policy, runtime detection, isolation	Fully real. Nothing here needs an accelerator	A Gatekeeper deny and a Falco alert behave identically on any node
GPU utilization, GPU memory, SM occupancy, power draw	Designed, wired, no data. DCGM exporter is in the manifests, panels exist and read `No data`	The queries and dashboards are written and reviewable. They light up the day hardware arrives
Cost per 1M tokens on GPU	Modelled. Published price per GPU hour, estimated tokens per second	The formula and the measurement harness are real. One input is an estimate, and it is labelled
OOM kills and memory pressure	Fully real, and easier to trigger. A 1.5B model in fp32 on a 6 GiB node OOMs enthusiastically	Container OOM handling, restart loops and probe interaction are identical
GPU memory exhaustion, KV cache eviction under load	Designed drill, not yet run. Written up, never executed	Labelled as such. I will not pretend to have run a drill I have not run
Node loss	Fully real. `multipass stop` on a worker is a genuine node failure	k3smp uses real VMs, so this is not a container being paused

Why the real VM thing matters more than it sounds

This is the payoff for having built k3smp instead of reaching for a containerised cluster.

When I run the node loss drill in chapter four, I stop a Multipass virtual machine. The kubelet stops reporting. The node goes NotReady after the node monitor grace period. Pods get tainted, then evicted after tolerationSeconds. The scheduler tries to place them elsewhere and fails if the remaining nodes lack the memory. Persistent volumes bound to that node do not move.

Every one of those behaviours is the real Kubernetes behaviour, with real timers. That is the difference between practising an outage and reading about one.

How this volume is organised

Four chapters, in the order I would build them on a real platform. Observability first because everything else reads from it. Reliability last because it is the exam.

#	Chapter	What you can do when you finish it
1	Observability and LLM telemetry	Answer "what is happening, where, and why" for a single AI request. Prometheus and PromQL for metrics, Loki for logs, OpenTelemetry for traces, Langfuse for prompts, tokens, cost and retrieval quality, Grafana to tie it together. Includes recording rules and alerts you can paste.
2	Security and policy	Build defence in depth: Trivy in CI, OPA Gatekeeper at admission, Pod Security Standards on namespaces, Falco at runtime, secrets that never touch Git, an audit trail, and controls for the AI specific risks in the OWASP LLM Top 10.
3	Cost and efficiency	Compute cost per 1M tokens, per request and per tenant from telemetry you already collect. Then act on it: quantization, continuous batching, right sizing, autoscaling and scale to zero, model selection, spot capacity, and chargeback that changes behaviour.
4	Reliability and failure drills	Define SLIs, SLOs and error budgets for an AI platform, wire probes, PDBs and graceful degradation, then deliberately break eight things and write down what each one taught. Ends with a postmortem you can copy.

What carried over from the APM project, and what is new

I have done a chunk of this before. The APM project stood up a single node K3s cluster with k3smp and deployed SigNoz with the OpenTelemetry K8s Infra collectors, then used the OpenTelemetry Operator for auto instrumentation.

What carried over: the collector topology (a DaemonSet agent for node and pod signals, a gateway Deployment for enrichment and export), the k8sattributes processor habit of stamping every signal with namespace, pod and workload, the OTLP wire format, and the general discipline that instrumentation is a platform responsibility rather than a per team pull request.

What is different: AIForge does not use SigNoz. It uses the Prometheus, Grafana, Loki, OpenTelemetry, Langfuse stack, so the backend is composed of separate best in class pieces instead of one integrated product, and there is a fifth signal type SigNoz has no concept of: the content of the generation itself. Chapter one is honest about the tradeoff. Composing is more work than installing.

The gate that ends this volume

I do not consider Volume 4 finished when four documents exist. I consider it finished when all of these are true on my own cluster.

A single RAG backed chat request can be traced end to end: gateway, retrieval, embedding, Qdrant, generation, with per span timings
The four golden signals for every platform component are on one Grafana dashboard, and I know which panel to look at first
Token counts, prompt and completion text, latency and modelled cost are visible per request and per tenant in Langfuse
Recording rules exist so no dashboard runs an expensive raw query, and every alert has a runbook link in its annotations
GPU panels exist, read No data, and are labelled as awaiting hardware rather than quietly hidden

The last two tabs are the actual exam

Everything before them is a checklist, and checklists are easy to satisfy dishonestly.

If I reach the end of this volume and every drill went exactly as designed, I should be suspicious rather than proud. It would almost certainly mean I designed the drills to match what I already knew, instead of designing them to find out what I did not.

The same goes for the security chapter. Any write up that lists controls and no residual risk is marketing. Naming what you are still exposed to is the part that requires actual expertise.

What this volume deliberately leaves alone

So you know where the edges are:

Multi cluster and multi region. One k3smp cluster. Failover between clusters, global load balancing and cross region replication are out of scope, and I will say so again where it matters.
Backup and disaster recovery in depth. PostgreSQL and Qdrant durability get touched in the reliability drills, but a full DR plan with tested restores is its own volume.
Model evaluation and quality scoring. Langfuse can hold scores and evaluations. Building a real offline eval harness is a modelling problem, not an operations one, and it belongs with the serving layer.
Compliance frameworks. No SOC 2 or ISO mapping. The controls here are engineering controls, and mapping them to a framework is paperwork that depends on your auditor.
Real GPU numbers. Repeated for the third and last time, because it is the single most important caveat in the volume.