Where this volume sits in the project

Volume 1 built the K3s cluster with k3smp. Volume 2 put models on it: vLLM serving, LiteLLM in front, Qdrant and a RAG pipeline behind.

Both of those volumes were operated by me, with kubectl and Helm. Volume 3 is where that stops being acceptable and the platform starts operating itself.

Volume 3 The Platform Control Plane

At the end of Volume 2 the lab could serve a model. A developer who wanted their own model, their own knowledge base and their own API key had exactly one path to get it: message me, and wait for me to write YAML.

That is not a platform. That is me with extra steps.

This volume replaces me with software. By the end of it there is an HTTP API that accepts a deployment request, checks who is asking, checks whether their tenant is allowed to ask, writes the intent to PostgreSQL, and then drives Kubernetes until the cluster matches what was requested. Around that API there is identity, tenant isolation with teeth, infrastructure as code, GitOps, and a CI pipeline that will not let an unscanned image reach the cluster.

This is the volume that decides whether AIForge is a real project

Anybody can helm install an inference server and screenshot a chat response. The interesting engineering is one layer up: turning a sentence a developer would say into infrastructure, safely, repeatably, for more than one tenant.

That is the part hiring managers actually probe, and it is the part I did not want to fake.

1. What an Internal Developer Platform actually is

The term is used badly enough that it is worth defining from first principles.

An Internal Developer Platform is a product whose users are engineers inside your own company. It exists because the raw infrastructure underneath is powerful, correct, and hostile. Kubernetes will happily let you deploy an inference server with no resource limits, no NetworkPolicy, no probes, no metrics labels, an image pulled from a stranger's registry, and a Service exposed to the world. Kubernetes is not wrong to allow that. It is a kernel, not an opinion.

The platform is the opinion.

The developer wants to say	Without a platform they must know	With AIForge they need to know
"Serve Qwen2.5 1.5B Instruct for my team"	Deployment, Service, probes, vLLM flags, model cache volume, ingress, DNS	The model name and the replica count
"Give it my documentation as context"	Embedding model, chunking strategy, Qdrant collection, vector dimensions, upsert API	Which knowledge base to attach
"Only my team can call it"	OIDC client setup, token validation, NetworkPolicy, RBAC, API key storage	Nothing. It is the default
"Tell me when it is slow"	ServiceMonitor labels, metric names, dashboards, alert rules	Nothing. It is wired in at creation time

Read the right hand column again. That is the whole business case. The platform is not saving developers typing, it is saving them a body of knowledge they should not have to acquire to ship a feature. Every row also happens to be a row where a developer improvising alone would get it slightly wrong, in a way nobody notices until the incident.

The three properties that make it a platform and not a script

Property	What it means concretely	What breaks without it
Self service	A developer gets the thing without a human in the loop, at 2am, on a Sunday	The platform team becomes a ticket queue and the bottleneck moves onto people
Guardrails, not gates	Unsafe requests are rejected by validation and policy, safe ones need no approval	Either everything needs review, or nothing is reviewed. Both are bad
Convergence	The platform keeps working after the request returns, until reality matches intent	You built a deploy button, and the cluster drifts away from what anybody believes

That third property is the one that separates the two things people call the same name, and it deserves its own section.

2. A wrapper versus a control plane

Here is the part everybody gets wrong, including me on my first attempt.

The obvious way to build an Internal Developer Platform is to write an HTTP endpoint that renders a template and shells out. Request comes in, you generate YAML, you kubectl apply, you return 200 OK. It demos beautifully. It is a wrapper, and it fails in a specific, predictable way.

A wrapper is stateless about the outcome. It knows what it asked for. It does not know what happened.

Preparing diagram

The difference is not code volume. It is where the truth lives.

Question	Wrapper answer	Control plane answer
Is the deployment healthy right now?	It returned 200 once, so probably	Observed from the cluster and stored with a timestamp
Somebody hand edited the Deployment. Now what?	Nothing. The change survives	Detected as drift and either corrected or reported
The apply failed halfway through five objects	Partial state, no record of which ones landed	Request stays in a non terminal state and is retried
The API pod was restarted mid request	The intent is lost	The intent is in PostgreSQL and gets picked up again
What did tenant B ask for last Tuesday?	Grep the logs and hope	A row, with the actor, the payload and the outcome

The failure mode has a name and you have probably seen it

It is called a lying dashboard. The platform UI is green, the pods are in CrashLoopBackOff, and the developer trusts the UI because you built it and told them to.

A wrapper cannot avoid this, because it never looks again. A control plane is defined by the fact that it looks again.

This is not a novel idea, it is exactly how Kubernetes itself works. Every controller in the cluster is a loop that reads desired state, reads actual state, and does one step to close the gap. AIForge is a controller for a slightly higher level object called an AI application. Volume 3 chapter 1 builds that loop in Python and is honest about where it stops short of a real Kubernetes operator.

3. From developer intent to Kubernetes resources

Concretely, here is the translation the control plane performs. This one request is the spine of the entire volume:

Eight lines of developer intent become the following, in the tenant's own namespace, with labels and owner references that let the platform find them again later:

Intent field	What the platform creates or configures	Decision the developer never makes
`model`	A Deployment running vLLM with the right served model name, context length and CPU friendly flags	Which serving runtime, which flags, which image digest
`replicas`	Deployment replica count, plus a PodDisruptionBudget when replicas is above one	That a PDB exists at all
`resources`	Requests and limits, validated against the tenant ResourceQuota before the API answers	The LimitRange defaults, the quota arithmetic
`knowledge_base`	A Qdrant collection binding, retriever configuration, and the embedding model endpoint	Vector dimensions, distance metric, chunk size
`name`	Service, LiteLLM model route, Traefik ingress path, and a model alias scoped to the tenant	DNS, routing, gateway registration
implicit	NetworkPolicy, ServiceAccount, probes, ServiceMonitor labels, Langfuse project tag	All of it, which is exactly the point

And here is the same request as a sequence, including the parts that happen after the developer's HTTP call has already returned:

Preparing diagram

Notice step 6. The API answers 202 Accepted, not 200 OK. That single choice is what makes the rest of the design honest: the platform is admitting that creating infrastructure takes longer than an HTTP request should, and giving the caller something to poll instead of pretending.

4. The honest state of the lab

Two constraints shape every code sample in this volume, and I would rather put them on the landing page than let you discover them in chapter 3.

There is no GPU in this lab, and I am not going to pretend otherwise

Everything in AIForge that touches inference currently runs on CPU. The models are small instruct models: Qwen2.5 0.5B and 1.5B Instruct, TinyLlama, and a small sentence embedding model for the RAG path.

That has three concrete consequences for Volume 3:

Deployment specs request cpu and memory only. No nvidia.com/gpu is requested anywhere, because nothing would satisfy it.
The tenant ResourceQuota ships with requests.nvidia.com/gpu: "0". That is deliberate: a GPU request fails fast at admission instead of leaving a pod Pending forever with a scheduling message nobody reads. The GPU shaped quota is designed and written down, it is simply not exercised.
Latency numbers in this volume are lab numbers on a laptop. They prove the plumbing, not the throughput.

The important claim, and the one I will defend: the GPU path is a configuration change, not a redesign. The control plane already carries a GPU field through validation, quota and the pod template. Adding a GPU node means installing the NVIDIA device plugin, raising one quota value, and letting the same code request one more resource type.

The cluster is real, and that changes what the examples are worth

The lab is a K3s cluster built by k3smp, which provisions genuine Multipass virtual machines and installs K3s on them. Nodes are separate machines with separate kernels and separate network stacks, not containers pretending to be nodes. The APM project already used the same tool to stand up a single node cluster and run SigNoz on it, so this is not a first outing.

Cluster build details live in the lab on k3smp. What matters here is what K3s hands you and how it colours the examples.

K3s default	Consequence for Volume 3
Traefik as the bundled ingress controller	Ingress objects and the Keycloak and API hostnames are written for Traefik, and the ingress class is explicit rather than assumed
local-path as the default StorageClass	PostgreSQL and Qdrant volumes are node local. A PVC binds a pod to one node, which is a real single point of failure I name rather than hide
ServiceLB, the klipper load balancer	`type: LoadBalancer` works without a cloud provider, using host ports on the nodes
An embedded NetworkPolicy controller based on kube-router	NetworkPolicy is genuinely enforced, so the isolation tests in chapter 2 fail closed for real. This is not true of a bare flannel install
No admission webhooks, no service mesh	Policy enforcement in this volume is Kubernetes native plus application level. OPA and Gatekeeper are Volume 4 material

The namespace convention used throughout the volume:

Namespace	Contents	Who can touch it
`aiforge-system`	Control plane API, reconciler, PostgreSQL, Keycloak, LiteLLM, Qdrant	Platform engineers and the platform's own ServiceAccount
`aiforge-tenant-a`	Tenant A workloads created by the control plane	Tenant A's ServiceAccount, read mostly. Never tenant B
`aiforge-tenant-b`	Tenant B workloads, the control group for every isolation test	Tenant B's ServiceAccount only

Two tenants exist because one tenant proves nothing

A single tenant platform is a deployment tool with extra vocabulary. Tenant B exists so that every isolation claim in chapter 2 can be tested by trying to break it from the other side rather than by reading the YAML and feeling reassured.

If a control cannot be proven with a command that fails, I do not count it as a control.

5. How this volume is organised

Three chapters, in build order. Chapter 1 gives you an API that can create things. Chapter 2 makes it safe to give that API to other people. Chapter 3 makes the whole thing reproducible from Git.

#	Chapter	What you have when you finish it
1	The Python control plane	A FastAPI service with Pydantic validated requests, a PostgreSQL state model, the Kubernetes Python SDK doing server side apply, and a reconcile loop that keeps looking after the response is sent
2	Identity and multi tenancy	Keycloak issuing tokens, the API verifying them properly, and tenant isolation enforced in six layers with a test that proves each one
3	IaC, GitOps and CI/CD	Terraform for infrastructure, Helm for packaging, Argo CD reconciling the platform from Git, and a GitLab pipeline that lints, tests, builds, scans with Trivy and promotes by digest

Read chapter 2 before you ship chapter 1 anywhere

The order is not decorative. Chapter 1 finishes with an API that will create Kubernetes objects for anybody who can reach it, because authentication has not arrived yet. That is a perfectly reasonable intermediate state on a laptop and a catastrophic one anywhere else.

I sequence it that way because building authorization before you know what you are authorizing produces abstract permissions that fit nothing.

What you will be able to do at the end

Not "will have read about". Will be able to do.

Explain what a reconcile loop is, why it beats a fire and forget apply, and where a Python loop stops being enough and a real operator with a CRD starts.
Design a request model where invalid input is rejected by the type system before any business logic runs.
Set up a Keycloak realm, a confidential client and a public client, and say precisely which OIDC flow each one uses and why.
Validate a JWT correctly: signature against JWKS, issuer, audience, expiry, and the claim that carries the tenant.
Enforce tenant isolation in layers, and prove each layer with a command that is supposed to fail.
Draw the boundary between Terraform, Helm, Argo CD and CI, and explain what breaks when two of them own the same object.
Read an Argo CD Application manifest and predict what it will do, including sync waves and self healing.

6. The gate that ends this volume

Volume 3 is not finished when these three documents exist. It is finished when all of the following are true on my own cluster.

POST /v1/deployments with a valid body returns 202 Accepted and a Location header, and a row exists in PostgreSQL before the response is sent
A malformed body returns 422 with a field level error, produced by Pydantic and never reaching business logic
Deleting the Deployment by hand with kubectl results in the reconciler recreating it, and the transition is visible in the deployment's status history
Killing the API pod mid request loses the HTTP response but not the intent: the record is picked up and driven to READY
Every object the platform creates carries app.kubernetes.io/managed-by: aiforge and an owner reference, so cleanup is a delete of one parent object

The one item on that list I care about most

It is in the Isolation tab: a pod in tenant A cannot reach a pod in tenant B, verified from inside the pod.

Everything else on the list is craft. That one is the difference between a multi tenant platform and a single tenant platform with two namespaces and good intentions. Isolation you have not attacked is isolation you do not have.

What Volume 3 deliberately does not do

Being clear about the edges is part of being honest about the middle.

Left out	Why, and where it goes
A real Kubernetes operator with a CRD and controller runtime	The reconcile pattern is what teaches the lesson. Chapter 1 explains exactly when the Python loop should be replaced and what you gain
OPA and Gatekeeper admission policy	Policy as code sits with runtime security in Volume 4
Prometheus, Grafana, Loki, Langfuse wiring	Volume 3 emits the labels and hooks. Volume 4 collects and alerts on them
Vault and external secret operators	Chapter 3 states the rule that secrets never enter Git and shows the minimum viable handling. The full story is Volume 4
Autoscaling and cost attribution	Both need real load and real prices. Volume 4
GPU scheduling, MIG, time slicing	No GPU exists yet. The design is recorded, the exercise is postponed, and I will not write a benchmark I did not run

If you have not read the layer diagram yet, the reference architecture shows where the control plane sits relative to the serving layer built in Volume 2. After this volume, Volume 4 takes the platform this one builds and asks the harder question: what does it do at 3am when something is broken and it costs money per hour.