Projects · AIForge
The Python control plane
Building the AIForge control plane: FastAPI request handling, Pydantic as the first policy engine, PostgreSQL for platform state, the Kubernetes Python SDK doing server side apply, and a reconcile loop that keeps looking after the response is sent.
The Python control plane
This is the chapter where AIForge stops being a pile of Helm charts and becomes a platform.
The job is narrow and it is worth stating in one sentence before any code appears: accept a developer's intent over HTTP, decide whether it is allowed, remember it durably, and then drive Kubernetes until reality matches it. Four verbs. Every design decision in this chapter serves one of them, and when I was tempted to add a fifth I usually regretted it.
A FastAPI service in aiforge-system that:
- validates deployment requests with Pydantic before any business logic runs
- writes intent to PostgreSQL in a transaction, then answers
202 Accepted - runs a reconcile loop that server side applies Kubernetes objects and reads back real status
- survives its own restart without losing a request
- tells the truth about a deployment even after somebody edits it by hand
Authentication arrives in the next chapter. Until then, this API trusts everybody, which is fine on a laptop and nowhere else.
1. What the control plane is responsible for
A control plane is a translator with a memory. Draw the boundaries wrong and you either build a thin YAML printer or you accidentally rebuild Kubernetes badly.
Two things in that diagram matter more than the boxes.
First, the arrow from Kubernetes back into the reconciler is dotted and it points backwards. The control plane reads from the cluster as much as it writes to it. A design where information only flows left to right is the wrapper from the volume overview.
Second, the reconciler does not talk to FastAPI. They share PostgreSQL and nothing else. That is what lets me restart, scale or crash either one without the other noticing.
The control plane owns: the meaning of an AI application, which fields are legal, what a tenant may consume, what Kubernetes objects express an intent, the current believed state of every deployment, and the audit trail of who asked for what.
The control plane does not own: scheduling (that is the kube scheduler), health checking (that is kubelet probes), rollout mechanics (that is the Deployment controller), routing (that is Traefik and LiteLLM), or vector search (that is Qdrant).
Every one of those was tempting to reimplement, and each time the honest answer was that Kubernetes already does it better than I would. A good platform is mostly a good delegation strategy.
2. FastAPI, and what it actually does for you
FastAPI is an ASGI web framework, which means it speaks the asynchronous Python server protocol and runs under a server like Uvicorn. If you have never used it, here is the accurate mental model: it is a router plus a very aggressive type driven request parser.
The part that matters for a platform is not speed. It is that the function signature is the contract.
Four things happen in that signature before the first line of my code executes.
| Declaration | What FastAPI does with it | What I would otherwise write by hand |
|---|---|---|
payload: DeploymentCreate | Parses JSON, coerces types, validates constraints, returns 422 with a field path on failure | Fifty lines of defensive if statements per endpoint, forgotten in three places |
Depends(get_principal) | Runs the auth dependency first and short circuits with 401 or 403 | An auth check copied into every handler, missing from the newest one |
Depends(get_uow) | Opens a scoped database session and guarantees teardown even on exceptions | Leaked connections under load, discovered when the pool exhausts |
response_model=DeploymentRead | Filters the outbound object to declared fields only | Accidentally serialising an internal column into a public API |
response_model is the reason a database column called internal_notes or keycloak_client_secret cannot leak by accident. The response is built from the declared output schema, so adding a column to a table does not silently widen the API.
Response schemas are an allowlist. I have watched a project without them expose a password hash through an ORM object that was just handed to the JSON encoder.
FastAPI also generates an OpenAPI document from those same annotations, which is not a nice extra for a platform. It is how the CLI, the UI and other teams' CI jobs get a client without me writing documentation that drifts.
Async, and the one mistake that kills it
FastAPI handlers are async. The Kubernetes Python SDK is synchronous and blocking. Put the two together naively and you block the event loop, at which point your concurrency is one.
My first reconciler ran inside the API process and called the Kubernetes SDK directly from an async def handler. Every apply blocked the event loop for the duration of the API round trip. With one caller it was invisible. With two it looked like a network fault: health checks timing out for no reason a log could explain.
Two fixes, and I use both:
- The reconciler is a separate process, so nothing about it can stall HTTP.
- Where synchronous work is unavoidable inside the API, it runs through
anyio.to_thread.run_syncinstead of directly on the loop.
If you take one operational lesson from this chapter: an event loop is a single lane road and a blocking call is a parked truck.
3. Pydantic, the first policy engine
Pydantic is a data validation library that builds a validator from type annotations. In a platform it is doing something more important than parsing: it is the layer where bad requests stop being possible instead of being handled.
Look at what each construct buys.
| Construct | The class of bug it removes |
|---|---|
extra="forbid" | Silent typos. A request with replica: 3 is rejected instead of quietly deploying one replica. This is the single highest value line in the file |
Literal for the model | Arbitrary model names becoming arbitrary image pulls. The allowlist is the type |
Field(ge=1, le=6) | A tenant asking for 500 replicas and finding out via the quota error path instead of the validation path |
The regex on name | An invalid Kubernetes name failing halfway through an apply, leaving three of five objects created |
| The reserved prefix check | A tenant naming their app something that collides with a platform object |
model_validator | Combinations that are individually valid and jointly stupid. Field validation cannot see across fields, this can |
Pydantic answers "is this request well formed and within platform limits". It cannot answer "is this caller allowed to do this to this tenant", because it has never seen the caller.
I keep them in separate layers on purpose. Schema rules are public, documented in OpenAPI, and identical for everyone. Authorization rules depend on identity and live in the dependency described in the next chapter.
If your validator needs to know who is calling, it is not a validator.That gpu: int = 0 field is the GPU story of this entire project in one line. There is no GPU in the lab. Nothing can satisfy an nvidia.com/gpu request, and the tenant quota pins it to zero so an attempt fails at admission rather than sitting Pending.
I still carry the field end to end: schema, database column, quota arithmetic, pod template. The reason is that the shape of the code is the expensive part, not the value. When a GPU node joins, the change is a device plugin install and one quota edit. The control plane does not get rewritten. That is the claim I want to be able to make, and keeping the field wired is how I keep it true.
4. PostgreSQL, the platform's memory
The control plane needs to remember things that Kubernetes has no opinion about: which human asked, which tenant they belong to, what they originally requested versus what got applied, how much of their quota is committed, and what the platform believed at each point in time.
PostgreSQL holds that, reached through SQLAlchemy 2.0.
The three columns that carry the design
| Column pair | Why it exists |
|---|---|
spec and observed | Desired versus actual, side by side, in one row. Every reconcile is a comparison of these two. Storing only one of them is how you end up with a lying dashboard |
generation and observed_generation | Straight out of the Kubernetes playbook. When a developer updates a deployment, generation increments. If observed_generation is behind, the reported status describes the old spec and must not be trusted as current |
state and state_reason | A machine readable state for logic and a human sentence for the developer. DEGRADED plus "0 of 2 replicas ready, container OOMKilled" is actionable. DEGRADED alone is a shrug |
Three properties of this workload point at a relational database, and I want to be concrete rather than say "it is the standard choice".
- Transactions. Reserving quota and inserting a deployment must be one atomic act. Two concurrent requests that each fit the quota but jointly do not must not both succeed. A transaction plus a constraint solves that. Application level checking does not.
- Constraints. The unique constraint on
(tenant_id, name)is enforced under concurrency by the database. Myif existscheck is not, and never will be. - Row locking.
SELECT ... FOR UPDATE SKIP LOCKEDturns the deployments table into a safe work queue for multiple reconciler replicas, with no extra infrastructure.
A ConfigMap in the cluster was genuinely tempting because it removes a dependency. It gives you no transactions, no constraints, no queries and a 1 MiB ceiling. Platform state is business data that happens to be about infrastructure. Treat it as business data.
Why PostgreSQL and Qdrant are two different stores
This question comes up every time, and answering it with "one is SQL and one is vector" is not an answer.
| Property | PostgreSQL, platform state | Qdrant, vector state |
|---|---|---|
| What it holds | Tenants, users, applications, deployments, quotas, API keys, audit records | Embeddings and chunk payloads for retrieval |
| Access pattern | Small reads and writes by exact key, joins, transactions, ordered listings | Approximate nearest neighbour search over high dimensional vectors |
| Correctness requirement | Exact. A lost deployment row is a lost workload nobody can find | Approximate by design. ANN search trades recall for speed on purpose |
| If you lose it | The platform has amnesia. Restore from backup is the only path | Re embed the documents. Slow and annoying, not fatal |
| Is it derived data? | No. It is the source of truth | Yes. It is derived from documents that live elsewhere |
| Who writes it | The control plane only | The ingestion pipeline, per tenant collection |
That difference drives everything downstream: backup frequency, restore drills, blast radius, and how much I panic when a pod restarts. Merging them into one store would mean applying the strictest requirement to both, which is expensive, or the loosest, which is negligent.
The vector side is built in Volume 2: see the RAG pipeline and Qdrant. In this chapter, Qdrant appears only as a name the control plane binds a deployment to.
K3s ships local-path as its default StorageClass. That means a PersistentVolume is a directory on one specific node, and the pod using it is pinned to that node forever.
For PostgreSQL in this lab, that is a genuine single point of failure and I am not going to dress it up. If that node dies, the platform loses its memory until I restore. What makes it acceptable is that I know it, the backup path is exercised, and the production answer is a StorageClass with real replication or a managed database. What would not be acceptable is discovering it during an incident.
5. Talking to Kubernetes from Python
The Kubernetes Python client is the official SDK. It is generated from the OpenAPI spec of the Kubernetes API, which is useful context: it is not a hand crafted library with a friendly design, it is a faithful mirror of an HTTP API.
Two things about using it well.
Server side apply, and the gotcha that silently downgrades it
The naive approach is create, catch a 409 conflict, then replace. It works and it is wrong, because replace overwrites fields that other controllers legitimately own, and it loses races. Server side apply is the correct primitive: you declare only the fields you own, the API server records that ownership under a field manager name, and it tells you when somebody else owns a field you are trying to set.
Server side apply is selected by an HTTP content type, application/apply-patch+yaml. The typed clients in this SDK, the ones like AppsV1Api.patch_namespaced_deployment, do not send that content type. Pass field_manager to them and you get a strategic merge patch that accepts the argument and ignores the semantics.
The symptom is horrible: everything appears to work, fields I removed from my manifest are never pruned from the live object, and stale configuration survives forever. There is no error. The object simply accumulates history.
The fix is to use the dynamic client, which sets the right content type. If you cannot point at the content type your apply is sending, you do not know whether you are applying.
force_conflicts=True deserves a sentence of justification, because it is a loaded flag. It means "if another field manager owns a field I am declaring, take ownership anyway". For fields the platform genuinely owns, that is correct and prevents a deployment stuck in conflict because somebody once ran kubectl edit. For fields the platform does not own, the right answer is to not declare them at all. Chapter 3 has the concrete example of this going wrong: the platform, Argo CD and an autoscaler all trying to own spec.replicas.
Ownership and cleanup, including a footgun
When a developer deletes a deployment, five or six objects must go. Chasing them individually is fragile, so Kubernetes offers owner references: mark children as owned by a parent, delete the parent, the garbage collector takes care of the rest.
This one is subtle and it bites hard. An owner reference is namespace scoped. A namespaced object can only be owned by an object in the same namespace, or by a cluster scoped object.
My instinct was to make a record in aiforge-system the owner of workloads in aiforge-tenant-a. The API server accepts that manifest without complaint. Then the garbage collector evaluates it, cannot resolve the owner in the dependent's own namespace, concludes the owner does not exist, and deletes the dependent. Your workloads disappear minutes after a successful apply, and the only clue is an event about a cross namespace owner reference.
What I do instead: the platform creates one small ConfigMap named aiforge-app-<name> inside the tenant namespace, holding the rendered spec hash. That ConfigMap is the owner of the Deployment, Service, PDB and NetworkPolicy. Deleting it cascades everything. The labels above are the belt to that braces: even if garbage collection ever misbehaves, a label selector finds every object the platform ever made.
6. Reconciliation versus fire and forget
Now the core of the chapter.
Fire and forget: receive request, apply objects, return success. Reconciliation: receive request, record intent, and separately run a loop that repeatedly compares intent to reality and does one step to close the gap.
The properties that come for free once you accept the loop:
| Situation | Fire and forget | Reconcile loop |
|---|---|---|
| Kubernetes API is briefly unavailable | Request fails, developer retries, or does not | Next iteration succeeds. Nobody notices |
| API pod is killed mid apply | Partial objects, no record of what landed | Intent is in the database, applied again from scratch. Apply is idempotent |
Somebody runs kubectl delete deploy | Gone. Platform still says ready | Recreated on the next pass, drift recorded |
| Pods cannot schedule due to quota | Apply succeeded, so the platform says success | State becomes PROGRESSING then FAILED with the real reason |
| Two API replicas get the same request | Race, duplicate objects, possibly both partial | Unique constraint rejects one, loop is single writer per record |
Here is the loop. It is shorter than people expect, which is part of the point.
Everything above classify is plumbing that any competent engineer writes the same way. classify is where you either translate Kubernetes reality into something a developer can act on, or you emit DEGRADED and make them read pod events.
The OOMKilled branch is my favourite because it is domain specific. On CPU inference the model weights sit in RAM, so the memory limit is not an abstract number, it is roughly "does this model fit". A platform earns trust by explaining failures in the vocabulary of the person who caused them.
Where this loop stops being enough
I would rather name the limits than let a reviewer find them.
| Limitation of a polling Python loop | What a real operator does instead |
|---|---|
| Polls every 10 seconds, so drift can persist for 10 seconds | Watches the API with informers and reacts in milliseconds |
| Re reads every non terminal record on every pass | Keeps a local cache and only processes objects that changed |
Desired state lives outside Kubernetes, so kubectl cannot see it | A CustomResourceDefinition makes the intent a first class cluster object |
| Backoff and requeue are hand rolled | A workqueue with rate limiting, provided by the framework |
| Leader election is my problem | Built in leases, so extra replicas are standby not competing |
Two honest reasons, and neither is that operators are hard.
The teaching reason. Writing the loop is how you internalise that a controller is desired state, actual state, one corrective step, requeue. Adopting a framework first teaches you the framework's API and hides the idea. Once you have written reconcile_one, every controller in Kubernetes reads like something familiar.
The scope reason. The platform API is HTTP first, with a UI and a CLI in front of it. A CRD would add a second write path with its own validation, its own RBAC surface and its own consistency question about which store wins. That is the right trade for a product, and the wrong one for a lab volume that also has to teach identity, tenancy and GitOps.
The upgrade path is real, and it is exactly this: promote DeploymentCreate to a CRD schema, replace the polling query with an informer, keep classify and the renderer unchanged. Those two functions are the actual value, and they are framework agnostic.
7. The deployment request lifecycle, end to end
Putting the pieces together. This is the walkthrough I would give at a whiteboard.
The quota precheck at step D is worth a note. Kubernetes will reject an over quota pod at admission anyway, so this check is not the security control. It exists so the developer gets a useful answer synchronously instead of a 202 followed by a mysterious FAILED thirty seconds later. The cluster remains the enforcement point. The API is being polite.
Here is the request against the running lab:
And here is what landed in the cluster, all of it labelled and owned:
Proving the loop actually loops
A reconcile loop that has never been attacked is a claim. So attack it: delete the Deployment behind the platform's back and see whether the platform notices, fixes it, and admits it happened.
That sequence proves the loop converges, proves the apply is idempotent, and proves the drift is recorded rather than silently healed. The third one is the one people forget: a platform that fixes things without telling you is also a platform that hides a colleague repeatedly breaking something.
Note also what did not happen. The pods came back with the same spec, the same labels and the same owner. That is server side apply plus a renderer that is a pure function of the stored spec.
8. API design decisions I would defend in a review
Design choices, each with the failure it prevents.
| Decision | Reasoning |
|---|---|
Version in the path, /v1/ | Header negotiation is more elegant and worse to operate. A path version can be curled, cached, logged and routed by Traefik without anyone thinking |
| 202 for creation, not 201 | The resource is accepted, not created. The Location header gives the caller something to poll. Returning 201 would be a claim I cannot honour |
| Idempotency-Key header | A retried POST after a timeout must not create a second deployment. The key is stored with a unique constraint, so a replay returns the original record |
| Tenant is never in the body | It comes from the token. A body field would be a privilege escalation waiting for a missing check. Detailed in the next chapter |
| Cursor pagination, not offset | Offset pagination skips and duplicates rows when the underlying set changes, which it constantly does. A cursor on (created_at, id) is stable |
| PATCH is a new generation, not a live edit | An update bumps generation and lets the reconciler converge. The API never mutates the cluster directly, so there is exactly one writer |
| DELETE is soft first | State goes to DELETING, the reconciler tears down, then DELETED. A failed teardown is retried instead of leaking orphans |
| Errors are RFC 9457 problem documents | One error shape for the UI and the CLI. Free form error strings become a parsing exercise for every client |
An error, in full, because error design is where APIs quietly become unpleasant:
Compare with what most platforms return: 403 quota exceeded. That sends the developer to Slack to ask a human what their quota is, which is the exact interaction the platform exists to remove.
The response above lets them fix it themselves in one attempt. It costs four extra fields. Every error message is a small piece of documentation delivered at the moment somebody actually needs it.
Both stores show up in the database, which is worth one look because it makes the audit story concrete:
experiment-x asked for a GPU. The tenant ResourceQuota sets requests.nvidia.com/gpu to zero, so admission refused the pod immediately and the reason is a full sentence rather than a mystery.
That is deliberate, and it is the honest state of a lab with no GPU in it. The alternative, leaving the quota unset, produces a pod that sits Pending forever with a scheduling message nobody reads. Failing fast with a quota message is better behaviour, and the day a GPU node exists the fix is one number in one manifest. The quota design is in identity and multi tenancy.
9. What went wrong, and what I watch
The honest section. Every item cost me real time.
| What happened | Root cause | What I changed |
|---|---|---|
| Applies appeared to work but stale fields never disappeared | Typed client sends a merge patch, so field_manager was accepted and server side apply semantics were not applied | Dynamic client only, and a startup assertion that the applier's content type is application/apply-patch+yaml |
| Tenant workloads vanished a few minutes after a successful apply | Owner reference pointed at a record in aiforge-system, and cross namespace owners are treated as missing by the garbage collector | A per application ConfigMap owner inside the tenant namespace, plus label based cleanup as a fallback |
| API health checks timed out with only two concurrent callers | Blocking Kubernetes SDK calls on the asyncio event loop | Reconciler split into its own process. Any remaining sync work goes through a thread |
| Two reconciler replicas fought over the same records | Plain SELECT with no locking, so both claimed everything | FOR UPDATE SKIP LOCKED, which makes the table a work queue |
| A tenant deployed one replica when they asked for three | Typo in the JSON body, replica instead of replicas, silently ignored | extra="forbid" on every inbound model. A typo is now a 422 naming the offending key |
| Connection pool exhausted after a burst of requests | Sessions opened per handler and not always closed on the exception path | Session as a FastAPI dependency with guaranteed teardown, and a pool size that is a stated number rather than a default |
The platform is not the only thing that writes to tenant namespaces. Argo CD writes platform components, and in the future an autoscaler will want to write spec.replicas on the same Deployments the control plane owns.
Server side apply makes the conflict visible instead of silent, which is a real improvement, but visible conflicts still need an owner decided by a human. My current rule is deliberately blunt: the control plane owns everything it renders, Argo CD owns platform components only, and no autoscaler touches a control plane managed Deployment until the ownership question is answered properly.
Two systems that both believe they own a field will fight forever, and the loser is whoever is on call. Chapter 3 has the concrete boundary rules.
What I watch on the control plane itself, which is a preview of Volume 4:
- Reconcile lag, the age of the oldest unreconciled record. If this grows, the loop is behind and every status in the UI is stale.
- Drift corrections per hour. Nonzero is healthy. A spike means a human is fighting the platform and needs a conversation, not a stricter policy.
- Time from PENDING to READY, split by model, because CPU inference startup is dominated by loading weights from disk.
- 422 rate by field. A field that everybody fails is a documentation bug or a bad API, not a user problem.
- Records stuck in DELETING. Every one is a resource still costing something that nobody believes exists.
Next
The API now creates real infrastructure for anybody who can reach it, which is the most dangerous state this project will ever be in. Identity and multi tenancy fixes that: Keycloak issues the tokens, the API verifies them properly, and tenant isolation is enforced in layers with a test for each one.
After that, IaC, GitOps and CI/CD makes the control plane itself reproducible, so the thing that manages everything is not the one component I installed by hand.