Projects · AIForge
IaC, GitOps and CI/CD
Terraform for infrastructure, Helm for packaging, Argo CD for GitOps with drift detection and sync waves, and a GitLab CI pipeline that lints, tests, builds, scans with Trivy and promotes by digest. Plus the boundary rules that stop the four of them fighting.
IaC, GitOps and CI/CD
At this point AIForge works. There is an API that turns intent into workloads, Keycloak issues tokens, and tenants are isolated in six layers.
There is also a problem I have been carefully not mentioning: I built all of it by hand.
The namespaces were kubectl apply. The quotas were a file I edited in place. Keycloak was configured by clicking through the admin console, which means the realm exists in exactly one place and its configuration is undocumented. If the cluster dies, the platform does not come back. It gets rebuilt from memory, differently.
This chapter fixes that with four tools. The interesting part is not any one of them. It is the boundaries between them, because that is where real platforms rot.
Terraform creates the things Kubernetes cannot create for itself. Helm decides how a Kubernetes application is packaged. Argo CD is the only thing allowed to apply platform manifests to the cluster. GitLab CI produces and verifies artifacts and never touches the cluster at all.
Four tools, four questions, no overlap. Every serious delivery outage I have seen came from two of these owning the same object.
1. Four tools, one question each
Before any HCL or YAML, the ownership map. This diagram is the most useful thing in the chapter and I would put it in a team wiki.
| Tool | The question it answers | What it must never do |
|---|---|---|
| Terraform | Does this thing exist, and is its configuration what I declared? | Manage objects that change many times a day. Every change is a state file mutation and a lock |
| Helm | How is this Kubernetes application templated and parameterised? | Be the thing that installs into the cluster in normal operation. Templating and applying are different jobs |
| Argo CD | Does the cluster match Git right now, and if not, what do I do? | Build images, run tests, or hold secrets in its manifests |
| GitLab CI | Is this artifact correct, safe, and traceable to a commit? | Hold cluster credentials or run kubectl apply. See the callout below |
| The control plane | What do tenants want right now? | Touch platform components. It owns tenant workloads only |
The traditional pipeline ends with kubectl apply from a runner holding a kubeconfig. It is the default in most tutorials, and it has three properties I am not willing to accept.
- The blast radius is every merge request. A CI runner with cluster write access means anybody who can influence a pipeline, including through a dependency, can write to the cluster.
- Nobody can answer what is running. The cluster reflects the last successful pipeline, which is not a thing you can read or diff.
- Rollback is a rebuild. Reverting means re running an old pipeline and hoping it still passes.
With GitOps, CI's final act is a commit. Argo CD, running inside the cluster, pulls that commit. The credential never leaves the cluster, the desired state is a file with a history, and rollback is git revert.
2. Terraform: making infrastructure a declaration
Terraform reads a description of what should exist, compares it against a state file recording what it created last time, calls provider APIs to close the gap, and writes the new state back. Two ideas do all the work:
| Concept | What it actually is | Why it matters here |
|---|---|---|
| Provider | A plugin translating HCL resources into API calls for one system | The Kubernetes, Helm, Keycloak and PostgreSQL providers let one language describe four systems |
| State | A JSON record mapping declared resources to real object identifiers | It is how Terraform knows the difference between create, update and destroy. It is also the thing that ruins your day when it is wrong |
Terraform owns the platform substrate: namespaces with their labels, quotas, limit ranges, the RBAC bindings from the previous chapter, the Keycloak realm, and per tenant database objects. All of it is derived from one map, so onboarding a tenant is a few lines rather than a checklist.
Keycloak gets the same treatment, which is how the realm stops being a thing I clicked together once:
Adding a tenant is now: one entry in var.tenants, one terraform apply. Out comes a namespace with the right labels, a quota, a LimitRange, RBAC bindings, a Keycloak group carrying the tenant attribute, a database schema and a Qdrant collection.
Before this, onboarding was a nine step runbook that I performed slightly differently each time, and step seven was always forgotten. The value of IaC is not automation, it is that the second tenant is identical to the first.
Terraform does not manage tenant workloads. Every deployment a developer requests would be a state file mutation and a lock, and a lock means one developer at a time. It also means a laptop with a state file becomes load bearing for a production API. Workloads are the control plane's job, driven by PostgreSQL.
Terraform does not manage platform application manifests either. The Helm provider exists and works, and using it here would put Terraform and Argo CD in a fight over the same objects. Terraform stops at the substrate.
And the honest gap: my lab state is a local file. That is genuinely wrong for anything shared. No locking, no history, and it lives on the machine most likely to be reinstalled. The production answer is a remote backend with locking and encryption. My mitigation is small and stated: the state file is backed up with the rest of the lab, and everything in it can be recreated from HCL if I lose it, because none of it holds data. A state file with no remote backend is a single point of failure you carry around in a laptop bag.
3. Helm: packaging, and only packaging
Helm is two things wearing one name, and separating them is the key to using it well.
- A template engine that turns a chart plus a values file into Kubernetes YAML. This part is excellent.
- A release manager that installs that YAML and records what it did in a Secret in the cluster. This part I hand to Argo CD.
The problem Helm solves is real. Without it, "the same application in two environments" means two copies of nine manifests that drift apart, and the drift is discovered in production. A chart makes the difference between environments a values file, which is a diff a human can read.
A template excerpt, because the interesting part of a chart is where it refuses to be flexible:
{{- fail "image.digest is required" }} makes a mutable image tag a rendering error. Not a policy document, not a code review convention, an error that stops the sync.
Tags are mutable. v1.4.2 can be re pushed to point at different bytes, which means a tag is a promise rather than an identity. Digests are content addressed and cannot lie. Making the safe thing the only thing that renders is worth more than any amount of documentation.
Same idea as extra="forbid" in Pydantic: put the guardrail where the mistake happens, not in a wiki page.
- No drift detection. After
helm install, Helm has no idea what happened to those objects. Somebody edits a Deployment and Helm still reports the release as deployed - Release state lives in the cluster. In a Secret in the namespace. Lose it and Helm cannot upgrade or roll back the release it created
- Someone has to run it. Which means credentials somewhere, and a human or a CI job with cluster write access
- Hooks are not ordering. Helm hooks work for pre install jobs, and they are invisible to anything that renders the chart with
helm template. Argo CD does exactly that, so I use sync waves instead
Helm answers "what should this application look like". It does not answer "is that what is running right now". Templating and convergence are different problems.
4. Argo CD: the cluster pulls, nobody pushes
Argo CD is a controller that runs inside the cluster and does one thing forever: render the manifests in a Git repository, compare them to what is live, and report or fix the difference.
If you have read the control plane chapter, this is a familiar shape. It is the same reconcile loop, with Git as the desired state store instead of PostgreSQL. Argo CD is to platform components what the AIForge reconciler is to tenant workloads.
Four properties fall out of that loop, and each one replaces a manual practice.
| Property | How it works | What it replaces |
|---|---|---|
| Git is the source of truth | Nothing reaches the cluster except through a commit | "Who applied this and when", answered by asking around |
| Drift detection | Continuous diff between rendered Git and live objects | Finding out during an incident that production is special |
| Self healing | An out of band change is reverted automatically | A quick fix that nobody wrote down and everybody forgot |
| Rollback | git revert, or pin the Application to an older revision | Re running an old pipeline and praying |
The Application object is the unit of everything:
Sync waves, or how to stop a chart from racing itself
Kubernetes has no dependency graph. Apply ten manifests and ten controllers start working at once. Usually fine. Occasionally the API pod starts, finds a database schema three migrations old, and crash loops while everything reports as applied successfully.
Argo CD's answer is an annotation. Resources are grouped into waves, and a wave does not start until the previous wave is healthy.
The wave order above encodes exactly one real dependency: the migration Job must finish before the API starts, because an API talking to an old schema fails in confusing ways.
The temptation is to keep adding waves until the install is a sequential script, which throws away the parallelism Kubernetes gives you and makes every sync slow. Worse, it hides missing readiness probes: if service B needs service A, the correct fix is usually a probe and a retry in B, not a wave.
My rule: a wave boundary needs a sentence explaining what breaks without it. If I cannot write the sentence, the wave comes out.
App of apps
One Application can point at a directory of other Applications. That gives you one root object that installs the entire platform, and it means adding a component is a file rather than a kubectl apply somebody has to remember.
There is an obvious circularity: Argo CD manages everything, so who manages Argo CD?
The answer is a deliberate one time act. Argo CD is installed once, by hand or by Terraform, and then it is handed an Application that manages itself from Git. From that point on, upgrading Argo CD is a commit.
I keep exactly one documented manual step in the whole platform: install Argo CD and apply the root Application. Two commands. Everything else in this project descends from those two, and I would rather have one honest bootstrap step than a clever recursive story that nobody can follow at 3am.
Proving self heal, which is the same style of test as the drift test in the control plane chapter:
That terminal is a good outcome and an uncomfortable one. My deliberate scale to seven replicas was reverted in eight seconds, without asking.
The uncomfortable half: self heal will also revert your emergency fix. At 3am, scaling something up to survive a traffic spike gets undone by a controller that is technically correct and unhelpfully literal.
So the emergency procedure has to be part of the design, not improvised. Mine, in order of preference: commit the change (fastest correct path, roughly a minute with a protected branch), or disable auto sync on the one Application, fix, then reconcile the change back into Git before re enabling.
If your incident response requires fighting your GitOps controller, you have not finished designing your GitOps controller.5. GitLab CI: producing an artifact worth trusting
GitLab CI is the last piece. Its job is narrow: take a commit, decide whether it is any good, produce an immutable artifact, and record a pointer to that artifact in the GitOps repository.
It does not deploy. It writes a commit and Argo CD does the rest.
| Stage | What it prevents reaching the cluster |
|---|---|
| lint | Style churn in reviews, and with mypy --strict, whole classes of shape bug in a codebase built on typed models |
| test | Logic regressions, plus a broken isolation control, which is tested as behaviour against a real API server |
| build | Unreproducible artifacts, and a privileged runner with a Docker socket, which is root on the host |
| scan | Known exploitable CVEs, and committed secrets, caught before the image is promoted |
| publish | Nothing on its own. It adds a human readable tag next to the immutable digest |
| promote | Untraceable deployments. The GitOps commit ties a running digest to a source commit and an author |
Trivy deserves a paragraph, because scanning is easy to do theatrically. It reads the packages in an image, the language dependencies inside it, and known secrets, then matches them against vulnerability databases. The judgement is not in running it, it is in what you fail on:
Four HIGH findings. Two are unfixable in this pipeline: no patched package exists, and one is marked will_not_fix upstream. Failing on those means a permanently red pipeline, and a permanently red pipeline gets bypassed by Friday. That is why --ignore-unfixed is on the gating command and off the reporting command: everything is recorded, only actionable things block.
Two are fixable, and one of them is a JWT algorithm confusion bug in the library that validates my tokens. That is precisely the class of attack the previous chapter pins algorithms=["RS256"] against, and it is a good reminder that a defence in my code and a patched dependency are both required.
A scanner that fails on everything teaches people to ignore scanners. Gate on what can be fixed, report everything, and review the ignored list on a schedule instead of forgetting it exists.
Secrets, briefly and firmly
The GitOps repository is a plaintext, replicated, permanently archived copy of the desired state. A secret committed there is a secret that exists forever in the history of every clone.
| Secret | Where it lives |
|---|---|
| PostgreSQL password | Generated by Terraform, written to a Kubernetes Secret. The chart references it by name and never contains a value |
Keycloak client secret for aiforge-ci | A masked, protected GitLab CI variable. Never in a file |
| GitOps push token | A project access token, scoped to one repository, write only to main |
| Registry credentials | GitLab's job token, which is short lived and scoped to the pipeline |
The lab uses Kubernetes Secrets created by Terraform. They are base64, not encrypted, and readable by anybody with the right RBAC, which is why tenant Roles in the previous chapter deliberately exclude secrets.
Production wants one of: the External Secrets Operator pulling from Vault, or Sealed Secrets so encrypted material can safely live in Git. Both are real work and both belong with the rest of the security story in Volume 4.
What I will not do is put a plaintext secret in the GitOps repository as a temporary measure. There is no temporary in Git history.
6. The boundary rules, stated as rules
The full loop, one commit from a keyboard to a running pod:
Notice there are two reconcile loops in that picture, and they never touch the same objects. Argo CD owns platform components from Git. The control plane owns tenant workloads from PostgreSQL. That separation is the single most important boundary in the volume, and it is what makes the loops safe to run at the same time.
| Rule | What it prevents |
|---|---|
| Exactly one system writes any given Kubernetes object | Two controllers reverting each other forever, which looks like a flapping deployment and is diagnosed as a network problem |
Terraform creates namespaces. Argo CD has CreateNamespace=false | A race where Argo CD creates a namespace without labels, so NetworkPolicy selectors silently match nothing |
| Helm templates, Argo CD applies | Two release managers with two opinions about the same objects, and a Helm release Secret nobody trusts |
| CI never holds a kubeconfig | Cluster write access reachable from any pipeline, including a compromised dependency |
| Images are referenced by digest | A mutable tag meaning different bytes tomorrow, and an unanswerable question about what is running |
| The control plane touches only tenant namespaces | A bug in my Python reconciler taking out Keycloak or Argo CD |
Fields that another system may own are in ignoreDifferences | An endless diff between Git and a live value an autoscaler legitimately changed |
You never see any of the four tools. Your loop is:
- Open a merge request against the control plane repository.
- The pipeline tells you whether lint, types, tests, isolation tests and the vulnerability gate passed.
- A reviewer approves and merges.
- A few minutes later the change is live, and the GitOps commit tells you the exact digest running.
If you want an AI application rather than a platform change, you never open a merge request at all. You call the API, and the control plane does the work.
Two things you cannot do, on purpose: deploy an image that failed a scan, and hand edit a platform object and have it survive. Both are guardrails rather than gates, because neither needs a human to approve anything.
7. What went wrong, and what I would watch
| What happened | Root cause | What I changed |
|---|---|---|
| A Deployment flapped between two and three replicas for an hour | Argo CD self heal reverting to the Git value while I scaled it by hand. Two systems, one field | /spec/replicas in ignoreDifferences, and a rule that scaling is a commit |
| NetworkPolicies selected nothing and isolation quietly did not exist | Argo CD created the namespace before Terraform did, so the tenant labels the selectors match were missing | CreateNamespace=false. Namespaces are Terraform only, and the label set is part of the contract |
| The API crash looped after a deploy that reported success | Migration Job and API Deployment applied in the same wave, so the API met an old schema | Sync waves, with the migration in wave 1 and the API in wave 2 |
| A rollback deployed the wrong code | The chart referenced a mutable tag, which had been re pushed | Digests only, enforced by fail in the template so a tag cannot even render |
| The Trivy gate was disabled within a week of being added | It failed on unfixable base image CVEs, so it blocked every pipeline and became noise | --ignore-unfixed on the gate, full severity report kept as an artifact, ignored list reviewed monthly |
| Deleting an Argo CD Application left every object it had created | Missing resources-finalizer annotation, so deletion was not cascading | Finalizer on every Application, plus a label based sweep to find orphans |
| Terraform wanted to destroy and recreate a namespace over a label | Some Kubernetes fields are immutable, so a small edit becomes a replacement | Read every plan before applying. A destroy line for a namespace full of workloads is a full stop, not a rubber stamp |
It looks like a deployment that occasionally has the wrong replica count. It looks like a config value that reverts every few minutes. It looks like flakiness, so it gets debugged as flakiness, and people spend days in the wrong logs.
The reason this is so dangerous is that both systems are working perfectly. Argo CD is enforcing Git. The autoscaler is enforcing load. Neither is buggy, and no single log line is wrong.
The fix is never technical, it is editorial: write down which system owns which field, and make the others explicitly ignore it. Server side apply and ignoreDifferences are how you express that decision. They cannot make it for you.
What I watch on the delivery path, which becomes alerting in Volume 4:
- Argo CD sync status per Application. Anything
OutOfSyncfor more than a few minutes is either a failing sync or a human editing production. - Self heal events per hour. Nonzero is fine. A spike means somebody is working around the platform and needs a better path, not a stricter one.
- Time from merge to running. If this creeps past a few minutes, people start batching changes, and batched changes are riskier changes.
- Pipeline failure rate by stage. A gate that fails constantly is about to be bypassed, whatever the policy says.
- Age of the oldest ignored CVE. The number that quietly grows until it is the reason for an incident.
- Terraform drift. A scheduled
planthat should produce no changes. Anything else means the substrate was edited outside HCL.
Every gate in section 5 checks that the artifact is well formed. It lints, it tests, it scans, it is pinned to a digest. Not one of them checks that what the artifact produces is still correct.
Volume 4 showed why that gap matters. A prompt template refactor went through this entire pipeline green at every stage, deployed cleanly, and silently stopped inserting retrieved context into the prompt. Every signal stayed healthy because every signal was measuring plumbing rather than output. Requests returned 200, latency was normal, retrieval itself worked fine. The answers just quietly got worse, and nothing anywhere went red.
What belongs in this pipeline is a canary assertion: one known question with a known answer, run against the built image, asserting that a specific phrase from a specific test document shows up in the output. It costs one stage and it would have caught that change before it shipped. A pipeline for an AI platform has to test behaviour, not just that the container starts. The incident is written up in reliability and failure drills.
Next
That closes Volume 3. The platform now has an API that turns intent into infrastructure, an identity provider issuing verifiable tokens, tenant isolation proven by commands that fail, and a delivery path where every change to the platform is a commit with an author.
What it does not have is any idea how it is doing. There are no dashboards, no alerts, no traces through a request, no LLM level telemetry, and no notion of what any of it costs. Deliberately: those are all things that need a working platform first.
Volume 4 picks it up from here. Prometheus, Grafana, Loki, OpenTelemetry and Langfuse for observability, OPA and Falco for policy and runtime security, deliberate failure injection, and honest cost accounting for AI infrastructure. It is also where the GPU conversation finally gets its numbers, on the day there is a GPU to measure.
If you want to revisit what runs underneath all of this, Volume 1 has the cluster and Volume 2 has the serving layer. The cluster tool itself is documented in k3smp.