Identity and multi tenancy

The previous chapter ended with an API that creates real Kubernetes workloads for anybody who can reach it. That is a fine place to be on a laptop for an afternoon, and an unforgivable place to stay.

This chapter fixes it in two halves.

The first half is identity: who is calling, proven cryptographically, using a token this platform did not issue and cannot forge. The second half is isolation: given that we know who is calling, making it structurally impossible for tenant A to reach tenant B, in enough independent layers that any single mistake is survivable.

The mental model I want you to leave with

Multi tenancy is not a feature you enable. It is a property that emerges when several independent mechanisms all happen to agree about where the boundary is.

A namespace, on its own, stops nothing. RBAC without NetworkPolicy stops the API and not the network. NetworkPolicy without application scoping stops packets and not queries. Isolation is a stack, and the useful question is always which layer is doing the work.

1. Why identity comes before everything else

Every authorization decision in a platform is a sentence of the form "this actor may do this action to this resource". Miss the actor and the other two are decoration.

The naive approach is a shared API key in an environment variable. It works, and here is what it costs you: no idea which human acted, no revocation without rotating everyone, no roles, no expiry, and no way to answer the only question that matters after an incident, which is who did this.

So the platform needs an identity provider. Concretely it needs something that can:

authenticate humans against a credential store, ideally with a second factor
authenticate machines (CI jobs, the CLI, other services) without a human present
issue short lived, signed, verifiable tokens that carry the caller's tenant and roles
revoke access centrally when somebody leaves
speak a standard protocol so the API does not implement authentication itself

That last bullet is the important one. I do not want to write authentication. Password hashing, session fixation, token rotation, brute force lockout, account recovery: all of it is a specialist domain where the cost of a small mistake is total. Delegating it to a dedicated component is not laziness, it is the single highest leverage security decision in the project.

Keycloak is that component here. It is an open source identity and access management server that implements OpenID Connect and OAuth 2.0, runs as a container, and stores its state in PostgreSQL.

What it would cost me to not have Keycloak

I priced this out honestly before adding another stateful component to a laptop cluster.

Without it I would be writing: a user table with password hashing, a login endpoint, session or token issuance, refresh and rotation, a group and role model, an admin UI to manage all of it, and eventually SSO integration when somebody asks for it. That is a project, not a feature, and every line of it is security sensitive code I would maintain forever.

With it I write one dependency function that verifies a JWT signature and reads claims. That is the whole integration. The trade is a stateful service I must operate against a body of code I must never get wrong. I will take the stateful service.

2. Keycloak in plain terms

If you have never used Keycloak, the vocabulary is the main obstacle. Five words, and once they click the rest is configuration.

Keycloak concept	What it actually is	In AIForge
Realm	A fully isolated identity universe: its own users, its own clients, its own signing keys. Two realms share nothing	One realm, `aiforge`. The `master` realm is for administering Keycloak itself and holds no platform users
Client	An application that participates in authentication. Either something that asks for tokens, or something that consumes them	`aiforge-ui`, `aiforge-cli`, `aiforge-api`, `aiforge-ci`
User	A human account with credentials, attributes and memberships	`alice`, `bob`, `charlie`. The tenant lives in group membership, not in a free text attribute
Group	A container of users that can carry roles and attributes. Hierarchical	`/tenants/tenant-a` and `/tenants/tenant-b`. Group membership is how tenancy is assigned
Role	A named permission label. Realm roles are global, client roles are scoped to one client	`platform-admin`, `tenant-admin`, `developer`, `viewer`

The client types, and why there are four

This is where people get stuck, so here is the rule: a client's type is decided by whether it can keep a secret.

Client	Type	Flow	Why
`aiforge-ui`	Public	Authorization code with PKCE	It is JavaScript in a browser. Any secret shipped to it is public. PKCE replaces the secret with a per request proof
`aiforge-cli`	Public	Authorization code with PKCE, loopback redirect	Installed on a laptop, so it also cannot hold a secret. It opens a browser and listens on `127.0.0.1` for the code
`aiforge-api`	Bearer only	None. It issues nothing	It is a resource server. It exists in Keycloak so tokens can carry it as an audience, which is what makes audience validation meaningful
`aiforge-ci`	Confidential	Client credentials	A GitLab runner with no human present. It holds a real secret in a masked CI variable and exchanges it for a token

Never use the password grant, even though it is the easiest thing in the world

The resource owner password credentials grant means your application collects the username and password and posts them to Keycloak. It is four lines of code and it is the wrong answer.

It teaches users to type platform credentials into anything that shows a login box, it cannot support multi factor authentication or any federated login, it forces every client to handle raw credentials, and it is deprecated by the OAuth security best current practice.

Authorization code with PKCE is more moving parts and it means no component of AIForge ever sees a password. That property is worth the extra work, and it is not close.

The OIDC flow, once, concretely

Here is a developer logging into the UI and creating a deployment. Every arrow is a real HTTP request.

Preparing diagram

Two points about that diagram that are easy to miss.

Step 12 happens once, not per request. The API fetches Keycloak's public keys and caches them. Verification after that is local CPU work: no network call, no dependency on Keycloak being up. That is the entire argument for JWT over opaque tokens plus introspection. The cost is that a revoked token stays valid until it expires, which is why token lifetimes are short.

The API never handles a credential. It receives a signed assertion from a component it trusts and checks the signature. That is the whole security model, and it is why the integration is small.

The three tokens, and the one people misuse

Token	Audience	Purpose	Lifetime here
Access token	The API, `aiforge-api`	Authorize API calls. This is the only one the API accepts	5 minutes
ID token	The client, `aiforge-ui`	Tell the client who logged in, so it can render a name and avatar	5 minutes
Refresh token	Keycloak's token endpoint	Obtain a new access token without a new login	30 minutes idle, 10 hours absolute

Sending the ID token to the API is the most common OIDC mistake there is

Both tokens are JWTs, both are signed by the same realm, both contain a sub. A naive verifier accepts either. So people grab whichever token is convenient in their frontend state and send it, and it works.

It is still wrong. The ID token's audience is the client, and its purpose is to describe an authentication event to that client. It is not a capability to call an API, and it may carry claims that the API has no business making decisions on.

The defence is one line: validate the audience. The API accepts tokens whose aud contains aiforge-api and rejects everything else. Skip that check and you have also accepted every access token issued to every other client in the realm, which is the version of this bug that actually hurts.

Here is an access token as the API sees it, with the claims that matter:

aiforge_tenant is not a Keycloak default. It comes from a protocol mapper on the client scope that flattens group membership into a single claim, so the API does not have to parse group paths:

Why the tenant is a claim and not a lookup

I could load the user's tenant from PostgreSQL on every request. One extra query, always current, no mapper configuration.

I put it in the token instead, for one reason: the tenant then arrives signed. It cannot be tampered with in transit, it cannot be spoofed by a request body, and the authorization decision needs no database round trip. The cost is staleness: moving a user between tenants takes effect when their token refreshes, up to five minutes.

That trade is only acceptable because the lifetime is short. Long lived tokens with authorization claims inside them are how you end up unable to revoke anything. Five minutes of staleness I can live with. Eight hours I could not.

3. Authentication versus authorization

Two words that get used interchangeably and mean completely different things. The distinction is not pedantry, it maps directly onto two different pieces of code and two different HTTP status codes.

	Authentication	Authorization
Question	Who are you, and can you prove it?	Are you allowed to do this, to this thing?
Owner	Keycloak issues, the API verifies	The API, entirely. Keycloak has no idea what a deployment is
Input	A signed token	The verified principal, the action, and the target resource
Failure	`401 Unauthorized`, meaning "I do not know who you are"	`403 Forbidden`, meaning "I know exactly who you are and no"
Frequency	Once per request, cheap and local	Once per decision, and there are several per request

Keycloak can do authorization too, and I chose not to use it for this

Keycloak has a full authorization services feature: resources, scopes, policies, permission evaluation. It is genuinely powerful.

I keep authorization in the API anyway, because platform authorization questions are data dependent. "May Bob scale this deployment" depends on which tenant owns the deployment, which is a row in my database that Keycloak cannot see without me pushing my entire data model into it and keeping it in sync.

The boundary I settled on: Keycloak answers who you are and what roles you hold. The API answers what that means for this specific object. Coarse grained in the token, fine grained in the code, and the fine grained part sits next to the data it needs.

In FastAPI this becomes one dependency for authentication and a small factory for role checks:

Used at the route, where a reviewer can see it:

Four validation checks, and skipping any one of them is a vulnerability

Signature against JWKS. Without it, anybody can write any claims they like. This is the whole game
Algorithm pinned to RS256. Trusting the token's own alg header enables the classic alg: none and HMAC confusion attacks, where the attacker signs with the public key
Audience. Without it, a token issued to any other client in the realm is accepted here
Issuer. Without it, a token from any Keycloak realm anywhere, including one an attacker controls, is accepted

Every one of those has shipped in production somewhere. The jwt.decode above passes all four explicitly rather than relying on library defaults, because defaults change between versions and this is not a place for surprises.

The rule that prevents the worst bug in multi tenant software

Say it out loud: the tenant comes from the token

This is the bug class that produces breach reports. It shows up as a helpful tenant_id field on a request model, added because an admin endpoint needed it once, and then never removed. Any authenticated user can now act as any tenant.

Two structural defences, because a rule nobody enforces is a preference:

DeploymentCreate has extra="forbid" and no tenant field. A body carrying one is a 422, not an ignored value
The repository layer takes the tenant as a required constructor argument. There is no query method that can be called without one, so "forgot to filter" is not expressible

Make the insecure version fail to compile, or at least fail to instantiate.

404 or 403 for a cross tenant read? I return 404, and log it as an attack

If Bob in tenant A requests a deployment id that belongs to tenant B, there are two defensible answers.

403 Forbidden is honest: the object exists and you may not see it. It also confirms the object exists, which turns the endpoint into an enumeration oracle for another tenant's inventory.

404 Not Found leaks nothing. From Bob's position, tenant B's objects and objects that never existed are indistinguishable. That is exactly the property I want.

The repository above produces 404 naturally, because the query returns nothing. The important addition is on the logging side: a 404 for a well formed UUID that exists under another tenant is not a typo, it is a signal, and it gets logged at warning with the actor's subject. Quiet on the wire, loud in the audit trail.

4. Multi tenancy, enforced in layers

Identity tells us who is calling. Now make the boundary real. Six layers, each independently sufficient to stop a different class of attack, and none of them trusted alone.

Preparing diagram

Layer	Threat it actually stops	What it does not stop
Identity	Anonymous or forged callers, expired sessions	A legitimate user doing something they should not
Application authorization	A valid token reaching another tenant's records through the API	Anything that bypasses the API and talks to Kubernetes directly
Namespace	Name collisions, and it scopes the layers below	Nothing else. See the callout immediately below
RBAC	A leaked ServiceAccount token being used against another namespace	Network traffic, which does not go through the Kubernetes API at all
NetworkPolicy	Pod to pod connections across tenants, and unwanted egress	Anything at layer 7. It cannot read an HTTP path or check a token
Quota and LimitRange	Resource exhaustion, deliberate or accidental	Data access. A tenant inside quota can still misbehave

A namespace is not a security boundary by itself

This is the most expensive misconception in Kubernetes, and it is worth being precise about what a namespace is.

A namespace is a scope for object names and a target for policy. That is all it is. Out of the box, with nothing else configured:

A pod in aiforge-tenant-a can open a TCP connection to any pod in aiforge-tenant-b. Pod networking is flat by default
Any ServiceAccount with a broad ClusterRole can read Secrets in every namespace
Pods from both tenants share a node, a kernel and a container runtime
One namespace can consume every CPU cycle on the cluster and starve the other

The namespace only becomes a boundary once RBAC, NetworkPolicy and ResourceQuota are attached to it. The namespace is the handle. The policies are the boundary.

And note the last bullet in that list: even with all six layers, tenants share a kernel. Real hostile multi tenancy needs node pools per tenant, or virtual machine level isolation. In this lab the tenants are trusted teams in the same organisation, not the public internet, and I would rather state that than imply a guarantee I have not built.

5. Layer 3 and 4: namespaces and RBAC

The namespace, created by Terraform in chapter 3 rather than by hand, with labels the other layers select on:

Kubernetes RBAC is four object types and one rule. The rule: a Role is a set of permissions, a RoleBinding attaches it to a subject. Role and RoleBinding are namespaced. ClusterRole and ClusterRoleBinding are cluster wide.

The control plane needs write access in every tenant namespace, and this is where a genuinely useful RBAC detail lives:

Why a ClusterRole in a RoleBinding, which looks like a contradiction

A ClusterRole referenced by a RoleBinding grants its permissions only inside the binding's namespace. This is the standard pattern for "the same permission set, applied separately per namespace", and it matters here for a specific reason.

I define the workload manager permissions once as a ClusterRole, then bind it into each tenant namespace individually. The control plane can therefore create Deployments in aiforge-tenant-a and aiforge-tenant-b because those bindings exist, and nowhere else because no other binding does. Onboarding a tenant adds a RoleBinding. Offboarding removes one.

The lazy alternative is a ClusterRoleBinding, which grants the permission in every namespace including kube-system. A compromised control plane with a ClusterRoleBinding owns the cluster. With per namespace bindings it owns exactly the tenants it manages. Same code, radically different blast radius.

Proving RBAC does what the YAML claims. kubectl auth can-i --as asks the API server to evaluate a permission for another subject, which means the answer comes from the authorizer rather than from my reading of the file:

Asking the API server directly, instead of trusting the YAML

$▊

The last two commands are the ones I would show in an interview

The control plane can create Deployments in a tenant namespace and cannot create them in kube-system. That single pair demonstrates the whole least privilege argument: the component with the most power in the platform is still bounded, and the bound is testable in one command.

Anybody can write RBAC YAML. kubectl auth can-i is how you find out whether it means what you thought.

6. Layer 5: NetworkPolicy

RBAC governs the Kubernetes API. It has nothing to say about a pod opening a socket, because that traffic never touches the API server. By default, Kubernetes pod networking is flat: every pod can reach every other pod in every namespace.

NetworkPolicy changes that, with two rules worth internalising before writing any:

A pod selected by no policy allows all traffic. Policies are additive allowlists, so the absence of policy is not deny
Once a pod is selected by any policy for a direction, everything not explicitly allowed in that direction is denied

Which is why the first policy in every namespace is a deny all that selects everything:

That immediately breaks everything, which is correct. Now allow only what the workload genuinely needs:

K3s enforces NetworkPolicy, which is not true of every small cluster

This deserves saying because it is the difference between testing a control and admiring it. NetworkPolicy is an API object that any cluster will happily accept. Whether it is enforced depends entirely on the CNI plugin.

K3s ships flannel plus an embedded NetworkPolicy controller based on kube-router, so policies are enforced with real iptables rules. A hand rolled cluster with plain flannel and no policy controller accepts every policy above and enforces none of them, silently.

If you are ever unsure which situation you are in, do not read the docs. Apply a deny all and try to open a connection. A control you have not attacked is a control you do not have.

The attack, from inside a pod, which is the only place the answer is trustworthy:

Trying to reach tenant B from inside a tenant A pod

$▊

Three results, and the third is the one people forget to test

Cross tenant traffic times out. Platform traffic works. And egress to the public internet also times out, because the default deny covers egress and nothing allows it.

That third result is a real control, not a side effect. A model serving pod that cannot reach the internet cannot exfiltrate a prompt, cannot phone home, and cannot pull a payload if the container image turns out to be compromised. It also means the day a tenant legitimately needs an external API, that access is a written policy with a reviewer rather than an assumption.

Note that it times out rather than refusing. A dropped packet gives an attacker no information, and it also means an application with a short timeout reports this as slowness. That is worth knowing before you debug it at 2am.

What NetworkPolicy cannot do

Limitation	Consequence here
Layer 3 and 4 only	It cannot say "tenant A may call `/v1/chat` but not `/v1/admin`". Path level control is LiteLLM's and the API's job
No identity, only labels and IPs	Any pod that lands in `aiforge-system` is trusted by every tenant policy. Who can create pods there is therefore a critical RBAC question
Selectors are labels, and labels are mutable	Whoever can edit a namespace label can change what a policy matches. Namespace labels are Terraform managed and not tenant writable
No DNS name based egress in this implementation	Allowing one external API means allowing a CIDR, which is coarser than anybody wants. A CNI like Cilium would do better
No encryption	Traffic that is allowed is still plaintext inside the cluster. mTLS is a service mesh question, and it is not in this volume

7. Layer 6: ResourceQuota and LimitRange

The last layer is about fairness and blast radius rather than confidentiality. Without it, one tenant deploying six replicas of a model that wants 6 GiB each takes the cluster down for everybody, and no amount of RBAC notices.

ResourceQuota caps aggregate consumption per namespace and is enforced at admission.

The GPU line, stated plainly, because this is the constraint that shapes the whole project

There is no GPU in this lab. I do not own one. Every model AIForge serves today is a small instruct model running on CPU: Qwen2.5 0.5B and 1.5B Instruct, TinyLlama, and a small embedding model for retrieval.

So requests.nvidia.com/gpu is pinned to "0", and that is a design decision rather than a placeholder. Here is the reasoning.

With the quota set to zero, a pod requesting a GPU is rejected at admission with a message naming the quota, the resource and the limit. The control plane surfaces that sentence to the developer, as you saw in the previous chapter. With the line absent, the pod is admitted and then sits Pending forever, because no node advertises nvidia.com/gpu, and the only clue is a scheduling event nobody reads.

Failing fast with a sentence beats hanging quietly.

What changes on the day a GPU node exists: install the NVIDIA device plugin so nodes advertise the resource, raise this one number, and set requests equal to limits because extended resources require that. The schema field, the database column, the quota arithmetic and the pod template already carry it. That is what I mean when I say the GPU path is a configuration change rather than a redesign, and I would rather be measured against that claim than against a benchmark I cannot run.

Setting a compute quota changes the rules for every pod in the namespace, including ones you did not write

This surprises people and it broke my lab the first time. Once a namespace has a quota on requests.cpu or requests.memory, every pod must specify that resource explicitly or be rejected.

The symptom is brutal. You add a sensible quota, everything running keeps running, and then some unrelated deployment fails with must specify requests.cpu. It is usually not your application. It is a sidecar, a job, or a debug pod nobody templated properly.

The fix is not to loosen the quota. It is to pair every quota with a LimitRange that supplies defaults, so a pod with no resources stated gets sensible ones rather than a rejection. Quota and LimitRange are a pair. I have never seen one deployed correctly without the other.

Field	What it prevents
`defaultRequest` and `default`	Unspecified pods being rejected by the quota, and unbounded containers eating a node
`max`	A single container claiming the whole tenant allocation, leaving no room for a second replica
`min`	Absurdly small requests that get scheduled anywhere and then thrash
`maxLimitRequestRatio`	The noisy neighbour pattern: schedule small, burst enormous, and make somebody else's latency graph ugly

LimitRange only applies at creation, which changes how you roll it out

A LimitRange is an admission time mutation. Existing pods are not touched, and there is no reconciliation.

So tightening one does nothing until pods are recreated, and the change surfaces at the worst possible moment: the next unrelated rollout, or a node reboot at 3am, when a pod that ran for months suddenly cannot be admitted.

My rule after learning this the annoying way: change a LimitRange and then deliberately roll the namespace, in working hours, while watching. Do not let an admission policy change take effect during an incident.

8. Tenant isolation testing

Everything above is a claim until a command fails. This is the suite I run after any change to a tenant's configuration, and each test maps to exactly one layer.

#	Test	Layer proven	Expected result
1	Call the API with no token, an expired token, a wrong signature, and a wrong audience	Identity	Four `401` responses, four distinct log reasons
2	Read tenant B's deployment id with a valid tenant A token	Application authorization	`404`, plus a warning log naming the actor
3	POST a body containing a `tenant` field	Application authorization	`422` from `extra="forbid"`, never an ignored value
4	`viewer` role attempts a create	Application authorization	`403` naming the required roles
5	Tenant A ServiceAccount lists pods in tenant B	RBAC	`Forbidden` from the API server
6	Tenant A pod opens a socket to a tenant B pod IP	NetworkPolicy	Timeout, from inside the pod
7	Tenant A pod calls the public internet	NetworkPolicy	Timeout
8	Deploy past the CPU or memory quota	ResourceQuota	`Forbidden` with the arithmetic in the message
9	Request a GPU	ResourceQuota	`Forbidden` immediately, not `Pending` forever
10	Create a pod with no resources specified	LimitRange	Admitted, with the defaults visible in the applied spec
11	Query tenant B's Qdrant collection with tenant A's credentials	Data isolation	Rejected by the collection scoped key

Quota and LimitRange behaviour, observed rather than assumed

$▊

Four controls, four observed outcomes, zero trust in my own YAML

The quota is genuinely counting. The over quota deployment is refused at pod creation with the full arithmetic. The GPU request is refused instantly rather than left Pending. And the pod with no resources specified came out of admission carrying the LimitRange defaults, which I can read back from the applied spec.

That last one is the test people never write, and it is the one that proves the quota is survivable rather than merely present.

And the identity layer, which is the one an attacker actually meets first:

Four ways to fail authentication, four distinguishable answers

$▊

Vague on the wire, precise in the log

Notice the asymmetry. The caller gets invalid token and a 404. The log records audience_mismatch, token_expired, and a cross tenant access attempt with the actor's identity attached.

That is not paranoia theatre. Detailed error responses are a free debugging tool for an attacker: signature_invalid versus audience_mismatch tells them exactly which knob to turn next. The person who legitimately needs the detail has access to the logs. Verbose to the operator, terse to the caller.

And the last line is the whole reason identity comes before isolation. Without a verified actor, that log entry would read "somebody tried something", which is not an audit trail. It is a rumour.

9. What went wrong, and what I would watch

What happened	Root cause	What I changed
Every pod in the tenant namespace broke the moment I applied the deny all policy	Egress deny includes DNS, and no policy allowed port 53	The DNS allow ships in the same commit as the deny all, always. They are one change, not two
An unrelated deployment started failing hours after I added a quota	A quota on `requests.cpu` makes that field mandatory for every pod, and a sidecar did not set it	Quota and LimitRange are applied as a pair by Terraform. Neither exists alone
A GPU request sat `Pending` for two days before I noticed	No GPU exists, and the quota did not mention GPUs at the time, so admission had no reason to object	`requests.nvidia.com/gpu: "0"`, which converts silent hanging into a clear refusal
The UI's token worked against the API when it should not have	Audience validation was off, so any token from the realm was accepted	`audience="aiforge-api"` passed explicitly, plus a test that sends an `aiforge-ui` token and asserts `401`
Users kept being logged out mid task	A 5 minute access token with no silent refresh in the frontend	Kept the 5 minute lifetime and implemented refresh properly. Extending the token was the tempting wrong fix
A cross tenant read returned `403` and confirmed the object existed	Authorization checked ownership after loading the row by id	Tenant scoped repository, so the row is never loaded. `404` falls out naturally
Keycloak's own database was the least backed up thing in the platform	I treated it as infrastructure rather than as state	It is state. Realm export in version control, database in the same backup schedule as the platform database

The limitation I would raise first if somebody asked me to defend this design

Tenants share a kernel. Six layers of policy do not change that, and I am not going to imply otherwise.

Namespaces, RBAC, NetworkPolicy and quotas are Kubernetes level controls. A container escape, a kernel vulnerability, or a node level compromise walks straight through all of them, because they all live above the boundary that was broken. Pod Security Standards at baseline make that harder. They do not make it impossible.

The honest framing: this design isolates cooperating teams inside one organisation. It is not built for hostile multi tenancy. Running untrusted tenant code would need node pools per tenant, or a sandboxed runtime like gVisor or Kata Containers, or separate clusters. That is a different architecture with a different cost, and pretending policy YAML substitutes for it is exactly the kind of claim I am trying not to make in this project.

What I watch, which becomes alerting in Volume 4:

Authentication failure rate by reason. A spike in signature_invalid is somebody probing. A spike in token_expired is my refresh logic being broken.
Cross tenant access attempts. Should be exactly zero. Anything else is either an attack or a client bug, and both need a human.
Quota utilisation per tenant. Above 80 percent is a conversation before it is an incident.
NetworkPolicy denied connections. A new nonzero number means either an attack or a legitimate dependency nobody declared.
Pods rejected at admission, by reason. This is where quota, LimitRange and policy problems surface first.
Keycloak token issuance latency. It is on the login path, so when it is slow the whole platform feels broken.

An asterisk on this chapter that only appeared once Volume 4 broke things

Everything above argues that authentication does not hang on any one component staying up, because Keycloak's signing keys are cached and verification after that is local CPU work.

That argument holds for the control plane API and it does not hold for the gateway. LiteLLM caches tenant API keys in memory, and the source of truth for those keys is PostgreSQL. Scaling Postgres to zero during the database drill did not break authentication, which looked like a win until I worked out why it survived. The cache was already warm. Had the gateway restarted at any point during that outage it would have come up empty, been unable to reach Postgres, and every tenant would have lost authentication at the same moment.

So the independence I claim here has a condition attached: it holds as long as nothing restarts. The fix is a bounded on disk cache with a documented staleness window, and it is not built yet. Details in reliability and failure drills.

IaC, GitOps and CI/CD does that. Terraform creates the namespaces, quotas and the Keycloak realm, Helm packages the control plane, Argo CD reconciles it all from Git, and a GitLab pipeline makes sure nothing unscanned gets that far. It also settles the ownership question left open in the previous chapter: who is allowed to write which object.