Projects · AIForge
The reference architecture
Every AIForge layer, what each component is actually responsible for, the call rules that keep it debuggable, one request traced end to end, and the alternatives I rejected.
The reference architecture
A list of tools is not an architecture.
I could write "Kubernetes, vLLM, LiteLLM, Qdrant, Keycloak, Prometheus" and it would tell you almost nothing useful. It would not tell you who calls whom, what happens when one of them is down, where a request spends its time, or which component you go look at first when latency doubles.
So this document does the part that actually matters: responsibilities and boundaries. For every component I answer three questions. What is it responsible for. What is it explicitly not responsible for. What breaks when it is gone.
Read it once top to bottom to get the layer model. Then come back to section 8 whenever you are debugging, because that is the request path traced end to end, and knowing which hop you are on is most of the work.
1. The layer model
Six layers. Every component belongs to exactly one. If I cannot place something cleanly, that is a design smell and I go fix the design rather than fudge the diagram.
Observability, security, and delivery are not a layer you pass through. They are properties every other layer must have.
Drawing them as a box at the bottom of a stack is the most common architecture diagram lie, because it implies you can add them later. You cannot. A workload that was not built observable does not become observable by installing Prometheus.
The call rules
This is the part I care most about, and the part most architecture documents skip.
| Layer | May call | Must never |
|---|---|---|
| Experience | The control plane API, and Keycloak for tokens | Talk to Kubernetes, a model server, or Qdrant directly |
| Control plane | Kubernetes API, PostgreSQL, Keycloak admin, MLflow registry | Sit in the inference request path, or hold model weights |
| AI platform | Other AI platform components, and its own storage | Decide who a user is, or create Kubernetes resources |
| Execution | Compute and storage | Know anything about models, tenants, or prompts |
| Day two | Scrape, collect from, and gate every other layer | Be optional, or be bolted on after the workload exists |
It is tempting to route inference through the platform API, since the platform already knows tenants and quotas. Do not. If you do, then every control plane deploy, migration, or restart becomes an inference outage, and your platform's uptime becomes the ceiling on your product's uptime.
The control plane configures the path, then gets out of it. Inference goes through LiteLLM. That single boundary is worth more than most of the rest of this design.
2. Layer 1, experience and access
The surface where intent is expressed and identity is proven. Nothing here is optional and nothing here does real work.
Keycloak
Keycloak is the identity provider. It owns users, groups, roles, realms, clients, and the OIDC flows that issue tokens.
Responsible for: authentication. Proving that the caller is who they claim to be, and issuing a signed token containing claims about them.
Not responsible for: deciding what they may do inside AIForge. That is application authorization, and it lives in the control plane using claims from the token.
What breaks without it: you write your own auth. Which means you write your own password hashing, session handling, token expiry, refresh logic, group membership, and eventually SAML federation for an enterprise customer. Every one of those has a well known way to get subtly wrong.
Authentication answers "who are you". One system should own it, and it should be a system built by people who think about token replay for a living.
Authorization answers "may you do this specific thing to this specific resource". This is business logic. It depends on tenants, quotas, workload ownership, and role. It belongs in your application, close to the resource.
Push authorization into your identity provider and you end up encoding your entire domain model in realm roles. Pull authentication into your app and you own a security surface you did not want. Volume 3 shows exactly where I draw the line.
The API, UI, and CLI
Three faces on the same control plane API. The REST API is the real interface. The UI and CLI are clients of it, with no privileged path of their own.
If the UI can do something the API cannot, the platform is not automatable, and an internal developer platform that cannot be scripted has missed its own point.
So the rule is simple: the UI never gets a special endpoint. Whatever the UI does, a CI pipeline can do with a token and a curl.
3. Layer 2, the control plane
This is the part that makes AIForge a platform rather than a bundle of installed software.
The translation service
Python, FastAPI, Pydantic for validation, and the Kubernetes Python client for the write path.
Responsible for: accepting a deployment intent, validating it hard, checking tenant quotas, resolving a model reference into a concrete image and configuration, creating the Kubernetes resources, and recording what it did.
Not responsible for: serving inference, storing vectors, or deciding scheduling. It writes desired state and lets Kubernetes do what Kubernetes is good at.
A request as small as this:
becomes a Deployment, a Service, a ServiceAccount, a NetworkPolicy, a ServiceMonitor, a LiteLLM model registration, and a row in PostgreSQL. Seven artifacts from one intent, and the developer authored none of them.
Every field a developer can send is a field an attacker can send. Replica counts, memory strings, model names, and knowledge base identifiers all arrive as untrusted input.
Pydantic gives me a typed schema at the edge, so invalid input is rejected with a clear error before any Kubernetes call happens. The alternative is discovering that "replicas": 9999 was accepted because nobody checked, and the scheduler is now the thing enforcing your business rules.
PostgreSQL, and why it is not Qdrant
PostgreSQL holds platform state: tenants, users, applications, deployments, models, API keys, quotas, configuration, and audit records.
This trips people up, so let me be explicit about the split.
| PostgreSQL | Qdrant | |
|---|---|---|
| Holds | Business and platform state | Embeddings and document payloads |
| Query style | Exact, relational, transactional | Approximate nearest neighbour, similarity |
| Question it answers | "Which deployments does tenant A own, and are they within quota?" | "Which text chunks are semantically closest to this question?" |
| Consistency need | Strong. A quota check that is eventually correct is a broken quota check | Relaxed. An index rebuilt slightly behind is usually acceptable |
| Loss impact | Catastrophic. This is the platform's memory | Painful but recoverable. It can be rebuilt from source documents |
Look at the last row, because it drives the backup strategy.
PostgreSQL is the only place where certain facts exist. If it is gone, I cannot reconstruct who owned what. Qdrant is a derived store: every vector in it was computed from a source document, so worst case I re run ingestion and get it back.
Two databases with two different recovery postures. Treating them the same would mean either over engineering the recoverable one or under protecting the irreplaceable one.
4. Layer 3, the AI platform
The layer that does the actual AI work. Five components, each earning its place.
| Component | Responsible for | What breaks without it |
|---|---|---|
| LiteLLM | One OpenAI compatible front door. Routing, virtual keys, budgets, rate limits, retries, fallbacks, usage tracking | Every application hardcodes a model server address. Swapping a model becomes a coordinated deploy across every consumer |
| vLLM | The inference engine. Loading weights, managing the KV cache, continuous batching, generating tokens | You serve with a naive loop, and throughput collapses because requests queue instead of batching |
| KServe | A declarative serving contract. InferenceService, serving runtimes, autoscaling, scale to zero | You hand roll Deployment, Service, HPA, and probes per model, and they drift |
| RAG service | Ingestion, chunking, embedding, retrieval, reranking, context assembly | The model answers from training data only, and confidently invents your internal policies |
| Qdrant | Vector storage and filtered similarity search | Retrieval becomes a linear scan, and tenant filtering has nowhere to be enforced |
| MLflow | Model registry, versions, artifacts, stages | "Which model is in production" is answered by reading a container tag and hoping |
Fair challenge, and I want to answer it directly rather than collect tools.
They solve different problems. vLLM is an engine: it makes token generation fast. KServe is a contract: it makes model deployment declarative and gives you autoscaling and scale to zero for free.
You can run vLLM as a plain Deployment with no KServe at all, and for a single always on model that is genuinely simpler. KServe earns its complexity once you have many models with uneven traffic, because scale to zero on an idle model is the difference between paying for it and not.
AIForge documents both paths on purpose, because knowing when an abstraction is not worth it is a platform engineering skill. Volume 2 does the comparison properly.
5. Layer 4, execution
Kubernetes. Deliberately treated as an execution engine, not as a user interface.
| Primitive | What AIForge uses it for |
|---|---|
| Namespaces | The tenant boundary that every other control hangs off |
| RBAC | What each tenant service account may do, scoped to its own namespace |
| NetworkPolicy | Default deny, then explicit allows. Tenant pods cannot reach each other |
| ResourceQuota and LimitRange | Ceilings per tenant on CPU, memory, storage, and GPU, plus sane per pod defaults |
| Scheduling, requests and limits | Placing AI workloads on suitable nodes and keeping them from starving neighbours |
| Persistent volumes | Model weight caches, Qdrant storage, PostgreSQL data |
Worth internalising, because a lot of "multi tenant" Kubernetes is exactly this mistake.
A namespace is an organisational boundary. On its own it stops nothing: pods in different namespaces can reach each other over the network by default, and a permissive RBAC binding crosses it without effort.
A namespace becomes a boundary only when you add RBAC, a default deny NetworkPolicy, a ResourceQuota, and application level checks on top. Four controls, all required. Volume 3 builds them in that order and then attacks them.
6. Layer 5, compute and storage
Where the physics lives, and where my honesty about hardware belongs.
Real Multipass virtual machines running K3s, built with k3smp. Real kernels, real node boundaries, real scheduling.
Compute is CPU and memory only. Models are small instruct models: Qwen2.5 0.5B and 1.5B Instruct, TinyLlama 1.1B, and small embedding models such as all-MiniLM-L6-v2.
Storage is the K3s local-path provisioner, which means volumes are node local. That is a real constraint and it is in the deviation register, because node local storage plus a lost node equals lost data.
7. Layer 6, day two
Cross cutting, and covered in depth in Volume 4. Here is the map.
| Concern | Tools | The question it answers |
|---|---|---|
| Metrics | Prometheus and Grafana | Is it healthy, is it fast, is it saturated |
| Logs | Loki | What exactly happened in that one bad request |
| Traces | OpenTelemetry | Where did the time go across services |
| LLM telemetry | Langfuse | What was the prompt, what came back, how many tokens, was the retrieval any good |
| Policy | OPA Gatekeeper | Should this resource be allowed to exist at all |
| Vulnerabilities | Trivy | What known CVEs are we shipping |
| Runtime threats | Falco | Is something happening inside a container that should not be |
| Delivery | Terraform, Helm, Argo CD, GitLab CI | How does a change get from a commit to the cluster, reproducibly |
Prometheus, Loki, OpenTelemetry, and Grafana will happily tell you a request took 4.2 seconds and returned HTTP 200.
They cannot tell you the answer was wrong. They cannot tell you retrieval returned three irrelevant chunks, or that the prompt template silently truncated the context, or that this tenant burned 40,000 tokens on one conversation.
LLM observability is a different signal type, not a dashboard variant. That is the whole argument for Langfuse in this stack, and Volume 4 makes it concretely.
8. One request, traced end to end
This is the section to bookmark. When something is slow or wrong, knowing which hop you are on is most of the diagnosis.
Where time actually goes, and what each hop tells you
| Hop | If it is slow here | Go look at |
|---|---|---|
| Key validation and budget check | The gateway is struggling or its state store is slow | LiteLLM metrics and its backing store latency |
| Embedding the question | The embedding model is undersized or cold | Embedding service CPU saturation and pod restarts |
| Vector search | Index configuration, collection size, or a missing payload index on the tenant field | Qdrant query latency and index parameters |
| Rerank and assemble | Too many chunks, or a reranker doing more work than it is worth | Retrieval span duration and chunk count in traces |
| Queue time inside the engine | Saturation. More requests than the batch can absorb | vLLM queue depth and running sequence count |
| Token generation | Hardware limits, output length, or a model that is simply too large | Tokens per second, output token counts, and the hardware itself |
In this lab, token generation dwarfs every other hop. That is the direct consequence of running a language model on CPU.
Which creates a real analytical trap: on this hardware, every latency investigation ends at "generation is slow", and I could easily stop learning anything from latency work at all.
So I measure the other hops independently rather than as a share of total. Vector search latency, embedding latency, and gateway overhead are each meaningful on their own, they are hardware independent enough to be worth tuning, and they are the numbers that will still be true when a GPU shrinks the generation bar.
9. What I rejected, and why
Every choice above had alternatives. Here are the real forks.
| Decision | Chosen | Rejected, and why |
|---|---|---|
| Gateway | LiteLLM | Direct connections from apps to model servers. Simpler on day one, and it makes every model change a multi team deploy |
| Vector store | Qdrant | pgvector inside PostgreSQL, which is genuinely attractive at small scale. Rejected because I wanted the platform state store and the derived index to have different failure and recovery profiles, and because payload filtering is the security control here |
| Identity | Keycloak | Application managed users. Cheaper until the first enterprise SSO request, then a rewrite |
| Control plane style | API service that writes Kubernetes objects | A custom Kubernetes operator with CRDs. More idiomatic and more powerful, and a much larger build. Noted as the natural evolution, not the starting point |
| Delivery | Argo CD pulling from Git | CI pushing with kubectl apply. Faster to set up, no drift detection, and the cluster state becomes whatever the last pipeline run did |
| Cluster | K3s on real VMs via k3smp | kind or k3d. Faster, and containers as nodes cannot give me honest node loss drills |
A Kubernetes operator with a AIWorkload CRD would be the more idiomatic design, and if AIForge grows, that is where it should go.
I did not start there because an operator adds reconciliation loops, CRD versioning, and a much harder debugging story, and I wanted the platform boundaries proven first. Getting the responsibilities right with a plain API service is a prerequisite for getting them right in a controller. Volume 3 explains where the current design would hurt at scale, which is exactly the argument for the operator later.
10. What I would watch for
Layer bleed. The most likely way this design rots is the control plane slowly acquiring runtime responsibilities because it is convenient. Every time something wants to move into the request path, that is the moment to push back.
Gateway as single point of failure. LiteLLM sitting in front of everything is exactly the kind of thing that takes the whole product down. It needs replicas, probes, a PodDisruptionBudget, and a tested failure path. Volume 4 drills it.
Retrieval filter regressions. The tenant filter on vector search is a security control that looks like a query parameter. Refactor the retrieval code carelessly and you have a silent cross tenant leak. It needs a test that actively tries to break it, not a comment.
Too many components for one operator. This architecture has real surface area. In a real org that is a team. As one person, the honest risk is depth suffering across the breadth, and the mitigation is that every volume names what is running versus what is designed.
Next
You have the argument from the previous chapter and the design from this one. Time to build the thing it runs on.
The lab on k3smp provisions the cluster, sets the resource budget, creates the namespaces and quotas, gets a small model answering on CPU, and writes the deviation register that keeps the rest of this project honest.