Projects · AIForge
AIForge
Production AI infrastructure as an internal developer platform. Kubernetes, LLM serving, RAG, multi tenancy, observability, security and cost, built and documented end to end on a real cluster.
AIForge is a platform engineering project, not a machine learning project. I am not training models here. I am building the infrastructure that runs other people's models in production, and documenting every decision along the way.
If you only read one page, read this one. It explains the problem, the architecture, and where each volume takes you.
AIForge
Here is a sentence I have heard, in some form, in every engineering org that started using LLMs.
"The demo works. Why is it taking three months to ship?"
The demo works because a demo is one Python file talking to one API. Shipping takes three months because production is not one Python file. Production is compute, GPU scheduling, model serving, retrieval pipelines, vector storage, authentication, tenant isolation, observability, security controls, CI/CD, infrastructure automation, cost accounting, and somebody who wakes up at 3am when the thing falls over.
Every team rebuilds that stack. Badly. Separately. Three times in the same company.
AIForge is my answer to that. It is a cloud native platform that turns all of the above into something a developer can consume, while the platform team keeps control of the parts that matter.
AIForge makes production AI infrastructure consumable as a platform instead of making every team build it themselves.
The gap this project is really about
Let me show you the gap, because it is bigger than people expect.
This is the prototype everyone has:
And this is the same idea, in production, serving real users under a real SLO:
And that diagram is still missing everything that makes it survivable: infrastructure as code, GitOps, CI/CD, metrics, logs, traces, LLM telemetry, policy enforcement, image scanning, runtime detection, secrets, quotas, autoscaling, cost tracking, and disaster recovery.
Not at the model. At the seventeen things around the model.
Teams underestimate this because the model is the interesting part and the rest feels like plumbing. Then they discover that nobody can tell which tenant burned the GPU budget, no one can reproduce last Tuesday's deployment, and the retrieval quality regression that shipped on Friday was invisible until a customer complained on Monday.
What developers see, and what the platform absorbs
The whole design rests on one separation. Developers should express intent. The platform should own complexity.
A developer should be able to say this:
Deploy this model with two replicas, connect it to this knowledge base, expose an API, monitor it, isolate it from other tenants, and make the whole thing reproducible.
And that is it. No Helm chart authoring, no NetworkPolicy debugging, no arguing with a GPU scheduler.
- Create an AI application
- Pick a model
- Pick a knowledge base
- Pick resources and replicas
- Deploy, then read logs, metrics, and traces for their own workload
- Get an API endpoint and a key
That is the entire surface. Everything else is somebody else's problem, and that somebody is the platform.
The honest part: I do not have a GPU
I want this on the first page, not buried in an appendix.
AIForge is designed for GPU workloads. Right now I do not own one, so the entire lab runs small instruct models on CPU: Qwen2.5 0.5B and 1.5B Instruct, TinyLlama 1.1B, and small embedding models such as all-MiniLM-L6-v2.
That changes what the lab can prove, so let me be precise about it rather than vague.
What CPU still proves, genuinely: the wiring, the API contracts, the scheduling behaviour, tenant isolation, autoscaling reactions, observability coverage, security policy enforcement, failure handling, and the whole developer experience. All of that is real, running, and testable.
What CPU cannot prove: throughput at scale, PagedAttention efficiency under load, tensor parallelism, and real euros per million tokens on real hardware.
What I did about it: the GPU path is designed, written down, and parameterised. NVIDIA GPU Operator, nvidia.com/gpu resource requests, GPU quotas per tenant, and DCGM metrics all exist in the manifests and dashboards. They are configuration, not architecture. When a GPU arrives, I change values, not the design.
I would rather show you an honest platform with a documented hardware gap than a fake benchmark. Every volume marks clearly which parts are running and which parts are designed and waiting.
The lab it runs on
The cluster is not Docker containers pretending to be nodes. It is real virtual machines running real K3s, built with k3smp, a tool I wrote for exactly this reason.
Node isolation, scheduling, disruption budgets, and node loss drills only teach you something when the nodes are actually separate machines. In containers-as-nodes, half the interesting failures cannot happen.
I already used this approach in the APM project, where a k3smp cluster carried a full SigNoz and OpenTelemetry stack. That project is prior art here: it is where I proved the bootstrap path and the telemetry wiring before pointing them at AI workloads.
The full lab build, resource budget, and the honest list of what is real versus collapsed lives in the lab on k3smp.
Who this platform is for
A platform with one type of user is a script. AIForge deliberately serves five, and each one gets a different view of the same system.
| Persona | What they care about | What AIForge gives them |
|---|---|---|
| Developers | Models, knowledge bases, API access, deployment status, logs | Self service deployment and a stable OpenAI compatible endpoint |
| Platform engineers | Kubernetes, automation, capacity, GPU utilization, reliability | One reproducible stack instead of eight bespoke ones |
| SRE and operations | SLOs, latency, availability, incidents, alerts, recovery | Metrics, logs, traces, LLM telemetry, and rehearsed failure drills |
| Security | Identity, RBAC, isolation, vulnerabilities, runtime threats, audit | Policy at admission, scanning in CI, detection at runtime, isolation in layers |
| Management | Adoption, cost, utilization, availability, business value | Cost per tenant, cost per million tokens, and utilization that is measured |
Reference architecture in one picture
Six layers. Each one has a clear job, and each one is covered by a volume.
The layer by layer walkthrough, including what I rejected and why, is in the reference architecture.
How this project is organised
Four volumes. They build on each other, so read them in order the first time.
| Volume | What it covers | What you can do when you finish it |
|---|---|---|
| 1 | Platform foundations | Explain why AI needs a platform, read the full reference architecture, and stand up the K3s lab that everything else runs on |
| 2 | The AI serving layer | Serve an LLM with vLLM, understand KServe and MLflow, front everything with LiteLLM, and build a working RAG pipeline on Qdrant |
| 3 | The platform control plane | Build the FastAPI control plane that turns intent into Kubernetes objects, wire Keycloak identity and real multi tenancy, and automate it with Terraform, Helm, Argo CD and GitLab CI |
| 4 | Operations, security and cost | Instrument the platform end to end, enforce security in depth, measure what AI actually costs, and break the thing on purpose to learn how it fails |
Because the four map onto the four questions a hiring manager or a staff engineer will actually ask you.
Can you design it. Can you serve models on it. Can you make it self service and multi tenant. Can you run it on a Tuesday when it breaks.
Everything else is a subsection of one of those.
The technology stack
Nothing exotic. These are the tools teams genuinely run, which is the point.
| Layer | Technology | Why it is here |
|---|---|---|
| Execution layer | Kubernetes, K3s in the lab | Scheduling, isolation, scaling, self healing, and a stable API to build a platform on |
| GPU enablement | NVIDIA GPU Operator | Drivers, device plugin, and metrics as a managed stack. Designed in, not yet exercised |
| LLM serving | vLLM | Continuous batching and PagedAttention, the throughput engine of the platform |
| Model serving abstraction | KServe | InferenceService as a contract, autoscaling and scale to zero |
| Model gateway | LiteLLM | One OpenAI compatible front door with keys, budgets, routing, and fallbacks |
| Vector database | Qdrant | Semantic retrieval with payload filtering, which is how tenant scoped RAG stays scoped |
| Model lifecycle | MLflow | Versions, artifacts, and a registry, so "which model is in production" has an answer |
| Control plane | Python and FastAPI, PostgreSQL | Where developer intent becomes validated Kubernetes objects |
| Identity | Keycloak | OIDC, realms, groups, and roles, so authentication is not homemade |
| Delivery | Terraform, Helm, Argo CD, GitLab CI | Infrastructure, packaging, GitOps reconciliation, and the pipeline that feeds them |
| Observability | Prometheus, Grafana, Loki, OpenTelemetry, Langfuse | Metrics, dashboards, logs, traces, and the LLM specific signals the first four cannot give you |
| Security | OPA Gatekeeper, Trivy, Falco | Policy at admission, vulnerabilities in the pipeline, threats at runtime |
The principles I am actually holding myself to
Principles are cheap when they are a bullet list, so each one here has a consequence attached.
| Principle | What it costs me in practice |
|---|---|
| Automation first | If I do something manually twice, the third time has to be code, even when code is slower today |
| Infrastructure as code | The lab must survive a full destroy and rebuild. If it does not, it was never reproducible, it was just alive |
| GitOps | No kubectl apply from my laptop into a tracked namespace. Drift is a bug, not a shortcut |
| Security by design | Policy, isolation, and scanning land with the feature, not in a hardening sprint that never gets scheduled |
| Observable by default | A workload with no metrics, logs, and traces is not finished, regardless of whether it responds |
| Multi tenant by design | Isolation is written and then actively attacked in tests. Retrofitting tenancy is a rewrite |
| Cost aware | Every workload has a cost per request and a cost per million tokens, or it is not measured at all |
| Failure oriented | I break the platform on purpose, on a schedule, and write the postmortem even when nobody is watching |
Failure oriented engineering. It is genuinely uncomfortable to delete your own working pod at 11pm to see what happens.
But an untested recovery path is not a recovery path. It is a belief. Volume 4 turns those beliefs into drills with recorded outcomes, including the ones that went badly.
What this project is not
Worth saying out loud, because scope honesty is part of the engineering.
- It is not a claim that every component is hyperscale production ready. It is a working platform at lab scale with a documented path upward.
- It is not model research. I do not train, fine tune, or benchmark model quality.
- It is not a GPU benchmark. See the hardware callout above.
- It is not finished. It is a build log, and volumes get corrected when the cluster proves me wrong.
A complete, honest, end to end demonstration of designing, building, securing, observing, costing, and operating an AI platform on Kubernetes, with every decision written down and every limitation named.
Next
Start with Volume 1, platform foundations. It sets up the problem properly, walks the architecture layer by layer, and ends with a running K3s cluster that the next three volumes deploy onto.
If you want to see the cluster tooling first, k3smp is the provisioning tool, and the APM project shows it carrying a real observability stack before AIForge ever touched it.