Projects · AIForge
KServe and the model lifecycle
The InferenceService CRD, serving runtimes, autoscaling and scale to zero, and the honest tradeoff against a plain Deployment. Then MLflow as the registry that answers which version is in production and where its artifacts live.
Two tools, one question: who owns the fact that a model is running, and who owns the fact that it is version 4.
Right now the answer to both is "a YAML file I edited by hand", and that is fine for exactly one model and one person. KServe takes over the first question, MLflow takes over the second.
I am also going to argue, at the end of section 6, that KServe is not obviously worth it for a small platform. Adding a tool and then defending it is easy. Adding a tool and staying honest about its cost is the part worth reading.
KServe and the model lifecycle
The previous chapter ended with a model server running in aiforge-system. It works. It answers requests. And it has four problems that no amount of tuning inside the pod will fix.
It runs forever, whether anybody is using it or not. It has no idea what version it is serving beyond a string in an argument. Deploying a second model means copying 90 lines of YAML and remembering to change six of them. And the moment a second person is involved, "which model is in production" becomes a question answered by reading manifests in a git repository and hoping the cluster agrees with them.
Those are lifecycle problems, and they are what this chapter fixes.
1. Where a plain Deployment stops being enough
Let me be fair to the Deployment first, because it gets you further than tool vendors like to admit.
A Deployment plus a Service gives you scheduling, rolling updates, health probes, restart on failure, and horizontal scaling through an HPA. For a stateless HTTP service that is genuinely most of what serving needs. I have shipped worse things to production and slept fine.
Here is where it stops:
| Need | Plain Deployment | Why it matters for models specifically |
|---|---|---|
| Scale to zero when idle | Not possible. An HPA will not go below one replica | An idle GPU is the most expensive idle thing in the building. A model nobody is querying should cost nothing |
| Fetch model artifacts from object storage before start | You write the init container yourself, per model | Every model needs it, and everyone writes a slightly different version of the same 20 lines |
| Scale on concurrency rather than CPU | Awkward. HPA thinks in CPU and memory by default | An inference pod at 100 percent CPU may be perfectly healthy. Queue depth is the real signal |
| One consistent shape for many model types | Every model is a bespoke manifest | A platform that generates YAML needs a small, stable, validated schema to generate |
| Canary a new model version by traffic percentage | Two deployments and something in front of them | Model quality regressions are subtle. Ten percent of traffic is how you find them safely |
| Reuse one runtime definition across many models | Copy and paste | Engine flags, images and probe tuning should be defined once by the platform team, not per tenant |
A Deployment describes a process you want running. An InferenceService describes a model you want available.
Those sound like the same sentence. They are not. "Available" permits zero replicas right now, a cold start on the next request, a canary split between two versions, and an artifact fetched from a registry at start time. The abstraction is about intent, and intent is what a platform generates from a developer request.
Which is exactly the shape Volume 3 needs, because a control plane that has to generate a bespoke Deployment per model is a control plane that has to know about engine flags. It should not have to.
2. What KServe actually is
KServe is a set of controllers and CRDs for model serving on Kubernetes. The important object is InferenceService, and its job is to take a short declaration of intent and expand it into the pods, services, routes and autoscaling policy that intent implies.
Three pieces of vocabulary carry most of the weight.
InferenceService is the user facing object. It names a model, points at artifacts, optionally names a runtime, and sets resources and scaling bounds. It is short on purpose.
ServingRuntime and ClusterServingRuntime describe how to run a class of model: which image, which arguments, which model formats it supports, which ports it listens on. Namespaced or cluster wide. This is the platform team's object, and tenants never write one.
Storage initializer is an init container KServe injects when storageUri is set. It fetches artifacts from S3 compatible storage, GCS, Azure, an HTTP URL or a PVC, drops them in /mnt/models, and exits. The model container starts with its weights already on local disk and no credentials of its own.
This is the single biggest thing to understand before installing KServe, and it is where people get surprised by the size of the dependency.
Serverless mode builds on Knative Serving. You get scale to zero, concurrency based autoscaling, revisions and traffic splitting for canaries. You also get Knative's controllers, an activator, an autoscaler, and a networking layer, all of which are real components you now operate.
RawDeployment mode skips Knative entirely. KServe creates a plain Deployment, Service, Ingress and HPA. Far less to install and understand. You keep the CRD, the runtimes and the storage initializer. You lose scale to zero and concurrency based scaling.
The annotation is one line: serving.kserve.io/deploymentMode: RawDeployment.
The interesting part is that this choice is not really about KServe. It is a choice about whether scale to zero is worth an extra control plane. With a GPU that costs money by the hour, the answer is usually yes. In my CPU lab, section 5 measures what it actually buys, and the answer is less flattering.
3. Serving runtimes, where the engine knowledge lives
KServe ships built in runtimes for the classic model formats: scikit-learn, XGBoost, PyTorch, Triton, and a Hugging Face runtime that can use vLLM as its backend. For AIForge I define my own, because I want the exact engine flags from the previous chapter under platform control rather than defaults.
Every hard won detail from the previous chapter lives here once. The KV cache sizing. The concurrency cap. The thread count. The memory limit that makes a runaway cache kill one pod and nothing else.
A tenant deploying a model does not get to override those and does not need to know they exist. When I learn something new, for example that max-num-seqs should be 6 rather than 8 on this hardware, I change one object and every model deployed after it inherits the fix.
4. The InferenceService itself
Now the tenant facing object. Compare its length to the 90 line Deployment from the previous chapter.
Twenty five meaningful lines, and none of them mention an image, an engine flag, a probe, a Service or a port. That is the abstraction doing its job: the tenant declares what, the runtime owns how.
The pod says 2/2, not 1/1. The second container is Knative's queue proxy. It sits in front of the model container, counts in flight requests, reports concurrency to the autoscaler, and enforces containerConcurrency. It is also a real hop on the request path, and therefore a real thing that can be misconfigured.
The revision is named 00001. Knative creates an immutable revision per configuration change. That is what makes traffic splitting between two model versions a percentage in a spec rather than a load balancer project. It is also why a careless edit loop can leave you with a graveyard of old revisions.
5. Autoscaling and scale to zero, measured
This is the headline feature, so it deserves numbers rather than adjectives.
The activator holding the request is the piece that makes this usable. The client does not get a 503, it gets a slow response. Whether that is acceptable depends entirely on how slow, so I measured it.
Here is the breakdown from the pod events and logs, with a warm image and a warm object store:
| Stage | Time |
|---|---|
| Autoscaler reacts, pod scheduled | about 2s |
| Storage initializer pulls 3.09 GiB of weights | about 51s |
| Engine starts, loads weights, profiles memory | about 39s |
| Health check passes, request forwarded and answered | about 6s |
Now the uncomfortable question: who is willing to wait 98 seconds for a first answer? Almost no interactive user. A batch job, absolutely. An internal tool used twice a day, probably.
Scale to zero is not free. It converts a money cost into a latency cost. That trade is excellent for an expensive idle GPU and terrible for a chat interface with impatient humans in front of it.
The mitigations, in the order I would reach for them:
- Cache the artifacts on a node local PVC rather than pulling from object storage every cold start. Removes about 51 seconds here, at the cost of node affinity.
- Keep
minReplicas: 1for interactive models and reserve scale to zero for batch and rarely used models. This is a per model policy decision, and it belongs in the control plane's model catalogue rather than in a human's memory. - Raise the retention period so brief gaps in traffic do not trigger a scale down that a user pays for two minutes later.
- Pre pull images with a DaemonSet so a cold node is not also a cold registry pull, which adds minutes rather than seconds.
Scale to zero gets the attention, but the more broadly useful feature is scaling on concurrency instead of CPU.
An inference pod pinned at 100 percent CPU may be perfectly happy: it is decoding, that is what decoding looks like. Meanwhile a pod at 40 percent CPU with twelve requests queued is in trouble. CPU based autoscaling gets both of those backwards.
Setting containerConcurrency: 8 to match --max-num-seqs=8 means the autoscaler and the engine share one definition of full. When the queue proxy sees more than eight in flight per pod, it adds a pod. That is the correct signal, and it is the same signal as vllm:num_requests_waiting from the previous chapter, observed from outside.
6. The honest tradeoff
Time to answer the question rather than dodge it.
| Dimension | Raw Deployment | KServe, RawDeployment mode | KServe, Serverless mode |
|---|---|---|---|
| Components to install and operate | None beyond Kubernetes | KServe controller, CRDs, cert manager | All of that plus Knative Serving: controller, autoscaler, activator, networking layer |
| Lines of YAML per model | About 90, mostly copied | About 25 | About 25 |
| Scale to zero | No | No | Yes, at the cost of a cold start |
| Concurrency based autoscaling | No, CPU and memory only | HPA metrics, so still awkward | Yes, and it is the right signal |
| Artifact fetching | Write your own init container | Built in storage initializer | Built in storage initializer |
| Canary by traffic percentage | Roll your own | Limited | Yes, through revisions |
| Extra request hop | None | None | Queue proxy, and the activator while scaled to zero |
| Debuggability when it breaks | Excellent. It is a pod | Good. One controller to reason about | Hardest. A failure can live in KServe, Knative, or networking, and the events are spread across three places |
| Fit for a platform that generates manifests | Poor. The generator must know engine flags | Good. Small validated schema | Good, same schema |
If you run one or two models and a human deploys them, use a plain Deployment. KServe will cost you more in operational surface than it returns. I mean this. Reaching for a CRD to manage two pods is how platforms become unmaintainable.
If you run many models on expensive hardware, with a control plane generating deployments and idle GPUs burning money, KServe in Serverless mode pays for itself quickly.
AIForge is the second case by design, because Volume 3 is a control plane whose entire job is turning a developer request into a running model. A small validated CRD is a far better generation target than a 90 line Deployment full of engine specific flags.
So in the lab I keep both paths alive on purpose: the raw Deployment from the previous chapter, and the InferenceService here, serving the same model with the same public name behind the gateway. Being able to demonstrate both, and to say precisely what the second one costs, is worth more to me than picking a side. A platform engineer who cannot articulate the cost of their abstraction is just someone who installed a tool.
7. MLflow, and what a registry is actually for
Switch topics. KServe answers "is it running". Nothing so far answers "what is it".
MLflow is usually introduced as an experiment tracker for data scientists, which is true and which makes platform engineers assume it is not their problem. The part that is very much their problem is the Model Registry.
Strip it to its structure:
| Concept | What it is | Why the platform cares |
|---|---|---|
| Experiment and run | One execution, with parameters, metrics and tags | The provenance trail. Where a model came from and how it scored |
| Artifact | Files produced by a run: weights, tokenizer, config, evaluation reports | The bytes that actually get deployed. This is what storageUri points at |
| Registered model | A named entity, for example qwen2.5-1.5b-instruct | The stable public name a tenant asks for. Never changes |
| Model version | An immutable numbered version under that name | The thing that is actually deployed. Version 4 is version 4 forever |
| Alias | A movable pointer such as production or champion | The indirection that lets a promotion be a metadata change rather than a redeploy |
Older MLflow used fixed stages: None, Staging, Production, Archived. They were built in and they were a straitjacket. You got exactly those four words, and any team whose lifecycle did not match had to lie about which stage something was in.
Modern MLflow deprecates stages in favour of aliases plus tags. An alias is a name you choose, pointing at exactly one version, and you can have as many as you like: production, canary, tenant-a-pinned, last-known-good.
That last one matters more than it looks. A rollback is now moving a pointer, not finding out which version was running before the incident by reading a chat log at two in the morning.
MLflow needs two backing stores, and confusing them is a classic first deployment mistake:
| Store | Holds | In the lab |
|---|---|---|
| Backend store | Metadata: runs, params, metrics, versions, aliases. Small rows, many queries | PostgreSQL in aiforge-system, the same instance the control plane uses, different database |
| Artifact store | The actual files. Gigabytes per version | MinIO, S3 compatible, so storageUri works unchanged against real S3 later |
Every MLflow quickstart uses SQLite and a local directory. It works instantly, which is exactly the problem.
Then the pod restarts on a node without that volume and the entire registry is gone. Not corrupted. Gone. Every version, every alias, every pointer to every artifact.
The artifacts usually survive, sitting in object storage, and that is somehow worse: you have 40 GB of model files and no record of which one is version 3. I have watched a team spend a day matching checksums to reconstruct that mapping.
Postgres from the first day. It is twenty extra lines of manifest.
8. Registering an LLM, honestly
Here is a nuance most write ups skip. MLflow was designed around models you train, where the run that produces the artifacts and the registered version are naturally connected. I am not training anything. I am serving open weights models that someone else trained.
So what does registration mean here? It means the registry becomes the place that records which exact bytes we serve, and what we know about them. That is still enormously valuable, arguably more so, because there is no training pipeline providing provenance for free.
source_revision is the commit hash of the model repository. "We run Qwen2.5 1.5B" is not a reproducible statement. A hash is. Model repositories get updated, quietly, and the file you downloaded in March is not always the file you download in August.
context_length because it is a serving constraint that lives in the model's metadata but is enforced by the engine. When someone asks why a 6,000 token prompt was rejected, this is the answer.
eval_faithfulness because it is our measurement on our golden set, not a benchmark from a model card. The evaluation harness that produces it is built in the RAG chapter.
9. Connecting the registry to the serving layer
Two systems, one join. And the join has to be one directional or you build a distributed lock by accident.
MLflow is consulted at deploy time and never on the request path. Not for routing, not for lookups, not for a version check.
The temptation to do otherwise is real, because a live lookup means the running system is always consistent with the registry. It also means every user request now depends on the availability of your metadata database. A registry outage becomes a serving outage, and you have coupled the least critical system to the most critical one.
The version that is serving is recorded where it is being served: in the annotations on the InferenceService. That is queryable, it is local, and it works when MLflow is down.
The registry says production is version 4. Tenant B is running version 3. Those two facts do not contradict each other, because promotion in the registry does not push anything anywhere.
That is intentional. Tenant B may be pinned during a validation window, or may have opted out of a rollout, or may have hit a regression. What matters is that the discrepancy is visible in one command rather than being a surprise.
A registry that force pushes to production the moment somebody moves an alias is not a registry, it is an unreviewed deployment pipeline with extra steps.
10. What went wrong, and what I watch
| Symptom | Actual cause | How I found it |
|---|---|---|
InferenceService stuck with Ready false, no pod ever created | No ServingRuntime matched the requested modelFormat | kubectl describe isvc. The condition message names the format it could not resolve, and it is the only place that says so |
| Storage initializer fails with a 403 against MinIO | The S3 credentials secret was not annotated for KServe, so it was never attached to the init container | Init container logs. The model container never starts, so its logs are empty and misleading |
| Pod scales to zero in the middle of a long generation | Knative counted the request as finished. Long streaming responses plus a short retention window | Client saw a truncated stream. Fixed by raising timeout and the retention period |
| Autoscaler adds pods that immediately go idle | containerConcurrency left at 0, meaning unlimited, so the concurrency signal was meaningless | Pod count oscillating while vllm:num_requests_running stayed low on every replica |
| MLflow UI empty after a pod restart, artifacts still in MinIO | SQLite backend store on ephemeral storage | Painful. Section 7 exists because of this |
| Two tenants served different weights under the same model name and nobody noticed for a week | Exactly the version drift in section 9, before the annotation existed | A tenant reported different answers to an identical prompt. Now it is one kubectl command |
In the previous chapter the failures were about resources: memory, time, throughput. Every one of them had a number attached.
Here they are about agreement. Two systems that each believe something reasonable and different: the runtime that did not match, the credential that was not attached, the autoscaler and the engine disagreeing about full, the registry and the cluster disagreeing about the version.
Lifecycle bugs are consistency bugs. Which is why the useful diagnostic is almost always a command that prints what two systems each believe, side by side.
What I would watch
InferenceServiceobjects with Ready false for more than five minutes. This is the one alert that catches most of the table above.- Cold start duration as a histogram, per model. It drifts upward as models grow and nobody notices until a user complains.
- Any deployed model version that does not match a registry alias. Drift is fine, silent drift is not.
Next
The cluster now has models with a lifecycle. They can be deployed from a declaration, scaled to zero, versioned in a registry, and traced from a running pod back to the exact bytes it serves.
What it still does not have is a front door. Right now a client needs to know a Kubernetes service name, there is no authentication worth the word, no budget, no rate limit, and no answer to what happens when the pod behind that name is gone.
LiteLLM the model gateway puts one OpenAI-compatible endpoint in front of everything built so far, and makes both of this volume's serving paths, the raw Deployment and the InferenceService, look like the same model to a tenant.