Projects · AIForge
Cost and efficiency
Unit economics for an AI platform: cost per 1M tokens, per request and per tenant, computed from telemetry. Utilization traps, quantization, continuous batching, right sizing, autoscaling and scale to zero, spot capacity, and chargeback, with every GPU input honestly labelled as modelled.
Cost and efficiency
Here is the sentence that reframes everything in this chapter.
A GPU costs the same whether it is busy or idle.
That is it. That single property, which is boring and obvious when you say it out loud, is why AI infrastructure cost behaves completely differently from web infrastructure cost, and why the discipline that works for one fails for the other.
On a web platform, cost broadly follows traffic. Quiet night, small bill. On an AI platform, you rent an accelerator by the hour, and at 4am when nobody is asking your model anything, that accelerator is drawing full price and doing nothing. Idle is not free. Idle is the default state of a badly run AI platform, and it is where most of the money goes.
I do not own a GPU. So let me be exact about what is real in this chapter and what is not, because a cost chapter with unlabelled numbers is worse than no cost chapter at all.
Real and measured: the token counters, the throughput measurements on CPU, the utilization queries, the attribution pipeline, the autoscaling behaviour, the cold start timings, and the entire method.
Modelled: every euro figure that involves a GPU. Those use a published price per GPU hour and an estimated tokens per second from vendor and community benchmarks. The formula is real. Two inputs are estimates and they are labelled inline every single time.
The number that changes on real hardware is tokens per second, and it changes by two orders of magnitude. Everything else in the method survives contact with a real invoice.
1. Where the money actually goes
Before optimising anything, get the proportions right, because engineers reliably optimise the wrong line. Here is the shape of the bill for a self hosted LLM platform serving a moderate load.
Read that bottom split carefully, because it is the whole argument.
The accelerator line is 70 to 90 percent of the bill, and it divides into work you were asked to do and time you paid for and threw away. On a typical first pass AI deployment, that split is somewhere between 10/90 and 30/70 in favour of the waste. A model server sitting at 15 percent duty cycle is not unusual, it is the normal outcome of deploying two replicas because two felt safer than one.
Which produces the single most important consequence for how you spend your time:
Moving your workload to a provider that is 20 percent cheaper per GPU hour saves you 20 percent.
Moving your duty cycle from 15 percent to 60 percent saves you 75 percent of the same line item.
Both are real. One of them is a procurement conversation and the other is an engineering problem, and the engineering problem is where the order of magnitude lives. Efficiency work beats price negotiation by roughly a factor of four on an AI platform, and it is entirely within your control.
2. The three numbers, and why you need all three
"What does the platform cost" is not a useful question. These three are.
| Number | Definition | The decision it drives | Who asks for it |
|---|---|---|---|
| Cost per 1M tokens | Infrastructure cost divided by tokens produced, per model and per hardware configuration | Build versus buy. Which model to run. Whether quantization or batching paid off | Platform engineering, and anyone comparing you to a hosted API |
| Cost per request | Cost per 1M tokens times the average token count of a request, plus a share of fixed platform cost | Product pricing. Whether a feature is viable at the volume it will get | Product and finance |
| Cost per tenant | All of a tenant's consumption attributed to that tenant, plus their share of shared infrastructure | Chargeback, quota setting, and identifying the one tenant generating 60 percent of the bill | Whoever pays, and whoever has to have the difficult conversation |
The formula, once
Two inputs. One comes from a price list, one comes from measurement. The measured one is the one that matters, because it varies by more than an order of magnitude depending on choices you make.
Here is that formula applied. Every row is labelled, and the labels are the point of the table.
| Configuration | Price per hour | Output tokens per second | Cost per 1M output tokens | Status |
|---|---|---|---|---|
| CPU lab, Qwen2.5 0.5B Instruct, 2 vCPU | €0.05 (notional, priced as a small cloud instance) | 18.4 | €0.75 | Throughput measured. Price notional |
| CPU lab, Qwen2.5 1.5B Instruct, 2 vCPU | €0.05 (notional) | 6.9 | €2.01 | Throughput measured. Price notional |
| 1x L4 24GB, 7B fp16, concurrency 1 | €0.75 (published on demand) | ~45 | €4.63 | Modelled. Throughput estimated |
| 1x L4 24GB, 7B fp16, continuous batching at high concurrency | €0.75 (published on demand) | ~900 | €0.23 | Modelled. Throughput estimated |
| 1x L4, same, but at a 12 percent daily duty cycle | €0.75 (published on demand) | ~108 effective | €1.93 | Modelled. Duty cycle from the lab pattern |
| 1x L4 spot, 60 percent discount, high concurrency | €0.30 (published spot indication) | ~900 | €0.09 | Modelled. Both inputs estimated |
| 1x A100 40GB, 7B fp16, high concurrency | €2.80 (published on demand) | ~2600 | €0.30 | Modelled. Throughput estimated |
Three findings jump out of that table, and they are the reason it is worth building even from modelled inputs.
Continuous batching is worth 20x. Same hardware, same price, same model. €4.63 versus €0.23. Nothing else in this chapter comes close to that multiple, which is why section 5 spends more words on batching than on anything else.
Idle destroys the win. The batched L4 at 12 percent duty cycle costs €1.93 per million tokens, which is eight times worse than the same setup kept busy. You can do all the batching work correctly and still throw away most of the benefit by leaving the thing running overnight.
The bigger GPU is not the more expensive one per token. The A100 costs 3.7 times more per hour and produces 2.9 times more tokens per second, so its cost per token is only 30 percent higher, and it can serve models the L4 cannot hold at all. Cost per hour is a procurement number. Cost per token is the engineering number, and they rank hardware differently.
Published API prices for small hosted models sit in roughly the €0.10 to €0.60 per 1M output token range, with no operational work, no cold starts and no on call rotation.
My modelled batched L4 is €0.23. Which means self hosting is, at best, roughly competitive on price, and only when the GPU is kept genuinely busy. At 12 percent duty cycle it is several times worse than just calling an API.
So if someone tells you they self host LLMs to save money, ask for their duty cycle. The real reasons to self host are data residency, latency control, model customisation, and not having a hard dependency on a vendor's roadmap. Those are good reasons. "It is cheaper" is usually a claim that has never been divided by a token count.
3. Building the attribution pipeline
Cost per tenant is not a report you write monthly. It is a metric, derived from telemetry you already collect in chapter one, computed continuously.
The trick that makes this work in PromQL is to expose prices as metrics. A tiny exporter reads a ConfigMap of prices and publishes them as gauges, which lets you join price against usage inside a query rather than hardcoding numbers in a dashboard where they go stale and nobody notices.
Queries 2 and 3 tell you what usage should cost according to your price book. Query 4 tells you what it actually cost, by dividing real hardware spend by real tokens produced.
The gap between those two numbers is your utilization loss, expressed in euros, continuously, on a graph.
In my lab, the price book says the 1.5B model costs €2.01 per 1M tokens. Query 4 says the realised figure is €11.40, because the pod runs all day and produces tokens for about eighteen minutes of it.
One graph, and the entire cost conversation stops being abstract. Nobody argues with a five and a half times gap.
Two things worth reading in that output.
Mean TPOT of 144 milliseconds means the model produces about 7 tokens per second per stream. That is roughly a slow but readable typing speed, which is genuinely usable for a chat interface and completely unusable for anything that needs a 2000 token answer quickly. Knowing that number is what lets you tell a product team what is possible before they design around it.
Tenant A costs six times what tenant B costs, which is a fact I did not know before building the pipeline and which is the entire justification for building it. You cannot manage a distribution you have never looked at.
4. Measuring utilization, and the four traps
Utilization is the number that drives every efficiency decision, and it is remarkably easy to measure wrongly in a way that makes you feel good. Four traps, all of which I have fallen into.
DCGM_FI_DEV_GPU_UTIL does not mean "fraction of the GPU's compute capacity in use". It means "fraction of the sampling window in which at least one kernel was executing".
A single tiny kernel that occupies 3 percent of the streaming multiprocessors, running continuously, reports 100 percent utilization. The GPU is busy and almost entirely idle at the same time.
This is why a fresh deployment often shows a comforting 90 percent GPU util while producing terrible throughput, and why people conclude they need more GPUs when they need better batching.
What to use instead: DCGM_FI_PROF_SM_ACTIVE and DCGM_FI_PROF_PIPE_TENSOR_ACTIVE from the profiling metrics, which measure actual SM and tensor core occupancy. And above all, tokens per GPU second, which is an outcome rather than a proxy.
Status in this lab: designed, no data. No GPU present.
5. The levers, in order of how much they pay
Now the actual work. Ordered by the return I would expect, which is not the order most people try them in.
5.1 Continuous batching, which is not optional
If you take one thing from this chapter, take this.
Static batching waits for a batch of requests, runs them together, and returns them together. Everyone in the batch waits for the slowest member to finish generating, so a request wanting 20 tokens waits for the one wanting 800. Throughput is mediocre and tail latency is dreadful.
Continuous batching, sometimes called iteration level scheduling, works at the granularity of a single decode step. After every token, the scheduler evicts finished sequences and admits waiting ones. The batch is never idle waiting for a straggler.
The enabling mechanism is PagedAttention. KV cache is allocated in fixed size blocks, like operating system memory pages, rather than one contiguous slab per sequence. That removes the fragmentation that would otherwise make admitting and evicting sequences mid flight impossible, and it is why you can hold far more concurrent sequences than a naive allocator would allow.
vLLM does this by default, which is the main reason it is the serving engine in Volume 2. But default on does not mean tuned:
Turn max-num-seqs up and aggregate throughput goes up, cost per token goes down, and everybody's individual request gets slower because they are sharing the accelerator with more neighbours.
Turn it down and each request is fast and your cost per token gets worse.
There is no correct value. There is only the value that satisfies your latency SLO at your expected concurrency, which means you cannot tune this until the reliability chapter has defined the SLO. That dependency is real and it is why these two chapters sit next to each other.
5.2 Quantization
Weights are normally stored as 16 bit floats. Quantization stores them in fewer bits, which shrinks memory and, because decode is memory bandwidth bound, usually speeds things up as a side effect.
| Precision | 7B model weights | Typical throughput effect | Quality effect | When I would use it |
|---|---|---|---|---|
| fp16 or bf16 | ~14 GB | Baseline | Baseline | Default. Anything where quality is the product |
| fp8 or int8 | ~7 GB | 1.3x to 1.8x faster | Very small, often not measurable on general tasks | The default I would actually reach for. Nearly free win on hardware with fp8 support |
| int4, AWQ or GPTQ | ~3.5 GB | 2x to 3x faster, and fits far smaller hardware | Noticeable. Worst on long context, multi step reasoning and code | When memory is the binding constraint, and after you have measured quality on your own task |
The second order effect is bigger than the throughput number and people miss it. Smaller weights leave more room for KV cache. A 7B model at int4 frees roughly 10 GB on a 24 GB card, and that space becomes concurrent sequences. More concurrency means better batching means better cost per token. The compounding is where most of the benefit lives.
Published quantization benchmarks are averages over general benchmarks. Your task is not a general benchmark.
I have seen int4 be indistinguishable from fp16 for short factual RAG answers and clearly worse for anything requiring multi step reasoning. Same model, same quantization, opposite conclusions, because the tasks stress different capabilities.
So build the evaluation before you build the optimisation. A held out set of 100 real queries with expected answers, run against both precisions, scored consistently. Cost savings that degrade your product are not savings, they are a price cut you did not agree to. This is the one place where the cost chapter needs the quality tooling that Volume 4 explicitly does not build, and I would not ship a quantization change without it.
5.3 Right sizing, which is unglamorous and free
Requests and limits on an AI workload are more tractable than on a normal service, because the memory profile is largely deterministic: weights plus a pre allocated KV cache plus a modest runtime overhead. You can compute it rather than guess.
This asymmetry is deliberate and it is one of the more useful pieces of Kubernetes judgement I have.
Memory is incompressible. A process that needs memory and cannot get it does not slow down, it dies. So a memory limit is a blast radius control: it means one leaking pod cannot take out every other pod on the node.
CPU is compressible. A process that wants more CPU than is available simply waits. A CPU limit does not protect anybody, it throttles the container even when the node is completely idle, via the CFS quota. For a decode loop that runs continuously, throttling is directly visible as latency, and I have watched a p99 double because of a CPU limit that was never needed.
Request CPU so the scheduler places you correctly. Do not limit it. Limit memory so a leak is contained.
5.4 Autoscaling, and the honest truth about scale to zero
The Kubernetes HorizontalPodAutoscaler scales on CPU by default, and CPU is close to useless as a scaling signal for an inference server. A model server waiting on memory bandwidth during decode can show modest CPU while being completely saturated in the only sense that matters.
Scale on the queue instead. vllm:num_requests_waiting is the correct signal, because it is the direct measurement of demand exceeding capacity.
KEDA rather than a plain HPA for two reasons: it reads Prometheus directly with no metrics adapter to maintain, and it can scale to zero, which an HPA cannot.
Fifty two seconds for the first request after scale to zero. Then 0.9 seconds. That is the whole tradeoff, measured, in one terminal session.
Fifty two seconds is not a latency spike, it is a timeout. Most HTTP clients give up long before that, so from the user's point of view the platform is down, not slow.
So my rule, which I would defend anywhere:
Scale to zero: yes for development and staging models, for tenants with an agreed batch or best effort SLO, and for anything a human is not waiting on. Scale to zero: no for anything with an interactive latency SLO. There, the minimum is one replica and you accept paying for idle as the price of not being down.
If you genuinely need both, the answer is not a smarter autoscaler. It is a small always warm model handling the first token while a larger one loads, or a queue that gives the user an honest "starting up, one moment" instead of a hanging connection. Cold start is a product design problem wearing an infrastructure costume.
5.5 Model selection, the biggest lever nobody pulls
A 0.5B model is roughly fourteen times cheaper per token than a 7B on the same hardware, in my measurements. If it answers the question adequately, that is a 93 percent saving that no amount of infrastructure tuning can match.
The catch is that "adequately" requires evaluation, and evaluation is the thing teams skip. So the pragmatic version is routing with a defined escalation path, which LiteLLM supports directly:
5.6 Spot and preemptible capacity
Spot instances are 60 to 80 percent cheaper and can be taken away with 30 seconds to two minutes of notice. For stateless inference replicas that is a genuinely good trade, and for anything stateful it is a trap.
| Workload | Spot? | Why |
|---|---|---|
| Stateless inference replicas above the first one | Yes | Losing one replica degrades capacity, not correctness. Keep at least one on demand as the floor |
| Batch embedding and document ingestion | Yes, ideal | Nobody is waiting. Restartable by design. This is what spot was invented for |
| Fine tuning and training | Yes, with checkpointing | Only if you checkpoint frequently. Without checkpoints you are gambling a multi hour job on a 60 percent discount |
| PostgreSQL, Qdrant, the control plane | No | Stateful, and the recovery cost of an eviction dwarfs the saving |
| The last remaining inference replica | No | An eviction with no capacity left is an outage, and eviction is a certainty rather than a risk |
To use spot properly you need three things wired: a PodDisruptionBudget so voluntary evictions cannot take the last replica, a handler for the termination signal that drains in flight requests, and topology spread so all your replicas are not on the same doomed node. All three are built in the reliability chapter, which is not a coincidence. Spot capacity is a reliability engineering problem that happens to save money.
Status in this lab: designed only. Multipass VMs on my laptop have no spot market.
6. Chargeback and showback
You have the numbers. Now the harder half: making them change behaviour.
Showback shows each tenant what they consumed and what it cost, with no money moving. Chargeback actually bills them.
Start with showback, always, and the reason is not political caution. It is that your attribution is wrong at first and you need to find out before anyone's budget depends on it. My first per tenant report attributed all embedding cost to the platform rather than the tenant whose documents were being embedded, because ingestion ran under the platform service account. Tenant A looked 30 percent cheaper than it was. Finding that in a showback report is a correction. Finding it in an invoice is a credibility event you do not recover from quickly.
| Element | How I attribute it | The honest difficulty |
|---|---|---|
| Inference tokens | Directly, from gateway counters tagged with the tenant | Easy and accurate. This is the bulk of the cost, which is fortunate |
| Embedding and ingestion | By the tenant owning the target collection, not the invoking service account | Requires the ingestion job to carry a tenant label. Got this wrong first time |
| Vector storage | By collection size, sampled hourly | Fair, but it charges for storage a tenant may consider a one off cost. Needs explaining |
| Idle model capacity | Split across tenants using that model, weighted by their token share | The genuinely contentious one. A small tenant sharing a model with a large one pays a share of idle they cannot influence |
| Shared platform: control plane, gateway, observability | A flat platform fee per tenant, stated in advance | Crude and defensible. Precise attribution of a shared control plane costs more to build than it saves |
That fourth row is the one that generates arguments, and it deserves a real answer rather than a formula.
A tenant sending 200 requests a day shares a model server with a tenant sending 20,000. The server runs 24 hours regardless. If you split idle by token share, the small tenant pays almost nothing, which is generous to them and unfair to the large tenant who is effectively subsidising the platform's availability. If you split idle evenly, the small tenant's bill is dominated by capacity they never asked for and they will, correctly, ask why.
My answer: idle capacity is a platform cost, not a tenant cost, unless a tenant demanded dedicated capacity. Then it is theirs entirely, at cost, and stated in the agreement.
That decision makes the platform's own efficiency the platform team's problem, which is exactly where the incentive belongs. If I make tenants pay for my idle, I have removed my own incentive to fix it.
7. What went wrong, and what I watch now
Two replicas by reflex, for eleven weeks. I deployed the 1.5B model with replicas: 2 because two feels like production and one feels like a toy. Measured duty cycle across both: under 4 percent. One replica would have handled every request the lab ever received with room to spare. What I watch now: replica count is justified in the manifest with a comment naming the reason, and the only two acceptable reasons are a measured capacity need or an availability requirement with an SLO behind it.
A CPU limit that halved my throughput. I set cpu: "2000m" as a limit as well as a request, because symmetry looked tidy. Tokens per second dropped by roughly 45 percent and I spent an evening convinced the model was the problem. It was CFS throttling: container_cpu_cfs_throttled_seconds_total was climbing steadily and I was not looking at it. What I watch now: that metric is on the model serving dashboard, and CPU limits are off by default on inference workloads.
Prometheus retention was quietly the third biggest storage line. Fifteen second scrape interval, thirty day retention, a lot of series, on a local-path volume on a laptop. It was consuming more disk than every model weight file combined. What I watch now: retention is 7 days in the lab, scrape interval is 30 seconds for everything that is not latency sensitive, and prometheus_tsdb_storage_blocks_bytes is on a dashboard. Observability cost is a real line item and it belongs in the model.
My first cost per token number was wrong by a factor of five and I nearly published it. I divided price by theoretical throughput from a vendor benchmark instead of by realised throughput from my own counters. That is not a cost model, it is a best case scenario with a euro sign on it. What I watch now: every cost figure has a basis field saying whether it is measured, modelled or notional, and the dashboard shows realised alongside theoretical so the gap is impossible to miss.
Scale to zero broke a tenant's demo. I enabled it on what I believed was a development deployment. It was being used for a live demonstration, and the first request took 52 seconds and timed out in front of an audience. Entirely my fault and entirely preventable. What I watch now: scale to zero requires an explicit label on the deployment, aiforge.io/scale-to-zero: allowed, set by whoever owns the workload rather than by me.
Real GPU cost per token. Every GPU euro in this chapter is modelled. The formula is right, the price is published, the throughput is an estimate from other people's benchmarks. On real hardware the throughput number will move and every derived figure moves with it.
Whether quantization is safe for your task. I have no evaluation harness. The advice in 5.2 is to measure before you ship, and I am giving that advice from a position of not having measured it myself.
How any of this behaves at real scale. My peak measured load is a handful of concurrent requests. Continuous batching, autoscaling and spot handling all have failure modes that only appear at volume, and I have not seen them.
Multi GPU and multi node inference economics. Tensor parallelism, pipeline parallelism and the interconnect costs that dominate them are entirely outside what this lab can explore.
Next
Finish the volume with reliability and failure drills. Several threads here end there on purpose: max_num_seqs cannot be tuned without a latency SLO, spot capacity needs a PodDisruptionBudget and a drain handler, and the cold start penalty measured above is exactly the number that decides whether graceful degradation is worth building.
For the serving parameters referenced throughout, see serving LLMs with vLLM and LiteLLM, the model gateway. For the token counters every number here depends on, observability and LLM telemetry.