Observability and LLM telemetry

A tenant sends me a message that says: "the chatbot feels slow today."

That is the entire report. No timestamp, no request ID, no idea whether slow means two seconds or twenty. This is not a hypothetical, it is the normal shape of an AI platform complaint, because the user experience of a language model is inherently fuzzy. Nobody says "your p95 time to first token regressed by 340 milliseconds." They say it feels slow.

Everything in this chapter exists so that I can turn that sentence into an answer in under five minutes.

The definition I actually use

Monitoring tells you whether the system is healthy against questions you thought of in advance. Observability is whether you can answer a question you did not think of in advance, without shipping new code.

That difference is not academic. "Is vLLM up" is monitoring. "Why were tenant B's requests four times slower than tenant A's between 14:10 and 14:25, and was it retrieval or generation" is observability. The second question is the one you get asked.

1. The four signal types, and the fifth one nobody warned me about

Every observability conversation drowns in tool names before anyone establishes what the tools are for. So here is the map first. Each signal type answers a different kind of question, and no signal type can substitute for another.

What it is: a number, sampled at a regular interval, with a small set of labels. vllm:num_requests_waiting at 14:05 was 7 in namespace aiforge-tenant-a.

The question it answers: how much, how fast, how often, right now and over time.

Why you cannot live without it: metrics are cheap enough to keep for every component forever and to alert on. A single time series costs roughly 1 to 2 bytes per sample on disk after compression, so a year of one metric at 15 second resolution is a couple of megabytes. That price point is what makes "graph everything, alert on the important bits" affordable.

What it cannot do: tell you about one specific request. Metrics are aggregates. The moment you try to make them per request by adding a request_id label, you have destroyed the thing that made them cheap.

Tool: Prometheus.

This is the paragraph I would tattoo on a platform engineer

An AI platform can be green on all four classic signals and still be broken, and it is the only kind of platform I have worked on where that is routine.

Latency normal. Error rate zero. CPU fine. Pods running. And the answers are garbage because someone re indexed a knowledge base with the wrong chunk size last Tuesday and top similarity scores quietly fell from 0.78 to 0.34.

There is no infrastructure metric on Earth that catches that. That is why Langfuse is not a nice to have in this stack, it is the difference between operating an AI platform and operating a web service that happens to contain a model.

2. Prior art: what the APM project taught me and what I changed

I did not start from zero. The APM project built a single node K3s cluster with k3smp and deployed SigNoz, then used the OpenTelemetry Operator to auto instrument workloads with pod annotations instead of code changes.

Three things carried over more or less unchanged.

The collector topology. A DaemonSet agent on every node for host metrics, container logs and kubelet signals, plus a gateway Deployment that does enrichment and export. That split exists because node local collection has to be node local, while anything expensive or stateful belongs somewhere you can scale independently.

The k8sattributes habit. Every single signal gets stamped with namespace, pod, workload and node before it leaves the cluster. In the APM project that was how the SigNoz infrastructure views populated automatically. Here it is how I correlate a Loki log line with a Prometheus series with an OTel span, because they all carry the same k8s.namespace.name and k8s.pod.name.

Instrumentation is a platform responsibility. Auto instrumentation by annotation, not by pull request. If enabling tracing requires a tenant to modify their code, tracing coverage will be about 20 percent forever.

Two things are genuinely different, and one of them is a real tradeoff I want to name honestly.

Dimension	APM project, SigNoz	AIForge, composed stack	The honest tradeoff
Backend	One product, ClickHouse underneath, all three signals in one UI	Prometheus, Loki, Grafana, plus Langfuse. Four things to run	SigNoz was faster to stand up and correlated signals for free. The composed stack gives me PromQL, the Prometheus alerting model, and the ecosystem of exporters, at the price of wiring correlation myself
Metrics query language	ClickHouse SQL and a builder UI	PromQL, which is what every Kubernetes exporter, HPA adapter and KEDA scaler already speaks	PromQL is the lingua franca of Kubernetes autoscaling. That alone decided it for a platform that needs to scale on custom metrics
LLM awareness	None. A generation is an HTTP span with a duration	Langfuse understands prompts, completions, tokens, models and cost as first class objects	No tradeoff. This is a straight capability gap that Langfuse fills
Resource cost in the lab	ClickHouse alone wanted more RAM than my whole observability budget	Prometheus at roughly 700 MiB, Loki at 300 MiB, Grafana at 150 MiB, Langfuse plus its Postgres at roughly 600 MiB	This mattered a lot. On a CPU only lab, the observability stack competes with the model for memory

The resource row is not a footnote, it is a design constraint

On a laptop hosted k3smp cluster, my observability stack costs roughly 1.8 GiB of RAM. The model server serving Qwen2.5 1.5B Instruct on CPU wants about 4 GiB.

So observability is consuming close to a third of what the workload consumes. That ratio is absurd in production and completely normal in a lab, and it forced two decisions I would not otherwise have made: Prometheus retention is 7 days rather than 30, and trace sampling is head sampling at 20 percent with a tail rule that keeps 100 percent of errors.

Both are documented deviations, not accidents. A constraint you wrote down is engineering. A constraint you forgot about is a future incident.

3. The stack, drawn once

Preparing diagram

Two details in that diagram are deliberate and worth calling out.

Prometheus scrapes directly, it does not go through the collector. The OpenTelemetry Collector can absolutely receive and forward metrics, and in the APM project it did. Here I let Prometheus pull /metrics itself, because the pull model gives me a free liveness signal: if a target stops answering, up goes to 0 and I know the component is gone. Push through a collector loses that. It is a small thing that has caught real outages.

Langfuse gets data from two directions. The RAG API talks to it directly through the SDK, because that is how you attach a prompt, a completion and a retrieval score to a generation. The collector also exports OTLP to it, which picks up spans from components that are only OTel instrumented. Belt and braces, and Langfuse deduplicates on trace ID.

4. Prometheus, and what it is actually doing

Almost everyone using Prometheus has never thought about how it works, and then is surprised by cardinality. So, briefly, under the hood.

Prometheus pulls. On a schedule, usually every 15 or 30 seconds, it makes an HTTP GET to /metrics on every target it has discovered, and parses a plain text exposition format. Targets in Kubernetes come from kubernetes_sd_configs, which watches the API server, or from ServiceMonitor custom resources if you run the Prometheus Operator, which is what I do.

Each unique combination of metric name and label values is one time series. This is the single most important sentence in this section.

That is one series. Change any label value and it is a different series. The number of series a metric produces is the product of the cardinality of its labels, which is why an innocent looking decision can multiply your memory footprint by a thousand.

Samples land in an in memory head block, are simultaneously appended to a write ahead log for crash recovery, and every two hours the head is compacted to an immutable block on disk. Compression is delta of delta on timestamps and XOR on float values, which is why you get down to roughly 1.3 bytes per sample. Prometheus is genuinely, remarkably efficient at the thing it is designed for.

The cardinality mistake I nearly shipped

My first pass at tenant aware metrics in the AIForge control plane had these labels on the request counter: tenant_id, model, api_key_id, endpoint, status.

Do the arithmetic. 50 tenants, 6 models, and here is the killer, one API key label per key, of which a tenant may hold dozens. Call it 400 keys. 8 endpoints, 6 status codes. That is 50 times 6 times 400 times 8 times 6, which is 576,000 series from a single counter. Prometheus would have needed several gigabytes of RAM for the head block alone, on a cluster where I had budgeted 700 MiB for the whole thing.

api_key_id went in the logs and in Langfuse, where per request identifiers belong. The metric kept tenant_id, model, endpoint and status, which is 14,400 series worst case and entirely fine.

The rule: if a label can take an unbounded or per user number of values, it is not a metric label. It is a log field or a trace attribute.

Here is what verifying a scrape target actually looks like. I do this every single time I add a component, because a target that was never scraped produces a dashboard full of No data that looks identical to a broken workload.

Confirming vLLM is exposing metrics and Prometheus is scraping it

$▊

Look at that last line: dcgm-exporter is 0, and it stays 0

up equals 0 for the Prometheus job that scrapes the NVIDIA DCGM exporter, because there is no GPU in this lab and therefore no exporter pod.

I left the ServiceMonitor and the dashboards in place on purpose, with an explicit alert inhibition so it does not page me. Two reasons. First, the day a GPU arrives, the panels light up with zero work. Second, and more importantly, the queries are written down and reviewable now, which means the design decision about what to measure on a GPU is made while I have time to think, not during the rush of new hardware.

Look at the TTFT histogram above though, because it is the honest counterpoint. Those buckets are real measurements from CPU inference. 341 requests under one second, 1877 under two and a half. Latency, throughput and saturation are fully observable without an accelerator.

5. PromQL, and the queries that actually earn their keep

PromQL confuses people because it looks like SQL and behaves like nothing else. Two ideas unlock most of it.

An instant vector is one sample per series at one moment. A range vector is a window of samples per series, written with brackets, like [5m]. Most functions consume one and produce the other. You cannot graph a range vector directly, which is the error message everybody hits on day one.

Counters only go up, so you almost never graph them raw. You wrap them in rate(), which gives per second average over the window and correctly handles counter resets when a pod restarts. rate() on a counter is probably 60 percent of all useful PromQL.

Here are the queries that live on my dashboards. These are the real ones, not illustrations.

And here are the GPU queries, written now, returning nothing today.

A trap in the availability query that cost me an afternoon

Look at clamp_min(..., 0.001) in the availability SLI.

Without it, when traffic is zero the denominator is zero, the division produces NaN, and my error rate panel shows a gap. Which is defensible. But my alert on that expression also evaluated to nothing, which meant that during a genuine outage where the gateway stopped accepting connections entirely, traffic went to zero, the error ratio became NaN, and the alert resolved itself.

The outage silenced the alarm. That is the worst failure mode an alert can have, and it is entirely invisible until it happens to you.

Every ratio alert needs a companion absolute alert on the denominator. Mine is a rule that fires when request rate drops below 20 percent of its one hour average, which catches "everything is broken and quiet" as reliably as the error ratio catches "everything is broken and loud".

Recording rules, because dashboards should not do arithmetic

A histogram_quantile over a rate over five minutes across dozens of series is not free. Put six of those on a dashboard that eight people have open during an incident and Prometheus starts spending real CPU on rendering rather than on ingesting.

Recording rules evaluate an expression on a schedule and write the result back as a new series. The dashboard then reads a single pre computed series.

Note the naming convention: level:metric:operations. It is the Prometheus community convention and it means that six months later I can tell at a glance that aiforge:ttft_seconds:p95 is a recorded aggregate and not something an exporter produced. Small discipline, large payoff.

Alerts, and the rule I apply to every single one

My one non negotiable rule about alerts

Every alert must have a runbook annotation, and every alert must be actionable by a human at 3am.

If I cannot write a runbook for an alert, the alert is not ready. Nine times out of ten the reason I cannot write the runbook is that the alert does not actually describe a user impacting condition, it describes a number I found interesting.

Alerts that are interesting rather than actionable are how you train a team to ignore alerts. And a team that ignores alerts is strictly worse off than a team with no alerts at all, because it has all the noise and none of the trust.

Check the rules before you ship them. promtool ships inside the Prometheus image, and it catches the syntax errors that would otherwise silently disable an entire rule group.

Validating rules, then confirming the recorded series exist

$▊

Those throughput numbers deserve a sentence, because they are the honest heartbeat of this lab. 18.4 tokens per second for the 0.5B model and 6.9 for the 1.5B, on two vCPUs of a Multipass VM. A single L4 would do two orders of magnitude better on a larger model. That gap is exactly why the cost chapter is careful about which inputs it measures and which it models.

6. Grafana, and what a dashboard is for

Grafana is where the signals meet a human. Which means the design constraint is not "show everything", it is "be readable by a stressed person at 3am who did not build this".

I keep three dashboards and I refuse to add a fourth without deleting one.

Dashboard	Audience and moment	Top row, which is all most people read
Platform health	The one you open when paged. Answers "how bad and where"	Four stat panels: gateway availability, p95 TTFT, tokens per second, requests waiting. Red or green, no interpretation needed
Model serving detail	The one you open second. Answers "which layer"	TTFT and TPOT percentile bands, queue depth, preemption rate, KV cache usage, restarts, plus the GPU row that reads `No data`
Tenant and cost	Weekly review, not incidents. Answers "who is using what and what does it cost"	Tokens per tenant, requests per tenant, modelled cost per tenant, top models by spend

Here is one panel specified properly, because "make a Grafana dashboard" is useless advice and a panel spec is not.

Panel: p95 time to first token by model

Type: time series, with a stacked threshold band rather than a raw line, so the SLO boundary is visible without reading the axis
Query A: aiforge:ttft_seconds:p95 , legend {{model_name}}
Query B: vector(3) , legend SLO target, styled as a dashed red line
Unit: seconds, decimals 2
Thresholds: green below 2, amber 2 to 3, red above 3, applied as an area fill at 15 percent opacity
Min: 0, hard set. Grafana's autoscaled y axis makes a 40 millisecond wobble look like an outage, which is how you teach people to panic at noise
Legend: table mode, showing last and max, sorted descending by max. During an incident I want to know which model is worst, immediately
Annotations: an Argo CD deploy annotation query overlaid on the time axis, so "what changed" and "what got worse" appear on the same picture

That last bullet is the highest value line in this entire section. Overlaying deploys on latency graphs has diagnosed more incidents for me than any other single dashboard feature. It converts a five minute investigation into a two second glance, roughly half the time.

Dashboards are for orientation, Explore is for investigation

A dashboard cannot answer a question you did not anticipate, by definition, because someone had to build the panel in advance.

So I stopped trying. My dashboards orient: they tell me which component and which time window. Then I switch to Grafana Explore and write PromQL or LogQL freehand against the actual question.

Teams that try to make the dashboard answer everything end up with 90 panel monsters that nobody can read and that still miss the thing that broke. Three readable dashboards plus fluency in the query languages beats thirty dashboards every time.

7. Loki, and why its design forces you to think about labels

Loki looks like a log search engine and is architecturally nothing like one. Understanding the difference is what separates people who find Loki fast from people who find it maddening.

Loki does not index log content. Elasticsearch tokenises every word in every line and builds an inverted index, which is why it is fast at arbitrary text search and why it costs what it costs in CPU, RAM and disk. Loki indexes only the labels, and stores the log lines themselves as compressed chunks in object storage.

A LogQL query therefore has two stages. The label selector uses the index to pick which chunks to fetch, and it is fast. Everything after that is a brute force scan over decompressed text, and it is only fast because the selector already threw away 99 percent of the data.

The practical consequence is one rule: your label selector must be narrow, and your labels must be low cardinality. Same discipline as Prometheus, same reason.

The Loki label mistake everybody makes once

Somebody will suggest putting request_id or trace_id in a Loki label, reasoning that labels are indexed and therefore fast.

Do not. Loki creates a separate stream, with its own chunks, for every unique label combination. A per request label means a stream per request, which means millions of tiny chunks, which means the index explodes and query performance collapses. Loki will tell you off about this in its logs, and by then you have a mess to clean up.

trace_id belongs in the line, where |= finds it with a brute force scan that is genuinely fast once the selector has narrowed things down. Labels are for dimensions with tens of values: namespace, app, container, level. Not thousands.

8. OpenTelemetry, traces, and the collector

Metrics tell me the p95 is 6 seconds. Logs tell me nothing was wrong. OpenTelemetry tells me where those 6 seconds went.

Context propagation is the whole trick

A trace is only a tree because every service passes the context along. Concretely, the caller puts a W3C traceparent header on the outgoing HTTP request:

The callee reads it, creates its spans as children of that parent span ID, and passes a new traceparent onward with its own span as parent. Break that chain anywhere and you get two disconnected traces instead of one, which is the single most common tracing bug in existence.

In practice the chain breaks in exactly two places on an AI platform, and both are worth knowing before you spend an evening on them:

Async work and queues. A background embedding job started from a request loses the context unless you serialise the traceparent into the message and restore it on the worker side. This has to be done by hand.
Any library that builds its own HTTP client without instrumentation. The auto instrumentation hooks the standard client. A vendor SDK that rolls its own does not get hooked, and the span silently becomes a root.

The collector config

The gateway collector is where enrichment, sampling and fan out happen. This is the real config, trimmed of TLS boilerplate.

Why tail sampling instead of head sampling, in one paragraph

Head sampling decides at the start of the request, before anything has happened. It is cheap and it is stupid: it throws away errors at exactly the same rate as successes, so the traces you most want are the ones you are least likely to have.

Tail sampling buffers the completed trace for decision_wait seconds and then decides with full knowledge. I keep 100 percent of errors, 100 percent of traces over 4 seconds, 100 percent of LLM generations, and 20 percent of everything else.

The cost is memory in the collector, which is why memory_limiter is the first processor in the pipeline rather than an afterthought. On a lab with 320 MiB budgeted for the gateway, that limiter is load bearing.

9. Langfuse, the fifth signal

This is the part that makes AIForge an AI platform rather than a web platform with a model in it.

Langfuse models the world in objects that mean something to an LLM application, which no general purpose observability tool does:

Langfuse object	What it holds	The operational question it answers
Trace	One end to end user interaction, with tenant, user and metadata	What happened for this specific person at this specific moment
Span	A non model step: retrieval, reranking, a tool call, a guardrail check	Which stage of my pipeline was slow or wrong
Generation	A model call. Input messages, output text, model name and version, parameters, prompt and completion token counts, latency, computed cost	What did the model see, what did it say, how much did it cost
Session	Many traces grouped as one conversation	Did quality degrade as the context grew over a long chat
Score	An evaluation attached to a trace or generation, from a human, a heuristic or a judge model	Is output quality getting better or worse over time
Prompt	A versioned, labelled prompt template fetched at runtime	Which prompt version was live when quality dropped

Here is the instrumentation on the AIForge RAG path. This is the shape that matters: the retrieval span carries the similarity scores, because that single field is the difference between debugging a RAG system and guessing at one.

The one field that pays for the entire Langfuse deployment

scores on the retrieval span.

A tenant reported that answers had "got vague". Every infrastructure signal was clean. In Langfuse I filtered to that tenant, sorted by the retrieval_confidence score ascending, and the top similarity had fallen from a typical 0.71 to 0.78 band down to 0.29 to 0.36 starting at a specific hour.

Cause: a document re ingestion had run with a changed chunk size, splitting sentences mid clause and wrecking the embeddings. The model was working perfectly. Retrieval was feeding it fragments.

No metric, log or trace in the classic stack could have surfaced that. Latency was normal, status codes were 200, tokens were in range. The system was healthy and the product was broken, and only content level telemetry can tell those two states apart.

Tracing one RAG call, drawn

Preparing diagram

Read the timings. Embedding 41 milliseconds, vector search 18 milliseconds, generation 4.1 seconds. On CPU inference, generation is 98 percent of the wall clock, every single time. That number is the reason the cost chapter cares about tokens per second and almost nothing else, and it is the reason a latency incident on this platform is a generation incident until proven otherwise.

10. What went wrong, and what I watch now

The honest section. Five things this stack got wrong in my hands.

The alert that resolved itself during the outage. Covered in section 5 and it is the worst of the five. A ratio based error alert went NaN when traffic hit zero, so a total gateway failure silenced its own alarm. Fixed with the AIForgeTrafficDisappeared companion rule. What I watch now: every ratio alert gets an absolute companion, and I test alerts by breaking things rather than by reading the expression.

Cardinality nearly ate the head block. api_key_id as a metric label would have produced 576,000 series from one counter. Caught before shipping, by accident, while reading a dashboard query. What I watch now: prometheus_tsdb_head_series on the platform health dashboard, and an alert if it grows more than 30 percent week over week. Cardinality growth is silent right up until it is an OOM.

Traces broke at the async boundary and I blamed the collector. Background document ingestion produced root spans with no parent, so ingestion never appeared inside the request that triggered it. I spent an hour on collector config before realising the traceparent was simply never serialised into the job payload. What I watch now: a panel counting root spans by service. A service that produces only root spans is a service whose context propagation is broken.

Loki got slow because a label was too clever. I labelled log streams with model_name, which felt low cardinality and then was not, once tenants started pinning specific model versions. Stream count went up by a factor of nine and queries crawled. What I watch now: loki_ingester_memory_streams, and a hard rule that new labels need a written justification of their maximum cardinality.

Langfuse and its Postgres were not in the backup plan. For three weeks the only copy of every prompt version and every generation record lived on a single local-path volume on one Multipass VM. That is not observability, that is a single point of amnesia. What I watch now: Langfuse's database is in the same backup story as the control plane database, and the reliability chapter's database drill covers it explicitly.

What this stack still does not tell me

Naming the gaps, because a write up with no residual risk is a brochure.

Whether an answer was correct. Langfuse holds scores, but nothing populates them automatically. Retrieval confidence is a proxy for retrieval health, not a measure of truth. A real evaluation harness is not built.

Anything about GPU behaviour. Utilization, framebuffer, SM occupancy, power, thermal throttling, all designed, all empty. The queries are written. That is all I can claim.

Cross cluster anything. One cluster, one Prometheus, no federation, no long term remote write store. Retention is 7 days and then the data is gone.

Traces at real volume. Tail sampling with 20,000 buffered traces is untested above a handful of requests per second, because my lab cannot generate more than a handful of requests per second.