Projects · AIForge
Reliability and failure drills
SRE practice for an AI platform: SLIs, SLOs and error budgets, probes that do not kill a busy model server, PodDisruptionBudgets, graceful degradation, then eight deliberate failure drills with symptom, detection, diagnosis, mitigation and lesson, ending with a real postmortem.
Reliability and failure drills
Everything so far in Volume 4 has been about seeing, protecting and measuring the platform.
This chapter is the exam. I break it on purpose, eight times, and write down what happened.
That is not showmanship. It is the only way to find out whether the controls from the previous three chapters actually work, because a control you have never watched fire is a control you believe in rather than a control you have. My alert rules, my probes, my PodDisruptionBudget and my fallback chain were all correct in my head. Roughly half of them were wrong in the cluster, and I only know which half because I broke things while watching.
The AIForge cluster is built with k3smp, which provisions K3s onto real Multipass virtual machines. Not containers pretending to be nodes.
So when the node loss drill runs, I execute multipass stop and a real machine goes away. The kubelet stops posting status. The node controller waits out its grace period. Taints appear, tolerations expire, pods get evicted, the scheduler tries to place them and fails on memory. Persistent volumes bound to that node stay exactly where they are.
Every timer is the real Kubernetes timer and every behaviour is the real behaviour. That is the difference between practising an outage and reading about one, and it is why building the tool was worth it.
1. SLIs, SLOs and error budgets, applied to something probabilistic
The Google SRE books are the source for this vocabulary and they are worth reading properly. The short version, because the words get used loosely:
An SLI is a service level indicator: a measurement of one aspect of service quality, expressed as a ratio of good events to valid events. An SLO is the target for that indicator over a window. An error budget is the arithmetic complement of the SLO, which is the amount of failure you have deliberately decided to permit.
That last one is the idea that changes behaviour, so it is worth stating properly. An error budget is a permission, not a threat. A 99.5 percent monthly availability SLO says you may be unavailable for 216 minutes a month. If you have used 12 of them, you should be shipping faster and taking more risk, because unspent budget is a sign you are being too cautious. If you have used 210, you stop shipping features and fix reliability. The number turns an argument about feelings into a decision with a rule.
Choosing SLIs when correctness is fuzzy
Here is the AI specific problem. On a normal service, "good" means a 2xx status code within a latency threshold. On an AI platform, an HTTP 200 with a p50 latency can still be a completely useless answer, which is a thing I established in the observability chapter and which now has to survive contact with an SLO.
So AIForge has five SLIs, and the fifth one is the interesting one.
| SLI | Good event | Valid event | SLO target | Error budget, 30 days |
|---|---|---|---|---|
| Gateway availability | Response is not a 5xx | Every authenticated request to the gateway | 99.5 percent | 216 minutes, or 0.5 percent of requests |
| Time to first token | TTFT under 3 seconds | Every streaming chat request | 95 percent | 5 percent of requests may be slow |
| Stream completion | Stream ends with a stop reason, not a disconnect | Every streaming request that got a first token | 99.0 percent | 1 percent of streams may break mid answer |
| Control plane availability | Deployment API responds under 2 seconds without a 5xx | Every control plane API call | 99.0 percent | 432 minutes. Deliberately looser than the data path |
| Retrieval health | Top chunk similarity at or above 0.40 | Every RAG request that reached retrieval | 97 percent | 3 percent of answers may be built on weak context |
The first four SLIs would fit any web service. The fifth would not exist anywhere else and it is the one I would fight to keep.
It does not measure whether an answer was correct. Nothing I have built can measure that. What it measures is whether the model was given anything worth working with, which is a genuine leading indicator: when top similarity collapses, answer quality collapses shortly afterwards, and every infrastructure signal stays green throughout.
It is a proxy, and I want to be clear about that. But it is a proxy for the thing users actually care about, and a rough SLI on something that matters beats a precise SLI on something that does not.
Note the deliberate asymmetry between rows one and four. The control plane has a looser SLO than the data path, and that is a considered decision rather than laziness. If the deployment API is down for ten minutes, a developer waits ten minutes to create a model deployment, which is annoying. If the inference gateway is down for ten minutes, every end user of every tenant application is broken. Same platform, different blast radius, different target, and writing that down stops someone treating them as equally urgent at 3am.
The error budget arithmetic
A 30 day month is 43,200 minutes. So:
Read that last line and understand what it commits you to. Four minutes of unavailability per month means no human can be in the recovery path. Detection, decision and remediation must all be automatic, because a human being paged, waking up and reading a dashboard has already spent the entire budget. Every additional nine is not a slightly harder engineering problem, it is a different engineering problem.
My lab targets 99.5 percent, which is honest for a platform running on a laptop, and I would rather state a target I can hold than a number that looks impressive.
Burn rate alerts, which are how you page on an SLO
Alerting on "error rate above 1 percent" is bad in two directions at once: it pages you for a harmless blip, and it stays silent through a slow leak that drains your whole month.
Burn rate fixes this. A burn rate of 1 means you are consuming budget exactly fast enough to exhaust it precisely at the end of the window. A burn rate of 14.4 means you will exhaust a 30 day budget in about 50 hours, which means the last hour consumed 2 percent of the month.
The multi window, multi burn rate pattern uses a long window to establish significance and a short window to confirm the problem is still happening right now.
2. The mechanics that keep it up
Before breaking things, the four controls that determine how gracefully they break.
2.1 Probes, and the one that kills busy model servers
Kubernetes has three probes and they answer three different questions. Getting them confused on an inference workload does real damage, and this is the single most common serious mistake I see.
| Probe | Question | Consequence of failure | The AI specific trap |
|---|---|---|---|
| startupProbe | Has it finished booting? | Container is restarted after the failure threshold | Model load takes 40 to 70 seconds. Without a startup probe, the liveness probe kills the pod mid load, forever. Guaranteed crash loop |
| readinessProbe | Should it receive traffic right now? | Removed from Service endpoints. Not restarted | Should reflect capacity, not just liveness. A server with a 40 deep queue is honestly not ready |
| livenessProbe | Is it broken beyond recovery? | Container is killed and restarted | The dangerous one. A busy server that answers slowly is not a dead server, and killing it makes everything worse |
This is the cascading failure that catches almost everyone once, and it is worth walking through slowly because it is not obvious.
Traffic rises. The model server's queue fills. The event loop is busy and /health starts answering in 4 seconds instead of 20 milliseconds. Your liveness probe has timeoutSeconds: 2 and failureThreshold: 3, so after 3 slow responses Kubernetes concludes the container is dead and kills it.
Now you have one fewer replica serving the same traffic. Every remaining replica gets more load, answers /health even more slowly, and gets killed too. Then each restart takes 60 seconds of model loading during which it serves nothing at all.
Three rules that prevent it. Liveness must be generous, far more generous than feels right. Liveness must hit an endpoint that does not queue behind inference work. And readiness is the probe that should react to load, because shedding traffic is recoverable and killing a process is not.
2.2 PodDisruptionBudget
A PDB constrains voluntary disruptions: node drains, cluster upgrades, spot reclamation, anything going through the eviction API. It does nothing about a node catching fire, which is an involuntary disruption and which no PDB can help with.
Set minAvailable: 1 on a Deployment with exactly one replica and you have built a permanent block on node drains.
The eviction API is asked to remove the only pod. Doing so would leave zero available, which violates the budget. So the eviction is refused, forever, and kubectl drain hangs politely and indefinitely while you wonder what is wrong with your cluster.
The correct fix is two replicas, at which point the PDB does its actual job. The wrong fixes are removing the PDB, which gives up the protection, or using --disable-eviction, which bypasses every safety check you have.
This is the tension from the volume overview appearing in a concrete YAML file: reliability wants two replicas, cost wants one, and a PDB on a single replica workload gives you the costs of both choices and the benefits of neither.
2.3 Graceful shutdown, which matters more with streaming
A normal HTTP request finishes in milliseconds, so shutdown ordering is forgiving. An LLM stream can run for 30 seconds or more, which makes termination handling a user visible feature rather than a detail.
2.4 Graceful degradation, the design decision behind the fallbacks
The interesting reliability question on an AI platform is not "how do I stay up". It is "what do I serve when I cannot serve the good thing", and answering it well is mostly product judgement expressed in infrastructure.
Look at the no context branch again, because that box contains an argument I had with myself for a while.
When Qdrant is down, I can still answer from the model's own parametric knowledge. The response will be fluent, plausible and completely unmoored from the tenant's documents. A user asking "what is our leave policy" gets a confident answer about a generic leave policy that has nothing to do with their company.
That is worse than an error. An error is honest and the user retries. A confident wrong answer gets acted upon.
So degradation is only acceptable when it is visible. The banner is not decoration, it is the entire reason this branch is allowed to exist. If you cannot make a degraded mode obvious to the user, fail instead.
3. The drills
Eight drills. One consistent shape for every one: what I broke, symptom, detection, diagnosis, mitigation, lesson. The lesson is the part that matters, and in three cases it is a control that turned out not to work.
| # | Drill | Detected by | Time to detect | Status |
|---|---|---|---|---|
| 1 | Model server crash | Kubernetes restart, then queue alert | ~9 s to reschedule, 71 s to ready | Run |
| 2 | OOM kill under concurrency | AIForgeModelServerDown, restart counter | 2 min 20 s | Run |
| 3 | Vector database down | Retrieval health SLO burn | 38 s | Run |
| 4 | PostgreSQL down | Control plane 5xx alert | 1 min 05 s | Run |
| 5 | Latency spike from CPU contention | AIForgeTTFTSLOAtRisk | 10 min, too slow, fixed | Run |
| 6 | Bad deploy and rollback | Nothing. A user told me | ~26 min | Run, and the worst result |
| 7 | Traffic spike | Queue backlog alert | 5 min 12 s | Run |
| 8 | Node loss, real VM shutdown | Node NotReady alert | 52 s to NotReady | Run |
| 9 | GPU memory exhaustion | Designed against DCGM metrics | Not applicable | Designed, never run. No GPU |
Drill 1: the model server crashes
What I broke. Deleted the vLLM pod with no warning. The crudest possible drill and the right one to start with, because it establishes the recovery baseline that every other drill is measured against.
Symptom. For 71 seconds there was no healthy model server. Requests arriving in the first 62 of those seconds got 503 from the gateway, because there was nothing to route to.
Detection. Kubernetes noticed instantly, since it was the one doing the deleting. The interesting question is whether I would have noticed, and the answer is that the readiness gap was visible on the dashboard in under 10 seconds but no alert fired, because AIForgeModelServerDown has for: 2m and the incident lasted 71 seconds.
Diagnosis. Trivial in a drill. Worth noting what the logs give you for free: 28.9 seconds loading weights from the PVC, 9.1 seconds building the KV cache and warming up. That split is genuinely useful, because it tells you whether a slow start is a storage problem or a compute problem.
Mitigation. None needed. Kubernetes did the whole thing. Which is the point of drill 1.
Lesson. Two, and the second is the real one.
The self healing works and it costs 71 seconds. That number is now the input to every other decision: it is why cooldownPeriod on the KEDA scaler is 600 seconds rather than 60, and it is why scale to zero is unacceptable for interactive workloads.
And a single replica means every restart is a user visible outage. I knew that abstractly. Watching 62 seconds of 503s made it concrete, and it is the single most persuasive argument for the second replica that the cost chapter argues against. Both chapters are right. The resolution is that the answer depends on the SLO, which is exactly why the SLO comes first.
Drill 2: OOM kill under concurrency
What I broke. Raised max-num-seqs from 32 to 128 and left the memory limit at 6 GiB. More concurrent sequences means more KV cache, and on CPU that cache is ordinary process memory subject to the cgroup limit.
Symptom. A crash loop with RESTARTS 3. Between restarts, throughput collapsed from 6.9 tokens per second to 2.2 while the KV cache thrashed at 99.9 percent and the scheduler preempted sequences on every iteration.
Detection. AIForgeModelServerDown fired at 2 minutes 20 seconds. But AIForgeKVCachePreemption fired 90 seconds earlier, and that is the alert that would have been useful. Preemption is the leading indicator. OOM is the lagging one.
Diagnosis. Exit code 137 is SIGKILL, and combined with Reason: OOMKilled there is no ambiguity: the kernel OOM killer, acting on the cgroup limit. The --previous logs are essential here and easy to forget, because the current container has no memory of what killed the last one.
Mitigation. Rolled max-num-seqs back to 32. The correct long term answer is a memory limit computed from the concurrency setting rather than the two being configured independently by different people at different times.
Lesson. The one that surprised me: the last 30 seconds before the OOM were worse for users than the OOM itself. Throughput at 2.2 tokens per second with a 34 deep queue is functionally unusable, and it lasted longer than the restart. A crash is fast and honest. Slow death is neither.
Which changes what I alert on. AIForgeKVCachePreemption is now a page rather than a warning, because sustained preemption means users are already suffering and a crash is coming.
Drill 3: the vector database goes away
What I broke. kubectl scale statefulset qdrant --replicas=0. Qdrant is the retrieval layer for every RAG tenant.
Symptom. RAG requests kept returning HTTP 200. Latency actually improved, because retrieval was failing fast instead of doing work. Answers were fluent, confident and completely disconnected from tenant documents.
Detection. This is the drill that justifies the retrieval health SLI. The chain of what fired and what did not:
| Signal | Did it fire? | Why |
|---|---|---|
| Gateway availability SLO | No | Every response was a 200. Nothing was failing from HTTP's point of view |
| TTFT SLO | No | Latency improved. Skipping retrieval is faster than doing it |
Qdrant up == 0 | Yes, at 38 s | The scrape target vanished. This is the fast, boring, correct signal |
| Retrieval health SLO burn | Yes, at 15 min | Slower, but it is the signal that catches degradation without an outage, such as a bad re index |
| Langfuse retrieval scores | Visible immediately | Chunk count 0 on every trace. Unmistakable once you look |
Diagnosis. Ten seconds, because up == 0 on a component names the component.
Mitigation. Scaled Qdrant back up. Recovery took 34 seconds, dominated by loading the HNSW index into memory.
Lesson. Two, and the first is the most important lesson in this whole chapter.
My degradation path was silent, and I had convinced myself it was not. I had written the "no context" fallback and I believed it set a banner. It did not: the banner was implemented in one client path and the API returned a normal response body with no indicator at all. So the platform quietly served plausible nonsense for eleven minutes while I watched a green dashboard.
That is a control that existed in my head and not in the cluster, and I would never have found it by reading the code, because I had read the code and concluded it was fine.
Second: the boring signal beat the clever one. up == 0 detected this in 38 seconds. My sophisticated retrieval health SLO took 15 minutes. Both belong, but I had been quietly proud of the clever one and it was the slower of the two.
Drill 4: the database goes away
What I broke. Scaled PostgreSQL to zero. Postgres holds platform state: tenants, deployments, API keys, quotas, audit records.
Symptom. This is the drill where the architecture earned its keep. Inference kept working perfectly. Every existing tenant continued to send chat requests and get answers. What broke was the control plane: creating a deployment, listing applications, rotating a key, all 500.
Detection. Control plane 5xx alert at 1 minute 5 seconds. The gateway availability SLO was untouched, correctly, because the data path was genuinely healthy.
Diagnosis. Control plane logs were unambiguous: sqlalchemy.exc.OperationalError: connection to server ... failed: Connection refused. Thirty seconds.
Mitigation. Scaled Postgres back up. The control plane's connection pool recovered on its own after roughly 20 seconds without needing a restart, which was a pleasant surprise and worth verifying rather than assuming.
Lesson. The separation between control plane and data path is real, and this drill is the proof. A developer platform whose control plane outage takes down every running workload is not a platform, it is a very large single point of failure. Volume 3 designed for this and I had never actually tested it.
The uncomfortable part: LiteLLM caches API keys in memory, which is why authentication kept working. If the gateway had restarted during the database outage, it would have started with an empty cache, been unable to reach Postgres, and every tenant would have lost authentication. My data path independence has an asterisk on it, and I only found the asterisk by running the drill. The fix is a bounded on disk cache with a documented staleness window, and it is not built yet.
Drill 5: a latency spike from CPU contention
What I broke. Ran stress-ng --cpu 2 --timeout 600s inside the Multipass VM hosting the model server. This simulates the entirely realistic case of a noisy neighbour, a batch job, or my own laptop deciding to index something.
Symptom. p95 TTFT went from 2.4 seconds to 11.8 seconds. Tokens per second fell from 6.9 to 2.1. No errors, no restarts, no crashes. Just a platform that felt broken while reporting itself healthy.
Detection. AIForgeTTFTSLOAtRisk fired after 10 minutes, because it has for: 10m. That is far too slow for a threefold latency regression, and it was the most useful process finding of the drill.
Diagnosis. The sequence that got me there in about four minutes: TTFT up, queue depth up, container_cpu_cfs_throttled_seconds_total flat, node CPU saturated. Throttling flat plus node saturated means the contention is outside the container, not a limit problem, so it is a neighbour.
Mitigation. Killed the stress process. In production the equivalents are CPU requests that reserve genuine capacity, a Guaranteed QoS class for the model server, and node affinity keeping batch work away from latency sensitive inference.
Lesson. My alert timing was tuned for stability rather than for users. for: 10m exists so a 30 second blip does not page anyone, which is reasonable. But a threefold sustained regression should page in under two minutes.
The resolution is two alerts rather than one compromise: a fast alert with a high threshold, p95 above 8 seconds for 2 minutes, and a slow alert with a low threshold, p95 above 3 seconds for 10 minutes. A single alert cannot be both sensitive and stable, so stop trying to make it both.
Drill 6: a bad deploy, and the worst result in this chapter
What I broke. Shipped a RAG prompt template change through Argo CD that referenced a variable the retrieval stage no longer produced. The template rendered with an empty context block. No exception, no error, no failed health check. Just a system prompt that said "use the following context" followed by nothing.
Symptom. Every answer became generic. Fluent, confident, and containing none of the tenant's information.
Detection. Nothing fired. A user told me, roughly 26 minutes later.
Every single signal was green. 200s, normal latency, normal token counts, healthy pods, and Qdrant was up and being queried successfully so even the retrieval health SLI was fine. Retrieval worked perfectly. The chunks were simply never inserted into the prompt.
Diagnosis. Four minutes once I looked in the right place, and the right place was Langfuse. The generation object shows the fully rendered prompt, and the empty context block was immediately visible. Then Argo CD's history named the commit. Without prompt level telemetry I would have been reading application logs for an hour, and application logs contained nothing wrong because nothing was wrong from the application's point of view.
Mitigation. argocd app rollback aiforge-rag 47. Back to correct answers in 90 seconds, most of which was the pod restart.
Lesson. The most important one in this chapter, and it is uncomfortable.
No infrastructure signal can detect a correct system doing the wrong thing. I had five SLOs, four signal types, and a dedicated LLM observability platform, and the thing that detected this outage was a human being who noticed the answers had got worse.
Three changes came out of it. A canary assertion in CI: a known question with a known answer, run against the built image, asserting that a specific phrase from a specific test document appears in the output. It would have caught this in the pipeline. A prompt length SLI: rendered prompt token count per model, alerting on a sudden drop, because a template that lost its context block gets dramatically shorter and that is trivially detectable. Argo CD deploy annotations on every Grafana dashboard, so "what changed" and "what got worse" are on the same picture.
Read the detection line again. Twenty six minutes, and the detector was a person.
That is what an AI platform failure looks like when it is not an infrastructure failure. Nothing crashed. Nothing was slow. Nothing returned an error. The product was simply wrong, and every dashboard I owned said it was fine.
The only telemetry that could have caught this is telemetry about the content of the generation. If you take one argument from Volume 4 for why an AI platform needs a fifth signal type, take this drill.
Drill 7: a traffic spike
What I broke. Ran 40 concurrent chat requests against a platform sized for about 4.
Symptom. Queue depth hit 36. p95 TTFT reached 41 seconds. Clients with a 30 second timeout gave up, and their abandoned requests kept generating, because nothing told vLLM the client had gone. So the server spent real capacity producing tokens for connections that no longer existed.
Detection. AIForgeInferenceQueueBacklog at 5 minutes 12 seconds, which is for: 5m doing exactly what it says. KEDA reacted faster, adding a replica at around 45 seconds.
Diagnosis. Straightforward. Queue depth up, TTFT up, tokens per second flat at the hardware maximum. Flat throughput at the ceiling with a growing queue is the signature of saturation rather than of a fault.
Mitigation. KEDA scaled to 4 replicas, which on a laptop meant the fourth was stuck Pending on insufficient memory. Real mitigation was rate limiting at the LiteLLM gateway: 429 with a Retry-After header, which is a much better user experience than a 41 second wait followed by a timeout.
Lesson. Autoscaling is not a load shedding strategy. Scaling up takes 71 seconds, needs capacity that may not exist, and on a fixed size lab it simply cannot help. Rate limiting is instant and always available. You need both, and you need to be clear that they solve different problems: autoscaling handles sustained growth, rate limiting handles spikes.
The abandoned request finding was new to me and worth propagating: without client disconnect handling, a traffic spike wastes capacity on requests nobody is waiting for, which makes saturation worse in a self reinforcing way at exactly the wrong moment.
Drill 8: node loss, for real
What I broke. multipass stop aiforge-w2. A real virtual machine, powered off, no notice, no drain. This is the drill that only works because the lab uses real VMs.
Symptom. Layered, and each layer is a different lesson.
The node went NotReady at 52 seconds, which is the node monitor grace period. Pods on it were marked for deletion but stayed Terminating indefinitely, because the kubelet that would confirm deletion is on the powered off machine. The vLLM replica on aiforge-w1 absorbed all the traffic and coped. And Qdrant never came back, because its local-path PersistentVolume physically lives on the dead node's disk.
Detection. KubeNodeNotReady at 52 seconds. Good enough.
Diagnosis. The 52 second delay is worth understanding rather than being surprised by. The kubelet posts node status every 10 seconds. The node controller waits node-monitor-grace-period, then applies node.kubernetes.io/unreachable:NoExecute. Pods carry a default 300 second toleration for that taint, so eviction begins roughly five minutes later. From power off to eviction is about six minutes, and if you did not know that number you would spend the first five of them thinking Kubernetes was broken.
Mitigation. multipass start aiforge-w2. Everything recovered in about 90 seconds once the kubelet re registered.
Lesson. Three, and the second one was a control I thought I had.
Stateless workloads survived node loss and stateful ones did not. That is not a failure of Kubernetes, it is local-path storage doing exactly what it says. But it means Qdrant on local-path is a single point of failure for every RAG tenant, and no amount of replica count fixes it. Real answers are replicated storage or Qdrant's own clustering with distributed collections. Neither is built.
My PDB was useless here, which I should have known. A PodDisruptionBudget governs voluntary evictions through the eviction API. A powered off VM is involuntary. The PDB was not consulted, did not help, and could not have. If your reliability plan for node failure is a PDB, you have no reliability plan for node failure.
And a minAvailable: 1 PDB on a single replica deployment blocks drains forever, which I discovered while trying to drain aiforge-w1 afterwards to test the graceful path. The command hung, politely, indefinitely. Two replicas or no PDB. Pick one.
Drill 9: GPU memory exhaustion, designed and not run
Status: designed, never executed. There is no GPU in this lab.
I am including it because the design is real work and because pretending to have run it would poison everything else in this chapter.
What I would break. Send requests with prompts near max-model-len at high concurrency, so KV cache demand exceeds the pre allocated framebuffer.
Expected symptom. vLLM preempts sequences with PreemptionMode.RECOMPUTE, so requests get paused and their KV cache thrown away and recomputed later. Tail latency degrades brutally while throughput looks acceptable in aggregate. In the worse case, torch.cuda.OutOfMemoryError and a crash.
Detection, written and untestable. AIForgeGPUMemoryPressure on the DCGM framebuffer ratio, plus vllm:gpu_cache_usage_perc above 0.95, plus the preemption rate alert that does work on CPU and did genuine work in drill 2.
Expected diagnosis. DCGM_FI_DEV_FB_USED near total, vllm:num_preemptions_total climbing, TPOT p99 far above p50 while p50 stays reasonable. That divergence between p50 and p99 is the fingerprint.
Planned mitigation. Lower max-num-seqs, lower max-model-len, enable --enable-chunked-prefill, or quantize the KV cache. Adding replicas does not help, because each replica needs its own full KV cache.
What I can honestly claim. The CPU version of this drill, drill 2, exercised the same scheduler code path and the same preemption logic, and it taught me that sustained preemption is the leading indicator. That transfers. The specific GPU memory behaviour does not, and I will not pretend otherwise.
4. Postmortems, and the practice that makes drills compound
A drill that produces a fix is worth something. A drill that produces a written postmortem is worth several times more, because the writing is what turns one incident into a class of incidents you have addressed.
Title. What broke and who it affected, in one line. Not "Qdrant incident".
Impact. Duration, tenants affected, requests affected, error budget consumed in minutes and as a percentage. Numbers, not adjectives.
Timeline. Timestamped, in UTC, from the change that caused it to the confirmed recovery. Include when a human first became aware, separately from when the alert fired. The gap between those two is often the most instructive number in the whole document.
Root cause. Not "human error". Human error is never a root cause, it is a description of the last step in a chain of missing controls. Keep asking why until you reach something you can change with engineering.
What went well. Genuinely. Which control worked, which alert fired correctly, which runbook was accurate. This is not morale management, it is how you learn which investments paid off.
What went badly. Which control did not work. Which alert was too slow. Which dashboard misled you.
Action items. Each with an owner, a due date, and a priority. Each either prevents the failure or detects it faster. Anything that is neither is not an action item, it is a feeling.
Lucky breaks. The bit everybody skips and the bit that predicts your next incident. What could have made this much worse and did not happen to?
5. What went wrong, and what I watch now
The pattern across eight drills, because the aggregate is more interesting than any individual one.
Three of my controls did not work. The silent degradation path in drill 3, the API key cache dependency in drill 4, and the PDB deadlock in drill 8. All three were designed correctly, reviewed by me, and wrong in the cluster. The only reason I know is that I ran the drills. A control you have not watched fire is a belief.
Two of my alerts were too slow. Drill 5 took 10 minutes to alert on a threefold latency regression, and drill 6 never alerted at all. Both were tuned for stability, which is a real goal, and both were tuned without asking how long a user would suffer first.
The worst outage was the one where nothing broke. Drill 6 had no errors, no crashes, no latency change and no SLO impact, and it was the longest and most damaging incident in the set. That is the defining property of AI platform failures and it is why the observability chapter is the longest in the volume.
What I watch now, specifically:
- Every SLI has both a fast, high threshold alert and a slow, low threshold alert. No more single compromise alerts.
AIForgeKVCachePreemptionpages instead of warning, because drill 2 proved slow death is worse than a crash.- Rendered prompt token count is an SLI, so a template that lost its content is visible without a human reading answers.
- Argo CD deploy annotations are on every dashboard, so "what changed" is always one glance away.
- Every degradation path has a test that asserts the user visible indicator is present, not just that the fallback executed.
- A quarterly drill calendar, because controls rot. The PDB that works today breaks the day someone changes a replica count.
Behaviour at real scale. My traffic spike drill was 40 concurrent requests. Real saturation, thundering herds, connection pool exhaustion and cascading timeouts appear at volumes this lab cannot generate.
Multi node and multi cluster failure. One control plane node. No etcd quorum loss drill, no split brain, no cross cluster failover. All out of scope and all real risks in production.
Data loss and recovery. I broke Postgres and Qdrant by scaling them to zero, which is availability, not durability. I have not corrupted a volume, lost an index, or performed a tested restore. An untested backup is not a backup, and mine are untested.
GPU failure modes. Drill 9 is designed and unrun. Thermal throttling, ECC errors, driver crashes and MIG partition failures are entirely unexplored.
Anything involving other humans. Every drill here had one operator who knew exactly what he had broken, because he had just broken it. Real incident response involves handoffs, unclear ownership, conflicting theories and communication overhead, and none of that is testable alone.
Next
That is Volume 4. Back to the volume overview for the gate checklist, and it is worth re reading now: the fourth and fifth tabs are the ones the drills actually tested, and two items on them are not yet satisfied.
Three threads from this chapter land in earlier volumes rather than a later one, and I would rather name them than let them dangle. Qdrant on local-path storage is a single point of failure that belongs in the RAG pipeline chapter. The LiteLLM API key cache dependency on PostgreSQL belongs with identity and multi tenancy. And the canary assertion in CI belongs in IaC, GitOps and CI/CD.
That is the honest shape of day two work: it does not produce a finished platform, it produces a corrected one, and it sends you back to the volumes you thought you had finished.