Projects · Mini Hostpapa
Volume 2 Lab Environment
Turn the Volume 1 network design into a running simulated datacenter on one laptop. OVS as the top of rack switch, nested KVM, NFS storage, and Apache CloudStack on top.
All of the planification in this document (and the rest of Volume 2) is implemented as runnable Ansible, Terraform, and CloudMonkey helpers in mini-HostPapa/virt-dc. Clone that repo, follow its README, and you get the fabric, the rack VMs, CloudStack, the zone path, and the first-instance gate without retyping these snippets by hand.
Volume 2 Lab Environment
Volume 1 ended with a complete network design. Traffic types, VLAN assignments, IP ranges, bridge names, port matrices, an IPv6 plan. Every number I needed was written down.
And not one packet had moved.
That is the honest situation at the end of Volume 1, and I want to name it clearly rather than glide past it. A design that has never been built is a hypothesis. It is a good hypothesis, researched against the official documentation and reasoned through carefully, but it has never once been contradicted by a machine. That is not the same thing as knowledge.
Volume 2 exists to make the Volume 1 design wrong in specific places, so I can fix it before it matters.
The output is not a document. The output is a running environment: Apache CloudStack installed, a zone created, a hypervisor host registered, system VMs green, and a real instance booting inside it.
Why this is not a Linux fundamentals volume
The original plan for Volume 2 was Linux administration. Users, groups, SSH, systemd, filesystems. I scratched it, and I want to say that plainly, because "I skipped the fundamentals" sounds like a red flag until you know where those fundamentals actually live.
Here is the decision in one sentence: this knowledge base already has a home for Linux fundamentals, and duplicating that material inside the HostKid project would make both places worse. Enormous docs that repeat the same systemd explainer twice help nobody. Volume 2 exists to get straight to the point — building the lab, wiring the network, standing up CloudStack — not to re-teach Linux from zero inside a project track.
If you need the fundamentals first, that is not a gap in this volume. That is what the Linux foundations section is for: a progressive path through the operating system itself, separate from any one project. Read that when you want the mental model. Come back here when you want the rack on your laptop.
This volume is not "Linux-free." By the end I will have configured netplan bridges and VLAN sub-interfaces, written systemd units, tuned kernel parameters through sysctl, managed NFS exports, debugged firewall forwarding, and read journald logs in anger.
The difference is scope. I assume you can open the foundations track when a concept needs depth. Here, every command serves the HostKid lab — nothing extra, nothing repeated from a textbook chapter you already have elsewhere.
The constraint, stated honestly
I am building a datacenter simulation on the laptop I also work on. Here is what I actually have, measured rather than assumed:
| Resource | Total | Reality check |
|---|---|---|
| CPU | 8 threads (Intel i7-1165G7, 4 physical cores) | VT-x present, so nested virtualization is possible |
| RAM | 15 GiB usable | Already sitting at 13 GiB used with my normal workload running |
| Swap | 11 GiB | Already 8.9 GiB used |
| Disk | 64 GiB free on a root filesystem that is 86% full | My self-imposed ceiling is 50 GiB, and I will land at 44 GiB |
Look at that RAM row again. 13 of 15 GiB were already in use when I measured, with 1.5 GiB available. A CloudStack management server plus a nested KVM hypervisor needs roughly 11 GiB.
The arithmetic does not work while my IDE, browser, and containers are running. So the lab is not designed as something that hums along in the background. It is designed as a mode I deliberately switch into, with a documented shutdown of everything else first.
I would rather tell you that on page one than have you discover it when the OOM killer reaps MySQL at 2am.
The whole resource ledger, and the discipline for living inside it, is worked out in lab architecture and resource budget.
What gets simulated, and what gets faked
The interesting engineering question in a lab like this is not "how do I make it run". It is which parts must be real and which parts may be pretend. Get that wrong in either direction and the lab is worthless: too much pretending and it teaches nothing, too much realism and it never boots.
My rule was this: anything that CloudStack touches, or that I would have to configure on real hardware, must be genuinely real. Anything that only exists because production has redundancy or physical separation may be collapsed.
| Layer | Lab treatment | Why |
|---|---|---|
| 802.1Q VLAN tagging | Real. Open vSwitch does actual tagging on actual trunk ports | This is what CloudStack manipulates. Faking it teaches nothing |
Bridges cloudbr0 and cloudbr1 | Real. Real Linux bridges, same names as production | CloudStack traffic labels must match Volume 1 exactly |
| Nested hypervisor | Real. KVM inside KVM, /dev/kvm present in the guest | The agent, the system VMs, and live migration all depend on it |
| NFS primary and secondary storage | Real. Real exports, real mounts, real MTU 9000 path | Storage mount failures are the most common CloudStack setup problem |
| Edge firewall and NAT | Real. nftables masquerade, real internet egress | The Secondary Storage VM must genuinely reach the internet |
| Top of rack switch | Collapsed. One OVS bridge on the laptop plays the ToR | A physical switch is not simulatable, but its behaviour is |
| Redundancy (bonds, A/B power, MLAG, dual firewall) | Faked or skipped | Redundancy protects against failures the lab cannot have |
| Physical separation of servers | Collapsed. The laptop is also the storage array | Saves roughly 1.5 GiB of RAM and buys no learning |
Every collapse above is recorded in a deviation register in the architecture document, with the production equivalent written next to it. That register is the thing that stops a lab from quietly becoming a lie you believe.
The three levels of nesting
This is the mental model that makes everything else in the volume readable. There are three floors, and confusing them is the single biggest source of "why is this not working".
When something breaks, my first question is always which floor am I standing on.
A VLAN problem at L0 is an ovs-vsctl problem. The same symptom at L1 is a netplan or bridge problem. At L2 it is a CloudStack traffic label problem.
Three completely different toolboxes, one identical symptom of "the ping does not work". Naming the floor first has saved me more time than any single command in this volume.
The software I chose, and the one-line reason
The full reasoning, including what I rejected, is in the architecture document. The short version so you know what you are installing:
| Job | Choice | The one-line reason |
|---|---|---|
| Hypervisor at L0 | KVM / QEMU / libvirt | Native, free, and the only sane way to get nested virtualization on Linux |
| Network fabric | Open vSwitch | It performs genuine 802.1Q tagging and trunking, so the VLANs are not a simulation |
| Fabric definition | libvirt network XML with OVS portgroups | Declarative VLAN access and trunk ports in a file I can version control |
| VM lifecycle | Terraform with dmacvicar/libvirt 0.9.8 | Declarative, and it warms up the Terraform muscle this project needs by Volume 7 |
| Guest OS config | cloud-init | The industry standard, and CloudStack templates use it too |
| Host and server config | Ansible | Idempotent, and it is already the Volume 21 choice |
| Rack server OS | Ubuntu 24.04 LTS | On the CloudStack 4.22 support matrix, and cloud images exist |
| Tenant instance OS | Alpine Linux 3.24 | A 190 MB image that boots happily in 192 MiB of RAM. This is what makes the budget work |
| The platform | Apache CloudStack 4.22 | The Volume 3 decision, deployed exactly as Volume 1 designed it |
This split is deliberate and I use the same rule in production.
Terraform owns whether a thing exists. Virtual machines, disks, networks, cloud-init seeds. Things with a lifecycle, where "destroy and recreate" is a sensible operation.
Ansible owns the state inside a thing that already exists. Packages, config files, services, mount points.
Mixing them produces the classic mess where nobody can answer "what happens if I run this twice". Keeping the boundary clean means I can always rebuild the whole lab from zero with two commands.
How this volume is organised
Four documents, in build order. They are meant to be executed in sequence, not browsed.
| # | Document | What you have when you finish it |
|---|---|---|
| 1 | Lab architecture and resource budget | The full design, the RAM and disk ledger that proves it fits, the deviation register, and the tool selection reasoning |
| 2 | Host preparation and network fabric | A laptop with nested virtualization on, an OVS ToR carrying real VLANs, working SVIs, NAT to the internet, and NFS exports |
| 3 | Provisioning the rack | Two booted, addressed, SSH-reachable servers built from a Terraform plan you can destroy and rebuild |
| 4 | Installing CloudStack and the first instance | The socle: CloudStack running, zone created, host added, system VMs green, and an Alpine instance with an IP |
The gate that ends this volume
I do not consider Volume 2 finished because the documents are written. I consider it finished when these are all true on my own machine:
terraform applyfrom an empty state produces two running, SSH-reachable servers with no manual stepsterraform destroyfollowed byterraform applyreproduces the environment, proving the lab is disposable rather than precious- The whole lab footprint stays under 50 GiB, verified with a single
du -sh - The lab starts and stops with one command each, so switching into lab mode is not a ritual
Three of those tabs are checklists. The last one is the actual point of the volume.
If I get to the end and nothing in the Volume 1 design turned out to be wrong, I should be suspicious rather than pleased. It would almost certainly mean I built the lab to match my document instead of building it honestly and letting it argue back.
Next
Start with lab architecture and resource budget. It is the only document in this volume with no commands in it, and it is the one that decides whether the other three work.
After this volume, Volume 3 revisits the platform decision with the benefit of having actually run the thing, and Volume 7 picks up the CloudStack API that this volume leaves authenticated and waiting.