Projects · Mini Hostpapa
Provisioning the rack
Two servers declared in Terraform against the libvirt provider, configured by cloud-init, attached to real VLAN access and trunk ports. Including the bridge design CloudStack requires.
Provisioning the rack
L0 is a switch, a firewall, and a storage array. Now it needs servers plugged into it.
Two of them: cs-mgmt-01 and kvm-host-01, with the addresses Volume 1 specified, attached to the portgroups defined in the previous document. By the end of this one I can SSH into both, and kvm-host-01 has cloudbr0 and cloudbr1 configured exactly as Volume 1 drew them.
I want to be honest that this is over-engineering measured against the immediate task. Two virt-install commands would produce the same two VMs faster.
Three reasons I did it anyway, and only the third is really about the lab:
One. I am going to break these servers. Repeatedly. A misconfigured bridge is much easier to fix by destroying and recreating than by unpicking. The value is not in creating the rack, it is in creating it for the eleventh time.
Two. Volume 7 is a Terraform project against the CloudStack provider. Meeting Terraform's failure modes now, on infrastructure I can destroy without consequence, is much cheaper than meeting them for the first time while also learning a new provider.
Three. The declaration is the documentation. A virt-install command lives in shell history and is gone. This file states what the rack is, and I can diff it against Volume 1.
The dmacvicar/libvirt provider had a complete schema rewrite at 0.9.0. The nested structures are generated from the libvirt XML schema now, so almost every tutorial and Stack Overflow answer you will find describes the old 0.8 schema and will not work.
The most common differences: source = { volume = { pool = ..., volume = ... } } instead of a flat source, portgroup became port_group, and value/unit pairs split into two attributes such as memory plus memory_unit.
Everything in this document is written against 0.9.8 and validated with terraform validate against the real provider schema. If you are on 0.8.x, none of it applies.
1. Base images
Two downloads, and one of them is doing more work than it looks.
ubuntu-24.04-cloudimg.qcow2 is roughly 600 MB on disk with a 3.5 GiB virtual size.
alpine-3.24-cloudinit.qcow2 is roughly 190 MB.
The Ubuntu image is shared as a backing file by both servers, so I pay for it once rather than twice. Each server's own disk stores only the blocks where it differs from that base, which for a freshly installed server is a few hundred megabytes.
This is why 44 GiB of allocated disk costs 13 GiB of real disk. Not a trick, just how qcow2 works, and the single reason this lab fits.
The base image is read only, forever, for as long as any overlay depends on it.
If I ever boot the base image directly to "just check something", or run qemu-img resize on it, every overlay built on top of it is silently corrupted. Not immediately: the guests keep running from page cache, and then produce filesystem errors later that look like disk failure.
The rule I follow: nothing ever writes to base/. If I want to poke at a base image, I make a throwaway overlay of it first. Mode 0644 and a directory called base are there to make the intent obvious to future me at 2am.
2. Project layout
libvirt has two daemons. qemu:///session runs as your user, qemu:///system runs as root.
They have completely separate sets of networks, storage pools, and domains. Document 2 created lab-tor and the minihp pool system-wide with sudo virsh.
Point Terraform at qemu:///session and it reports that the network lab-tor does not exist. Which is true, in that namespace, while sudo virsh net-list in another terminal cheerfully shows it as active. Two commands, two truthful answers, two different worlds.
For this to work without sudo terraform, your user needs to be in the libvirt group, which document 2 did. Log out and back in for it to take effect, or the connection is refused for a reason that looks like a URI problem.
3. The network configuration, which is the real content of this document
Everything else here is boilerplate. This is the part that has to be right, because it is Volume 1's hypervisor diagram turned into a file.
3.1 The management server, deliberately boring
There is no VLAN configuration. No tag, no sub-interface, nothing that mentions VLAN 20.
And yet this server sits on VLAN 20. The tagging happens in the mgmt-access portgroup on the OVS switch, which is what an access port means: the switch tags on the way in and strips on the way out, and the attached device never knows.
If this file mentioned VLAN 20, I would have built it wrong. This is the misconfiguration I have seen most often in real deployments: VLANs configured on servers that are plugged into access ports, which works by accident when the tag matches and breaks confusingly when it does not.
3.2 The hypervisor, where Volume 1 gets tested
One: cloudbr1 has no addresses key at all. Volume 1 was emphatic and the reasoning is worth restating: an address there would place the hypervisor itself inside a network that customer traffic crosses. A bridge with no IP is a bridge the host cannot be attacked through.
Two: stp: false and forward-delay: 0 on both. Spanning tree on a host bridge is pointless, because there is no loop to prevent, and forward delay means a newly attached interface sits in listening state for 15 seconds before passing frames. With CloudStack attaching interfaces constantly as VMs start, that delay turns into system VMs that appear to hang on boot. This one is easy to miss because the symptom is slowness rather than failure.
Three: trunk0 is MTU 9000 while cloudbr1 is MTU 1500. That asymmetry looks like a bug and is deliberate. The storage VLAN needs 9000 on the physical interface. Guest traffic wants 1500, because it crosses the internet where 1500 is the reality. Section 3.3 explains why the two do not conflict.
Four: the storage VLAN has no gateway. There is no routes key under trunk0.30. Storage traffic physically cannot leave the subnet, which is a stronger guarantee than a firewall rule.
3.3 Why a 9000 interface under a 1500 bridge is fine
This bothered me enough that I want to record the resolution, because it is the same kernel behaviour the architecture document dug into.
trunk0 is a member of bridge cloudbr1, and trunk0.30 is a VLAN sub-interface of that same trunk0. My worry was that frames arriving on trunk0 would all go to the bridge, so trunk0.30 would never see anything and 9000-byte storage frames would hit a 1500-byte bridge and be dropped.
That is not what happens, because of the ordering inside __netif_receive_skb_core: vlan_do_receive() runs before the bridge's rx_handler.
Follow the middle branch. When CloudStack needs public VLAN 40, the agent creates trunk0.40 plus a bridge for it. Those frames are claimed by trunk0.40 and never reach cloudbr1 either.
Which reframes what cloudbr1 actually is. It is not "the bridge carrying public and guest traffic". It is a declaration of which physical interface the agent should hang VLAN sub-interfaces off.
That is why the traffic label is just a string CloudStack writes down and uses later, and why cloudbr1 correctly has no IP address and, in a working zone, carries almost no traffic at all.
A bridge that looks idle is the correct state. I would have spent time investigating that.
3.4 The rest of cloud-init
CloudStack is genuinely intolerant of clock skew. The management server and the agent exchange timestamped messages, and certificate validation for the agent connection on port 8250 depends on the clocks agreeing.
The failure mode is memorable: the host registers, goes Up, and then flaps between Up and Down for no visible reason, or the agent fails TLS validation with an error about certificates that sends you looking for a certificate problem.
On a laptop this is more likely than in a datacenter, because suspending the lid stops the guest clocks and they resume convinced it is still yesterday. chrony is what makes suspend and resume survivable.
4. The servers
This is the line the entire volume rests on. The default CPU model presents a generic, portable virtual CPU to the guest, and a generic virtual CPU does not expose the VMX flag.
Leave it at the default and here is what happens: kvm-host-01 boots perfectly. Ubuntu installs perfectly. cloudstack-agent installs and registers, and the host shows Up in the CloudStack UI. Everything is green.
Then every system VM fails to start, and the error talks about the VM.
The failure appears four steps after the cause, in a component that is not at fault. migratable = false is required alongside it, because a passed-through host CPU cannot be guaranteed to exist on another machine, and libvirt refuses the combination otherwise.
Generated MACs change every time Terraform recreates a domain. The netplan config matches interfaces by MAC, so a regenerated MAC means netplan matches nothing, the server boots with no network configuration, and it is unreachable over SSH.
Pinning them means terraform destroy followed by terraform apply produces a server with the same identity. The last octets are not random either: 00:c9 is 201 and 00:65 is 101, matching the addresses. When I am reading a packet capture at 1am, a MAC address that tells me which host it is saves real time.
That second output exists so the number I am most likely to be wrong about is printed by the tool that is about to allocate it.
5. Applying it
terraform apply is about to ask for 11,264 MiB. Close the IDE and the browser, then check:
If that number is below about 13,600, do not apply. The VMs will start, and then the OOM killer will reap something at an unpredictable moment and you will debug the wrong problem.
Terraform finishes in seconds; cloud-init takes a couple of minutes. Watch it rather than guessing:
The first time I built this, kvm-host-01 never came up on SSH. With no console I would have been guessing between a netplan error, a cloud-init failure, and a switch problem.
The console showed cloud-init reporting a YAML parse failure on my network config, because I had a variable interpolating to an unquoted value that YAML read as something else.
A serial console turns an unreachable VM from a mystery into a log file. Five lines in the Terraform, and it is the difference between debugging and guessing.
6. The verification gate
Command two is the one that matters most. /dev/kvm exists and 8 CPUs report vmx. Nested virtualization genuinely works. If this fails, stop and fix host-passthrough before doing anything else, because everything downstream will lie to you about the cause.
Command three: cloudbr1 shows UP and nothing else. No address. An empty address column is the passing condition, and it looks exactly like a truncated command. If there is an address there, something assigned one and Volume 1's design has been violated.
Command four proves the bridges have the right members. A bridge can exist, be up, and be completely empty. That failure is invisible to command three.
Command five is the jumbo frame path, end to end, from inside the guest, across the OVS trunk, to the simulated array. Every hop agreed on 9000.
Command six proves both storage and internet work from the hypervisor, which are the two things the Secondary Storage VM will need in document 4.
The gate proves connectivity. It does not prove the trunk is a trunk. For that, watch a tagged frame arrive on the switch:
What proves it: output containing vlan 30, p 0, ethertype IPv4. The tag is on the wire between guest and switch, which means the rack-trunk portgroup is genuinely trunking rather than passing untagged frames that happen to work.
7. The fallback, if the provider fights you
The 0.9.x provider is a recent rewrite. If it misbehaves, the lab should not be blocked on it, so here is the same rack without Terraform.
Tooling is a means. If the provider has a bug on your version of libvirt, spending the evening debugging a Terraform provider instead of learning CloudStack is optimising the wrong thing.
You lose the destroy-and-rebuild property, which is a real loss. You do not lose the lab.
8. What went wrong
| Symptom | Actual cause | How I found it |
|---|---|---|
| Every Terraform example from the internet failed to parse | Provider 0.9.x is a generated-schema rewrite; the tutorials describe 0.8.x | terraform providers schema -json. Reading the actual schema was faster than trying examples |
Error: attribute "path" is required on the console block | Setting source = { pty = {} } means "a pty at this path" and the path is mandatory | Omitting source entirely lets libvirt allocate a pty, which is what I wanted |
Terraform says network lab-tor does not exist while virsh shows it active | Provider connected to qemu:///session, objects defined in qemu:///system | Two libvirt namespaces, both answering truthfully. See the callout in section 2 |
| Server boots with no network at all after a rebuild | libvirt generated a new MAC, so netplan's match matched nothing | Serial console showed the NIC present and unconfigured. Fixed by pinning MACs |
| System VMs would fail later with an unrelated error | CPU mode left at default, so no VMX in the guest | Caught early only because /dev/kvm is in the gate. This is why it is in the gate |
Document 2's failures were all the host having pre-existing opinions: NetworkManager, Docker, AppArmor.
This document's failures were all me holding a stale mental model: an old provider schema, an assumption about libvirt namespaces, an assumption that MACs are stable.
Different category, different fix. The first kind is solved by surveying the machine before building. The second is solved by reading the actual schema instead of the first example that looks close. Both are cheaper than debugging forward from the symptom.
Next
The rack is up. Two servers with the addresses Volume 1 specified, cloudbr0 and cloudbr1 configured exactly as it drew them, a working jumbo frame path to storage, and a hypervisor that can genuinely run hypervisor workloads.
Installing CloudStack and the first instance puts the platform on top: management server, zone, pod, cluster, the host added, system VMs green, and an Alpine instance with an IP address it got from a Virtual Router.