Projects · Mini Hostpapa
Host preparation and network fabric
Building the L0 layer on Ubuntu 24.04: nested virtualization, an Open vSwitch top of rack switch with real VLANs, SVIs, NAT to the internet, and NFS storage exports, all driven from one Ansible playbook.
Host preparation and network fabric
This is where the design meets a machine.
By the end of this document my laptop is no longer a laptop. It is a top of rack switch, an edge firewall, and a storage array wearing a trench coat, and it has nested virtualization turned on so it can pretend to be a rack of servers as well.
I could have done all of this with about fifteen commands typed into a terminal. It would have worked, and it would have taken twenty minutes.
I did not, and the reason is not tidiness. It is that I will get this wrong the first time. I will mistype a VLAN tag, discover it three hours later when a system VM will not start, fix it, and then have no record of what the working configuration actually is.
A fabric I can tear down and rebuild in ninety seconds is a fabric I am willing to experiment on. That is worth the extra effort of writing it as a playbook, and it is the difference between a lab and a house of cards.
My starting point, which is worth stating because it changes some commands:
| Fact about the host | Value | Consequence for this document |
|---|---|---|
| Distribution | Ubuntu 24.04.3 LTS (noble) | apt, and the CloudStack noble repository works later |
| Kernel | 7.0.0-28-generic (HWE) | Recent enough that nested virtualization and OVS need no special handling |
| CPU | Intel i7-1165G7, VT-x present | Nested virt is kvm_intel. On AMD substitute kvm_amd throughout |
| Network management | NetworkManager, active | This matters more than it looks. Section 5.1 is entirely about it |
| libvirt | 10.0.0 installed, service inactive | Already new enough for OVS portgroups, which need 1.1.10 or later |
1. The layout, so nothing ends up in a surprising place
One directory tree for everything, which is what makes the disk budget from the architecture document checkable with a single du.
My first instinct was /srv/minihp/images, so everything lab related sat in one place. That produces a permission denied error from QEMU that has nothing to do with permissions.
Ubuntu ships an AppArmor profile for libvirtd and qemu that whitelists specific paths. /var/lib/libvirt/images is on that list. An arbitrary directory is not, so QEMU is blocked from opening the disk it has been told to open, and the error message describes the file rather than the policy.
Putting VM disks in a subdirectory of the blessed path costs nothing and avoids writing an AppArmor override. The NFS exports can live anywhere, because they are reached over the network by a client rather than opened as files by QEMU.
2. The source of truth
Every number from the architecture document goes in one file. Nothing else in the lab hardcodes an IP address, which means a renumbering is one edit rather than a search.
primary is exported to 10.50.1.0/24 only. secondary is exported to 10.50.0.0/24 only. Neither lists both.
Volume 1 was emphatic about this and the reasoning is worth repeating, because the lazy version genuinely looks more convenient. Primary storage is mounted by the hypervisor, which sits on the storage subnet. Secondary storage is mounted by the Secondary Storage VM, which sits on the management subnet. Two different mounters, two different networks.
Export both to everything and it all works, right up until something on the management network writes to primary storage. Each export lists exactly the network whose clients have a legitimate reason to mount it. Anything broader is an availability decision disguised as a convenience.
The inventory is trivial, because at this stage there is exactly one machine:
The [rack] group is empty until document 3 creates those machines. Declaring it now means the CloudStack playbook in document 4 needs no inventory changes.
3. Packages, and enabling nested virtualization
If this one thing is not working, everything else in Volume 2 still appears to succeed. Packages install. The zone creates. The host registers and shows Up.
And then every single system VM fails to start, with an error about the VM rather than about the CPU.
So I verify it before building anything on top of it. This is the cheapest check in the volume and it saves the most time.
Here is the first half of the playbook. It installs what is needed and turns on nesting.
The assert task. Ansible would happily continue past a nesting failure and report a green run. That assert turns a silent misconfiguration into a loud stop, at the only point where the fix is obvious. Any prerequisite that fails invisibly deserves one of these.
The three bridge-nf-call-* sysctls. These say: when frames are switched by a bridge, do not haul them through the host's IP firewall. On a stock machine it changes little. On a machine that has ever run Docker it is the difference between a working lab and an afternoon spent debugging a network that is fine, because Docker's FORWARD chain is dropping frames that were never supposed to be its business.
Verify the foundation before going further:
Nothing later in this volume will work, so stop here.
Try sudo modprobe -r kvm_intel && sudo modprobe kvm_intel. If the module is busy, something is already using KVM: close VirtualBox, VMware, or any running VM, and try again. If it still says N, reboot so the modprobe.d file is read from a clean start.
Then check the firmware. Some laptop BIOSes expose VT-x but hide the nested extensions. grep -c vmx /proc/cpuinfo returning 0 means virtualization is disabled in firmware entirely, which is a reboot into BIOS rather than anything Linux can fix.
4. Getting NetworkManager out of the way
This section exists because of one line in my host survey: NetworkManager is active. On an Ubuntu desktop it manages every interface it can see, and it is about to see a switch.
NetworkManager notices tor-mgmt appear, decides it is an unconfigured Ethernet device, and helpfully starts a DHCP client on it.
There is no DHCP server, so after a timeout NetworkManager gives up and assigns a link-local 169.254.x.x address. Meanwhile my fabric script has already put 10.50.0.1/24 there. Now the interface has two addresses, one of which is wrong, and the routing table has an entry for 169.254.0.0/16 that was not there before.
Worse, it is intermittent. Sometimes my address wins because the script ran last. Sometimes NetworkManager's renewal cycle clobbers it minutes later. A lab that works and then stops working while you are not touching it is the single most demoralising kind of broken, and this is a common cause of it.
The fix is to tell NetworkManager that these interfaces are not its business:
And the matching handler:
tor-* covers the SVIs and lab-tor covers the switch, but vnet* is the pattern libvirt uses for the tap interfaces it creates when a VM starts.
Those come and go every time a VM boots. NetworkManager treating each one as a new network to configure produces log noise, occasional stalls while it probes, and desktop notifications about network changes every single time I start the lab. Excluding them is what makes the lab quiet.
5. The switch
Now the interesting part. One OVS bridge becomes sw-tor-01.
Before any commands, here is the whole thing on one page. This is the picture I wish someone had drawn for me before I started typing ovs-vsctl, because once you can see it, every command in the rest of this section is obviously just one line of this diagram:
The blue boxes are not VMs. tor-mgmt, tor-stor and tor-pub are OVS internal ports, which means the laptop's own network stack is plugged into the switch three times, once per VLAN. That is exactly what an SVI is on a real switch, and it is why the host can be the pod gateway, the firewall and the storage array all at once without any of those roles seeing each other's traffic untagged.
cs-mgmt-01 and kvm-host-01 are attached differently on purpose. The management server gets an access port and never knows VLANs exist. The hypervisor gets a trunk and does the tagging itself, because that is what CloudStack expects from a KVM host. Same bridge, two completely different contracts.
tor-stor is the only thing at MTU 9000. Jumbo frames live on the storage VLAN and nowhere else, which is the Volume 1 design. If you set 9000 everywhere you will spend an evening debugging why the internet half works.
5.1 Why this is a systemd service rather than a config file
I went back and forth on this. Ubuntu has netplan, netplan supports OVS, and a declarative YAML file would have been the elegant answer.
I chose a templated script behind a systemd unit instead, for three reasons:
| Reason | Detail |
|---|---|
| Netplan renders through NetworkManager here | Section 4 just spent effort telling NM to ignore these interfaces. Routing the fabric back through NM would undo that |
I need up and down as first class operations | Lab mode means starting and stopping the fabric deliberately. systemctl start and stop express that directly; netplan has no concept of "off" |
| OVS internal ports with access tags are awkward in netplan | Netplan models OVS VLANs as fake bridges. ovs-vsctl expresses "internal port, tag 20" in one line that reads like the switch config it is standing in for |
The script is generated from fabric.yml, so the source of truth is still declarative YAML. What runs is imperative; what I edit is not. That is the tradeoff I am happy with, and it is genuinely how most production network automation works.
5.2 The fabric script
--may-exist alone is not enough, and this is the mistake I would have made. If the port already exists with the wrong tag, --may-exist add-port succeeds and changes nothing, so editing fabric.yml and re-running would appear to work while the switch kept its old configuration.
The explicit set port is what makes the script converge rather than merely not fail. Same story with ip addr replace against ip addr add: the second one stacks a new address on every run until the interface has six of them.
Idempotent means "running it twice gives the same result as running it once", not "running it twice does not error".
5.3 The unit, and the tasks that install it
libvirt validates its network definitions at startup. The lab-tor network says "use the existing bridge lab-tor", and if that bridge does not exist yet, libvirt marks the network inactive and stays that way until something tells it to try again.
The symptom is a VM that refuses to start with a message about a missing network, on a host where ovs-vsctl show proves the bridge is right there. It exists now; it did not exist when libvirt looked.
Before=libvirtd.service makes the ordering explicit instead of leaving it to luck about which service systemd happened to start first.
The remaining playbook tasks:
6. The edge firewall
fw-edge-01 is a few nftables rules. The important design decision is where they live.
Volume 1 spent a long time on default deny, and here I am writing policy accept. That is not an oversight, and it is not laziness.
The firewall that matters in this lab is not this one. Tenant traffic is filtered by the CloudStack Virtual Router and by security groups, at L2, which is exactly where a real deployment filters it. Those are the rules Volume 1 designed and the ones worth testing.
This table's only job is to be the outer NAT, standing in for owning routable address space. Making it default deny would mean maintaining a second, redundant policy that can only ever produce false failures I would then debug as CloudStack problems.
Simulating a control you are not testing does not make a lab more realistic. It makes it noisier. This is deviation 8 in the register, and it is deliberate.
6.1 The Ubuntu trap Volume 1 warned about
Volume 1 flagged ufw and DEFAULT_FORWARD_POLICY. Here is the version that applies to L0, and it is worth checking before it costs an evening.
Read those three outputs together. ufw is inactive, so the obvious suspect is innocent. And forwarding is still policy DROP, because Docker set it, and Docker leaves it that way whether or not its daemon is running.
So the symptom is: the laptop reaches everything, the SVIs answer pings, and nothing routes between VLANs. Every instance can reach its own gateway and nothing beyond it.
I would have blamed CloudStack. The bridge-nf-call-iptables = 0 sysctl from section 3 protects bridged frames, but VLAN to VLAN traffic is genuinely routed, so it goes through FORWARD and gets dropped.
Two ways to fix it, and the choice matters:
Insert an accept rule for lab traffic ahead of Docker's chain, and leave the default policy alone:
This is the version I use, because it is narrow. Only interfaces whose names start with the lab prefix are affected, so the machine's actual security posture for every other interface is untouched.
Add these to the fabric script's up() so they come and go with the lab rather than living on the host permanently.
7. The storage array
nfs-store-01 becomes two directories and two export lines. This is short, and three of the export options are worth more than the brevity suggests.
no_root_squash is mandatory, not a shortcut. By default NFS maps a client's root to nobody. The hypervisor writes instance disk images as root, so with the default it cannot create a single volume. CloudStack documentation requires this and it is the most common reason primary storage "mounts fine but nothing works".
async is a deliberate lab tradeoff. The server acknowledges writes before they hit the platter, which is meaningfully faster and means a host crash can lose recent writes. Volume 1 specified sync, and production should keep sync. I take async here because nested virtualization is already slow and losing a lab volume costs me a terraform apply. This is a deviation and it belongs in the register.
no_subtree_check avoids a permission check that breaks when files are renamed while open. Standard for storage exports.
fsid gives each export a stable identifier. Without it, exports of directories on the same filesystem can confuse clients across a server restart, producing stale handle errors that look like network problems.
This is the bit I want to draw attention to, because a passing result looks like a broken one.
Asking 10.50.1.20 (the storage address) shows only primary. Asking 10.50.0.210 (the management address) shows only secondary.
If either command listed both exports, the client restriction is not working and I have quietly built the "export everything to everyone" configuration that Volume 1 argued against. Two commands with different output is the pass condition.
8. Teaching libvirt about the switch
The last piece of L0. This is the file that gives me declarative access ports and trunk ports, and it is the reason Open vSwitch won the tool selection.
<portgroup name='mgmt-access'> with a single tag. This is an access port, declared in a file. It is why cs-mgmt-01 has zero VLAN configuration anywhere in its cloud-init. From inside that VM, enp1s0 is an ordinary NIC on an ordinary network. The VLAN exists entirely in the switch, which is the correct division of responsibility and the one people most often get wrong by configuring VLANs on servers that do not need them.
<vlan trunk='yes'> with many tags. This is a trunk port, and it is the piece that makes CloudStack possible. The agent will create enp2s0.40, enp2s0.101, and so on, and those frames need to survive the trip to the switch. Every VLAN the agent might ever use has to be listed here in advance, which is why the guest range is in fabric.yml as a range rather than sprinkled through the config.
Suppose I set guest_vlan_last: 199 and later let CloudStack allocate VLAN 250.
CloudStack creates the network successfully. The Virtual Router starts successfully. The instance boots successfully. And there is no connectivity, with no error anywhere in any log, because the frames are tagged 250 and the trunk port silently drops what it was not told to carry.
Every layer reports success. The drop happens in the switch, which does not log it and which CloudStack cannot see.
Diagnostic if you ever meet this: ovs-vsctl get port <trunk-port> trunks and check the VLAN is in the list. sudo ovs-ofctl dump-flows lab-tor will show the drop counters increasing.
9. The verification gate
Volume 1 ended each design with a gate. Same principle here: do not build document 3 on top of an unverified document 2.
Command one proves three internal ports exist with the right tags. A port with tag=[] is a trunk port, and finding one of my SVIs in that state would mean it receives every VLAN instead of one.
Command two is MTU 9000 on the storage port only. Volume 1 was clear that MTU must be consistent along a path, and this is the first hop of that path.
Command four counts VLAN tags in the libvirt definition. 103 is the number to expect: one for VLAN 20 in mgmt-access, then 30, 40, and 100 through 199 in rack-trunk. A different number means my Jinja loop is wrong, which is far easier to catch here than by debugging a tenant network in three documents' time.
Command six is the one I would skip and should not. 13,940 MiB available against a requirement of 13,614. This is the moment to find out I am short, not after Terraform has allocated 11 GiB.
The claim is that VLANs 20, 30, and 40 are genuinely separate L2 domains. Since all three SVIs are on the same host, the routing table would let them talk regardless, so a plain ping proves nothing.
The honest test is to watch a tagged frame:
What proves it: tcpdump output containing vlan 30, p 0, ethertype IPv4. The tag is on the wire, which means OVS is doing 802.1Q rather than routing between interfaces that happen to have different subnets.
10. Running it, and the lab switch
For entering and leaving lab mode, two aliases save a lot of friction:
lab-up printing available memory is deliberate. It puts the number I am most likely to be wrong about in front of me at the moment I start the lab, rather than in a document I read once.
11. What went wrong, honestly
Recorded because the next person to build this, including future me, will hit the same things.
| Symptom | Actual cause | How I found it |
|---|---|---|
| SVI has a 169.254 address alongside the right one, intermittently | NetworkManager running DHCP on an OVS internal port | nmcli device status showed tor-mgmt as managed. Section 4 exists because of this |
| Everything pings, nothing routes between VLANs | iptables FORWARD policy DROP, left behind by Docker, with ufw reporting inactive | iptables -L FORWARD -n. I checked ufw first and it lied by omission |
virsh net-start lab-tor fails saying the bridge does not exist | libvirtd started before the fabric service | Bridge was present in ovs-vsctl show. Fixed with Before=libvirtd.service |
ping -M do -s 8972 fails on the storage VLAN | MTU set on the internal port but not on the OVS bridge datapath | ip link show lab-tor reported 1500. Fixed with mtu_request on the bridge as well as the port |
| QEMU permission denied on a disk with correct ownership | AppArmor, because images were outside /var/lib/libvirt/images | dmesg | grep -i apparmor. Section 1 exists because of this |
Not one of those was a mistake in the design. Every single one was something else on the machine having an opinion: NetworkManager wanting to manage interfaces, Docker having set a firewall policy months ago, AppArmor enforcing a path whitelist, systemd choosing a start order.
The lab was never fighting the design. It was fighting the host.Which is itself the lesson, and it transfers directly to real hardware. A production build fails for the same category of reason far more often than for a genuinely wrong architecture: some other subsystem was already doing its job and nobody accounted for it.
Next
L0 is real. There is a switch with genuine VLANs, a firewall that NATs to the internet, a storage array with correctly scoped exports, and a kernel that will let a guest be a hypervisor.
Provisioning the rack builds L1 on top of it: two servers, declared in Terraform, configured by cloud-init, attached to the portgroups defined in section 8.