Installing CloudStack and the first instance

This is the document where Volume 1 gets marked.

Every number in the zone wizard comes from the Volume 1 IP plan, typed in unchanged. If the design was right, the system VMs come up. If a VLAN, a traffic label, an export path, or an MTU was wrong, CloudStack will tell me, in the indirect and slightly hostile way that distributed systems do.

This plan is applied and ready to use

All of the planification in this document (and the rest of Volume 2) is implemented as runnable Ansible, Terraform, and CloudMonkey helpers in mini-HostPapa/virt-dc. Clone that repo, follow its README, and you get the fabric, the rack VMs, CloudStack, the zone path, and the first-instance gate without retyping these snippets by hand.

Why the Secondary Storage VM is the real exam

There is one moment in this document that matters more than the rest, and it is when the Secondary Storage VM reaches Running.

Look at what has to be simultaneously correct for that to happen. It needs an IP from the pod reserved range on the management network. It needs to mount secondary storage over NFS, which means the export, the client restriction, and the routing are right. It needs a public IP from VLAN 40, which means the trunk carries VLAN 40 and the CloudStack agent created the sub-interface. It needs to reach the internet through the simulated edge firewall NAT to download a template. And the management server needs to be talking to the agent over port 8250.

One VM reaching Running validates the entire stack from OVS tagging up to internet egress. Nothing else in this volume is such a good test, which is why the gate in section 8 is built around it.

1. What goes where

Component	Lands on	Notes
MySQL 8	`cs-mgmt-01`	Same host as the management server. Production separates them
`cloudstack-management` 4.22	`cs-mgmt-01`	Java 17. JVM heap must be capped or it will fight MySQL for 4 GiB
`cloudstack-agent` 4.22	`kvm-host-01`	Section 4 is entirely about Ubuntu 24.04 specific configuration
System VM template	Secondary storage on the laptop	Roughly 3 GiB. Seeded before the zone is created
`cmk` (CloudMonkey)	The laptop	So the zone build is a script rather than 40 clicks

2. The management server

Two numbers in there are the difference between working and mysteriously dying

-Xmx2048m. Without an explicit cap, the JVM sizes its maximum heap as a fraction of visible RAM and will happily plan to use most of the 4 GiB. MySQL is also on this box. They do not negotiate, and the loser is whichever process the OOM killer reaches first, which is usually MySQL because it has the larger resident set.

The symptom is brutal to diagnose: CloudStack starts, runs for a while, then the API stops answering and the log shows database connection errors. So you investigate the database, which is fine, because it is dead for a reason that has nothing to do with the database.

innodb_buffer_pool_size = 256M. Same fight from the other side. The lab has one user and a few dozen rows that matter, so a large buffer pool buys nothing and costs a lot.

Neither of these belongs in production. Both are in the deviation register.

On the passwords being in plain text

labroot and labcloud, written in a file, committed to a repository.

I want to be explicit that this is a deliberate lab-only decision, not an oversight, because unexamined shortcuts are how bad habits get established. The lab is unreachable from the internet, it holds no real data, and it exists to be destroyed.

Volume 22 covers Vault, and Volume 7's Terraform will need real secret handling for the CloudStack API keys. The rule I hold myself to is that a plaintext password must never outlive the thing it protects, and this environment is designed to be thrown away.

3. Seeding the system VM template

CloudStack cannot start a single system VM without this, and the mistake is easy: seed it before creating the zone.

Why the mount source is 10.50.0.210 and not 10.50.1.20

Both addresses are the same laptop. Using the wrong one still works, and that is precisely the problem.

Volume 1 established that secondary storage belongs to the management network, because the Secondary Storage VM lives there, and the export in document 2 restricts /srv/minihp/secondary to 10.50.0.0/24. Mounting via 10.50.1.20 would be refused, which is the export restriction doing its job.

This one line proves the whole two-export design is working. If I could mount secondary storage from the storage subnet, I would have built the "export everything to everyone" configuration and not noticed.

4. The KVM agent, and the Ubuntu 24.04 problem

This section is short and it is the one most likely to cost you an evening, because Ubuntu 24.04 needs configuration that older guides do not mention.

The two tasks that only exist because this is Ubuntu 24.04

Modern libvirt splits into per-driver daemons (virtqemud, virtnetworkd, virtstoraged) with socket activation, instead of one monolithic libvirtd. Ubuntu 24.04 ships that modular layout by default.

CloudStack's agent expects the monolithic daemon. Skip these two tasks and the agent starts, connects, and then fails to perform operations, with errors about connecting to libvirt on a host where virsh works perfectly from your shell.

This is the single most Ubuntu-24.04-specific thing in the whole volume, and it is documented in one sentence in the official install guide that is very easy to read past. If you are following an older tutorial, it will not be mentioned at all.

On disabling AppArmor for libvirt, which I did not enjoy

Turning off a security control is not something I want to do casually, so here is the honest position.

The official CloudStack documentation requires it. The reason is that CloudStack manipulates disk images, bridges, and VLAN interfaces in ways the stock libvirt AppArmor profile forbids, and virt-aa-helper cannot generate a profile that covers it.

Is this acceptable in production? It is what the project requires, so it is what deployments do. The mitigation is not AppArmor on the hypervisor, it is network segmentation, which is exactly what Volume 1 spent a whole document designing. The hypervisor is protected by being unreachable from anywhere untrusted, not by a local policy.

A control you have to disable to make the system work was never the control protecting you. Recognising that, and being able to name what actually is protecting you, is the difference between accepting a requirement and being careless.

5. The zone, from the Volume 1 plan

Now the part that grades the design. Every value below is copied from Volume 1.

Setting	Value	Volume 1 reference
Zone type	Advanced	Basic cannot do Source NAT, and the choice is permanent
Zone name / DNS	`minihp-zone-1` / `1.1.1.1`	Deviation 7: production uses `10.50.0.11` and `.12`
Hypervisor	KVM	The Volume 3 decision
Management traffic label	`cloudbr0`	Must match the bridge name exactly. No validation on typos
Public traffic label	`cloudbr1`	VLAN 40, tagged by CloudStack
Guest traffic label	`cloudbr1`	VLANs 100 to 199, tagged by CloudStack
Storage traffic type	Not defined	Deliberate. The SSVM then uses the management network, which is what the exports assume
Guest VLAN range	`100-199`	Deviation 5: production is 100 to 999
Pod gateway / netmask	`10.50.0.1` / `255.255.255.0`	The `tor-mgmt` SVI on the laptop
Pod reserved system IPs	`10.50.0.20` to `10.50.0.79`	Verbatim from Volume 1
Public IP range	`203.0.113.32` to `.60`, gw `203.0.113.1`, VLAN `40`	Deviation 6: production goes to `.190`
Primary storage	NFS `10.50.1.20:/srv/minihp/primary`	Storage VLAN, MTU 9000, mounted by the hypervisor
Secondary storage	NFS `10.50.0.210:/srv/minihp/secondary`	Management VLAN, mounted by the SSVM
Host	`10.50.0.101`, user `root`	CloudStack configures the agent over SSH

The Storage traffic type row is the one to slow down on

Leaving it undefined looks like an omission on a form that offers it. It is the decision Volume 1 argued for, and the reasoning is worth re-deriving rather than trusting.

The CloudStack traffic type called Storage has nothing to do with primary storage. It exists solely to tell CloudStack which network to attach the Secondary Storage VM's NIC to. Primary storage is mounted by the hypervisor using ordinary Linux routing, and CloudStack is not in that path at all.

Undefined means the SSVM lands on the management network. My secondary export is restricted to 10.50.0.0/24. Those two facts have to agree.

Define a Storage traffic type on VLAN 30 and the SSVM would get a 10.50.1.x address, then fail to mount an export restricted to 10.50.0.0/24. The error would be a permission denied on a mount, three layers away from the checkbox that caused it.

5.1 Doing it as a script rather than 40 clicks

The UI wizard at http://10.50.0.201:8080/client works, and I used it the first time because seeing the form teaches you what CloudStack is asking for. But I got it wrong twice, and re-clicking 40 fields is how you introduce a typo you cannot find.

Two ordering constraints in that script that are not obvious

The Virtual Router provider must be enabled before any guest network can work. It is not enabled by default and the UI wizard does it for you invisibly. Miss it in a script and networks create successfully, then no instance ever gets an IP address, because nothing exists to be its gateway.

allocationstate=Enabled goes last. Enabling the zone is what triggers system VM creation. Enable it before storage is registered and CloudStack tries to start system VMs with nowhere to put their disks, which produces failures you then have to clean up before retrying.

6. Global settings

A few defaults are wrong for a laptop. These need a management server restart.

The host setting is a documented production trap and it is worth internalising now

host tells every agent where to reach the management server. In this lab there is one server, so 10.50.0.201 is correct.

In production, Volume 1 specified two management servers behind VIP 10.50.0.200. If host points at 10.50.0.201, then everything works perfectly, and the second management server buys you nothing at all, because every agent is talking to the first one by name.

You discover this during the failover you built the pair to survive. This is the kind of thing that only becomes real once you have typed the setting yourself and seen what it does.

Optional but worth it, the system offering tuning from the architecture document:

7. Watching the system VMs, which is the actual exam

Enabling the zone starts the exam. CloudStack now tries to create the Secondary Storage VM and the Console Proxy VM, and to do it it has to exercise every single thing built in documents 2 and 3.

Preparing diagram

Watching the exam

adam@laptop:~$▊

Command four is the payoff for the whole VLAN design, and it is worth staring at

I never configured trunk0.40. I typed the string cloudbr1 into a form, and the CloudStack agent worked out that cloudbr1 sits on trunk0, then created a VLAN 40 sub-interface and a bridge for it.

This is the traffic label mechanism from Volume 1, working, visible, in one command. I had written that a traffic label is "just a string CloudStack writes down and uses later". Here is the later.

Note also that cloudbr1 itself is absent from any traffic. Volume 1 said it would hold no address and I found that odd at the time. Now it is obvious: nothing rides cloudbr1, because every VLAN gets claimed by its own sub-interface first.

If a system VM is stuck in Starting, check these in this order

The order matters, because it goes cheapest first and follows the diagram above.

Is the agent connected? On cs-mgmt-01: sudo grep -i 'agent' /var/log/cloudstack/management/management.log | tail -20. If the agent never connected, this is the libvirtd modular mode problem from section 4, not a networking problem.
Did the VLAN interface get created? On kvm-host-01: ip -br link | grep trunk0.40. Missing means the traffic label does not match a real bridge. Check for a typo like cloudbr01.
Is the template actually there? ls /srv/minihp/secondary/template/tmpl/1/3/ on the laptop. Empty means section 3 did not finish, and it takes a long time on a laptop uplink.
Can the SSVM reach the internet? SSH into it via the link-local control channel: ssh -i /var/lib/cloudstack/management/.ssh/id_rsa -p 3922 root@169.254.x.x, then try curl -sI https://download.cloudstack.org/. Failure here is the NAT or the FORWARD policy from document 2.
Is it just slow? Nested virtualization is 2 to 5 times slower. If a VM eventually goes Running after CloudStack has already declared a failure, raise the wait setting rather than chasing a configuration bug.

After fixing a traffic label, fixing it is not enough. System VMs created against the broken label must be destroyed so CloudStack rebuilds their interfaces: cmk destroy systemvm id=<id>. This caught me out, and Volume 1 had warned about it.

8. The first instance

Now the small part, which feels anticlimactic and is the point of everything before it.

Why the template URL points at 10.50.0.1:8000

The Secondary Storage VM downloads templates over HTTP, so the image has to be served from somewhere it can reach. 10.50.0.1 is the laptop's management SVI, so a throwaway web server does the job:

This is also why secstorage.allowed.internal.sites=10.50.0.0/24 was set in section 6. Without it the SSVM refuses to download from private address space, which is a sensible default that happens to block the only source available in a lab.

The first instance

adam@laptop:~$▊

Four things worth reading off that output

r-4-VM appeared. That is the Virtual Router, created automatically because the guest network needed a gateway. I did not ask for it.

trunk0.101 appeared. CloudStack allocated VLAN 101 from the 100 to 199 range and the agent created the sub-interface. The trunk portgroup from document 2 is carrying it, which is exactly the thing that would have failed silently if the range had been wrong.

The instance has 10.1.1.12 with gateway 10.1.1.1. That gateway is the Virtual Router, not the laptop. Tenant traffic is routed by CloudStack, and the laptop has no address in that network at all, which is what Volume 1's segmentation design asked for.

2,946 MiB still available inside kvm-host-01. The RAM ledger held. Room for roughly fifteen more Alpine instances, on a laptop that also runs my IDE at other times.

9. The Volume 2 gate

Passing looks like: 8 resources recreated with no manual steps, a ceiling under 50 GiB, an actual figure around 20 GiB, and /dev/kvm present.

10. What Volume 1 got right, and what it got wrong

This was the real purpose of Volume 2, so it deserves a straight answer rather than a victory lap.

Volume 1 claim	Verdict	What the lab actually showed
A traffic label is just a string CloudStack writes down and uses later	Correct, and I had not understood it	Watching the agent create `trunk0.40` from the string `cloudbr1` made the mechanism concrete. I could recite this before; now I can predict it
`cloudbr1` should hold no IP address	Correct, for a better reason than stated	Volume 1 gave a security reason, which is true. The deeper reason is that VLAN sub-interfaces claim frames before the bridge sees them, so `cloudbr1` carries almost nothing anyway
Storage on a VLAN sub-interface of a bridged trunk	Correct, and I doubted it	I was ready to file this as a bug. The kernel calls `vlan_do_receive()` before the bridge `rx_handler`, so it works, and that ordering is what CloudStack's whole KVM model depends on
Two exports, never overlapping client lists	Correct and load bearing	Proven by the fact that mounting secondary storage from the storage subnet is refused. A single permissive export would have hidden a real design error
Leave the Storage traffic type undefined	Correct, and more consequential than it read	On paper it was a checkbox. In practice it determines which subnet the SSVM lands in, which has to agree with the export restrictions
The `host` global setting must be the VIP	Correct, and now visceral	Typing a single server address and seeing every agent bind to it made obvious why an HA pair with this wrong is worthless
Port matrix: 8250 for the agent, 8080 for the UI	Incomplete	Volume 1 listed ports but not that libvirt must be in monolithic mode. On Ubuntu 24.04 the port being open is not sufficient, and this cost real time
Bridge parameters	Missing entirely	Volume 1 never mentioned `stp` or `forward-delay`. A 15 second forward delay makes system VMs appear to hang on boot. Volume 1 has been updated

The honest summary, including the part I did not expect

Volume 1's design held up completely. Not one IP address, VLAN, or export path needed changing, which I did not expect and am pleased about.

Volume 1's completeness did not. Two things were missing that stop a working design from becoming a working system: the libvirt daemon mode on modern Ubuntu, and bridge parameters. Neither is an architecture error. Both are the difference between a document that describes a system and a document you can build from.

And the biggest finding is not in that table at all. Three rows say "correct, and I had not understood it". I could have recited every one of those facts before starting, because I wrote them. What I could not do was predict what would happen if they were wrong, which is the only version of knowing that is worth anything when something breaks at 2am.

That gap, between reciting and predicting, is what this volume was for. Building the thing is what closed it.

11. What went wrong

Symptom	Actual cause	How I found it
Host adds successfully, then every system VM fails to start	`libvirtd` in modular mode on Ubuntu 24.04	Agent log showed libvirt connection errors while `virsh` worked fine from my shell. Fixed with `remote_mode="legacy"` and masking the socket units
Management server starts, then the API stops answering after a while	JVM heap and MySQL buffer pool competing for 4 GiB; OOM killer took MySQL	`dmesg \| grep -i "killed process"`. The CloudStack log only showed database errors, which sent me to the wrong component
SSVM stuck in Starting, no template download	`secstorage.allowed.internal.sites` unset, so it refused an RFC 1918 source	SSH into the SSVM over the link-local control channel and ran `curl` by hand
Instance boots, gets no IP, no error anywhere	Virtual Router provider never enabled on the physical network	`cmk list networkserviceproviders` showed `Disabled`. The UI wizard does this invisibly; my script did not
System VMs appeared to hang for 15 seconds on every boot	Bridge forward delay, the default 15 second listening state	`bridge link show` during boot showed `state listening`. Fixed with `forward-delay: 0`
Fixed a traffic label typo, system VMs still broken	Existing system VMs keep interfaces built against the old label	Volume 1 had warned about exactly this. `cmk destroy systemvm` and let CloudStack rebuild them

12. Where this leaves the project

The socle exists. Concretely:

A simulated datacenter on one laptop, inside 44 GiB of allocated disk and 11 GiB of RAM
Real 802.1Q VLANs on a real software switch, carrying management, storage, public, and tenant traffic
Apache CloudStack 4.22 with an enabled zone, a registered host, and both system VMs healthy
NFS primary and secondary storage on correctly scoped exports, with a verified MTU 9000 path
An Alpine instance routed by a Virtual Router, reaching the internet through a simulated edge firewall
An authenticated API endpoint, which is the only output that Volume 7 actually needs
A terraform destroy and terraform apply cycle that rebuilds all of it without manual steps

What I would do differently if I started again

Survey the host before designing anything. Document 2's failures were all pre-existing opinions on my machine: NetworkManager, Docker's firewall policy, AppArmor. Twenty minutes of nmcli device status, iptables -L FORWARD, and aa-status before writing a line would have saved hours.

Read the actual schema, not the first example that looks close. I lost time to the Terraform provider's 0.9 rewrite because I trusted tutorials. terraform providers schema -json answered it definitively in one command.

Put the memory check in the tooling, not the documentation. The lab-up alias prints available RAM because a number in a document is a number I will skip.

And build the lab earlier. Volume 1 would have been a better document if Volume 2 had existed while I was writing it. Two of the gaps in section 10 are things a running environment would have surfaced immediately, and no amount of additional reading would have found.

Volume 3 revisits the platform decision, which is a much more honest conversation now that CloudStack has been installed rather than only researched. Volume 4 goes deep into the platform itself. And Volume 7 picks up the API keys from section 9 and starts driving this zone from Terraform, which is where the provisioning engine begins.