Building and operating MacRack

Two repositories. This page is the public one: github.com/boualleiguie/macrack-operator, domain macrack.io. The DockerIOS repo is not a dependency, not a submodule, and not a replace directive.

1. Scaffolding

Same SDK, different domain, still multi-group, still go/v4. The ordering trap is identical: multigroup before the first API.

Webhooks on the objects that must refuse illegal states, which is most of them:

The defaulting webhook on the lease writes minimumDuration: 24h if it is unset, and the validating webhook rejects anything below that. Defaulting is not a substitute for validation. Both.

2. Layout, including the piece that is not Go

Why the agent is not in the operator repository

Virtualization.framework is Swift and Objective-C on Darwin. The operator is Go on Linux. Forcing them into one repo produces a go.mod that CI on GitHub-hosted Linux cannot build, and a signing pipeline mixed into a Go release.

More importantly, the public commercial artifact people install on the cluster is the operator bundle. The agent is an MDM payload. Different consumers, different release cadence, different signing identity.

They speak through the Kubernetes API, which is already the decoupling boundary. The agent is a client of MacNode and MacOSInstance. It does not import the operator's internals.

There is a third small repo, macrack-guest-agent, the launchd daemon inside the VM. vsock makes it simpler than DockerIOS's network agent. It still reports boot stage, Xcode versions, disk headroom, and ready-for-work.

3. RBAC, narrower than DockerIOS

No KubeVirt. No CDI. No subresources.kubevirt.io. The operator touches its own CRDs, Nodes (labels and taints), and placeholder Pods.

The agent uses a per-node ServiceAccount that can patch its own MacNode status and get, list, watch, update MacOSInstance objects assigned to it. It cannot list leases in other tenants. It cannot taint a node it does not own. Node taints are the operator's job, because a compromised agent should not be able to steal a chassis by rewriting tenancy.

4. Hardware bring-up

A Mac mini is not a PXE server away from being a node. The bring-up is MDM.

Step	What happens
Unbox, serial recorded	A `MacNode` is created in `Pending`, serial set, class set. Hardware is 1 TB SSD and 10 GbE, or it is not a commercial node
ABM / MDM enrolment	Host macOS, configuration profile, agent package, developer certificate trust
Agent starts, signed with `com.apple.security.virtualization`	Registers against the API, advertises capacity 2, taint `unleased`
Golden images pulled	Not preloaded onto every mini. Pulled over 10 GbE from the central store when a lease asks for that OS-plus-Xcode image. `MacOSImage.status.replicatedOn` gains this serial
Node `Ready`	Eligible for a lease

The signing identity is part of the product

Without a paid Apple Developer account and MDM, the host agent is a binary that will not be allowed to create VMs. That is not optional polish. It is a prerequisite, next to "own some Mac minis".

I would treat the Developer account, the MDM tenant (Jamf, Kandji, or Apple Business Manager plus a lightweight MDM), and the cluster as the three things that have to exist before operator-sdk run bundle is interesting.

5. The quota bug, as an operational loop

Apple's kernel tracks macOS VMs with hv_apple_isa_vm_quota. In the field, that counter sometimes does not decrement when a VM stops. The next start returns VZError code 6, "the maximum supported number of active virtual machines has been reached", while only one, or none, are actually running. The fix is a host reboot.

Preparing diagram

A reboot of a dedicated chassis is a tenant-visible outage. The operator has to:

refuse to start further instances on that node
stop the ones that are running, through the framework, so the tenant's disks are consistent
reboot
reattach
if the lease is still inside its 24-hour floor, give the chassis back to the same tenant, not into the free pool

Do not 'fix' the quota

There are boot-args and SIP-off tricks that raise hv_apple_isa_vm_quota. They are for Apple-internal kernels and for people writing blog posts about beating the limit.

MacRack will never ship them. Circumventing a kernel licence cap is the opposite of this product. A stuck counter is handled by reboot, and a genuine third-VM request is handled by rejection.

The moment the operator has a code path that asks the kernel for more than two, the commercial claim is gone.

6. Testing

Tier	Where	What it proves
Unit	CI, Linux	Lease state machine, 24-hour arithmetic, taint diffs, webhook reject tables
Integration, envtest	CI, Linux	A third instance is rejected, a cross-namespace schedule is rejected, a delete at +3h becomes Held, a delete at +25h releases
Agent unit	CI on a Mac runner, or a signed self-hosted Mac	Configuration rendering for `VZVirtualMachineConfiguration`. Does not start a VM
End to end	A real Mac mini, MDM-enrolled	Two VMs start, a third fails, vsock readiness, reboot recovery from a stuck quota

The tests that matter most run on Linux

Rule 2 and rule 3 are Kubernetes behaviour. I can prove, in envtest, that two namespaces cannot bind the same serial, and that a lease is not releasable before earliestReleaseAt.

Rule 1's scheduler layer is also Kubernetes. The kernel layer is not, and that is the Mac-only test. So the compliance claim is mostly CI, which is how I want it. A licence invariant that only holds when a particular Mac is plugged in is an invariant I will eventually break.

The webhook reject table is the test I would write first:

If those four ever go green for the wrong reason, the product is wrong.

7. CI and the decoupling check

No DockerIOS module. No DockerIOS import. No dockerios.io string in the MacRack codebase except, possibly, a comment in the README that says this is not DockerIOS. I would fail the pipeline on the comment too, to be pedantic, and put the explanation in docs outside the repo.

The same check exists in reverse on the DockerIOS repo.

8. Packaging

OLM bundle, AllNamespaces, required APIs limited to Kubernetes itself. MacRack does not require KubeVirt, which is the entire point of the split: a customer can install this on a cluster that has never heard of QEMU.

alm-examples should show a MacNodeClass, a MacNodeLease, and a MacOSInstance. Not an empty stub. The first thing a buyer sees has to make the dedicated-chassis model obvious, or they will ask where the shared VM pool is.

9. Order of work

Step	Deliverable	Why here
0	One Mac mini, MDM, a signed agent that starts two VMs with Virtualization.framework by hand, a third that fails with code 6	No operator. Proves the substrate and the kernel cap. Same philosophy as DockerIOS step 0
1	Agent registers a Node, advertises `macrack.io/macos-vm: 2`, heartbeats	The Mac is now visible to Kubernetes. Still no tenancy
2	`MacNodeLease`, taints, 24-hour floor, Held phase. Webhooks and envtest	The commercial invariants, proven in CI, before a single instance controller exists
3	`MacOSInstance` reconciled by the agent. vsock guest agent. Two-per-lease cap	First useful VM, already trapped inside a lease
4	Images: one golden per OS plus Xcode, 10 Gb pull, APFS clone, evict unused goldens, always wipe instance disks	Catalogue stays small. Provisioning is a cache hit or a two-minute pull, not a cartesian product of runners
5	Quota-stuck quarantine and reboot loop	The failure I know is coming
6	Audit export, bundle, OLM, metrics on leased chassis not on VM count	Operable by someone who is not me
7	Workload group: GitLabRunner as the first consumer	Only after a tenant can lease a Mac and run two VMs without a custom controller

Step 2 before step 3 is the whole lesson

DockerIOS had to prove a VM could boot before anything else, because booting was the hard part.

MacRack's hard part is not booting. Apple already boots macOS VMs. MacRack's hard part is refusing the profitable illegal states. So the lease, the taint, and the 24-hour floor come before the instance controller. If I build instances first, I will have a working demo that can place two customers on one Mac, and undoing that is harder than not doing it.

A demo that violates the SLA is worse than no demo.

10. What is still open

Billing integration: the lease emits the signals, a billing system has to subscribe. I have not picked one. Guest OS updates inside a running VM. Whether a Studio is worth a separate SKU when it still only carries two VMs. And the legal question I cannot close in this repository: whether a given customer's use of the two VMs fits the SLA's purpose language.

Where the four sections now sit

Section 01 is how Docker-OSX works. Section 02 is the supported-primitives floor on KubeVirt. Section 03 is the DockerIOS operator, for learning and for Intel Macs through 2028. Section 04 is MacRack, the operator I would put in front of a founder who sells infrastructure.

Ideas moved. Code does not. That was the constraint, and it is the reason a buyer of MacRack never has to see a QEMU flag.