Architecture on Apple silicon

The DockerIOS architecture was a Linux controller talking to KubeVirt talking to QEMU talking to OpenCore talking to macOS. MacRack is a Linux controller talking to a host agent that sits on a Mac mini and calls Virtualization.framework. That is the entire hypervisor stack, and Apple wrote it.

1. The shape of the cluster

The Kubernetes control plane stays on Linux. I am not putting etcd on a Mac mini, and I am not waiting for a production Darwin kubelet that does not exist.

Each Mac mini runs macOS, because Virtualization.framework is a macOS API and will not run anywhere else. The commercial node I am designing against is not a random mini off a desk. It is a 1 TB SSD and a 10 Gb Ethernet card, because image movement is the only slow thing left once Apple is the hypervisor.

On that Mac, a host agent process does three jobs:

registers the machine as a Kubernetes Node
advertises capacity
creates, stops, and reports on virtual machines by calling Virtualization.framework locally

Preparing diagram

Why this is virtual-kubelet, not a Darwin kubelet

A kubelet on Darwin would be a research project, and I am not in the business of maintaining one. The host agent implements the same contract virtual-kubelet implements for Fargate and similar: it appears to the API server as a Node, it accepts work assigned to that Node, and it reports status.

The work it accepts is not a Linux pod. It is a MacOSInstance. The agent translates that object into a VZVirtualMachineConfiguration and starts it. From the scheduler's point of view the Node has a capacity of two, an extended resource named macrack.io/macos-vm, and a taint that only one tenant can tolerate.

The operator never calls Virtualization.framework. It cannot. That API is not on Linux. The operator's job is admission, leasing, placement, and reconciling desired state. The host agent is the only process that ever talks to Apple's hypervisor, and it runs on Apple hardware, which is exactly where that call is legal.

2. What a guest looks like on this path

A macOS VM created by Virtualization.framework on Apple silicon is a different animal from the QEMU guest in DockerIOS. I want that contrast explicit, because a lot of the previous section evaporates.

Concern	DockerIOS, QEMU on x86	MacRack, Virtualization.framework
Firmware	OVMF plus OpenCore, a 384 MiB bootdisk I have to render	Apple's, inside the framework. There is no OpenCore
CPU	Emulated `Penryn`, feature bits I have to pick	The host's Apple silicon, virtualised. Native
SMC and OSK	Injected by a domain hook, the fifty-line seam	Does not exist. Apple silicon has no SMC of that kind
Identity	Generated SMBIOS serials, MLB, ROM, uniqueness I have to police	Apple-issued machine identity for the VM. Two VMs on one host must have different machine IDs and MAC addresses, which the framework requires
Disk	Emulated NVMe via a hook, or AHCI. TRIM chain I have to assemble	Framework-native block device. Sparse disk images Apple already supports
Network	Emulated e1000, no virtio-net, userspace packet processing	Framework virtio-net. A real virtio device, because Apple ships the guest drivers
Graphics	vmware VGA, or VFIO of an entire AMD card	Paravirtualised graphics through the framework
Guest agent channel	No vsock, no virtio-serial. Agent talks over the network	vsock is supported. The out-of-band channel I could not have on QEMU
Installer	Recovery fetch by board-id, DMG to qcow2	Official IPSW restore images via `VZMacOSRestoreImage`
Two-VM limit	Not enforced by anything. A licence I would be breaking	Enforced in the XNU kernel by `hv_apple_isa_vm_quota`. A third VM returns `VZError` code 6

The guest agent channel is the one I am happiest about

Section 03 spent a page explaining that on QEMU, macOS has no vsock and no virtio-serial, so the health channel shares a failure domain with the workload network. That was a real operational hazard.

Apple's framework gives the guest a vsock device. The MacRack agent talks to the in-guest agent over vsock, which means readiness does not depend on the tenant network being up. A CNI mistake can no longer make every machine in the fleet look dead at once. That alone is worth the substrate change, even before the licence argument.

3. The host agent, in more detail

The agent is a macOS launchd service, signed, and the only privileged process MacRack runs on the Mac. It is not a container, because Virtualization.framework wants to be called from a macOS process with the right entitlements, not from inside a Linux VM on the Mac.

On start it:

Reads its node identity from a bootstrap file placed at MDM enrolment.
Creates or updates a Kubernetes Node object, labelled with the chip (M2, M4 Pro), memory, macOS host version, and serial number of the chassis.
Advertises macrack.io/macos-vm: 2 as an extended resource, always two, never more, never a configuration knob.
Applies a taint macrack.io/unleased=true:NoSchedule until a MacNodeLease binds the chassis to a tenant.
Watches MacOSInstance objects assigned to this node and reconciles them against running VZVirtualMachine objects.
Heartbeats. A Mac that stops heartbeating is a NotReady node, and the operator does not migrate its VMs, because live migration of macOS VMs off Apple silicon is not a thing I have.

The entitlement that will bite the first person who packages this

Virtualization.framework requires the com.apple.security.virtualization entitlement. A launchd service running as a daemon needs to be signed with that entitlement, on a Mac that allows it.

This is an MDM and code-signing problem, not a Go problem. The host agent binary is built on a Mac, signed with an Apple Developer identity, and shipped through MDM. If you try to run an unsigned agent, the framework call fails in a way that looks like a capacity problem.

I would rather discover that in the hardware bring-up than in the first customer's onboarding.

4. Images: one golden per OS plus Xcode, nothing else

DockerIOS needed CDI, CSI clones, a golden qcow2, and a bootdisk factory. MacRack uses a central image store, a copy onto the Mac mini, then Apple's framework booting a local disk.

The image never goes inside a VM in order to create the VM. It lands on the host SSD. Virtualization.framework will not boot a disk that lives only on the network.

What a golden is allowed to contain

I ship one golden per macOS version, with that release's Xcode already installed, plus the MacRack guest agent. That is the entire catalogue rule.

In the golden	Not in the golden
macOS, one version, sealed	GitLab Runner, Jenkins agent, any CI binary
Xcode for that OS, first-launch agreements already accepted	Tenant SSH keys, certificates, extra CLIs, CocoaPods caches
The guest agent	A second Xcode, a beta OS, or a "plus runner" variant

So the store looks like sequoia-xcode16, tahoe-xcode26, maybe a previous point release. Not sequoia-xcode16-gitlab-16.9. Not twenty combinations.

Why I refuse to bake the runner into the image

Xcode is slow, licensed, and has a first-launch dance I do not want a tenant waiting on. GitLab Runner is a binary and a config file. If I bake both together I get a cartesian product: every OS times every Xcode times every runner version is another 80 to 150 GB file I have to build, store, replicate, and evict.

A tenant who wants a runner gets the matching OS-plus-Xcode golden, then the workload layer installs the runner on first boot of the clone. That is minutes, not another image. Same for extra SDKs and tenant keys.

The golden answers "which macOS and which Xcode". Everything else is provisioning, not imaging.

How it moves

Preparing diagram

The 10 Gb card is why this is a lease-start cost rather than a joke. A 100 GB golden on 10 GbE is a couple of minutes in the real world, not a fraction of an hour. The 1 TB SSD is why I can hold the golden the current tenant asked for, two copy-on-write instance disks that grow as they build, host macOS, and a little spare. I am not holding ten goldens on one mini. One, maybe two as an LRU cache. When the next tenant needs a different OS, the unused golden goes, the new one comes in.

Object	Where it lives	When it is deleted
Golden, OS plus Xcode	Central store forever. Cached on a mini while it fits	Evicted from the mini when a different image is needed and space is tight, or when it has not been used. Never while a clone of it is running
Instance disk	Only on the mini, APFS clone of the golden	Always when the lease ends. Tenant source, secrets, build dirs. Wipe it

Why the VM disk stays on the 1 TB SSD, not a SAN

A Mac mini's internal SSD is next to the CPU that will use it. A SAN would add a network round trip to every I/O for a guest that already shares the machine with at most one neighbour.

The 10 Gb link is for pulling goldens, not for running the VM's disk. The cost of local disks is that a dead Mac takes instance data with it. That is acceptable: instances are cattle with a 24-hour floor, and the durable artifact is the golden in the central store.

5. What disappears, which is most of section 03

I want this list in one place, because it is the payoff of changing substrate.

DockerIOS component	Why MacRack does not have it
KubeVirt, CDI, the domain hook, `isa-applesmc`	Different hypervisor. Apple's framework is the hypervisor
OpenCore, OVMF, the bootdisk builder, GPL isolation	No third-party bootloader. Nothing to isolate
`macserial`, IdentityPool, generated SMBIOS	Apple issues the VM identity. Uniqueness of machine ID is a framework requirement I obey, not a serial I mint
CPU pinning, hugepages, Topology Manager, `Penryn`	The framework and the Apple silicon memory architecture handle this. I expose vCPU count and memory size, not NUMA topology
VFIO, AMD GPUs, `vendor-reset`, NVIDIA dead-end	Paravirtualised graphics. No passthrough, no reset bug, no buying the wrong vendor
NVMe rewrite, TRIM chain, CSI UNMAP	APFS clone of a local disk image. Reclaim is deleting a file
e1000, vhost-net ceiling, Multus for the agent	virtio-net plus vsock. The management channel is not the tenant network
Silent TCG fallback	Structurally impossible. There is no software emulator in this path

That list is why MacRack is a smaller operator than DockerIOS

DockerIOS had to be clever because it was teaching an x86 hypervisor to pretend to be a Mac. MacRack is not clever about booting macOS. Apple already did that.

What MacRack has to be clever about is the commercial constraints: two VMs, one tenant per chassis, twenty-four hours. Those are the next page. Everything that made DockerIOS a deep project, the boot chain, the identity factory, the storage reclaim path, is not MacRack's problem.

I spent three sections learning a stack I am now allowed to leave behind. That is the correct use of DockerIOS, and it is why the two codebases do not share a line.

6. Capacity, which is now a property of the chassis

A Mac mini is not a 64-core dual-socket node. The arithmetic from section 03 does not transfer. What transfers is the idea that density is knowable in advance.

Chassis	What I would actually put on it
Mac mini, base	One VM, sized to leave the host comfortable. Two is legal and often unwise if the tenant's workload is a full Xcode build
Mac mini, Pro or Max class	The commercial SKU: 1 TB SSD, 10 Gb Ethernet, two VMs. Memory split so the host keeps a reserve. This is the default
Mac Studio	Still two VMs. More cores and memory per VM, not more VMs. The kernel will not let me have a third, and I would not ask

The operator never exposes a replica count higher than two per node, and the node never advertises more than two. A customer who wants four VMs is buying two Mac minis, dedicated, each with a 24-hour floor. That is the business model, and it is the same one AWS EC2 Mac already proved people will pay for. The difference is that I can put two VMs on a chassis the tenant already paid to dedicate, which is the one density trick the licence actually allows.

7. Failure, on this substrate

Failure	What happens
Host agent dies	Node goes NotReady. VMs may keep running, because they are framework objects, not agent children. On restart the agent reattaches. If it cannot, it reports Unknown and does not start extras
A third VM is requested	The scheduler refuses. If something bypasses the scheduler, the agent refuses. If something bypasses the agent, the kernel returns code 6. Three independent stops
Kernel quota counter gets stuck	Known XNU behaviour: after a VM stops, the quota sometimes does not decrement, and the next start fails with code 6 even though fewer than two are running. The agent detects this, marks the node unschedulable, and requests a host reboot through the operator. This is an operational fact I plan for, not a surprise
Mac mini dies	The two VMs are gone. No migration. The lease stays bound to that serial until it expires or an admin breaks it. Replacement is a new lease on a new chassis, which is a 24-hour clock restart
Operator restarts	Same rule as DockerIOS. Desired state is in the API. The agent, not the operator, holds the running VMs

The next page takes the three SLA rules and maps each one onto a Kubernetes primitive that cannot be turned off by a feature flag.