Projects · Dockerios
Rebuilding on supported primitives
Every constraint from the Docker-OSX teardown, answered with software that already exists and is already supported. KubeVirt as the substrate, NVMe and CSI cloning for storage, real pod networking, CPU pinning and hugepages, AMD GPU passthrough via VFIO, ACPI lifecycle, managed identity, and the seams that genuinely have no answer.
Only things that exist today and are supported today. Upstream features with documentation, not patches I would have to carry.
That rule is more restrictive than it sounds, and it kills several ideas that look obvious. It also means the honest answer to a few problems is "there isn't one", and those get their own section rather than a hopeful paragraph.
Rebuilding on supported primitives
The teardown produced a list. Configuration lives in shell string interpolation. Identity is generated at boot and never recorded. The only health signal is an SSH handshake. Storage is a file with no golden image or overlay story. Networking goes through a userspace NAT. Stopping the container is a power cut. One virtual machine per container, by construction.
Every one of those is the right call for a laptop, and every one of those breaks at the tenth machine.
So the question for this page is not "what would be nice". It is: for each of those constraints, what already exists that solves it properly, and where does the honest answer turn out to be "nothing"?
1. Three tiers, and why the distinction matters
Before any specifics, I want a vocabulary for how solid each answer is. Without it, a design document mixes "this is a documented feature" with "this could probably be made to work" and the reader cannot tell which is which.
| Tier | What it means | How much I trust it |
|---|---|---|
| Supported primitive | An upstream project ships this, documents it, and tests it. I write a field in a manifest | Fully. This is somebody else's maintenance burden |
| Supported extension point | The mechanism is documented and stable, but what I push through it is not something upstream tests. The hook is supported, the payload is mine | Mostly. It will keep working, and if it breaks it is my problem to debug |
| No answer exists | There is no driver, no feature, and no supported path. It has to be designed around | Not at all. These become architectural constraints, not backlog items |
Worth saying once, plainly: no tier here makes running macOS on non-Apple hardware supported by Apple. That is a licensing question, not a technical one, and no amount of good engineering changes it.
What the tiers describe is the quality of the infrastructure underneath. KubeVirt is genuinely supported software. Running a macOS guest on it is not a configuration KubeVirt tests. Solid foundation, unsupported payload, and I would rather name that clearly than blur it.
2. The substrate decision, which determines everything else
There is a fork right at the start, and it is the highest-leverage choice on this page.
Option A is what the Docker-OSX Helm chart does: run QEMU yourself inside a privileged pod. You keep total control of the command line, which means every tuning knob from the teardown is available immediately.
Option B is to hand the virtual machine lifecycle to KubeVirt and configure it declaratively.
I would take Option B, and the reasoning is not about elegance. It is about what you are signing up to maintain.
Every single one of these was an open item in the teardown table, and all of them are Tier 1 supported primitives:
Graceful ACPI shutdown with a termination grace period, which fixes the dirty APFS problem directly. A real VM lifecycle CRD, so "is it running" is a status subresource rather than an SSH attempt. Declarative CPU pinning through dedicatedCpuPlacement, hugepages, and NUMA. Disk attachment backed by any CSI driver you already run. PCI device passthrough through permittedHostDevices. Pod networking with a real tap device. Golden image distribution through CDI. Replica semantics through VirtualMachinePool.
The cost of Option B is real and I do not want to undersell it: KubeVirt's API models the virtual machines people normally run, which are Linux and Windows. macOS needs two devices that API does not have fields for. That is what the extension point is for, and it is section 3.
3. The one piece of glue you have to write yourself
macOS needs an isa-applesmc device carrying the OSK. KubeVirt has no field for it, and it never will, because it is not a device any supported guest needs.
KubeVirt's documented answer for exactly this situation is the sidecar hook. You enable the Sidecar feature gate, annotate the virtual machine with a hook sidecar image, and that container gets called with the libvirt domain XML before the VM starts. It returns modified XML.
What the hook injects is small and stable:
It is one device, one string, and a value that has not changed in over a decade. The hook has no state, no reconciliation loop, and no upgrade path to worry about. It is roughly fifty lines of Go that appends two XML elements.
Compare that to Option A, where the seam is the entire QEMU command line, the boot sequence, the health model, and the shutdown path.
Good architecture is not the absence of custom code. It is custom code confined to the smallest possible surface. Fifty lines behind a documented interface is a seam I am happy to own forever.
The second thing the hook does is storage, and that needs its own section.
4. Storage, which is where the real performance lives
The teardown's biggest finding was that the CPU is fine and storage is not. Emulated AHCI with ide-hd is a permanent tax on every I/O, and it also breaks discard, which is why disks only ever grow.
4.1 The NVMe question, and the honest seam
macOS has had a mature native NVMe driver since real Macs moved to NVMe SSDs in 2016, and QEMU's emulated NVMe controller is standards compliant. So NVMe is both faster than AHCI and, importantly, it implements deallocate properly so TRIM actually works.
Here is the seam, stated plainly. KubeVirt's disk.bus field accepts virtio, sata, scsi, and usb. macOS has a native driver for exactly one of those, sata. The fast option is not in the API.
| Bus | macOS driver | In KubeVirt API | Verdict |
|---|---|---|---|
virtio | None | Yes | Fast and unusable |
scsi, meaning virtio-scsi | None | Yes | Same problem |
sata | Native AHCI | Yes | Works, and it is the slow path Docker-OSX already uses |
| NVMe | Native IONVMeFamily | No | The one you want, reachable only through the hook |
So the same domain hook that injects the SMC also rewrites the system disk from an AHCI device to an emulated NVMe controller, with the discard options set:
Keep the OpenCore bootdisk on SATA. Boot ordering across a hook-injected controller is a good way to spend an evening on a machine that will not boot. The bootdisk is 384 MiB and read mostly once, so its device model is irrelevant to performance. Leave it where KubeVirt put it and only move the disk that does the work.
Benchmark rather than assume. I am confident NVMe beats emulated AHCI, because that is what the device models are. I am not going to put a multiplier in this document, because the honest number depends on your CSI driver, your host filesystem, and your workload's access pattern. This is the first thing I would measure, and measuring it is cheap.
4.2 Golden images, which is the change that matters most
Docker-OSX uses no qcow2 backing files anywhere, so every machine is a fresh install or a copied file. The supported answer is CDI, the Containerized Data Importer, which is part of the KubeVirt project.
Build the golden disk once, with macOS installed, Xcode present, and your runner agent baked in. Import it once as a PVC. Then every instance is a CSI-level clone of that PVC, which on a decent storage backend is a metadata operation rather than a copy.
You asked about reclaiming space when a guest deletes files inside a qcow2, and the mechanics of that are real. But look at what the golden image pattern does to the question.
If instances are ephemeral, you never reclaim anything. You delete the clone. The space returns completely and instantly, with no TRIM chain to configure, no compaction job to schedule, and no APFS tooling on Linux to fight with.
Disposal beats reclamation every time disposal is available. Reclaim is a problem for pets. Build runners are cattle, so the correct fix is not a better reclaim path, it is making sure nothing lives long enough to need one.
4.3 For the instances that genuinely must be long lived
Some will be. Somebody always needs a persistent machine for signing, or a stateful cache, or debugging. For those, the reclaim chain has to actually work, and now it can:
| Layer | What makes it work |
|---|---|
| Guest | APFS issues TRIM when the device advertises discard. Nothing to configure |
| Device | Emulated NVMe implements deallocate properly. This is the link that was broken on AHCI |
| QEMU drive | discard=unmap passes it down, detect-zeroes=unmap catches written zeroes |
| Storage backend | Use a CSI driver with real UNMAP semantics: Ceph RBD, LVM thin, a ZFS zvol. Not a monolithic file on a filesystem |
Thin provisioning means the sum of your virtual disk sizes will exceed physical capacity. That is the point of it, and it is also a correlated failure mode.
When a pool fills, every guest on that backend takes write errors at the same moment. A macOS guest that loses its disk mid-write is not a clean recovery, and you will be recovering all of them at once.
You need allocation monitored against physical capacity rather than against provisioned capacity, a headroom threshold that blocks new instance admission, and a documented answer to "what gets evicted first". Build that with the platform, not after the first incident.
5. Networking
Getting off QEMU's SLIRP is one of the easier wins, because KubeVirt does not use SLIRP in the first place. Its masquerade binding creates a real tap device inside the pod's network namespace and NATs through it, so the VM is a proper participant in pod networking.
The NIC model is the constraint, and it is worth being precise about the ceiling here rather than optimistic.
vhost-net, the kernel acceleration that makes virtual machine networking fast, only works with virtio-net. macOS has no virtio-net driver. So whichever supported NIC model you pick, packet processing happens in QEMU userspace and costs host CPU per packet.
SR-IOV does not rescue you either. You can absolutely pass a virtual function through with Multus and the SR-IOV device plugin, and then macOS needs a driver for that specific NIC. For the Intel and Mellanox cards that do SR-IOV, there is none.
So: emulated e1000 on a real tap device is the ceiling, and it is a meaningful step up from SLIRP but it is not fast networking. Budget host CPU for it, and if a workload is network bound, that workload does not belong in a macOS guest.
Multus is still worth having for a second interface, not for speed but for separation: a management network for the agent and a workload network for artifacts, with different policies on each.
6. CPU and memory, which is mostly just switching things on
This is the most satisfying section, because almost everything the teardown found missing is a field in the KubeVirt API. All Tier 1.
| Teardown finding | Supported fix |
|---|---|
| No CPU pinning, vCPUs migrate across sockets | dedicatedCpuPlacement with the kubelet static CPU Manager policy |
| QEMU I/O threads compete with the guest | isolateEmulatorThread |
| No hugepages | memory.hugepages.pageSize, with pages pre-allocated on the node |
| Fake multi-socket topology | Explicit sockets, cores, threads. One socket |
| Narrow instruction set, no SHA-NI or AVX2 | cpu.features with policy: require, so scheduling fails loudly rather than silently degrading |
| No NUMA awareness | cpu.numa.guestMappingPassthrough alongside pinning and hugepages |
| KVM falls back to TCG silently | Structurally impossible. KubeVirt requires KVM and the node has to advertise it |
This is my favourite one. In Docker-OSX, accel=kvm:tcg means a host without /dev/kvm boots anyway, in software emulation, with no warning, and the symptom is "macOS feels slow".
Under KubeVirt, virtualisation capability is a node resource that has to be advertised and requested. A node that cannot do KVM does not get scheduled a VM. The failure moves from a silent hundred-fold slowdown to a pod that stays Pending with a clear reason, which is exactly where you want that failure to live.
On the Penryn question from earlier: I left it in the manifest above because it is what is known to work, but it is a benchmark, not a constant. The default is incoherent, since an iMacPro1,1 SMBIOS claims Xeon W hardware while the CPUID reports a 2008 Core 2. Finding the newest CPU model that boots cleanly against your SMBIOS profile is a bounded experiment with a real payoff, because a wider feature set means fewer libraries falling back to SSE code paths.
7. GPU, and the density problem nobody mentions
You asked whether KubeVirt worker nodes with AMD GPUs would give you GPU-capable macOS machines. Yes, and the mechanism is Tier 1 supported. Configure the allowed devices at the cluster level:
Then request one per VM:
The host preparation is the usual VFIO story: IOMMU on the kernel command line, the card bound to vfio-pci rather than amdgpu, and the card alone in its IOMMU group.
The AMD reset bug, and this is the big one. Many Polaris and Vega cards cannot be re-initialised after a guest releases them. On a workstation you reboot and move on. On a node recycling VMs continuously, the second VM to land on that card gets a dead GPU, and it stays dead until the host reboots. The vendor-reset kernel module exists specifically for this and fixes most affected cards. For a recycling fleet it is not optional, and note that it is a third-party out-of-tree module, which drops it to Tier 2 on my scale.
One GPU, one VM, exclusively. There is no vGPU for macOS guests, no SR-IOV, and no virtio-gpu driver. So a GPU node's density is capped at the number of physical cards, on exactly the machines that cost the most. This is a capacity planning fact, not a tuning problem.
Live migration becomes impossible. A passed-through PCI device cannot migrate. GPU instances are pinned to their node for life, so node maintenance means shutting them down.
What is the GPU actually for?
If DockerIOS runs builds, signing, unit tests, and fastlane, then a GPU changes nothing. Compilation is CPU and I/O bound, and section 4 is where your money should go.
GPU earns its keep for Metal-based test suites, hardware video encode through VideoToolbox, and Simulator UI tests that need real rendering throughput. Those are real workloads, and they are a minority of them.
I would build the platform with no GPU support, add a GPU node pool as a distinct instance class when a workload demands it, and never make it the default. Otherwise you are paying for exclusive cards to run compiler jobs.
8. Lifecycle, and the bug that is quietly corrupting disks
The teardown found that stopping a Docker-OSX container kills QEMU, which is a power cut to the guest, which dirties APFS every single time. This is my favourite fix on the page because it is a one-line manifest change against a real correctness bug.
KubeVirt sends an ACPI power button event and waits. macOS treats it as a shutdown request, flushes, and exits cleanly. Only then is anything forced.
Everything else in the lifecycle column comes from the CRDs:
| Need | Supported mechanism |
|---|---|
| Declarative desired state, one instance | VirtualMachine with runStrategy |
| A fleet of identical runners | VirtualMachinePool, which gives replica semantics over VMs |
| Reusable shapes instead of copied YAML | VirtualMachineInstancetype and VirtualMachinePreference |
| Point in time state | VirtualMachineSnapshot, on a CSI driver with snapshot support |
| Clean shutdown, restart, pause | virtctl, and the same operations through the API |
The boot latency problem, and the supported answer
The teardown mentioned that restoring a saved memory snapshot turns a multi-minute boot into seconds. I have to be honest that KubeVirt does not expose that, so it is out of bounds for this page.
The supported answer is less clever and works fine: keep a warm pool. A VirtualMachinePool of already-booted, idle instances, with your controller assigning one to a job on demand and replacing it afterwards. You pay for idle capacity and you get assignment latency measured in seconds.
Memory snapshot restore is the technique that makes commercial macOS CI feel instant, and it is genuinely better. But it is not in KubeVirt, so choosing it means leaving the supported path and owning VM state management myself.
A warm pool trades money for latency using primitives that already exist. For a first version that is the right trade, and the pool size is a dial you can tune against real queue depth instead of a guess.
9. Identity, promoted from a side effect to a record
Docker-OSX mints serials at container start and forgets them. Across a fleet that produces duplicate identities and machines whose identity changes on every restart, which is precisely what Apple's services notice.
The fix is not a new technology, it is treating identity as state you own. A CRD, a pool, and an assignment:
Rendering the bootdisk from that record is a Kubernetes Job running the generator in a builder image, which is exactly what the existing tooling already does, just triggered by a controller instead of a container entrypoint. libguestfs was already the mechanism for building these images without privileges, so nothing new is needed.
Per-instance bootdisk generation exists so a machine can look individually real to iCloud, iMessage, and the App Store. Build runners never sign in to any of them.
So split the classes. Identity-agnostic runners share one immutable bootdisk, distributed as a KubeVirt containerDisk, which is read only with a discarded overlay. That is the same behaviour as Docker-OSX's snapshot=on, except versioned in your registry and cached by every node. No per-instance build Job, no 384 MiB PVC each, no controller work.
Only the machines that genuinely need iServices get a generated identity. One decision, and most of your fleet loses a whole provisioning step.
The other half of this is pinning the supply chain. Docker-OSX may fetch a plist from raw.githubusercontent.com and an installer from Apple at boot, unpinned. Neither belongs in a boot path: bake the plist into the builder image with a checksum, and make the macOS installer an input to golden image builds, never a runtime dependency.
10. Readiness and observability
Here the honest answer is mixed, and one part of it is genuinely not solved.
KubeVirt integrates with qemu-guest-agent for in-guest reporting, and there is no macOS build of it. So the richest source of guest state is unavailable, and you have to write the agent.
That is fine, because you want your own agent anyway. It lives in the golden image as a launchd daemon and reports what a build platform actually cares about: boot stage, agent version, Xcode version, disk headroom, whether a job is running, and whether the runner has registered with GitLab. Readiness becomes "the agent says the runner registered", which is a real signal rather than "a TCP handshake completed".
| Signal | Source | Tier |
|---|---|---|
| Is the VM scheduled and running | VirtualMachineInstance status | Supported primitive |
| Host side metrics, CPU, memory, block, network | KubeVirt's Prometheus metrics | Supported primitive |
| Block device statistics per disk | KubeVirt metrics, sourced from libvirt | Supported primitive |
| Guest boot stage and readiness | Your agent | You write it, no alternative exists |
| In-guest filesystem and process detail | Your agent | Same |
On the QEMU and OpenCore path, macOS has no vsock driver and no virtio-serial driver, so the tidy out-of-band channels other guests use are unavailable. Your agent talks over the network, on a management interface, and you should treat that interface as part of the platform rather than an afterthought.
Worth knowing that this is one of the places where Apple's own Virtualization.framework on Apple silicon is simply better: it gives macOS guests a supported vsock device. Different substrate, different constraints.
11. The seams that stay open
The most useful section, because a design document that claims everything is solved is not describing a real system.
| Seam | Why it has no answer | What I would do instead |
|---|---|---|
| Apple licensing | Not a technical problem. Virtualising macOS off Apple hardware is outside the agreement regardless of how well it is engineered | Name it in every document. For production, Apple silicon you own or rent. This platform is for understanding and for workloads whose legal position you have actually decided |
| The Intel sunset | macOS 26 Tahoe is Apple's last Intel release. There is no OpenCore for Apple silicon and emulating arm64 on x86 means no KVM, which means unusable | Treat the x86 path as a provider with a known end date. Keep the control plane's interface hardware agnostic so an Apple silicon provider can be added later |
| Network throughput | vhost-net needs virtio-net and macOS has no driver. SR-IOV needs a driver for the specific NIC and macOS has none | Accept emulated e1000 on a tap device, budget host CPU for packet processing, and keep network-bound work out of the guest |
| GPU density | No vGPU, no SR-IOV, no virtio-gpu for macOS. Passthrough is exclusive | A separate GPU instance class, never the default, sized to physical cards |
| Live migration | The hook-injected devices and any passthrough GPU are not migratable | Declare macOS instances non-migratable. Node drain means drain the queue, then shut down |
| Instant boot | Memory snapshot restore is not exposed by KubeVirt | Warm pools, paying idle capacity for assignment latency |
| Guest agent | No macOS qemu-guest-agent build exists | Write one. It is a launchd daemon and you want a build-aware agent regardless |
| APFS tooling on Linux | libguestfs support for APFS is thin, so host-side inspection and sparsifying of macOS disks is unreliable | Do disk work from inside the guest, or from a macOS builder. Never plan on mounting an APFS volume on a Linux node |
Six of those eight are ordinary constraints. You design around them and move on.
The licensing position and the Intel sunset are different in kind, because no amount of good work moves them. They decide whether this platform should exist and for how long, and that is a decision to make deliberately at the start rather than discover in year two.
Everything else on this page is engineering. Those two are a choice.12. What the whole thing looks like
Notice the proportions. Almost everything is somebody else's supported software. The custom surface is a controller, an agent, and fifty lines of XML manipulation.
Docker-OSX owns the entire stack from the QEMU command line upward, which is why it is so easy to run and so hard to scale.
This design owns three small things and delegates the rest to projects whose maintainers already solved lifecycle, storage, scheduling, and device passthrough. The interesting engineering is now the macOS-specific part, which is the only part nobody else was ever going to do for me.
13. What I would build first
Ordering matters, because several of these are worthless without the ones before them.
- Prove one macOS VM boots as a KubeVirt
VirtualMachinewith the domain hook injecting the SMC. Nothing else is worth designing until this works, because if the seam does not hold, the whole substrate decision is wrong. - Move the system disk to NVMe through the same hook and benchmark it against SATA. This is the largest single performance change and the number decides how much else is worth doing.
- Build the golden image pipeline. Install once, seal, import through CDI, then boot instances as CSI clones. This is what makes provisioning fast and makes the reclaim question disappear.
- Fix the lifecycle. Grace period, ACPI shutdown, and verify APFS is clean after a stop. A correctness fix, and cheap.
- Turn on the CPU and memory work. Pinning, emulator thread isolation, hugepages, correct topology, wider features. All fields, all measurable.
- Write the agent and make readiness mean something. This is where the platform stops being a VM launcher.
- Then the operator: identity records, warm pools, and instance classes.
GPU is deliberately absent from that list. It is an instance class to add when a workload asks for it, and adding it early buys exclusivity costs on your most expensive nodes to run jobs that never needed a GPU.
It does not design the operator's API, the instance class taxonomy, the runner registration flow, or the multi-tenancy model. Those are real and they are next.
What it does is establish the floor: every constraint the teardown found has a supported answer, except eight, and those eight are written down. Design on top of a known floor and you find out what you are actually building. Design on top of an unexamined one and you spend year two discovering that macOS has no virtio-net driver.