Projects · Dockerios
System architecture
What DockerIOS is as a system: the machine layer and the workload layer, the component inventory, the full path from a claim to a running macOS instance, exactly which parts of Docker-OSX and OSX-KVM I reuse, and the GPL boundary that reuse forces into the design.
System architecture
This page specifies the KubeVirt-on-x86 operator. API group dockerios.io. It shares no types with macrack.io. If you are here to sell macOS machines, start at MacRack instead.
DockerIOS is a Kubernetes operator that turns a macOS machine into a resource you can ask for. Something requests a Mac, the operator produces one that is booted, healthy, identified, addressable, and disposable, and when the requester is finished the machine goes away and its storage is reclaimed.
That is the entire product statement, and I want to keep it that short because the temptation with this kind of system is to let it become "a macOS CI platform". It is not. A CI platform is something somebody builds on this.
1. The two layers, and why I refuse to blur them
Every design decision on this page traces back to one split.
The failure mode here is obvious in hindsight and very easy to walk into. I need macOS machines to run GitLab runners, so the fastest path is a single CRD called something like MacOSRunner that boots a VM and registers a runner in one reconcile loop.
It would work, and I would ship it sooner. Then somebody asks for a Mac to run a Jenkins agent, or to reproduce a bug interactively, or to notarise a build outside any pipeline, and every one of those is now a fork of the runner logic.
The machine layer never learns the word "GitLab". A runner becomes a workload-layer resource that holds a claim on a machine, which means the interactive-debugging case and the Jenkins case cost nothing extra because they use the same claim API.
The practical test I apply to every field I am tempted to add to MacOSMachine: would a person who wants a Mac to click around in by hand need this? If not, it belongs in the workload layer.
2. Component inventory
Three categories, and I keep them separate in my head because they have completely different maintenance costs.
| Component | What it does | Who maintains it |
|---|---|---|
| The operator | Go, Operator SDK. Reconciles the machine-layer CRDs into KubeVirt and CDI objects. The only long-running process I write | Me |
| The domain hook | A KubeVirt sidecar that injects isa-applesmc and rewrites the system disk to NVMe. Roughly fifty lines, stateless | Me |
| The bootdisk builder | A container image that generates identity values and renders an OpenCore boot image. Runs as a Job, exits | Me, wrapping borrowed code |
| The guest agent | A launchd daemon inside macOS reporting boot stage, readiness, and disk headroom. Baked into golden images | Me |
| KubeVirt and CDI | VM lifecycle, ACPI shutdown, device plugins, golden image import | Upstream |
| CSI driver | Thin provisioning, volume cloning, snapshots, discard | Upstream |
| OpenCore, macserial, OVMF | Booting macOS at all, and the identity injection that makes it believe the hardware | Acidanthera and the OSX-KVM project |
Four things I maintain, and only one of them is a long-running process. That ratio is the whole point of section 02's substrate decision.
3. What I take from Docker-OSX, and what I leave
This is the part I most want to be explicit about, because the goal is to ship a working platform, not to prove I can rediscover a boot chain. Anything sickcodes and the OSX-KVM project already solved, I take.
The parts I take, essentially unchanged
| What I take | From | Why reimplementing it would be foolish |
|---|---|---|
generate-unique-machine-values.sh | osx-serial-generator | It already drives macserial correctly and already sources MAC prefixes from Wireshark's OUI database. The output format is exactly what I need to store in an identity record |
generate-specific-bootdisk.sh | osx-serial-generator | The plist templating is fiddly and already correct. Getting the placeholder set wrong produces a machine that boots but is subtly wrong, which is the worst failure class |
opencore-image-ng.sh | osx-serial-generator | A working libguestfs pipeline that builds a bootable GPT and EFI tree without root or loop devices. Rebuilding this is weeks of work for an identical outcome |
The pinned config-*.plist templates | osx-serial-generator | Including the picker and no-picker variants and the per-release ones. These encode a lot of hard-won OpenCore knowledge I do not have |
macserial | OpenCorePkg | The only tool that generates serials matching Apple's real encoding |
The OVMF and EFI tree, and fetch-macOS-v2.py | OSX-KVM | Firmware that is known to work, and the recovery fetch logic including the board-id distinction. Build-time only in my design |
| The known-good QEMU device combination | Docker-OSX and OSX-KVM | Not code, but the most valuable thing here. Knowing that q35 plus Penryn plus isa-applesmc plus the vmware display actually boots saves an unbounded amount of trial and error |
The identity and bootdisk pipeline is the single hardest part of running macOS on QEMU, and it is fully solved in code I can run today. Taking it means my first milestone is "one VM boots under KubeVirt" rather than "learn OpenCore".
The only reason I could write section 01 as a teardown rather than as a research project is that these parts exist. Reusing them is not laziness, it is the reason the project is feasible at all.The GPL boundary this forces, which is a real architectural constraint
Both Docker-OSX and osx-serial-generator are GPL-3.0. I checked, because it changes the design.
If I ported those scripts into my Go operator, or linked them as a library, I would be creating a derivative work and the operator itself would need to be GPL-3.0. That may or may not be acceptable, but the decision should be deliberate rather than accidental.
So I isolate them. The borrowed scripts live in a separate builder container image, licensed GPL-3.0, with its own source repository and attribution intact. The operator invokes it the way it invokes any other program: it creates a Kubernetes Job, passes arguments, and reads the artifact that comes out. No linking, no porting, arm's-length invocation.
I drew this line for licensing reasons and then noticed it is where I would have drawn it regardless.
Bootdisk generation needs libguestfs, a kernel image, macserial compiled from source, and a plist toolchain. None of that belongs in a long-running controller process, and all of it wants to run once and exit. A Job with a purpose-built image is simply the right shape.
A licensing constraint that pushes you toward the design you wanted anyway is a gift. I still have to keep attribution and the GPL text intact in that image, and I would want to do that on the merits.
The parts I deliberately leave behind
| What I drop | Why |
|---|---|
The Launch.sh generation and entrypoint model | Configuration through shell string interpolation cannot be validated, defaulted, or diffed. This is exactly what a CRD and a webhook are for |
| Runtime environment variables as the config surface | Same reason. Spec fields with an OpenAPI schema fail at admission instead of at boot |
| Identity generated at container start | Produces duplicates across a fleet and a different machine on every restart. Identity becomes a stored, allocated record |
docker commit as persistence | Replaced by golden images through CDI and CSI clones |
| SLIRP networking and SSH port forwarding | Replaced by KubeVirt's tap-based pod networking and real Services |
| SSH reachability as the health signal | Replaced by the guest agent, which can distinguish "booted" from "ready to accept work" |
| Fetching a plist from GitHub at boot | Unpinned network dependency in a boot path. Vendored into the builder image with a checksum |
Every one of those is something Docker-OSX does correctly for its own purpose. They are scope differences, not defects, and I said as much in section 01.
4. The path from a request to a running Mac
This is the flow I am building. Diagonal reading of this diagram is the fastest way to understand the operator.
Steps 4 to 9 and the shutdown at the end are the parts that cannot be a Helm chart.
Allocating an identity from a finite pool, holding it for the life of the machine, and releasing it on deletion is stateful coordination. So is sequencing a build Job, then a volume clone, then a VM, and handling each of them failing halfway. That is a reconcile loop with a phase machine and finalizers.
If the workflow were stateless, I would not need an operator, and I should be honest that quite a lot of "operators" are Helm charts with extra steps. This one is not, and identity allocation is the proof.
5. Where my responsibility stops
I find it useful to write this boundary down explicitly, because during implementation there is constant pressure to reach across it.
| Concern | Owner |
|---|---|
| Placing a VM on a node, respecting pinning and hugepages | KubeVirt and the scheduler |
| Starting and stopping QEMU, ACPI shutdown, grace periods | KubeVirt |
| Attaching volumes, exposing PCI devices | KubeVirt and its device plugins |
| Importing and cloning disk images | CDI and the CSI driver |
| Reclaiming deleted volumes | The CSI driver |
| Deciding which VM to create, with what shape, from which image | Me |
| Apple identity: generation, allocation, uniqueness, release | Me |
| Rendering the bootdisk artifact | Me |
| Knowing whether the guest is ready for work | Me, via the agent |
| Warm pools, claims, binding, recycling | Me |
| The two devices KubeVirt cannot express | Me, via the hook |
Read the right-hand column and the operator's actual job is visible: identity, artifacts, shape, readiness, and pooling. Everything else is delegation.
6. The guest agent, because readiness is the thing everyone gets wrong
Section 02 established that no macOS build of qemu-guest-agent exists, so this is code I have to write. I want to be clear about why I would write it even if one did.
A qemu-guest-agent answers "is the operating system alive". A build platform needs the answer to "can this machine accept a job", and those are separated by several minutes and a long list of ways to fail. Xcode's first launch, a mounted cache that is not there yet, a runner binary that has not registered, a disk that is at 96 percent.
So the agent reports a progression rather than a boolean:
That maps onto conditions on MacOSMachine, which means kubectl get macosmachine shows why something is not ready instead of just that it is not.
On the OpenCore and QEMU path, macOS has no virtio-serial and no vsock driver, so the tidy out-of-band transports other guests use are closed. The agent talks over the network.
That has a consequence I have to design for rather than discover: the health channel shares a failure domain with the workload network. A network policy mistake makes every machine in the fleet look unhealthy simultaneously.
So the agent gets its own interface through Multus, with its own policy, and the machine controller treats "agent unreachable but VMI running" as a distinct condition from "guest unhealthy". Conflating those two is how you build a control plane that mass-terminates a working fleet during a CNI upgrade.
7. What happens when things break
An architecture document that only describes the happy path is describing a demo. These are the failure modes I know about now, and each one is a requirement on the controller.
| Failure | What the operator must do |
|---|---|
| Bootdisk Job fails | Retry with backoff, keep the identity allocated, surface the Job's failure on the machine's conditions. Do not leak the identity |
| Identity pool exhausted | Hold the machine in Pending with a clear reason and emit an event. Never generate an unrecorded identity as a fallback |
| DataVolume clone fails or storage is full | Fail the machine, release the identity, and stop admitting new machines against that storage class |
| VM boots but the agent never reports | Time-bounded Starting phase, then fail. Never sit in Starting forever holding an identity and a GPU |
| Guest panics or hangs at runtime | Ready condition flips false, machine drains out of the pool, pool creates a replacement. Preserve the disk if the class asks for it |
| Node dies with a passed-through GPU | The VM is not migratable, so the machine is lost. Fail fast and replace rather than waiting for a node that is not coming back |
| Operator itself restarts mid-provision | Every phase must be re-derivable from the observed state of owned objects. No progress may exist only in operator memory |
| Machine deleted while a workload holds it | Finalizer ordering: drain the workload, ACPI stop, delete volumes, release identity, then remove the finalizer |
"Operator restarts mid-provision" is the requirement I would design around from the first commit, because retrofitting it is painful.
It means no phase may be inferred from anything the controller remembers. Every reconcile has to look at the world, list owned objects, and work out where it is. A controller that tracks progress in a local map works perfectly in testing and then loses a hundred half-built machines the first time it is rescheduled.
The status subresource is the only memory the operator is allowed to have, and even that has to be treated as a cache of observable facts.8. Where this goes next
The workload layer is out of scope for this section, but the shape of its contract is not, because it constrains the machine layer's API.
A future GitLabRunner resource holds a MacOSMachineClaim, waits for the machine to report ready, executes provisioning steps in the guest, and registers the runner. It never creates a VirtualMachine, never touches an identity, and never knows a bootdisk exists. Same for XcodeToolchain, and same for whatever comes after.
I will know the machine layer is done when I can write the GitLab runner resource without changing a single machine-layer CRD.
If adding the first workload forces a new field onto MacOSMachine, the boundary was in the wrong place, and better to find that out with one workload type than with four.
The API surface page is where that contract gets specified in detail, and the cluster requirements page covers what has to exist underneath before any of it runs.