Projects · Dockerios
The API surface
The DockerIOS CRD design: five API groups and why the boundaries fall where they do, every resource with its spec and status, cluster versus namespace scoping decisions, the ownership graph, the machine phase machine and conditions, and the contract the workload layer will plug into later.
The API surface
They are not v1 of a pair with MacRack. MacRack has its own kinds under macrack.io, documented in The MacRack API. No conversion webhook, no shared storage version, no shared Go module. A field that looks similar was rewritten, not imported.
This is the part of the design I expect to live longest. Controller internals can be rewritten quietly; an API that people build against cannot. So I would rather spend the time here and ship the controllers slightly later.
1. The rules I designed against
I wrote these down before drawing any resources, and then used them to reject several fields I wanted.
| Rule | What it rules out |
|---|---|
| One resource per independent lifecycle. If a thing is created, becomes ready, and is deleted on its own schedule, it is a resource. Otherwise it is a field | A BootDisk CRD. Its lifecycle is exactly an identity's lifecycle, so it is a field on MachineIdentity |
| Admin-shaped and consumer-shaped things are separate resources. Whoever chooses CPU pinning is not whoever asks for a Mac | Putting hardware shape on MacOSMachine. It lives on MacOSMachineClass, which admins own |
| The machine layer never learns a workload's vocabulary | Any field containing the word GitLab, Xcode, or runner |
| No raw passthrough escape hatch | A vmTemplateOverrides field holding an arbitrary KubeVirt patch |
| Status is observable facts only | Storing progress in status that cannot be re-derived by looking at owned objects |
The tempting field is a raw KubeVirt patch on the machine class. It would unblock every unforeseen requirement instantly, and I have wanted exactly that field on other people's operators.
The problem is that it becomes the API on the day somebody uses it. I cannot validate it, cannot default it, cannot reason about what a class does by reading it, and cannot change my own template generation without breaking whatever someone patched. Every future feature request gets answered with "just patch it", and the CRD stops meaning anything.
So the answer to unforeseen requirements is a new typed field, and the cost of that is my problem rather than the API's. The domain hook is a deliberate exception: it is one image, controlled by me, doing two known things, rather than an open patch surface for users.
2. Five groups
operator-sdk create api derives the group from --group plus the project --domain, so with a domain of dockerios.io the groups come out as follows.
A single group would work and I nearly did it. Three things changed my mind.
RBAC. Groups are the natural unit of permission. Letting a team create claims and machines while nobody outside platform engineering can touch identity or images is a two-line role with this split, and a per-resource enumeration without it.
Independent versioning. The machine group will churn. Identity should not. Separate groups can reach v1 on their own schedules instead of the least stable resource holding everything back.
Blast radius. The identity group holds the only data in the system that is genuinely hard to reconstruct. Isolating it makes accidents less likely.
The cost is that the project has to be multi-group from the start, which is operator-sdk edit --multigroup=true and a directory layout change. Doing it on day one is trivial; migrating later means moving every package and rewriting imports.
3. Scoping decisions
Scope is not cosmetic here, so this is the reasoning rather than just the answer.
| Resource | Scope | Reasoning |
|---|---|---|
PlatformConfig | Cluster, singleton | Describes operator-wide behaviour. A namespaced version would let namespaces disagree about the hook image, which is nonsense |
MacOSImage | Cluster | Golden images are expensive and shared. Per-namespace copies of a 200 GiB image is the outcome I am designing against |
IdentityPool | Cluster | Uniqueness has to be enforced fleet-wide. A namespaced pool cannot guarantee two namespaces do not mint the same serial |
MachineIdentity | Cluster | Same reason. An Apple serial is a globally unique value, so the resource holding it should be too |
MacOSMachineClass | Cluster | Mirrors StorageClass. Admin-defined shapes, referenced from any namespace |
MacOSMachine | Namespaced | Owns namespaced objects, and consumes namespaced quota. Tenants need to see their own |
MacOSMachinePool | Namespaced | Creates machines, so it lives with them |
MacOSMachineClaim | Namespaced | The tenant-facing resource |
Kubernetes allows a namespaced object to have a cluster-scoped owner, but not the reverse. So a namespaced MacOSMachine cannot own a cluster-scoped MachineIdentity, and garbage collection will not release identities for me.
That means identity release is a finalizer, and the relationship is a two-way reference rather than an owner reference. MachineIdentity.status.claimRef points at the machine; the machine's status points back.
This is precisely the PersistentVolume and PersistentVolumeClaim pattern, which is reassuring: the same constraint produced the same design in core Kubernetes. It also means I inherit that pattern's known hazard, which is a released identity whose claim vanished. The identity controller needs to detect a dangling claimRef and reclaim it according to the pool's policy, and that is a real code path I have to write rather than a theoretical one.
4. MacOSMachineClass
The admin-owned shape. Everything from section 02's tuning work lands here, which means tuning is a class change rather than a per-machine argument.
The GPU variant differs only in the parts that matter, which is the point of having classes at all:
bus: nvme is my API accepting a value that KubeVirt's disk API does not have. The class expresses intent; the controller renders sata into the VirtualMachine and the domain hook rewrites it to an emulated NVMe controller before boot.
I could have leaked the seam into the API by naming the field something like useNvmeHook. I would rather the API describe the machine the user gets, and keep the mechanism in the controller where it can change without an API break.
If the day comes when KubeVirt does model NVMe natively, the class manifests stay identical and the hook loses a job. That is the test of whether an abstraction was drawn in the right place.
5. MacOSImage
Images are immutable by convention and enforced by webhook: changing spec.source on a ready image is rejected. A new image gets a new name, classes are repointed deliberately, and rollback is repointing back.
Nothing in this API builds a macOS installation. spec.source imports something that already exists.
Building a golden image means running a macOS installer, driving Xcode's first-launch agreements, and baking in an agent. That is a long, interactive, failure-prone process, and wrapping it in a CRD in v1alpha1 would produce a controller whose main job is babysitting a ninety-minute install.
So golden image production is a pipeline that emits an artifact, and the operator's job starts when the artifact exists. A MacOSImageBuild resource is a plausible v1beta1 addition, and it should be added only once the manual process is boring.
6. IdentityPool and MachineIdentity
An identity is never generated on the provisioning path. The pool controller generates ahead of demand; the machine controller only ever allocates something that already exists.
The reason is the failure mode of the alternative. If a machine can mint an identity when the pool is empty, then a burst of a hundred machines produces a hundred concurrent generator Jobs, and any bug in uniqueness checking becomes a hundred duplicate serials in the fleet.
An exhausted pool has to be a visible Pending machine with a clear event, never a fallback code path. I would rather explain why a machine is waiting than why two machines share a serial.
7. MacOSMachine
Deliberately thin. Almost everything interesting is in the class it points at.
Phases and conditions
Phase is a single word for humans and dashboards. Conditions are the machine-readable truth, and they are what I would actually build alerting on.
AllocatingIdentity back to Pending on an exhausted pool. Not to Failed. An empty pool is a temporary capacity condition and the machine should wait, visibly, with an event explaining why. Failing would make a routine capacity blip destroy work.
Starting to Failed on readinessTimeout. Bounding this phase is the difference between a platform and a resource leak. A guest that boots but never becomes ready is holding an identity, a cloned volume, pinned cores, hugepages, and possibly an exclusive GPU. Without a timeout those accumulate silently until a node is full of machines that will never work.
8. MacOSMachinePool and MacOSMachineClaim
The pool is where warm capacity lives, and it carries the scale subresource so it behaves like other replica-shaped things.
A claim is one more resource, and for a while I thought it was ceremony. It is the resource that makes the workload layer possible.
A workload that creates a MacOSMachine directly is coupled to provisioning: it has to understand phases, identity allocation, and boot timing, and it cannot benefit from a warm pool because it always creates something new.
A workload that creates a claim asks for a machine and waits for one reference to appear. That indirection is what lets me put warm pools, recycling, TTL reaping, and quota behind the API without any consumer knowing. It is the same reason nobody provisions a PersistentVolume by hand.
The TTL is not decoration either. Claims are made by automation, and automation dies mid-run. Without a TTL, every crashed pipeline permanently strands a machine.
9. Ownership and cleanup
Solid arrows are owner references, so Kubernetes garbage collection handles them. Dashed arrows are the two places it cannot help me, and both need explicit code:
The machine holds a finalizer that must, in order, wait for any workload to release the machine, request a graceful ACPI stop and wait out the grace period, delete the DataVolume unless the class says retain, clear the identity's claimRef, and only then let itself be deleted. Getting that order wrong is how a machine deletion corrupts an APFS volume it was told to keep.
10. Printer columns
Cheap to add with a Kubebuilder marker, and they determine whether the platform is pleasant to operate.
Exposing the guest's boot stage in kubectl get macs means the most common support question answers itself.
"My machine is not ready" becomes a column reading Booting, or LoginWindow, or AgentStarting, and the difference between "wait thirty seconds" and "this is stuck" is visible without anyone opening a VNC session. On a platform where a machine legitimately takes minutes to become useful, that one column removes an enormous amount of pointless investigation.
11. Versioning
Everything starts at v1alpha1, and I mean it: no compatibility promises, and I expect to break it.
The specific things I expect to learn and get wrong are how much per-machine override is really needed, whether warm pools belong on the pool or in a separate policy resource, and whether GPU deserves its own class dimension rather than a nullable field.
The plan is to stabilise the identity and image groups first, since they are simpler and their data is the hardest to recreate, then the machine group once real workloads have pushed on it. A conversion webhook comes with the first v1beta1, not before, because maintaining conversions between two alpha versions nobody depends on is pure cost.
12. Where the workload layer attaches
Sketches only, to prove the contract holds. These are not v1alpha1.
Read that manifest and note what is absent. No storage class, no CPU topology, no identity pool, no image reference, no grace period.
It asks for machines of a named class and describes a runner. If I can implement that resource without adding a single field to MacOSMachine, the boundary in section 1 was drawn correctly. If I cannot, the machine layer is leaking and I would rather find out with one workload type than with four.
The next page turns all of this into scaffolding commands, controller structure, and a packaged bundle.