Projects · Dockerios
Cluster requirements
Everything a cluster must provide before the DockerIOS operator is worth installing: firmware and kernel command line, kubelet CPU and memory policies, KubeVirt and CDI configuration, CSI capabilities, GPU node preparation, node pool topology, and the capacity arithmetic that decides real density.
Cluster requirements
The operator is the small part. Almost everything that determines whether DockerIOS instances are fast, stable, and honest about their own health is decided by the cluster underneath, and most of it has to be decided before the first VM boots because it lives in firmware settings, kernel command lines, and kubelet policies that need node reboots.
So I am writing this page as a specification I can hand to whoever builds the cluster, and I am marking every item with how badly it hurts to skip.
Required means nothing works without it.
Required for correctness means things appear to work and are quietly wrong or quietly slow, which is worse. This is the category people skip and then spend a quarter debugging.
Required for GPU applies only to nodes in a GPU pool.
Recommended means I would do it and can defend the decision, but the platform functions without it.
1. Hardware and firmware
| Item | Grade | Detail |
|---|---|---|
| x86-64 Intel or AMD CPU with hardware virtualisation | Required | VT-x or AMD-V enabled in firmware. macOS guests are x86-64 only on this path, so there is no arm64 option |
| Bare metal nodes | Required for correctness | Nested virtualisation works and costs a large, variable amount of performance while making pinning and hugepages meaningless. Any benchmark taken on nested hardware is not a number I would trust |
| IOMMU: VT-d or AMD-Vi | Required for GPU | Enabled in firmware. Also enable ACS if the platform offers it, since it affects how cleanly devices separate into IOMMU groups |
| A GPU macOS actually has drivers for | Required for GPU | AMD Polaris, Vega, or Navi. There is no macOS driver for any modern NVIDIA part, so an NVIDIA node pool cannot serve macOS guests no matter what the operator does |
| Generous RAM per core | Recommended | Instances are pinned and use hugepages, so memory cannot be overcommitted. Section 8 has the arithmetic |
| NVMe-backed storage on the node or a fast SAN | Recommended | Storage is the bottleneck this whole design attacks. Putting golden images and clones on spinning disks or a congested network volume undoes the work |
| Reliable NTP | Required for correctness | macOS is unusually sensitive to clock skew for anything touching certificates, signing, or Apple services. Clock drift produces failures that look like everything except a clock problem |
The NVIDIA GPU Operator is excellent, widely deployed, and irrelevant here. It manages drivers and device plugins so that containers and Linux guests can use NVIDIA hardware.
macOS dropped NVIDIA driver support years ago, and Apple never shipped drivers for modern architectures. So there is nothing on the guest side to talk to the card, and no amount of correct host configuration creates a driver.
If a node pool is going to serve GPU-accelerated macOS, it has to be AMD. This is a hardware purchasing decision, which is why it belongs on this page rather than in a tuning guide.
2. Host OS and kernel command line
The kernel command line carries several settings that cannot be changed at runtime, which is why getting this right early avoids a rolling reboot later.
| Item | Grade | Why |
|---|---|---|
/dev/kvm present and usable | Required | KubeVirt will not schedule a VM to a node that cannot advertise virtualisation. This is the fix for Docker-OSX's silent software-emulation fallback and it only works if the node genuinely has KVM |
| 1 GiB hugepages reserved at boot | Required for correctness | Reduces page-table pressure for large guests. Reserving after boot frequently fails on a fragmented host, so this belongs on the command line |
isolcpus for the guest core range | Recommended | Pinning tells the guest which cores it owns. Isolation stops the host putting other work on them, which is what actually delivers the latency benefit |
| IOMMU on, pass-through mode | Required for GPU | No IOMMU means no VFIO means no passthrough |
Card bound to vfio-pci, host driver blacklisted | Required for GPU | If amdgpu binds first, the device is not available to pass through |
The vendor-reset module | Required for GPU | Many AMD cards cannot reinitialise after a guest releases them. On a fleet that recycles machines this turns the second VM on that card into a dead GPU until the host reboots. Note this is a third-party out-of-tree module, so it needs a build pipeline against your kernel |
Validate whether kvm.ignore_msrs=1 is needed | Recommended | macOS guests probe model-specific registers that KVM may not implement. I would treat this as a knob to test on the target CPU rather than a blanket setting, since masking MSR faults can hide real problems |
3. kubelet policies
This is the least visible section and the one I would check first when instances are slow, because every default here is wrong for pinned virtual machines.
dedicatedCpuPlacement: true on a KubeVirt VM requires the kubelet static CPU Manager policy. If the policy is left at its default, the field is accepted, the VM starts, and no pinning happens.
So the manifest says the machine is pinned, the API agrees, and the guest's vCPUs are being moved around by the host scheduler the whole time. There is no error anywhere.
This is the single most likely reason a correctly written DockerIOS class performs badly. The operator cannot detect it either, which is why I put a node-level check in the preflight list in section 9 rather than trying to validate it at admission.
One operational note: changing the CPU Manager policy requires draining the node and removing the kubelet's CPU manager state file. Switching it in place without that produces a kubelet that will not start.
4. Kubernetes and cluster components
| Component | Grade | Notes |
|---|---|---|
| Kubernetes with CRD v1, server-side apply, and the CSI snapshot APIs | Required | Any currently supported minor version. I would match whatever version KubeVirt and CDI both list as supported rather than being adventurous |
| KubeVirt | Required | Installed via its own operator. Configuration detail in section 5 |
| CDI, the Containerized Data Importer | Required | Golden image import and DataVolume cloning. Also installed via its own operator |
| A CSI driver with cloning and snapshots | Required | Detail in section 6. This is a real constraint that rules out some storage backends |
| external-snapshotter and a VolumeSnapshotClass | Required for correctness | Without it, CDI falls back to a host-assisted copy instead of a CSI clone, and provisioning goes from seconds to many minutes per instance |
| Multus | Recommended | The agent's management interface should not share a failure domain with the workload network |
| cert-manager | Required for correctness | Serving certificates for the operator's admission webhooks. OLM can provide these instead, so this is required only when installing outside OLM |
| Prometheus with the Operator CRDs | Recommended | KubeVirt already exports useful per-VM metrics and the operator will export its own. A ServiceMonitor needs the CRDs present |
| A container registry | Required | Five artifacts live here: the operator, the domain hook, the bootdisk builder, any containerDisk bootdisks, and the OLM bundle |
| Pod Security Admission set to privileged on the KubeVirt namespace | Required | The virt launcher legitimately needs privileges. Worth noting the DockerIOS operator itself does not, and I would not accept a design where it did |
5. KubeVirt configuration
Installing KubeVirt is not enough. Three settings matter to DockerIOS specifically.
The Sidecar feature gate has been stable and documented for a long time, so I am confident about that one.
Host device passthrough has moved between being feature-gated and being generally available across KubeVirt releases. The permittedHostDevices mechanism is the right one, but whether your version also wants a feature gate alongside it is a release-note question. I would rather flag that than state a gate name that may have been graduated.
The same applies to timer and TSC settings. Those exist and matter for guests sensitive to clock behaviour, and the specific fields are worth reading in the docs for the version you deploy rather than copying from an architecture page.
Nodes also need the KubeVirt node-labeller to have run, which is what publishes CPU model and virtualisation capability labels. The operator relies on those labels indirectly: a machine class that requires CPU features it cannot find will leave the VM unschedulable, and that is the correct, visible outcome.
6. Storage
This section has more power over the platform's feel than anything else on the page.
| Capability | Grade | Why DockerIOS needs it |
|---|---|---|
| CSI volume cloning | Required | Every instance is a clone of a golden image. Without native cloning, CDI copies hundreds of gigabytes per instance |
| Volume snapshots | Required for correctness | Both the efficient clone path and any future machine-snapshot feature |
volumeMode: Block | Required for correctness | Give the guest a raw block device instead of an image file on a host filesystem. Removes a whole filesystem layer from every I/O and is what makes the NVMe work pay off |
| Thin provisioning with working UNMAP | Required for correctness | The reclaim chain from section 02 ends here. Ceph RBD, LVM thin, or a ZFS zvol. A single large file on a filesystem is the configuration that made Docker-OSX disks grow forever |
| ReadWriteOnce is sufficient | Note | Shared access is only needed for live migration, and macOS instances are not migratable, so I am not paying for RWX |
CDI's StorageProfile is where the clone strategy is selected, and it is worth setting explicitly rather than hoping the default is right:
Thin provisioning means provisioned capacity will exceed physical capacity. That is the point of it, and it is also a single event that takes out every instance at once.
When the pool fills, every guest on that backend takes write errors simultaneously, and a macOS guest losing its disk mid-write is not a clean recovery.
So before the first production instance: alert on physical pool utilisation rather than provisioned, set a headroom threshold that makes the operator refuse to admit new machines, and decide in advance what gets evicted. I would rather have new machines stuck in Pending with a clear reason than a fleet-wide disk failure.
7. Node pools
I would not run a homogeneous cluster for this. Three pools, with taints so that ordinary workloads cannot land on expensive prepared nodes.
| Pool | Prepared with | Carries |
|---|---|---|
control | Nothing special | The operator, KubeVirt and CDI control planes, cert-manager, monitoring |
macos-compute | KVM, hugepages, static CPU Manager, isolated cores, fast local storage | The bulk of instances. Tainted so only VMs land here |
macos-gpu | All of the above plus IOMMU, vfio-pci binding, vendor-reset, AMD cards | GPU instance classes only. Density is capped at the number of physical cards, so this pool is small and expensive and should stay that way |
The bootdisk builder Job is worth calling out separately: it needs libguestfs, which wants /dev/kvm for its appliance backend, so those Jobs belong on the compute pool rather than the control pool. That is a scheduling detail with a real failure mode, since the Job will simply be slow or fail elsewhere.
8. Capacity arithmetic
Pinned CPUs and hugepages cannot be overcommitted. That single fact makes DockerIOS density predictable, which is pleasant, and lower than people expect, which is not.
Worked example on a dual-socket node with 64 physical cores, 128 threads, and 512 GiB of RAM:
| Quantity | Value | Where it goes |
|---|---|---|
| Physical cores | 64 | — |
| Reserved for host and kubelet | 8 | reservedSystemCPUs |
| Available for guests | 56 | The isolated range |
| Cores per instance class | 8 | Plus an emulator thread, isolated separately |
| CPU-bound instance count | 6 | 56 divided by 8, then one more core per instance for the emulator thread |
| Hugepages per instance | 32 GiB | Cannot be overcommitted |
| Memory-bound instance count | 14 | 448 GiB reserved as hugepages, 64 GiB left for the host |
| Actual limit | 6 | CPU binds first, so buying more RAM for this node changes nothing |
Six instances per large node is not exciting, but it is knowable in advance, and it does not degrade as the node fills.
The alternative, which is what unpinned overcommitted virtual machines give you, is higher nominal density and build times that get worse as neighbours get busy. On a build platform that produces the worst possible support conversation, because "the pipeline is slow today" has no answer.
Predictable and slightly expensive beats dense and variable, when the thing you are selling is build times. This arithmetic is also what should drive the instance class catalogue: fewer, well-chosen shapes that divide a node cleanly, rather than arbitrary sizes that leave stranded cores.
9. Preflight validation
I want these as an automated check rather than a wiki page, because several of them fail silently. My intention is that the operator ships a preflight command and a node DaemonSet that surfaces results as node conditions.
| Check | How | Silent if wrong? |
|---|---|---|
| KVM usable on the node | ls -l /dev/kvm and the KubeVirt schedulable label | No, VMs stay Pending |
| Static CPU Manager policy active | Inspect the kubelet config and its CPU manager state file | Yes. Pinning is silently ignored |
| Hugepages allocatable | kubectl describe node, look for the hugepages resource | No, the VM will not schedule |
| Topology Manager set to single-numa-node | Inspect the kubelet config | Yes. Cross-socket placement just performs badly |
| Sidecar feature gate enabled | Read the KubeVirt CR | No, the guest fails to boot without the SMC |
| CSI clone actually native | Clone a test volume and time it | Yes. A host-assisted copy works, just very slowly |
| Block volume mode honoured | Inspect a provisioned PVC | Yes. Filesystem mode is a quiet performance tax |
| Discard reaching the backend | Fill, delete, TRIM, and measure pool usage | Yes. Disks grow forever |
| vendor-reset loaded on GPU nodes | lsmod, plus a VM restart test on the same card | Yes. Fails on the second VM, not the first |
| Clock synchronised | chronyc tracking | Yes. Surfaces later as certificate and signing failures |
Look at the right-hand column. The majority of these misconfigurations produce a cluster that runs macOS instances successfully while being slow, growing without bound, or breaking on the second VM rather than the first.
That is why I want preflight as shipped code rather than documentation. A platform whose performance depends on ten invisible node settings needs to assert them continuously, because it only takes one node added by a different process, or one kubelet config drift, to produce a fleet where a third of the machines are inexplicably slower than the rest.
10. What this means for the operator's own requirements
By contrast, the operator itself is unremarkable, and I consider that a feature.
It needs a namespace, a service account with RBAC on its own CRDs plus KubeVirt and CDI resources, a webhook service with a certificate, and roughly the resources of any controller. It does not need privileges, host mounts, /dev/kvm, or a node label. It runs happily on the control pool with two replicas and leader election.
All of the hard requirements are on the substrate: firmware, kernel, kubelet, KubeVirt, CSI. The operator is an ordinary Kubernetes controller.
That is the shape I was aiming for in section 02, seen from the infrastructure side. The privileged, fragile, carefully-tuned parts are all somebody else's maintained software, and my code is a normal deployment that reconciles CRDs.
The next page specifies those CRDs.