CloudStack does not create the physical bridges on your hypervisors. You create them, by hand or by automation, before you add the host. If your traffic label points at a bridge that does not exist, nothing helpful happens. No error that names the problem, no validation at zone creation, just system VMs that never come up and logs that describe symptoms rather than causes.

Read that last row again. The traffic type is called Storage, and it has nothing to do with primary storage.

The official documentation is unambiguous on this point: while labeled storage, this is specifically about secondary storage traffic, and it does not affect traffic for primary storage.

Here is how I finally got this straight in my head: the Storage traffic type exists because a system VM needs a network interface to talk to the NFS export. The secondary storage VM is a virtual machine, so CloudStack has to give it a NIC, so CloudStack needs to know which network that NIC lives on. That is the whole reason the traffic type exists.

Primary storage never needs this, because primary storage is mounted by the hypervisor itself, using whichever host interface happens to share the storage subnet. CloudStack tells the hypervisor to mount it. CloudStack does not carry the bytes and does not need a traffic type for it.

Straight from the official topology requirements: the secondary storage NFS export is mounted by the secondary storage VM, and secondary storage traffic goes over the management traffic network, even if there is a separate storage network. Primary storage traffic goes over the storage network, if one is available.

The consequence you must design for: if you decide to put your secondary storage NFS server on the storage network, you must make sure a route exists from the management network to the storage network.

At first I found this completely backwards, and here is the trick that made it click: the secondary storage VM's default home is the management network, because that is where every system VM has to live in order to reach the management server. Giving it a second NIC on the storage network is an addition, not a relocation. So if the NFS box is only reachable on the storage side, the traffic that starts on the management side has to be routed there. It is not a bug. It is the consequence of the system VM being a management network citizen first.

Straight from the official docs: each zone has either basic or advanced networking, and once the choice of networking model for a zone has been made and configured in CloudStack, it cannot be changed. A zone is either basic or advanced for its entire lifetime.

There is no migration tool. There is no conversion procedure. There is no support ticket that fixes it. The remedy is build a new zone and migrate every customer into it, which for a hosting provider means a scheduled maintenance window per customer and a real risk of losing some of them.

This is the single most expensive mistake available in this entire document, and it is available on the second screen of the zone creation wizard, which is a genuinely unkind piece of interface design.

No VPCs and no isolated networks. In security group enabled advanced zones and in basic zones, creation of VPCs and isolated networks is not supported. This is the big one. You cannot offer a shared-network-with-security-groups tier and a VPC tier in the same zone if the zone is security group enabled.

No multiple VLAN ranges in a security group enabled shared network.

No mixed gateway or netmask. Two IP ranges with the same VLAN but a different gateway or netmask are not supported, in either security group enabled shared networks or account specific shared networks.

Security groups in advanced zones are hypervisor limited to KVM and XenServer or XCP-ng with the network backend configured as bridge. On KVM, which is our platform, this is fine.

There is no validation. Type cloudbr01 when you meant cloudbr1 and the zone creates successfully. The failure arrives later, when CloudStack tries to attach a system VM to a bridge that does not exist, and the message you get describes a system VM that will not start rather than a label that is wrong.

Worse, fixing the label is not enough. System VMs that were created against the broken label have to be destroyed and recreated so that CloudStack rebuilds their interfaces against the corrected value. This is why the label appears in our pre install verification gate at the end of this document.

You can configure a zone with multiple physical networks carrying the guest traffic type. In such zones, you must tag the additional physical networks.

And the rule that follows: you must have one physical network that is not tagged, so that isolated and layer 2 network offerings which are not configured with any tags still work. The default network offerings fall into exactly that category.

At first I found this confusing because the word tag is doing double duty, and here is the trick: the VLAN tag is a number in an Ethernet frame. A physical network tag is a text string CloudStack uses to match a network offering to a physical network. They are unrelated concepts that share a word. If a network offering has no tag, CloudStack needs an untagged physical network to put it on, and if every physical network is tagged, the default offerings have nowhere to go.

The interfaces used to configure the bridges must match one of these patterns:

eth*, bond*, team*, vlan*, em*, pp, ens*, eno*, enp*, enx*

Otherwise the KVM agent will not be able to configure the bridges properly.

Why our design is already safe: we use bond0 and bond1, which match bond*, and the VLAN sub interfaces CloudStack creates come out as bond1.40 style names which match vlan* handling. This is not luck. It is one of the reasons Doc 3 chose bonded interfaces with predictable names instead of renaming NICs to something friendly like mgmt0.

And that is the actual lesson here: the temptation to rename interfaces to something human readable is strong, and on a CloudStack KVM host it is a trap. Keep the boring names.

The same field carries the VLAN range in a VLAN zone and the VNI range in a VXLAN zone. CloudStack treats them as the same kind of thing, which is a segment identifier, and only the isolation method on the physical network decides how it is used.

This is genuinely useful to know, and here is why: it means the mental model does not change when you migrate to VXLAN later. You are still allocating segment identifiers from a range. You are just allocating from a range with 16 million values instead of 4094. The concept is stable across the migration, which makes the migration a lot less frightening than it sounds.

There is a global setting in CloudStack that governs how the platform partitions its internal identifier space, and it forces a choice: a higher ceiling on VLANs per zone means a lower ceiling on running instances per zone, and vice versa.

You do not get both maxed out. The default favours instance count, which is correct for the majority of deployments and is not necessarily correct for a per tenant VLAN product model, where VLAN consumption grows with customer count rather than with instance count.

The action item: decide this deliberately, at zone creation, based on how you sell. If every customer gets an isolated network, then customers times one VLAN is your growth curve, and you need to check that ceiling against your three year business plan before it becomes an emergency. Verify the current setting name and default against your exact CloudStack version, because this is the kind of internal setting that gets adjusted between releases.

This is worth working through slowly because getting it backwards produces the single most annoying class of bug in networking: everything works except large transfers.

When a VXLAN interface is created, the kernel reads the current MTU of the physical interface or bridge and creates the VXLAN interface exactly 50 bytes smaller. The subtraction happens automatically, and it happens downward from the parent.

So the chain is:

Physical interface at 1500, the ordinary default, gives you a VXLAN interface at 1450. Your customer's instance now has 1450 bytes available and any software assuming 1500 starts producing mysterious stalls.

Physical interface at 1550 gives you a VXLAN interface at exactly 1500. This is the documented minimum, and it means normal instances see a normal MTU.

Physical interface at 9000 gives you a VXLAN interface at 8950, and instances can be set to 8950.

Physical interface at 9216 gives you 9166, which comfortably contains a clean 9000 byte jumbo frame inside guest networks.

The documented recommendation: use an MTU of at least 9000 and preferably 9216, since most VXLAN capable cards and switches support up to 9216.

At first I found this backwards and here is the trick I use to keep it straight: the overlay is a passenger inside the underlay, so the passenger is always smaller than the vehicle. You never raise the overlay MTU. You raise the underlay MTU and the overlay follows, 50 bytes behind, forever.

This one deserves its own callout because the failure message tells you nothing about the cause.

Every multicast based VXLAN interface joins a multicast group. The kernel parameter net.ipv4.igmp_max_memberships controls how many multicast groups a host may join, and its default value is 20.

So on a default Linux host you cannot start more than 20 VXLAN interfaces. Attempt 21 and the CloudStack agent log shows:

That message says nothing about multicast, nothing about VXLAN, and nothing about a limit. It reads like memory exhaustion, which sends you looking in entirely the wrong place. I am flagging it here so that if you ever see it, you already know the answer.

The fix is to raise the parameter to something appropriate such as 100 or 200. The documentation states it plainly: if you need to operate instances from more than 20 different client networks on a host, this change is required.

And here is the business consequence: 20 networks per host is far below any interesting density for a hosting provider. This is not an optimization. It is a mandatory step, and it belongs in host build automation rather than in a runbook a human might skip.

Everything else in that table is configuration. This one is architecture, and it directly contradicts the bonded design we built in Doc 3.

Here is the reasoning, and once I understood it I stopped resenting it: bonding solves link redundancy at layer 2. The two physical links pretend to be one logical link, and the switches cooperate to make that work, which is where MLAG comes from and where a good deal of vendor specific complexity lives.

EVPN solves the same problem at layer 3. Instead of two links pretending to be one, you run two independent BGP sessions over two independent links to two independent switches. If a link dies, BGP withdraws the routes through it and traffic uses the other. No MLAG, no vendor cooperation between switches, no shared state between them at all.

It is a strictly better failure model, because there is no shared control plane between the two switches to go wrong. It is also a completely different rack build, a different switch configuration, and a different skill set to operate.

Which is exactly why we start with VLAN. Not because VXLAN with EVPN is wrong. Because it is a project, and projects need to be scheduled rather than discovered.

The documentation states this as a warning and so will I: VNI must be unique per zone and no duplicate VNIs can exist in the zone. Exercise care when designing your VNI allocation policy.

With 16.7 million values available, the temptation is to allocate carelessly. Do not. Design the allocation scheme before you need it, ideally encoding meaning into the number, for example a range per pod or a range per product tier. A flat sequential allocation is the option that works fine until you need to reason about it during an incident.

This one sentence from the documentation drives the entire management IP plan: hosts in the same pod are in the same subnet.

The consequence, stated as a design rule: your pod boundary is your management subnet boundary. One pod, one management subnet. When you add a second pod, you add a second management subnet, and it needs a route to the first one, which brings us to the console proxy requirement in section 6.

Here is the thing I find genuinely elegant about this constraint: it means CloudStack's logical hierarchy and your physical rack layout naturally line up, because a rack is a switch is a subnet is a pod. Doc 3 built one rack. So we build one pod. The model and the metal agree, and when they agree, nobody has to hold a translation table in their head during an incident.

Read that second row again. The VLAN range allows 900 guest networks. The guest IPv4 space at 10.1.0.0/16 with a /24 per network allows 256.

So the real ceiling is 256, not 900, and it is set by an address plan decision rather than by the isolation method everybody argues about.

This is the trap of planning two constraints in two separate documents. The VLAN counting felt like the answer because it was the number I had just worked out in section 4, and it was the looser of the two constraints all along.

The fix is trivial once you see it: use 10.0.0.0/8 for guest space rather than 10.1.0.0/16, which gives 65,536 possible /24 networks and puts the ceiling comfortably back on the VLAN range where our section 4 counting expects it. Private IPv4 space is not scarce. Public IPv4 space is. Being frugal with RFC 1918 addresses is a habit imported from a place where it made sense, and it does not make sense here.

Restarting a VR the way you would restart any other Linux VM, from virsh or the hypervisor console, wipes its firewall rules.

The rules live in memory, written there by CloudStack. A restart the platform did not orchestrate brings the router back with an empty ruleset, which is a security event and not merely an inconvenience: a customer's network comes back with its firewall gone.

The correct action is always to restart the network from CloudStack, which recreates the router and reapplies the rules from the database. Use the platform's own restart, not the hypervisor's.

Destroying a VR is a legitimate operation when you have redundant routers and you are replacing one. Destroying the only router in a network takes the network's gateway, DHCP and DNS away in one action.

The habit worth building: when a VR misbehaves, restart the network, do not destroy the router. Restart is idempotent and rebuilds cleanly. Destroy is an outage that happens to sometimes fix things.

This is the requirement that surprises people when they add a second pod, and it is stated directly in the documentation: the console proxy VMs connect to all hosts in the zone over the management traffic network. Therefore the management traffic network of any given pod in the zone must have connectivity to the management traffic network of all other pods in the zone.

Read that consequence carefully. Pods are supposed to be independent subnets, one per rack. But because a single CPVM must reach the VNC port of a VM on any host in the zone, every pod's management subnet must be routable to every other pod's management subnet.

Here is why I think this catches people, and it caught me: the pod abstraction reads like a failure domain. Separate subnet, separate rack, separate switch, so you naturally assume separate blast radius. But the console proxy quietly stitches all the pods in a zone into one routable management domain.

So the pod boundary is a routing and addressing boundary, not an isolation boundary. If you want genuine isolation between racks, the unit is a zone, not a pod. That is a real architectural insight and it costs nothing to know now, when we have one pod, versus discovering it during the second pod build.

This is the diagnostic discipline that saves the most time, and it is worth understanding the logic rather than just memorizing the order.

The Virtual Router is created per guest network, on demand, with configuration derived from a customer's choices. There are many of them and each is shaped differently.

The SSVM and CPVM are created per zone, from your infrastructure configuration, with no customer input at all. There is one of each and they are shaped entirely by the zone's public and management network definitions.

So the SSVM and CPVM are a purer test. If they cannot reach the internet, the fault is in the zone's own network configuration, which means the public range, the gateway, the VLAN, or the trunk. Nothing a customer did is involved.

1. Can the SSVM reach the internet and the NFS export?
2. Can the CPVM reach the management network and the hosts?
3. Only then look at Virtual Routers.

If steps one and two pass and only VRs fail, you have a guest network or offering problem, which is a much smaller search space. If step one fails, every VR in the zone will also fail, and debugging them individually is time spent chasing a symptom.

You will find this advice in forum threads: log into the system VM, flush its iptables rules, and connectivity starts working.

Here is what actually just happened. The system VM's rules are generated by CloudStack from your network configuration. If flushing them fixes connectivity, then CloudStack generated rules that block traffic you need, which means the network settings you gave CloudStack do not describe the network you actually built. The rules are a correct implementation of an incorrect input.

And it is not even a durable fix. The rules are regenerated when the system VM restarts, when the network restarts, and when the platform decides to reconcile state. So you have traded a diagnosable problem for an intermittent one that returns at an unpredictable time, which is strictly worse than the problem you started with.

Treat a successful flush as a diagnostic result, not a remedy. It tells you the problem is in the zone's network definition, and section 11 lists the specific definitions that most often cause it.

That is a direct quote and it appears in the security requirements section of the official topology documentation. I am quoting it rather than paraphrasing because it is the kind of statement that gets softened when retold.

Why 8250 is dangerous: it is the channel over which hosts and system VMs receive instructions. Reaching it from outside means reaching the mechanism that tells hypervisors what to do.

Why 8096 is worse: it is an API that requires no authentication. Not weak authentication. None. It exists so that scripts on the management server itself can call the API during upgrades and automation.

The history here is worth knowing because it explains why I insist on checking it rather than assuming it. Port 8096 was enabled by default in older CloudStack releases, and it was raised as a serious security concern by the community, with the eventual outcome that the default changed to disabled. Which means the safety of your installation depends on which version you installed and whether anybody ever turned it on for an upgrade and forgot to turn it off again.

The action, and it takes ten seconds: set the global configuration integration.api.port to 0, which disables it outright. Then confirm nothing is listening, rather than trusting the setting. Do not leave a no-credentials API listening on the theory that a firewall will always be correct.

NFS version 3 and its helper services historically use ephemeral ports assigned at daemon startup by the portmapper, which means the port numbers change when you reboot the storage server. You cannot write a firewall rule against a port number that changes.

The fix is to pin them to fixed ports in the NFS server configuration, then write rules against those fixed values. The services that need pinning are the mount daemon, the status daemon, the lock manager, and the quota daemon.

If you use NFS version 4 only, this problem largely disappears, because NFSv4 consolidates onto port 2049 and does not need the auxiliary services. Which is a genuinely strong argument for NFSv4 in a new build: not performance, but the fact that a firewall rule for it is one line instead of six lines and a runbook.

An NFS export with a permissive client list is a filesystem that anybody who can route to it may mount and write to. The documented consequence is not subtle: an out of pool client can destroy primary storage. Not corrupt a file. Destroy the pool, because a client that mounts primary storage sees the disk images of every running VM as ordinary files.

So restrict every export to explicit CIDRs, and here is the specific mistake to avoid: when management and storage are separate networks, remember to include both.

The reason traces straight back to section 1. Primary storage is mounted by hosts on the storage network. Secondary storage is mounted by the SSVM, whose traffic rides the management network. Two different mounters on two different networks. Export only the storage CIDR and secondary storage silently fails. Export only the management CIDR and primary storage fails. Export both to everything and you have the data loss scenario above.

Our exports: primary storage to 10.50.1.0/24 only, the storage subnet, which contains nothing but three hypervisors and the array. Secondary storage to 10.50.0.0/24 only, the management subnet. Neither export lists both. Each lists exactly the network whose clients legitimately mount it.

CloudStack writes packet filtering rules itself, on hosts and inside system VMs. A distribution firewall manager that also owns those rules will overwrite CloudStack's work, usually at a moment you are not watching.

On RHEL, CentOS and SUSE: firewalld is the default and it controls iptables. The documented guidance is that it is recommended that it be disabled, with systemctl stop firewalld and systemctl disable firewalld. On management servers specifically, the docs warn that firewalld will override all iptables rules set by the cloudstack-setup-management script, so either disable it or ensure explicit rules exist for 8080, 8250 and 9090.

On SUSE there is a second trap: iptables rules are not persisted across reboots, so the documented recommendation is to create iptables and ip6tables services to make them persist. Rules that vanish on reboot are worse than no rules, because you believe you are protected.

On Ubuntu 22.04 and newer, the one that silently breaks guest traffic: UFW's default policy for forwarding is DROP. It must be changed to ACCEPT in /etc/default/ufw:

This is my favourite bug in the whole document, because of how it presents. The host is fine. The management server is fine. System VMs start correctly. Instances boot correctly and get their DHCP leases. And no traffic passes between an instance and anywhere else, because a bridge forwards frames through the FORWARD chain, and the FORWARD chain is dropping them. Everything you would naturally check looks healthy, and the broken thing is a default policy in a file nobody opened.

This is the most consequential single configuration value in the entire document, so I want to state it precisely.

The administrator is responsible for setting the host global configuration value from the management server IP to the load balancer virtual IP address.

And the documented consequence of not doing it: if the host value is not set to the VIP for port 8250 and one of your management servers crashes, the UI is still available but the system VMs will not be able to contact the management server.

Sit with how that failure looks, because it is genuinely deceptive. You built two management servers and a load balancer specifically so that one can fail. One fails. You log into the UI and it works, because the load balancer is doing its job for HTTP. So you conclude the failover worked.

Meanwhile every system VM in the zone is orphaned, because they were told to connect to a specific server's address, that server is gone, and they have no idea a second one exists. Consoles stop working, template operations stop, and status reporting stops. The symptoms arrive gradually and none of them point at the cause.

Our value: host = 10.50.0.200, the VIP from the section 5 plan. Not .201, not .202. This is why that address exists as a distinct entry in the address table, and why it is worth verifying deliberately rather than assuming the installer did the right thing.

Step two is the one that catches experienced people, because the broken configuration looks correct and works most of the time.

Put two host interfaces in the same subnet, say 10.50.0.101 on cloudbr0 and 10.50.0.111 on the storage interface. Both are up. Both can reach the array. Storage traffic flows.

And Linux chooses which interface to use based on the routing table, which now has two equally specific routes to the same destination. It will pick one, deterministically for a given kernel and configuration, and it will not tell you which. So you have a 10 Gbps storage interface, a 10 Gbps management interface, and no way to know which one your storage traffic is actually using without capturing packets.

Then the failure: the interface it silently chose goes down. Traffic moves to the other one. Everything keeps working, so no alert fires, but now your storage traffic and your management traffic are sharing a wire and your jumbo frames are landing on an interface configured for 1500 bytes.

async is genuinely faster and the difference is measurable on small writes.

And I will not use it for primary storage, for a reason that has nothing to do with performance engineering. Consider explaining to a customer that their database lost the last few seconds of committed transactions because we chose a mount option that acknowledged writes before storing them. There is no version of that conversation that ends well, and no amount of measured throughput improvement that pays for it.

Use sync for primary storage. If the resulting performance is inadequate, the correct response is faster storage hardware or a write cache with battery backing, both of which make writes fast and durable. Buying performance with somebody else's durability is not an optimization, it is a transfer of risk to the person who trusted you.

async on secondary storage is a completely reasonable choice, because its contents are templates and snapshots which are reproducible. Different data, different risk, different answer.

This is the section where I am going to be straightforwardly cautious rather than confident, because the honest engineering answer is that VLAN tagged storage networks in CloudStack have had a history of agent side bugs, particularly around creating the storage bridge and specifically during system VM migration.

The pattern in the reports: the agent needs a bridge on a tagged storage VLAN, and under certain conditions and certain versions it does not create it correctly, which surfaces as a system VM that will not migrate or will not start on a particular host.

Our exposure is deliberately small, and it is worth seeing why the section 1 decision pays off here rather than only in complexity terms. Because we do not define a Storage traffic type, CloudStack is never asked to build a bridge on a tagged storage VLAN. Our bond1.30 is created by us, in the host build, with no CloudStack involvement, and CloudStack neither knows nor cares that it exists.

That is a security and reliability property that arrived as a side effect of a simplicity decision, which is my favourite kind. But I will not pretend it was the reason for the decision, because it was not. It is a bonus, and bonuses should be named as bonuses.

What we do anyway, because caution is cheap: record the exact CloudStack version in the as built document, and explicitly test system VM start and migration on every host before the zone carries a paying customer. If we ever add a Storage traffic type, this test becomes mandatory again at that time and on that version.

The CIDR size must be 64. Not 63, not 65, not 56. Exactly a /64.

It must be added without specifying start and end addresses, because the addresses assigned to networks are SLAAC based. CloudStack does not hand out addresses from a pool here, it derives them.

It must use the same VLAN as an existing public IPv4 range. The documentation is explicit that the public IPv6 CIDR must be added with the same VLAN as the public IPv4 range.

That third constraint is the one I want to dwell on, because it explains something that would otherwise look like a mistake in our plan. Our public IPv6 range is 2001:db8:0:40::/64 on VLAN 40, the same VLAN carrying 203.0.113.0/24. If you were designing from first principles you might reasonably give IPv6 its own VLAN for clean separation.

CloudStack will not let you, and the reason is sound once you see it: the Virtual Router's public interface is one interface. It sits in one VLAN. Dual stack means both protocols on that one interface. Separate VLANs would require a second public interface per VR purely for IPv6, doubling the interface count for no functional gain.

So the constraint is not arbitrary. It falls out of the VR's three interface architecture from section 6, and once you connect those two facts it stops being a rule to remember and becomes a consequence you can derive.

The rule: the CIDR size for the guest IPv6 prefix must be less than 64, and CloudStack carves exactly one /64 per guest network out of it.

Which means the prefix length you choose is the cap on dual stack guest networks in that zone. Count the /64 blocks and you have counted your product ceiling.

The /56 is the trap, and it is a trap specifically because 256 sounds like plenty. It is the number a careful engineer picks when applying IPv4 instincts to IPv6: big enough with room to spare, tidy, not wasteful. And it caps the zone at 256 dual stack networks, which a growing hosting provider reaches.

A /48 per zone out of our /32 costs us 1 of 65,536 available /48 blocks. Doc 2 said it and it bears repeating: being stingy in IPv6 creates work rather than saving anything.

The documentation is unambiguous on this: currently a guest network cannot be IPv6 only. It can only be IPv4 only or dual stack. The same applies to shared networks, where the docs state that a shared network cannot be IPv6 only and therefore you must configure an IPv4 range for the shared network with IPv6 addresses.

Sit with how much this constrains the strategy, because it undercuts a plan that sounds very attractive on paper. The obvious answer to IPv4 scarcity is to sell IPv6 only instances at a lower price and let the market move. CloudStack will not let you build that product. Every single guest network needs IPv4 addressing, so IPv4 planning is mandatory no matter how IPv6 first your thinking is.

What saves the plan is the distinction between private and public IPv4. The mandatory IPv4 is guest network addressing, which comes from RFC 1918 space, and there is plenty of that. What stays scarce is public IPv4 for source NAT and static NAT. So the product shape that works is: private IPv4 always, public IPv6 generously, and public IPv4 as a metered, priced resource.

That is exactly the conclusion Doc 2 reached from a completely different direction, which is a reassuring sign that both analyses are right.

Here is the operational catch that will shape your automation. The documentation states that IPv6 isolated networks and VPC network tiers only support static routing, which means the administrator must add upstream routes for routing to work inside the networks.

Read the consequence at production scale: every time a customer creates a dual stack network, somebody or something must add a route in the upstream router. If that is a person, then customer self service for IPv6 does not exist, because provisioning waits on a human.

Doc 2 proposed a /48 per customer, which is the registry default for an end site and a genuinely good plan.

CloudStack's model does not express it. The guest IPv6 prefix is a zone level setting, and CloudStack carves /64 blocks from it per network, not per account. There is no layer in between where a customer's /48 could live. A customer with three networks gets three /64 blocks, and there is no guarantee they are adjacent or aggregatable.

What survives, and it is the part that mattered: every customer network gets a full /64, which is 18 quintillion addresses, which is more than any customer will use. The generosity survives. The aggregation does not.

The honest correction, since Doc 2 is now partly wrong and I would rather say so than quietly leave it: the per customer /48 is the right model for a network you build yourself and for delegating to a customer who runs their own routing. It is not what CloudStack's isolated network model produces, and no configuration makes it produce that. If per customer aggregation becomes a real requirement, for example because a customer wants to announce their own space, the answer is a routed mode network with a registered subnet, which is section 10, not a different guest prefix setting.

This is the kind of behaviour that turns a routine change into a customer visible incident, so it belongs in a runbook rather than in somebody's memory.

If you update a BGP peer or the network settings, existing Virtual Routers are not reconfigured. A network or VPC restart is required for the change to take effect.

Now think about what that means at scale, because this is the part that reframed it for me. You change your upstream peer's address, perhaps because your transit provider renumbers your link. Every customer network using dynamic routing keeps its old configuration. Fixing it means restarting every affected customer network, and each restart is a brief interruption for that customer.

So a single upstream change becomes a scheduled maintenance affecting every dynamic routing customer. That is a real operational cost, and it needs to be understood before you sell dynamic routing to a hundred customers, not after.

Which is a decent argument for keeping the upstream peer set small and stable, and for treating BGP peer configuration as an infrequent, planned change with a communication plan attached.