A normal company defends against outsiders. The threat is outside, the assets are inside, and the firewall is the line between them.

A hosting company also defends against its own customers, because we are deliberately renting compute to strangers on the internet and taking their money for it.

Some of those customers will be compromised, running a vulnerable application that somebody else now controls. Some of them will be the attacker, having signed up with a stolen card specifically to abuse the platform. And some will be perfectly legitimate and simply incompetent, which produces similar traffic.

Your design has to assume all three, from day one, as normal operating conditions rather than incidents. That single assumption is what separates a hosting network design from a corporate one, and almost every decision in this document follows from it.

Edge firewall rules must not need to change when a customer is provisioned. CloudStack documents this itself, and it is the single most important architectural boundary in this document.

Why it matters so much: if selling a VPS requires an engineer to hand edit an edge firewall rule, then your business cannot scale, your change control becomes theatre, and your rule set rots into hundreds of permits nobody can explain. Within a year you have a firewall configuration that no living person is willing to clean up, because nobody knows which rule some customer depends on.

So: per tenant policy lives in the Virtual Router or in security groups, where it is created and destroyed by automation. The edge firewall expresses company policy, which is stable, reviewable, and small enough that a human can read the whole thing in one sitting.

If you ever find yourself adding a customer specific rule to the edge firewall, stop. Something upstream in the design is wrong.

"Management can reach guest" and "guest can reach management" are completely different statements with completely different risk.

The first is normal and necessary. The second is a tenant escape path.

Because firewalls are stateful, allowing management to initiate connections to guest networks does not allow guests to initiate connections back. Return traffic for an established session is permitted automatically. New inbound sessions are not.

That asymmetry is the whole mechanism, and if you do not have it clear in your head, you will write rules that are twice as permissive as you intended. Every cell in the matrix below is directional for exactly this reason.

Every firewall has a final rule. Ours drops and logs.

That sounds obvious and the failure mode is subtle: teams start with default deny, then hit a problem at 2 a.m., add a broad permit to make it work, and never come back. Six months later the broad permits are the policy and default deny is a comment at the bottom of the file.

So every exception carries three things, matching the exception rule from governance, risk and compliance:

A named owner. A person, not a team, because teams do not answer questions.

A documented reason, in the rule comment, written for someone who was not in the room.

An expiry date. This is the one that actually works. An exception that expires forces a conscious decision to renew it, which is how temporary access stops becoming permanent by neglect.

That framing is not pessimism, it is the design constraint. You are deliberately running deliberately vulnerable services and inviting attackers to succeed against them. Some of them will.

Therefore VLAN 60 is walled off from everything. No route to management, no route to storage, no route to guest networks, no shared credentials with anything real, and no trust relationship of any kind with production. Read the Deception row of the matrix again: outbound to the internet, rate limited and monitored, and deny to every internal zone.

The theory is appealing: an attacker must defeat two firewalls to reach the internal network.

The catch is that this only holds if the two firewalls are genuinely different. Two appliances from the same vendor running the same firmware share the same vulnerabilities, so the exploit that defeats the first defeats the second. You have doubled your cost and your operational burden for very little additional security.

Different vendors gives you real defense in depth and gives your team two products to master, two patch cadences to track, and two configuration languages to keep consistent. For a small team, that operational load is itself a risk, because misconfiguration is a far more common cause of breach than firewall exploitation.

Our position: one well configured, well monitored, promptly patched firewall beats two poorly maintained ones. We start with the three legged design and revisit this when the team is large enough to operate two products properly.

Port 8250, the management server agent port. This is how hypervisor hosts talk to the management server. It belongs on VLAN 20 and nowhere else. Exposing it offers attackers the channel that controls your hypervisors.

Port 8096, the integration API. Read this carefully: it is unauthenticated by design. It exists so local automation can call the API without credentials, and CloudStack ships it disabled or bound locally for that reason.

An exposed port 8096 is a complete, unauthenticated takeover of your entire cloud platform by anyone who finds it. Not privilege escalation, not a data leak. Full control, no password, no exploit required.

Our position: 8096 stays disabled. If some automation genuinely requires it, it binds to localhost only and is reached through an SSH tunnel from the jump host, never through a firewall rule.

And it goes in the verification checklist as an explicit external scan, because this is not a control you assume. Exact configuration lands in Doc 5.

One firewall appliance. It is a single point of failure for the entire perimeter, exactly like the single ToR switch in Doc 3.

We do the same three things we did there: write it down as an accepted risk with a named owner, monitor it properly so we know immediately when it degrades, and keep the configuration in version control so a replacement can be brought up from a known state rather than from memory.

The upgrade trigger is the same as the switch pair: the first customer with a contractual availability commitment. A risk with a documented trigger is a plan. Without one it is an excuse that ages badly.

Not "can cause problems". Not "may access some data". Owns.

With management plane access, an attacker can create and destroy any VM, attach any customer's disk volume to a machine they control and read it, snapshot any customer's data, reach every hypervisor, and delete the backups.

So this section is short, absolute, and has no convenience exceptions. Every rule here costs somebody a small amount of daily friction, and every one of them is worth it.

Volume 0 committed to one break glass CloudStack account for disaster recovery. Here is how it is protected, and the design principle is that we cannot prevent its use, so we make it impossible to use quietly.

Credentials stored offline, in a sealed physical envelope in a safe, or split so that two people are required.

Use triggers an immediate alert to management and security, automatically, with no way for the user to suppress it.

Every use is reviewed afterwards, regardless of who used it or why, as a scheduled review rather than an optional one.

Credentials rotate after every use, without exception.

The account is monitored for existence, so its removal or modification is itself an alert.

The principle: you cannot stop a legitimate emergency, and you should not try. What you can guarantee is that nobody uses it without everybody knowing. That is what turns a backdoor into a documented control.

Doc 3 established that BMCs are the most privileged and least monitored surface in the rack, and that BMC firmware has a poor security history.

The temptation, once you are tired, is to make OOB "just reachable from management" because that is easier during an incident.

Do not. Look at the OOB row in the zone matrix: it initiates to nothing, and nothing initiates to it except the jump host with separate authorization.

The reasoning: if management can freely reach OOB, then compromising management gives an attacker power control and pre operating system access to every machine you own, which is a level of control that survives every reinstall you might attempt as a response. Keeping OOB separate means a management compromise, catastrophic as it is, still leaves you a recovery path the attacker does not hold.

Enabling security groups in a zone is a zone level architectural decision, and CloudStack's own documentation is clear about the consequences: in a security group enabled zone you cannot use VPCs or isolated networks, and such shared networks come with single VLAN range restrictions.

Why that phrasing matters: this is not a setting you toggle later after you have customers. It shapes what your product catalogue is capable of offering, permanently, for that zone.

So the decision belongs here, in network design, before the first customer exists, rather than being discovered by a product manager who has already promised a customer a multi tier VPC.

A tenant escape or VM escape is when code inside a customer's VM breaks out into the hypervisor. From the hypervisor, an attacker reaches every other VM on that host.

VLANs do not protect you here, and neither do security groups, because both operate at the network layer and the attacker is now underneath it.

What follows from that, concretely: KVM and QEMU security updates get an expedited patch path rather than the normal cadence, live migration exists partly so hosts can be patched without customer downtime, and "we will patch it next quarter" is not an available answer for this specific class of update.

This is also the honest answer to a customer asking whether their VPS is as isolated as a dedicated server. It is not, quite, and the difference is this class of vulnerability. Saying so plainly builds more trust than claiming otherwise and being wrong later.

Think about what a hosting provider is: machines with substantial bandwidth, rented to people you have not met, on your address space, under your ASN.

If a tenant on your network can send spoofed packets, then your network is a reflection amplifier available for hire.

Walk through the consequences in order. The attacker rents the cheapest VPS you sell. They send a modest stream of small queries to public services, with the victim's address forged as the source. Those services reply, and the replies are many times larger than the queries and are aimed at the victim.

The victim sees a large attack. The services that replied see nothing unusual. And the traffic that started it all left your ASN.

This is why anti spoofing is not one control in one place. It is enforced at four layers below, on the principle that any single layer might be misconfigured and the others still hold.

The uncomfortable truth about anti spoofing is that a broken control is completely invisible.

If IP source guard is misconfigured, nothing breaks. No error appears, no alarm fires, no customer complains. Traffic flows normally. The only observable difference is that spoofed packets now succeed, and you will find out when a transit provider calls you.

Suppose we buy a 10 Gbps uplink, which is a serious circuit for a starter hosting business.

An attacker with a modest botnet or a rented attack service can generate hundreds of gigabits per second. Not theoretically. As a commodity service, priced in tens of dollars.

Your firewall could be infinitely fast and it would change nothing, because the congestion happens upstream of you, on your provider's link, before a single packet reaches equipment you control. You cannot filter traffic that has already saturated the pipe it would arrive through.

Every honest DDoS plan for a small provider starts by accepting that, and then asks the only useful question: what do we do about the part that is upstream? The answer is section 6's whole point, and it is not "buy a bigger firewall."

In the normal case, the server replies SYN ACK and reserves memory for a connection it expects to complete. That reservation is the vulnerability, and it is not a bug. It is how TCP has to work.

In the attack, the SYNs carry spoofed source addresses, so the server's SYN ACK goes to a machine that never asked for anything and will never reply. The connection sits half open, holding resources, until it times out.

The asymmetry is brutal: the attacker sends one small packet and never has to remember anything about it, while you allocate state and wait. A few thousand of these per second fills the table, and then legitimate connections are refused. Your server is not overloaded and not compromised. It has simply run out of room to accept anyone new.

Two things follow. First, this is why session table size and new sessions per second are on the appliance buying checklist rather than raw throughput. Second, SYN cookies are the defence: the server encodes the connection state into the sequence number it sends back, so it can validate a returning ACK without having reserved anything. No state held means nothing to exhaust. Enable it, verify it, and know why it works.

Read what a blackhole actually does one more time: it drops all traffic to that address.

So you have not saved the customer under attack. You have finished them off, on purpose, to save everybody else. Their service is now completely unreachable, and the difference between that and the attack succeeding is, from their point of view, nothing at all.

What you have saved is every other customer sharing that uplink, who were about to go down as collateral damage.

That is a genuine business decision disguised as a technical one, which is exactly why it needs a documented decision authority and a threshold agreed in advance, when nobody is panicking and nobody is on the phone. Improvising this at 3 a.m. produces either paralysis or an unfair outcome, and both are avoidable.

It looks like an operations or legal topic, and here is why it sits in Volume 1.

When a complaint lands, you need to answer four questions quickly: which customer was using that address at that time, what were they actually doing, can you prove it, and can you stop it right now without touching anything else.

Every one of those depends on infrastructure decisions. Flow telemetry has to already be collecting. Logs have to already be retained with accurate timestamps. IP assignment history has to be recorded. Per tenant network controls have to exist so you can throttle one customer without affecting neighbours.

None of that can be retrofitted during your first serious complaint, which is precisely when you will wish you had it. Design it now, use it later.

An unmonitored abuse@ address, or an out of date RIR contact, does not make complaints go away. It makes them escalate.

The reporter cannot reach you, so they go to your transit provider instead. Now the conversation is not "please look at this customer", it is your provider asking why they are receiving complaints about a customer who does not respond. That conversation has a much worse ending available to it.

Being reachable and responsive is, genuinely, most of abuse handling. A provider who replies within hours and acts within a day almost never escalates further, even when the underlying abuse takes time to resolve. The responsiveness is the reputation.

An abuse report says: "your IP 203.0.113.47 attacked us at 14:32 UTC on 2 August."

To act on that you must answer:

Which customer held that address at that moment? Requires IP assignment history with timestamps, not just current state.

What was that address actually doing? Requires flow telemetry, retained.

Is the report credible? Requires your own data to corroborate or contradict it.

Is it still happening? Requires current visibility.

And the one that catches people out: every one of those answers depends on accurate, synchronized clocks. If your timestamps are wrong by even a few minutes, you will identify the wrong customer, because public addresses get reassigned. Suspending an innocent customer based on a bad timestamp is a serious, avoidable failure.

So NTP across all infrastructure, in UTC, monitored for drift, is an abuse handling control. It looks like housekeeping and it is evidentiary integrity.

Doc 1 covered SPF, DKIM, DMARC, and the reputation economy of email. Here is the operational consequence for a hosting provider.

That is a shared fate problem, which makes it different from every other abuse category. A customer running a port scan harms the scan target. A customer sending spam harms every other customer you have, and they will not know why their invoices stopped arriving.

Port 25 blocked by default for new customers, opened on request after basic verification. This single control removes most casual abuse, because a spammer who has to ask and be verified usually goes elsewhere.

Outbound rate limits per customer, so a compromise is bounded rather than unlimited.

Reputation monitoring on our own ranges, so we discover a listing ourselves rather than through a confused customer's support ticket.

Separate our platform mail from customer mail, on different addresses, so a customer's mistake cannot stop our own password reset emails from being delivered.

Correct reverse DNS on every sending address, which Doc 2 set up the delegation for.

Blocking port 25 by default will annoy a small number of legitimate customers. It is still correct, and the honest way to handle it is a fast, friendly verification path rather than pretending the restriction does not exist.

This is the workflow, and it is only two commands.

The first output identifies the target. 203.0.113.47 is receiving 402 gigabytes in fifteen minutes at 3.8 gigabits per second, while the next busiest address is receiving fifty times less. That is not a busy customer. That is an attack, and the address is the target.

The second output identifies the shape. Filtering to just that destination and ranking by source, look at what you get: the top sources are all sending roughly identical volumes, around 5 megabits each, from completely unrelated networks.

And notice what you cannot do: there is no small set of sources to block, because the load is spread evenly across thousands. Which is precisely why the decision tree in section 6 routes a volumetric attack on a single customer address to RTBH rather than to a firewall rule. You have now confirmed with data that a firewall rule would be futile.

An attacker who compromises a machine will edit or delete its logs. That is not sophisticated behaviour, it is standard behaviour.

So logs ship off the host as they are written, to a collector the source host cannot modify or delete from.

The property you want is append only from the source's perspective. A compromised web server can stop sending new logs, and it cannot reach back and alter what it already sent. So your record of the compromise survives the compromise, which is the entire point.

An alert that fires constantly is not an alert, it is background noise with a notification attached. And the failure mode is not that people ignore it deliberately. It is that they stop seeing it, so the one time it matters, it looks like all the other times.

Two rules keep this list honest.

Every alert needs a documented action. If the answer to "what do I do when this fires" is "look at it", it is a dashboard metric, not an alert. Move it.

Tune thresholds against your baseline, which is why baselining is in this section rather than in a monitoring volume. Thresholds invented before you know your normal traffic will be wrong in both directions.

Alert routing and the reactive automation this feeds are developed in Volumes 17 and 18.

Every other test on that list is technical, and technical tests either pass or fail unambiguously. The abuse tabletop tests something harder: whether your organization can act.

Run it properly. Somebody plays the complainant and sends a realistic report to abuse@. Then the team has to actually identify the customer from the address and timestamp, actually find corroborating evidence, actually apply a throttle, and actually draft the customer notification.

You will discover missing pieces. Everybody does. Usually it is that nobody knows how to map an address and a timestamp back to a customer quickly, or that no notification template exists so the first one gets written under pressure and badly.