Technical Decisions & Stack Rationale

Before the architecture diagram, we owe the why.

This doc records the main technical decisions at executive level: what we picked, what we rejected, and what problem each choice solves. Not religious wars. Honest tradeoffs.

---

How I decide (owner with a technical profile)

I'm not only the business manager. I'm the owner, a big investor in this stack, and I still think like an engineer. That changes how decisions get made.

I put money where it buys the most robust product for downstream customers: the people who pay us and trust us with their sites, mail, and data.

At the same time, this is a starting phase. I'm an open source person when the solution is already benchmarked, battle-tested in production, and solid on security. Security is not a nice-to-have. It's the floor.

So most calls sit on this balance:

Lens	Question I ask
Customer outcome	Does the customer get reliability, clarity, and a secure service?
Cost vs quality	What costs me the least while keeping the best possible quality at the edge?
Open source	Is this OSS mature enough that I'm not paying with my customers' uptime?
Time to market	Can we sell and provision without building half the company in custom code first?
Future deals	Will a sales opportunity tomorrow break because we painted ourselves into one hypervisor corner?

That's the frame for everything below. First big decision: the hypervisor and the control plane.

---

Decisions map (overview)

Area	Decision	Primary alternative considered	Status
IaaS platform	Apache CloudStack on KVM (libvirt/QEMU)	Proxmox-only, VMware, custom provisioner on Proxmox	Documented (below)
Datacenter configuration	Terraform + GitOps on private Git (plan/apply via pipeline)	Manual CloudStack UI, imperative runbooks, backup-only drift recovery	Documented (below)
Corporate / billing ERP	RedOps (ERPNext) + Frappe Insights	HostBill, WHMCS, glued multi-tool stack	Documented (below)
Customer portal	Next.js hybrid (static + SSR), BFF pattern	Public SPA + exposed REST API	Documented (below)
Platform API	FastAPI on private network (not browser-exposed)	Customer-facing REST, GraphQL at edge	Documented (below)
Identity	Keycloak (OIDC), server-side sessions	ERPNext login, JWT in browser, roll-your-own auth	Documented (below)
Internal app runtime	k3s / Kubernetes (portal, API, workers)	Bare-metal systemd only, heavy multi-cluster day one	Documented (below)
DNS platform	TBD	TBD	To document
Email platform	TBD	TBD	To document

---

Platform: CloudStack, Proxmox, and the control plane

The picture: where we start

Picture the stack already funded at the datacenter layer: a serious provider, Tier IV facility, racks, routers, high-bandwidth internet. HostKid starts in Tunisia, first zone aimed at the Arab Maghreb (North Africa: Morocco, Algeria, Tunisia, Libya, Mauritania and the wider Maghreb market), then new regions and zones step by step as the business proves demand.

I need a provisioning plane that can grow in geography and in hypervisor diversity without rewriting the company every time sales brings a real requirement.

Phase 1 thinking: Proxmox

I did not start blind. I started where many engineers start: Proxmox VE. I knew it. ProxGen proved automation discipline on Proxmox. For a lab and for learning, it's excellent.

Then I asked the hosting-company question: what talks to our backend, our ops team, and our portal when a customer clicks Buy VPS?

Proxmox is a strong hypervisor cluster. It is not, by itself, the IaaS control plane a productized host needs: accounts, offerings, zones, quotas, multi-tenant API semantics, and a path for finance and support to trust what's running.

I needed a controller. A provisioner. Something our software team and operations could integrate once, and that could orchestrate creation, lifecycle, and network context at scale.

Building that entire layer myself on top of Proxmox only? Doable. Also how you spend a year before you sell VPS number one.

Why Apache CloudStack entered the room

Apache CloudStack is built as an IaaS orchestration layer. Hypervisor-agnostic by design: KVM, VMware vSphere, XCP-ng, and more in the native model. Zones, pods, clusters, accounts, domains, service offerings, templates, API-first operations: that's the vocabulary of a hosting company, not just a team cloning VMs.

For HostKid that maps cleanly to:

• Vast regions and zones as we expand beyond Tunisia
• Clusters of hypervisor capacity under one control plane
• One integration surface for portal, provisioning engine, and ops (Volume 19)
• Minimal custom control-plane code so time to market stays honest

"But CloudStack can control Proxmox now"

Fair. CloudStack 4.21+ ships an Extensions Framework, and there is a Proxmox VE orchestrator (Shell-based integration, not the same as deep native hypervisor support). ShapeBlue documents this clearly: Integrating Proxmox VE with Apache CloudStack using the Extensions Framework.

I read that. I take extensions seriously. I also read the limitations for production hosting semantics:

Limitation (Proxmox via extension)	Why it matters for a host
Reconfiguring NICs post-deployment not supported	Customer networking changes are normal, not edge cases
Migrate, scale instance not supported	Upgrades and capacity moves are core VPS product features
SSH key injection and User Data not supported	Cloud-init style onboarding is table stakes for developers
Host capacity metrics and utilisation not fetched in CloudStack	Capacity planning and oversell discipline need numbers

Proxmox stays in our story (ProxGen, lab history, possible edge cases later). It is not the foundation we sell on day one.

The sales scenario that locked the decision

Here's the moment that tipped it for me as owner-investor, not just architect.

Sales calls: a client needs a specific hypervisor or deployment constraint. Maybe VMware compatibility. Maybe a KVM-native offering with full cloud-init and resize. Maybe a migration story that requires live migration and capacity metrics in the panel.

If HostKid is Proxmox-only at the control layer, or Proxmox-via-extension with the limits above, we lose the deal. We lose revenue. We look amateur next to hosts who can say yes with a straight face.

What I want instead:

A dynamic architecture: new racks, different hypervisor types over time, but the same controller (CloudStack) so portal, billing, and ops keep one integration model.

That is not hypothetical luxury. That is sales insurance.

Final decision (production path)

Proxmox taught me automation. CloudStack gives me a hosting company control plane. KVM under CloudStack is the production bet. Extensions and alternate hypervisors stay on the roadmap as business cases appear, not as day-one shortcuts that cost deals.

Deep comparison and lab history: Volume 3, Virtualization & platform selection and Handbook: How we chose the platform.

---

Configuration model: Terraform, GitOps, and zero tribal knowledge

We already split the company on purpose. Infra IT owns the factory floor; Product Engineering owns the software that runs orders through it. This decision sits on the Infra IT side of the wall, but Product Engineering depends on it: if platform config drifts, their API integrations lie.

Why I refuse imperative configuration

Old-school ops is not stupid. Back when I started, Terraform, GitOps, and mature provider ecosystems were not the default. Person X clicks through the hypervisor UI for years, keeps the install knowledge in their head, and the company becomes hostage to one human's notebook. I don't blame that person. I blame building a business on unversioned clicks.

Imperative configuration is how you get configuration drift, untraceable changes, and a 2 a.m. incident where nobody knows who changed the service offering last Tuesday.

HostKid does not run that way. Not for hypervisor installs. Not for CloudStack objects. Not for "just this once" hotfixes that never get committed.

What "everything in Git" actually means

Every configuration operation that shapes our datacenter or our CloudStack control plane must live in a private Git repository (GitLab or GitHub), with branches, merge requests, and an approval hierarchy before anything hits production.

The workflow is boring on purpose. That's the feature:

Step	Who	What happens
1. Propose	Infra contributor	Edit Terraform in a branch (new plan, affinity rule, offering, zone object, etc.)
2. Plan	CI pipeline	terraform plan runs against saved remote state; output attached to the merge request
3. Review	Supervisor / Infra lead	Human reads the diff and the plan; approves or rejects
4. Apply	CI pipeline (protected branch only)	terraform apply runs with locked state; destroy paths equally gated
5. Audit	Git history + CloudStack events	Who changed what, when, and why is answerable without oral tradition

Access	Who	CloudStack UI	How changes ship
Standard Infra admin	All datacenter operators and supervisors	Read-only (view state, troubleshoot, correlate with Git)	GitOps repo: branch → plan → review → apply in CI
Break-glass backup user	One locked-down service account (not daily driver)	Read-write (DR and logged emergencies only)	Manual UI fix must be followed by Terraform catch-up in Git
CI pipeline identity	Automation user for terraform apply	API access via provider (not human UI clicks)	Protected branch merges only

How the Terraform pipeline stays secure

Someone will ask this in the first serious review. Fair question: if Git runs the changes, what stops a leaked token or a compromised runner from owning CloudStack?

The VCS can be GitHub or GitLab. That choice is flexible. What is not flexible: we own the runners, we isolate them, and only they talk to the CloudStack API.

Layer	Decision	Why it matters
Runners	Self-hosted on air-gapped VMs we own	No third-party runner pool holding our CloudStack credentials
Network	Egress/ingress limited to one VCS endpoint (GitLab or GitHub)	Shrinks attack surface; traffic is auditable and deny-by-default
CloudStack RBAC	Dedicated pipeline service account with least-privilege API role	Apply can only mutate objects Infra Terraform is allowed to own
Secrets	HashiCorp Vault or Sealed Secrets at runtime; never committed to Git	API keys and tokens are injected in the runner, not stored in repo history
SSDLC / CI gates	Syntax validation, linting, terraform validate, plan-only on merge requests	Broken or sloppy code never reaches apply
Secret scanning	Pre-commit hooks + server-side pre-receive hooks (gitleaks and peers)	Blocks passwords, tokens, and keys at commit time before they enter Git

Secrets in Git are a failure mode, not a workaround. Vault or sealed secrets at deploy time, gitleaks at the door, air-gapped runners at the keyboard. That is how we support secure SDLC without pretending humans never paste a token wrong once.

Implementation detail (runner hardening, Vault paths, firewall rules, hook config) lands in Volume 7, Infrastructure as Code and Volume 22, Security. Volume 0 records the decision; the volumes carry the wiring diagram.

Rollback is revert the commit and plan again, not "remember what the UI looked like." Backup and restore of CloudStack or storage? Last bullet. Critical disaster recovery only. Not our daily drift strategy.

This is the same muscle ProxGen trained on Proxmox: plan, apply, destroy, state that remembers. CloudStack is the new surface. The discipline stays.

Why CloudStack fits this decision

Choosing CloudStack was not only about hypervisor flexibility. It was also about mature automation support.

Apache CloudStack ships an official, Apache-licensed Terraform provider, maintained under the CloudStack project: apache/cloudstack-terraform-provider. It is published to the Terraform Registry as cloudstack/cloudstack, with documented resources for the platform objects Infra IT actually manages.

That provider is why "GitOps for the control plane" is a day-one requirement, not a future science project. Proxmox-as-primary with a half-feature extension would have fought us here too.

Detailed implementation is intentionally not in Volume 0. This page records the decision and the rules. The volumes carry the runbooks.

What Infra IT terraforms vs what the backend owns

Clear boundary. Company admins and Infra IT configure the platform catalog and topology. Customer instances flow through our backend and provisioning engine (Volume 19), not through manual VM clicks in the UI.

Three questions settle almost every argument:

Question	If yes →	If no →
Is this a platform object (zone, plan, template, host, affinity rule)?	Infra IT · Terraform in the GitOps repo	Keep reading
Did a customer trigger it from the portal (VM, disk, snapshot, key)?	Backend · provisioning engine	Keep reading
Is it audit or monitoring (events, alerts)?	Observe and correlate with Git; do not "configure" by hand	Escalate: probably a new platform object or product feature

The full CloudStack object catalog and Terraform resource mapping live in Volume 4, CloudStack Platform. Here are two everyday Infra IT paths so the GitOps vision is concrete.

Example 1: Sales wants a new VPS plan

Product asks for a 4 vCPU / 8 GB tier for the Maghreb zone. Infra IT does not open the CloudStack UI and click "Service Offering." They open a branch in the GitOps repo, add or update a cloudstack_service_offering resource (name, CPU, memory, storage tags), open a merge request, let CI run terraform plan, get supervisor approval, and merge so the air-gapped runner applies. The portal team maps the new SKU to that offering ID. Git history shows who approved the plan and when.

Example 2: Ops wants an anti-affinity rule for a noisy neighbor tier

A premium tier must not stack multiple VMs on the same host. Infra IT defines an affinity group in Terraform (cloudstack_affinity_group with anti-affinity policy), same pipeline: branch → plan → review → apply. Customer VMs on that tier still get created by the backend at order time; the backend attaches the offering and affinity group the platform already declared in code. Platform policy in Git; customer lifecycle in the provisioning engine.

CloudStack gives us the control plane. Terraform plus Git gives us memory, rollback, and team scale. Together they are why Infra IT can grow past one person who "just knows the clicks."

---

Corporate stack: RedOps, ERPNext, and why not HostBill

Picture the milestone: physical infra is ready, CloudStack is configured, Terraform pipelines behave, the first backend and frontend can sell a VPS. Automation on the IaaS side is real. And then you hit the wall every hosting founder hits late:

Payment confirmation and bill management are not back-office paperwork. They are the trigger that controls the entire customer resource lifecycle.

Invoice paid → entitlement exists → provisioning engine deploys. Subscription lapses → suspend → destroy or archive. Get that wrong and you either give away capacity or anger paying customers. The control plane can be perfect; if billing and provisioning disagree, the company still bleeds.

What I evaluated: mature billing-only tools

Tools like HostBill (and peers in the same class) are impressive. Large surface area, long production history, deep hosting-specific features: clients, invoices, gateways, addons, support hooks. I did not dismiss them lightly.

HostBill wins if your only problem is "bill for hosting." HostKid's problem is "run a hosting company," and that is bigger than invoices.

The glued-services trap (I've lived this)

In companies I worked across, I kept seeing the same architecture smell:

Need	Typical "solution"	What actually happens
Customer billing	HostBill / WHMCS / similar	Export CSV, webhooks, prayer
Internal accounting	Another ERP or spreadsheet	Manual reconciliation every month
HR, attendance, payroll hooks	Yet another HR tool	Employee exists in three places
Internal assets and procurement	Inventory or "we'll track in Notion"	No single asset truth
Business KPIs and owner view	BI tool #4	Dashboards that disagree with finance

Administration turns into fog: jump between glued micro-products, each with its own login, schema, upgrade cycle, and integration tax. Never forget the hidden line item: the cost of building and maintaining the glue code between them. Every sync job is a future 2 a.m. incident.

I refused that for HostKid. Same instinct as CloudStack over Proxmox-only and Terraform over UI clicks: one mature control surface beats five best-in-class silos.

RedOps: one secure place for the whole company

Here enters a project I architected and developed: RedOps.

RedOps is a SaaS company delivering a mature ERP for businesses that sell products or services. The core runs on ERPNext, which is the part that matters technically: one open source ERP with real modules for billing, accounting, CRM, HRMS, projects, inventory, and operations data in a single data model.

What RedOps adds on top is the product layer HostKid actually wants to operate:

• Unified business resource management in one place: HR and attendance, accounts and accounting, internal datacenter and asset context, performance records, sales, subscriptions, customers
• AI integrated across the stack, not bolted on as a chat widget: an employee asks in human language ("create a lead for this prospect," "show overdue hosting invoices," "what did we spend on racks this quarter?") and the assistant acts inside ERPNext with proper auth and audit
• Frappe Insights for dashboards and KPIs, with RedOps pushing further: AI-assisted live dashboards on demand instead of a static wall of charts nobody maintains

Rather than insanely limitless glued services, we get finance, HR, billing, CRM, and executive visibility in one secure system, with AI and Insights on the same data the provisioning engine trusts.

I'm not picking ERPNext because it is trendy. I picked it because I have ** shipped on this stack**, and RedOps is the shaped delivery model I trust for a host that intends to grow past ten people without admin schizophrenia.

How this connects to the customer lifecycle

The customer portal and backend (Volume 10, Volume 19) remain the customer experience. RedOps / ERPNext remains the commercial and corporate system of record.

Event	Corporate system (RedOps / ERPNext)	Platform
Customer buys VPS plan	Subscription + invoice + payment state	Portal creates order; ERP confirms entitlement
Payment confirmed	Invoice marked paid; SKU entitlement active	Provisioning engine deploys on CloudStack
Renewal fails	Dunning, suspension policy, finance visibility	Engine suspends or destroys per rules
Owner asks "are we profitable this month?"	Frappe Insights on ERPNext data	Same revenue numbers provisioning used, not a shadow spreadsheet

Final decision

CloudStack runs the factory. Terraform remembers the factory layout. RedOps/ERPNext remembers who paid to use the factory and whether the company itself is healthy.

Module setup, payment gateways, Insights cockpits, and AI workflow detail: Volume 11, Corporate. HR and fair evaluation tie-in: Core values & human engine.

---

Portal, platform API, identity, and runtime

CloudStack is configured. Terraform pipelines behave. RedOps/ERPNext knows who paid. The next gap is how the customer experiences the product and how Product Engineering orchestrates ERP, identity, and CloudStack without exposing an attack surface the size of a public hosting API.

Hybrid portal: static where we can, SSR where we must

Marketing pages, docs, and static assets ship as static content. The client portal (buy VPS, manage services, billing views) is server-side rendered. The browser gets HTML and cookies, not a JSON API surface to reverse-engineer.

Only the portal server calls the platform API internally. Create VM, edit service, view invoices: form post or server action → internal API → queue or sync response → HTML back to the customer. Same security instinct as CloudStack: humans and browsers on the outside, automation and orchestration on the inside.

Platform API: FastAPI, private, orchestration layer

The platform API (FastAPI, Python) is Product Engineering's integration hub. It talks to RedOps/ERPNext for billing and entitlements, enqueues work for the provisioning engine, and reads state from CloudStack through workers, not from the customer's browser thread.

API tier	Exposure	Purpose
Browser	None (HTML only)	Customer portal via SSR
Platform API	Private network only	Portal server, workers, payment webhooks
Partner API (later)	Separate public gateway if we need it	Resellers and power users, not the default portal path

Long work (provision, destroy, suspend) goes through a queue and workers (Volume 19). The SSR page shows status; it does not block on CloudStack deployVM finishing.

Identity: Keycloak, not ERP login in the browser

Customers authenticate with Keycloak (OIDC). ERPNext/RedOps remains the commercial record (Customer, subscription, invoice). Keycloak answers "who is logged in"; ERP answers "what did they pay for"; the platform links the two.

Server-side sessions (httpOnly cookies), MFA for customer accounts, step-up auth for destructive actions: sub-decisions, but the headline is fixed. No ERPNext API keys in the frontend. No JWT in localStorage theater.

Internal runtime: k3s for scale without day-one complexity

Portal, platform API, workers, and observability stacks run on k3s/Kubernetes (Volume 8). Not because we want Kubernetes cosplay. Because when Maghreb demand grows, we scale replicas and roll deployments without rebuilding the architecture.

k3s keeps the ops burden honest for a starting host: one internal platform cluster, ingress for the portal only, network policies so only the portal talks to the API and only workers talk to CloudStack management. Helm, cert-manager, and storage classes are implementation detail in Volume 8, not executive debate.

Final decision

CloudStack runs VMs. RedOps runs money. The portal is the storefront window. The platform API is the stockroom door, and it stays behind the building.

Topic	Volume
Portal SSR, BFF, billing UI integration	Volume 10, Customer Portal
Keycloak, OIDC, MFA, session model	Volume 9, Identity
Provisioning queue, CloudStack workers, lifecycle	Volume 19, Provisioning Engine
Partner/public API (if and when)	Volume 20, APIs
k3s, ingress, scaling portal/API/workers	Volume 8, Kubernetes Platform

---

Other stack choices

Placeholder. DNS (PowerDNS), email (Mailcow), monitoring, logging, shared hosting panel, object storage: brief rationale per line when decided.

---

Decision principles (summary)

1. Customer-facing robustness beats founder comfort with a favorite tool
2. Open source when mature, not open source for ideology
3. Security and compliance intent are non-negotiable
4. One control plane for provisioning; hypervisor diversity under it when needed
5. Declarative config in Git with Terraform; UI clicks are not the source of truth
6. Pipeline security by design: owned air-gapped runners, Vault/sealed secrets, gitleaks at commit
7. Document the rejection (Proxmox-only, HostBill-only) as honestly as the pick
8. One corporate system of record (RedOps/ERPNext) beats glued billing + HR + BI silos
9. Portal SSR, private platform API; browser never holds infra or ERP secrets
10. Arab Maghreb first, expand zones when metrics say go

---

What this doc does not cover

• VLANs, zones, Helm charts → High-level architecture
• CloudStack install and operations → Volume 4
• Terraform repos, pipelines, and modules → Volume 7
• Change approval and break-glass → Volume 27
• Portal, identity, platform API detail → Volumes 8, 9, 10, 19, 20
• Portal, identity, platform API detail → Volumes 8, 9, 10, 19, 20
• ERPNext module setup, RedOps integration, payment sync → Volume 11

---

Next: High-level architecture. The boxes and arrows, now that the why is on record.