Sizing & Capacity Planning

Every Marshal cluster splits its compute into two node pools that are sized — deliberately — for different jobs:

Control plane — the always-on services: API, gateway, web dashboard, marketplace, and Redis. Steady, predictable load.
Spaces (sometimes called “sessions”) — the on-demand pods that back each agent session. Bursty, short-lived, and the thing you’ll actually scale as usage grows.

This page gives you the reasoning and the rules of thumb to size both pools for your own cluster. For the exact Terraform variables and machine-type flags on a specific cloud, see Installing Marshal — this page stays cloud-agnostic. For how these pools fit into the rest of the system, see Architecture.

Control plane vs. session pods

Treat these as two independently-sized pools — never share a machine type, disk size, or node configuration between them.

	Control plane	Spaces (sessions)
Runs	API, gateway, web, marketplace, Redis	One pod per active agent session
Load shape	Steady, always-on	Bursty, scales with concurrent sessions
Typical starting machine	8 vCPU class (e.g. `e2-standard-8` / `m5.2xlarge`)	4 vCPU class (e.g. `e2-standard-4` / `m5.xlarge`)
Typical starting disk	Smaller — no per-session storage demand	Larger — sized around ephemeral storage, see below

Why control plane needs a real baseline, not a small machine. The platform’s own chart defaults to 2 replicas each of the API, gateway, web, and marketplace services — 8 pods before you’ve onboarded a single user. Summed together, that baseline already requests around 4.4 vCPU, before any horizontal autoscaling kicks in under load. A smaller 4-vCPU-class machine typically has under 4 vCPU of allocatable capacity once you account for node overhead, which doesn’t leave room for that baseline — pods land Pending with an Insufficient cpu event, and the control plane never comes up healthy. Start control-plane nodes at the 8-vCPU class (or your cloud’s closest equivalent) and treat going smaller as something to prove out under load, not a default to assume.

Why sessions get a different (usually cheaper) machine. Session pods don’t need the same per-pod CPU floor — the constraint that actually caps how many sessions fit on a spaces-pool node is disk, not CPU. That’s covered next.

Keep these pools isolated (via taints/tolerations or your cloud’s equivalent) once you’re past initial evaluation. It stops a burst of session traffic from starving the control-plane services that everything else depends on. See your cloud’s install guide under Installing Marshal for how to turn isolation on.

Ephemeral storage math (spaces pool)

Each session pod requests roughly 10–20 GiB of ephemeral storage, depending on the toolchain and workload it’s running. That’s the number that actually determines how many concurrent sessions fit on a spaces-pool node — not CPU or memory, which are usually roomier by comparison.

A chunk of every node’s boot disk is reserved by the platform for the OS and the container image cache before any of it is available to pods. In practice, plan on losing a meaningful fraction of the raw disk size to that overhead — don’t assume a 100 GB disk gives you 100 GB of schedulable ephemeral storage.

Rule of thumb:


usable ephemeral storage  ≈  node disk size − OS/image-cache overhead
sessions per node         ≈  usable ephemeral storage ÷ per-session request

Worked example: a 100 GB spaces-pool disk, with ~20–30 GB reserved for OS/image cache, leaves roughly 70–80 GB usable. At 15 GiB per session (the middle of the typical range), that’s around 4–5 concurrent sessions per node before the pool needs to scale out. Multiply by your max_node_count to get the pool’s ceiling, and work backward from your expected peak concurrent-session count to size the disk and node count together, rather than picking a disk size first and hoping it’s enough.

If you’re running heavier toolchains (large monorepos, big dependency caches, container-in-container builds), measure actual ephemeral-storage usage for your workload rather than assuming the low end of the range.

Cloud quota gotchas

Your cloud account’s CPU quota and disk quota are tracked separately, and either one can silently cap your cluster’s ability to scale — hitting one produces the same symptom as hitting the other (nodes fail to provision, pods sit Pending), so check both before you commit to a machine_type × max_node_count × disk_size combination.

CPU quota (e.g. GCP’s CPUS_ALL_REGIONS, or an AWS vCPU-based service quota) caps the total vCPUs your account can run across all node pools combined. This is the one people remember to check.
Disk quota (e.g. GCP’s regional SSD_TOTAL_GB, default 500 GB) caps total provisioned disk independently of CPU. It’s easy to forget because it’s tracked on a completely separate dimension — you can have plenty of CPU quota left and still fail to scale out because the spaces pool’s disk_size_gb × node_count has quietly used up your disk allowance.
Both pools draw from the same account-level quotas. If the spaces pool is already using most of your disk quota, scaling the control-plane pool’s node count (even without changing its machine type) can fail for a disk reason that has nothing to do with CPU.
When a scale-out fails, check both quotas before assuming it’s a CPU problem — the fix for insufficient CPU quota (bigger machine or a quota increase) is not the fix for insufficient disk quota (smaller disk, fewer max nodes, or a disk quota increase), and applying the wrong one wastes a round trip.

Putting it together

Start control-plane sizing from the chart’s own baseline, not from how much traffic you expect — 2 replicas each of 4 services already sets the floor, regardless of usage on day one.
Size the spaces pool from your target concurrent-session count, using the ephemeral-storage math above, not from CPU/memory alone.
Check CPU quota and disk quota as two separate line items for every pool, at your intended max_node_count — not just at today’s node count.
Re-check both quotas whenever you raise max_node_count on either pool — that’s the change most likely to hit a ceiling you haven’t looked at recently.

For the exact machine-type flags, disk parameters, and quota names on your specific cloud, continue to Installing Marshal.