Troubleshooting

Known failure modes, grouped by area. Each entry is a symptom you’d actually see, followed by the cause and the fix.

Registry & images

Bundled Redis pod stuck `ImagePullBackOff`

Cause: Bitnami retired every version-pinned tag from the free bitnami/* Docker Hub namespace. On chart versions that predate the fix, the bundled Redis (and Postgres) subcharts’ default image references 404.

Fix: Upgrade to a current chart version, which points at the fixed image references. If you’re stuck on an older chart in the meantime, override the image repository directly via the escape hatch:


values_overrides = [
  <<-EOT
  redis:
    image:
      repository: bitnamilegacy/redis
  EOT
]

bitnamilegacy/* is Bitnami’s own migration path and mirrors the exact same tags — it’s an unblock, not an upgrade. If you’d rather avoid this class of problem entirely, set redis_mode = "external" and point at a Redis instance you already run — Redis holds no durable state in Marshal, so this only costs you a small managed instance.

Missing EBS CSI driver — bundled Redis PVC never binds (EKS)

Cause: Unlike GKE (whose PD CSI driver ships enabled by default), EKS does not enable the EBS CSI driver out of the box. Without it, the PVC the bundled Redis subchart requests never binds, and the pod stays Pending.

Fix: Install the EBS CSI driver as an EKS addon before installing Marshal, or use the deploy_cluster = true provisioning path, which installs it for you automatically.

Missing AWS Load Balancer Controller — Ingress never gets an address (EKS)

Cause: terraform apply (or argocd app sync) succeeds even without the AWS Load Balancer Controller installed — but with nothing to reconcile the Ingress, no load balancer is ever created, and the Ingress simply never gets an address. There’s no error pointing at the missing controller.

Fix: Install the AWS Load Balancer Controller before installing Marshal onto a reused cluster, or use the deploy_cluster = true provisioning path, which installs it (with its own IAM role) for you.

Trying to set up an ECR pull-through cache for Marshal’s registry fails

Cause: ECR’s pull-through cache feature only accepts a fixed enum of known upstream registries (ECR, Docker Hub, GitHub/GitLab container registries, Quay, and a few others) — it validates the upstream URL against each type’s real hostname, so there’s no way to point it at a private registry like Marshal’s. This is a hard platform limitation, not a configuration mistake.

Fix: On EKS, use direct pull (the default) for a small install, or stand up a self-hosted proxy-cache (e.g. Harbor) in front of the registry if you need a shared cache — some EKS install flavors offer this as an opt-in. This limitation is specific to ECR; a GCP Artifact Registry remote repository does not have this restriction and can mirror the registry directly.

Ingress & TLS

Browser shows a certificate warning on every request

Cause: This is expected with the default TLS mode — a self-signed placeholder certificate (clearly marked in its subject so it’s unmistakable in a cert viewer) is generated automatically so HTTPS works out of the box, before you’ve supplied a real certificate.

Fix: Replace it once you’re ready, by switching to an inline certificate you provide, or to a Kubernetes Secret you manage yourself (e.g. via cert-manager). No other configuration needs to change either way. See Configuration Reference → Ingress & TLS.

Wildcard preview certificate request stuck pending, never issues

Cause: Two common causes, both specific to the wildcard cert a preview domain needs (*.<preview_domain>):

A GKE-managed certificate was used — Google’s managed-certificate product does not support wildcard domains at all, and simply never provisions for a *. entry.
A cert-manager ClusterIssuer was configured with the default HTTP-01 challenge solver — HTTP-01 can only prove control of one exact hostname per challenge, which is structurally impossible for a wildcard (there’s no single backend that can answer the challenge for every possible subdomain).

Fix: Don’t use a managed-certificate product for a wildcard. With cert-manager, configure a DNS-01 solver (e.g. your cloud DNS provider) for the preview domain’s zone instead of HTTP-01. If you don’t need this today, the default self-signed placeholder certificate already covers the wildcard correctly out of the box — this only matters once you’re replacing it with a real, trusted certificate.

Secrets & sessions

Sessions suddenly all invalidated, or every stored credential stops decrypting.

Cause: secrets.provider was left at (or reset to) generated on a GitOps-managed install. ArgoCD renders and applies manifests directly — it never runs helm install, so the chart’s normal mechanism for reusing an already-generated secret never runs. With generated, that means a fresh random signing key and encryption key are minted on every single sync: the signing key rotation invalidates every issued session and API token, and the encryption key rotation makes every previously-stored credential permanently undecryptable.

Fix: Set secrets.provider: existing (or externalSecrets) and seed a stable secret once, outside of ArgoCD’s management — this is the required posture for any GitOps-managed install, never generated. See Configuration Reference → Secrets management. There is no fix that recovers already-rotated credentials; the only way out is to stop the rotation from recurring and have users re-enter affected credentials.

Node pools & quotas

A node stays stuck Ready,SchedulingDisabled indefinitely during a control-plane node pool change.

Cause: A single-node control-plane pool being resized or otherwise changed gets cordoned and drained before Terraform replaces it — but the chart sets a PodDisruptionBudget requiring at least one replica of every control-plane component to stay available. With only one node, draining it means evicting the last remaining replica of every component at once, which the PodDisruptionBudget will never allow. The node sits drained but un-replaced, and the operation makes no further progress.

Fix: Either scale the control-plane pool to at least two nodes before making the change (so pods have somewhere to go during the drain), or temporarily delete the affected PodDisruptionBudgets to unblock the drain — control-plane pods will briefly all reschedule at once, which is fine for a planned maintenance window but not something to do without intending it. A subsequent chart reconcile recreates the PodDisruptionBudgets automatically.

Pods stuck `Pending` with “Insufficient cpu”

Cause: The control-plane node pool’s machine type is too small. The chart’s own baseline (two replicas each of the core control-plane components) already adds up to several vCPU of requests before any autoscaling kicks in — a machine type sized for a “small” pool often doesn’t actually fit that baseline.

Fix: Use the recommended (larger) default machine type for the control-plane pool rather than downsizing it — the space/session node pool can be sized independently and much smaller per node.

Node pool scaling or a rolling update stalls with no clear error

Cause: A regional disk (or CPU) quota was silently exceeded — the symptom looks identical to a generic stuck rollout, with nothing in kubectl describe pointing at “quota.”

Fix: Check your cloud project’s compute and disk quotas (e.g. regional SSD/EBS capacity, total CPU) against what your node pools’ machine_type × max_node_count × disk_size_gb actually require, before sizing them. If you’re tight on disk quota specifically, prefer capping the node count over increasing machine size further — the fix for insufficient CPU is a bigger machine, the fix for insufficient disk quota is fewer nodes.