tool calls
[
{
"capability": "jira.issue.read",
"args": {
"key": "MZCLD-3839"
},
"result": {
"key": "MZCLD-3839",
"summary": "Protect ArgoCD's public endpoints from abuse at the edge",
"status": "Backlog",
"description_text": "Background All three ArgoCD instances ( sandbox , webservices , dataservices ) run on public GCP load balancers fronted by IAP. IAP proves who a caller is; it does nothing about how much an authenticated caller sends. We hit that gap directly: a browser session whose IAP auth had gone stale began hitting the webservices endpoint at over 900 requests per second, enough to make ArgoCD flap. ArgoCD itself was healthy, and the root cause looks like a UI retry bug rather than a platform fault, but we had no control that would have damped it. The concern generalizes past that one bug. Anyone holding a valid IAP token, or any client looping on an error path, can saturate an endpoint and we are currently unprotected against it. As Nate put it: \"lets say someone with a valid iap token was smashing the interface accidentally we would be unprotected.\" All three realms share this exposure , so remediation is scoped to all three rather than to webservices alone. Each realm has its own external Gateway, pre-shared cert, IAP backend, and OAuth client ID, so each needs its own edge configuration even though the pattern is identical. Current edge configuration From argo/k8s/argocd-bootstrap/templates/ and argo/tf/ : ArgoCD builds its load balancers through Gateway API. Each realm's external Gateway uses gke-l7-global-external-managed with a pre-shared cert; a second internal Gateway uses gke-l7-rilb . IAP is attached per-Service via GCPBackendPolicy , not on the Gateway: the server service at path / gets iap.enabled: true plus the realm's client ID and OAuth secret. /api/webhook routes to a separate server-webhook service with no IAP , protected instead by the Cloud Armor policy argocd-webhook-policy , which allowlists GitHub webhook source IP ranges. Note this policy is defined once and shared across realms, unlike the per-realm internal policies. The external server backend sets timeoutSec: 1800 , with an in-repo comment citing SREIN-1621: the UI opens long-lived watch/SSE streams on /api/v1/stream/* , and GCP's 30s default cut every one of them with a backend_timeout 504. GCPBackendPolicy already supports a securityPolicy field for external services, so Cloud Armor can be attached to the ArgoCD backends today. Goal On every ArgoCD realm, an authenticated client that misbehaves, whether maliciously or through a UI bug, gets throttled before it can degrade ArgoCD for everyone else, and we can identify and rate limit a specific source without a code change. Approach Three child tickets, in dependency order: Spike to establish whether Fastly can sit in front of an IAP-protected backend without breaking the UI, the CLI, or GitHub webhooks. Proven in sandbox first, per the Admin Guide's rule that ArgoCD changes are tested there before anything else. This is genuinely unknown at Mozilla, so it gates the rest. Implementation across all three realms: front each endpoint with Fastly using terraform-modules//google_fastly_waf , add NGWAF rate-limiting rules, and lock each origin down so the load balancers cannot be reached around the edge. Rollout order sandbox \u2192 dataservices \u2192 webservices , leaving the busiest realm for last. Interim mitigation , independent of the above: Cloud Armor rate-based bans on the external backends of all three realms. Cloud Armor still sees real client IPs today because nothing fronts the load balancers, so this closes the specific gap from the incident in days rather than a quarter, and remains useful as defense in depth if Fastly lands later. Why this needs a spike first No service at Mozilla currently runs Fastly in front of IAP. Balrog is the only service with both a waf.tf and an iap.tf , and its entire IAP block is commented out (\"IAP disabled - resources commented out for Atlantis to clean up GCP state\"). MZCLD-3643 frames Fastly WAF and IAP as an either/or for Backstage and flags IAP as awkward for programmatic clients. Layering them is a new pattern for us. The two layers should compose in principle, since IAP terminates at the GCP load balancer rather than at Fastly, so Fastly is just a reverse proxy in front of an IAP-protected origin. The risks are in the details, and the spike covers them. Related MZCLD-3643 \u2014 Backstage: Fastly WAF or IAP. Same two technologies, opposite framing (either/or rather than layered). Whatever we learn here should inform that decision. MZCLD-3834 \u2014 ArgoCD documentation. If the edge topology changes, the architecture page and the runbook's load balancer log queries change with it.",
"comment_count": 0,
"comments": []
}
}
]