run_cd59bbe9af0b0127

input

{
  "request": "Triage Jira ticket MZCLD-3840: Spike: can Fastly front an IAP-protected ArgoCD endpoint without breaking the UI, CLI, or webhooks?",
  "references": [
    "https://mozilla-hub.atlassian.net/browse/MZCLD-3840"
  ],
  "sourceContext": {
    "issue": "MZCLD-3840"
  }
}

summary

MZCLD-3840 is a well-scoped spike (status: Backlog, no comments yet) asking whether Fastly can front an IAP-protected ArgoCD endpoint (sandbox realm first, then dataservices/webservices) without breaking the UI, CLI, or GitHub webhooks. The ticket description is unusually thorough — it already lays out the architecture (IAP terminates at the GCP LB, not Fastly), the six specific risk areas to validate (long-lived SSE streams/timeouts, client IP propagation, webhook allowlisting, the IAP OAuth/cookie/Host-header flow, gRPC-Web CLI compatibility, and edge-bypass prevention), and a concrete deliverable (go/no-go + Fastly module config + webhook plan + per-realm applicability). This reads as ready-for-work rather than needing clarification; triage should focus on prioritization, owner assignment, and flagging the one item most likely to be a hard blocker (long-lived streams vs. Fastly's default per-backend timeouts).

findings

• Ticket is fully specified and actionable as written — it names the exact configs to inspect (argocd-bootstrap values.yaml timeoutSec, Fastly module defaults for first_byte_timeout/between_bytes_timeout, argocd-webhook-policy Cloud Armor allowlist), references a working precedent (llm-proxy's override_host + shared-secret header pattern), and defines a clear deliverable format (go/no-go, module config, webhook plan, per-realm applicability). No further requirements-gathering is needed before an engineer picks it up.
• The highest-risk item is explicitly flagged by the reporter as the SSE/watch-stream timeout mismatch (Fastly default first_byte_timeout 60s / between_bytes_timeout 10s vs. ArgoCD's required 1800s timeoutSec, put in place per SREIN-1621 to avoid GCP's 30s default 504s). This is called out as the item that could invalidate the whole approach and should be validated first before the other five items are investigated in depth.
• No existing Mozilla precedent fully covers this combination: Balrog has both waf.tf and iap.tf but its IAP block is commented out/disabled, and MZCLD-3643 (Backstage) treats Fastly WAF and IAP as either/or rather than layered. This spike is genuinely novel work, not a duplicate of prior investigation.
• The GitHub webhook path (/api/webhook) has no IAP and currently relies solely on a Cloud Armor IP allowlist (argocd-webhook-policy) that is shared across realms — inserting Fastly breaks this allowlist as written and needs an explicit decision (move allowlist to Fastly, match on forwarded header in Cloud Armor, or bypass the edge for webhooks). This has blast-radius implications beyond the sandbox realm since the policy is shared.
• Attempted to pull related context from linked tickets MZCLD-3643 (Backstage Fastly/IAP either-or precedent) and SREIN-1621 (origin of the 1800s ArgoCD stream timeout) for additional evidence, but both reads were blocked by run-scope policy (only the triggering issue MZCLD-3840 is in scope for this run). Findings above rely solely on MZCLD-3840's own description.

proposed actions

tool calls

[
  {
    "capability": "jira.issue.read",
    "args": {
      "key": "MZCLD-3840"
    },
    "result": {
      "key": "MZCLD-3840",
      "summary": "Spike: can Fastly front an IAP-protected ArgoCD endpoint without breaking the UI, CLI, or webhooks?",
      "status": "Backlog",
      "description_text": "Question to answer Can we put Fastly in front of an ArgoCD realm's external endpoint while IAP stays enabled on the backend, without breaking the UI, the  argocd  CLI, or GitHub webhook delivery? Answer it in  sandbox , then state whether the same shape holds for  dataservices  and  webservices . This should compose in principle: IAP terminates at the GCP load balancer, not at Fastly, so Fastly is a reverse proxy in front of an IAP-protected origin rather than a competing auth layer. The module's VCL already suits an authenticated app, defaulting to  return(pass)  and bailing out of caching on  Set-Cookie . The risks are in the specifics below. No Mozilla precedent exists.  Balrog is the only service carrying both a  waf.tf  and an  iap.tf , and its IAP block is entirely commented out (\"IAP disabled\").  MZCLD-3643  treats Fastly WAF and IAP as an either/or for Backstage. Layering them would be new for us, which is the reason for this spike. Things to validate, roughly in risk order 1. Long-lived streams.  The biggest functional risk.  argo/k8s/argocd-bootstrap/values.yaml  sets  timeoutSec: 1800  on the external  server  backend specifically because the UI opens watch/SSE streams on  /api/v1/stream/* , and GCP's 30s default cut them with 504s (SREIN-1621). The Fastly module defaults to  first_byte_timeout  60s and  between_bytes_timeout  10s per backend. Determine what Fastly's actual ceiling is on holding an idle-ish 30 minute response, whether raising the per-backend timeouts is sufficient, whether it needs a Fastly support request, and whether an idle stream trips  between_bytes_timeout  regardless. If Fastly cannot hold these streams, say so plainly, since it likely changes the whole approach (for example routing  /api/v1/stream/*  around the edge, or accepting Cloud Armor only). 2. Client IP.  Behind Fastly, the load balancer, Cloud Armor, and IAP all see Fastly IPs rather than the caller's. Confirm which header carries the true client IP ( Fastly-Client-IP  versus  X-Forwarded-For ), that it survives to the ArgoCD server, and what that does to LB logging and to our ability to identify an abusive source during an incident. This is also why rate limiting has to happen at Fastly: it is the only layer that still sees the real client. 3. GitHub webhooks.   /api/webhook  has no IAP and relies on the Cloud Armor policy  argocd-webhook-policy  allowlisting GitHub source IPs. Behind Fastly that allowlist stops working as written. Decide whether the allowlist moves to Fastly, whether Cloud Armor can match on the forwarded header instead, or whether webhooks should bypass the edge entirely. Note this policy is shared across realms. 4. The IAP flow itself.  Confirm the OAuth redirect ( /_gcp_gatekeeper/authenticate ) completes with Fastly in the path, that the IAP cookie survives, that the public  Host  is preserved (the  HTTPRoute  matches on it, so this is load-bearing), and that  Authorization  /  Proxy-Authorization  headers are not stripped. Work out the origin addressing and TLS: whether Fastly targets the LB by hostname or IP, and how  ssl_sni_hostname ,  ssl_cert_hostname , and  override_host  need to be set against the pre-shared cert. 5. The CLI.   argocd  uses gRPC over HTTP/2 by default. Our Admin Guide already prescribes  --grpc-web  for the IAP path, which is HTTP/1.1-compatible, so this is probably fine, but confirm it, and check the paths that use WebSockets (terminal exec, and log streaming) since those have their own edge constraints. 6. Bypass.  The load balancer stays publicly reachable regardless of where DNS points, so establish how we prevent traffic from going around the edge. llm-proxy already demonstrates the pattern:  override_host  plus a shared secret header injected at the edge ( X-Fastly-Request ), which the origin can then require. Pair that with a Cloud Armor allowlist of Fastly ranges and decide how those ranges get maintained. Deliverable A short written finding, linked here and suitable to fold into the Admin Guide, that gives a go/no-go plus: Which of the six items above work as-is, which need configuration, and which are blockers. The concrete Fastly module configuration a realm would need (backend definition, timeouts, snippets, NGWAF level). What changes for the webhook path. Whether the same configuration holds for all three realms or whether any realm needs different treatment. If the answer is no, what we do instead. Notes Test in  sandbox , per the Admin Guide's rule that ArgoCD changes are validated there first. Fastly's staging environment (the module's  stage = true  flag, which requires a local  /etc/hosts  override) may let us test without moving DNS at all.",
      "comment_count": 0,
      "comments": []
    }
  },
  {
    "capability": "jira.issue.read",
    "args": {
      "key": "MZCLD-3643"
    },
    "result": {
      "error": "read blocked by policy: resource not in run subject",
      "args": {
        "key": "MZCLD-3643"
      }
    },
    "blocked": true
  },
  {
    "capability": "jira.issue.read",
    "args": {
      "key": "SREIN-1621"
    },
    "result": {
      "error": "read blocked by policy: resource not in run subject",
      "args": {
        "key": "SREIN-1621"
      }
    },
    "blocked": true
  }
]

log