run_a415f69d399ead6c

input

{
  "request": "Triage Jira ticket MZCLD-3841: Put all three ArgoCD realms behind Fastly with abuse-protection rules",
  "references": [
    "https://mozilla-hub.atlassian.net/browse/MZCLD-3841"
  ],
  "sourceContext": {
    "issue": "MZCLD-3841"
  }
}

summary

MZCLD-3841 is a well-scoped, detailed backlog engineering task to front all three ArgoCD realms (sandbox, webservices, dataservices) with Fastly + NGWAF (SigSci) for abuse protection, following the terraform-modules//google_fastly_waf pattern already used in webservices-infra and dataservices-infra. The ticket explicitly states it is blocked by a spike (MZCLD-3840) and must not be started until that spike returns a go/no-go decision. I was unable to read MZCLD-3840 or MZCLD-3834 directly (access to those keys was blocked by policy in this run), so the spike's current status/outcome could not be independently verified — this is the key open item for triage. No comments exist yet on MZCLD-3841, and no linked Confluence runbook/design doc was provided to fetch.

findings

• Ticket is explicitly gated on an upstream spike (MZCLD-3840) and should remain in Backlog/blocked until that spike resolves.
• Could not verify the current status or outcome of the blocking spike MZCLD-3840, or of the related follow-up ticket MZCLD-3834 (ArgoCD Admin Guide/runbook update), because reads to those issue keys were blocked by this run's access policy (only MZCLD-3841 was in scope).
• The ticket is thorough and implementation-ready: it specifies per-realm Fastly service + NGWAF site pattern, rollout order (sandbox → webservices → dataservices), staged ngwaf_agent_level (baseline_protection then block), rate-limit thresholds derived from real traffic data (median ~1.7 req/s, p90 ~31 req/s), origin lockdown via Cloud Armor allowlist + shared secret header, and explicit per-realm verification steps before proceeding to the next realm.
• Scope is deliberately narrow: it excludes documentation updates and upstream bug reporting, which are called out as separate follow-up tickets (one already exists: MZCLD-3834; another for ArgoCD's retry-storm behavior is not yet filed per the ticket text).
• No comments exist on the ticket yet, and status is Backlog, consistent with work not having started (appropriate given the blocking spike).

proposed actions

Triage note: This ticket is correctly held in Backlog pending a go/no-go from the blocking spike MZCLD-3840 — please don't start implementation until that spike's outcome is posted here or linked. Could someone confirm MZCLD-3840's current status/decision? If it's a go, recommend: (1) confirming the follow-up ticket for 'report ArgoCD's stale-session retry storm upstream' has been filed (only MZCLD-3834 for docs/runbook is referenced so far), (2) validating the rate-limit threshold math against the latest dashboard data before the sandbox rollout, and (3) explicitly documenting the Fastly IP-range refresh mechanism for the Cloud Armor allowlist as part of the origin-lockdown work item, since the ticket flags this as a to-decide item. No other blockers noted; scope and rollout plan (sandbox → webservices → dataservices) look sound and align with existing webservices-infra/dataservices-infra patterns.

tool calls

[
  {
    "capability": "jira.issue.read",
    "args": {
      "key": "MZCLD-3841"
    },
    "result": {
      "key": "MZCLD-3841",
      "summary": "Put all three ArgoCD realms behind Fastly with abuse-protection rules",
      "status": "Backlog",
      "description_text": "Scope Blocked by the spike ( MZCLD-3840 ); do not start until it returns a go. If it returns a no-go, close this and re-plan from the finding. Front each ArgoCD realm's external endpoint with Fastly and add rules that damp abusive traffic, whether malicious or the product of a client-side retry bug. All three realms are in scope :  sandbox ,  webservices ,  dataservices . Roll out in that order, so the busiest realm benefits from whatever the first two teach us. Each realm has its own external Gateway, pre-shared cert, IAP client, and hostname, so each gets its own Fastly service and NGWAF site, following the per-environment pattern used elsewhere in  webservices-infra  and  dataservices-infra . Work Fastly service per realm , using  github.com/mozilla/terraform-modules//google_fastly_waf : Backend pointing at the realm's load balancer, with TLS settings and  override_host  as determined by the spike, and the per-backend timeouts the spike establishes for long-lived streams. ngwaf_agent_level  set deliberately. Consider starting with  ngwaf_baseline_protection  (which disables immediate blocking in favour of attack-threshold alerts) on the first realm, then tightening to  block  once we have seen real traffic. ArgoCD's clients are internal and few, so a false positive is disruptive. Leave caching alone. The module's VCL defaults to  return(pass) , which is correct for an authenticated app; do not set  cache_header . Consider  ddos_protection  with  ddos_protection_alert  pointed at a CloudEng Slack channel, so we hear about volumetric events rather than discovering them in a page. Rate-limiting rules.  Add  sigsci_site_rule  resources with  type = \"rateLimit\" , following the pattern already in use (see  webservices-infra/monitor/tf/modules/waf_rules , which bans at 20 requests / 10 minutes for 3600s, and llm-proxy's  rate_limit_threshold ). For ArgoCD specifically: Identify clients by IP via  client_identifiers , since that is what we can act on mid-incident. Pick a per-IP threshold that a normal UI session (which does open several concurrent streams) stays well under, but which a 900 rps loop trips immediately. Derive the number from real traffic rather than guessing; the request-volume data in the dashboard and Wil's analysis give a baseline (median around 1.7 req/s, p90 around 31 req/s across the whole realm). Return something a browser handles sanely when throttled. llm-proxy's snippets show how to convert an NGWAF 429 into a custom response if the default is unhelpful. Origin lockdown.  Prevent traffic bypassing the edge by hitting the load balancer directly: a Cloud Armor allowlist of Fastly ranges on the external backend plus an edge-injected shared secret header, per the pattern in llm-proxy ( X-Fastly-Request ). Decide and document how the Fastly IP ranges are kept current. Webhook path.  Implement whatever the spike decided for  /api/webhook , keeping GitHub delivery working and its source-IP filtering enforced at a layer that sees the real client IP. Remember  argocd-webhook-policy  is shared across realms. Verification per realm , before moving to the next: UI loads and streams stay open,  argocd  CLI works over grpc-web, GitHub webhooks deliver and trigger syncs, a synthetic burst gets throttled, and the dashboard's LB-derived panels still report something meaningful. Follow-ups to file, not do here Update the ArgoCD Admin Guide and runbook for the new topology, including where responders now find real client IPs ( MZCLD-3834 ). The ArgoCD UI retry behaviour that produced 900 rps from a single stale session is worth reporting upstream. Rate limiting treats the symptom.",
      "comment_count": 0,
      "comments": []
    }
  },
  {
    "capability": "jira.issue.read",
    "args": {
      "key": "MZCLD-3840"
    },
    "result": {
      "error": "read blocked by policy: resource not in run subject",
      "args": {
        "key": "MZCLD-3840"
      }
    },
    "blocked": true
  },
  {
    "capability": "jira.issue.read",
    "args": {
      "key": "MZCLD-3834"
    },
    "result": {
      "error": "read blocked by policy: resource not in run subject",
      "args": {
        "key": "MZCLD-3834"
      }
    },
    "blocked": true
  }
]

log