run_30ffa237b9407757

input

{
  "request": "Triage Jira ticket SREIN-1621: ArgoCD webservices (webservices.argocd.global.mozgcp.net) returning intermittent 5XX",
  "references": [
    "https://mozilla-hub.atlassian.net/browse/SREIN-1621"
  ],
  "sourceContext": {
    "issue": "SREIN-1621"
  }
}

summary

SREIN-1621 already contains a thorough, well-evidenced investigation (GCP Log Explorer analysis, kubectl inspection) identifying two distinct root causes for intermittent 5XXs on webservices.argocd.global.mozgcp.net, plus concrete proposed fixes. The ticket is sitting in Backlog with only an automated 'reduced support' notice as a comment — no maintainer has acknowledged or actioned it yet. Triage recommendation: confirm scope/ownership, prioritize the low-risk GCPBackendPolicy timeout fix (addresses ~83% of errors) as a quick win, and track the HPA/preStop drain fix as a follow-up since it requires more care (HPA tuning, rollout).

findings

• Primary root cause (~83% of errors, 25/30 in sampled 24h window): 504 backend_timeout on ArgoCD's long-lived SSE/watch stream endpoint GET /api/v1/stream/application..., caused by the argocd-webservices-server GCPBackendPolicy having no timeoutSec set (defaults to GCP's 30s). Failures recur on a ~30s cadence matching the timeout, causing the UI to disconnect/reconnect repeatedly.
• Secondary root cause (5/30 in sample): 503 backend_connection_closed_before_data_sent_to_client during pod termination, caused by HPA flapping between 2-3 replicas (75 scale-ups on CPU, 107 scale-downs on memory over 32 days) combined with an empty preStop hook and only 30s terminationGracePeriodSeconds, giving the LB/NEG no time to drain before the pod exits.
• Concrete, low-risk remediation is already proposed for root cause 1: set spec.default.timeoutSec (1800 or 3600) on the argocd-webservices-server GCPBackendPolicy while preserving existing IAP/logging config. This is a config-only change with no code/deploy risk and should be actionable quickly.
• Remediation for root cause 2 is more involved: add a preStop sleep (30–60s), raise terminationGracePeriodSeconds to ~60, and add HPA scaleDown stabilization window / adjust memory target or minReplicas to stop 2↔3 flapping. Ticket also notes the exact chart/config source location is not yet confirmed ('to be located, likely outside the wikimo dir').
• Ticket is unowned/unactioned: it is still in 'Backlog' status with only one comment, an automated Cloud Engineering reduced-support-week notice (referencing MIR/Security Incident Response processes) — no engineer has picked it up, and no priority/assignee triage has occurred despite the investigation being complete and actionable.

proposed actions

tool calls

[
  {
    "capability": "jira.issue.read",
    "args": {
      "key": "SREIN-1621"
    },
    "result": {
      "key": "SREIN-1621",
      "summary": "ArgoCD webservices (webservices.argocd.global.mozgcp.net) returning intermittent 5XX",
      "status": "Backlog",
      "description_text": "Summary The external ArgoCD instance at  webservices.argocd.global.mozgcp.net  (namespace  argocd-webservices , cluster  gke_moz-fx-platform-mgmt-global_us-west1_global-platform-admin-mgmt ) is returning intermittent 5XX errors. Investigation identified  two distinct root causes . Evidence (GCP Log Explorer, project  moz-fx-platform-mgmt-global ,  resource.type=\"http_load_balancer\" , host filter on the argocd hostname, last 24h) Count Status statusDetails 25 504 backend_timeout 5 503 backend_connection_closed_before_data_sent_to_client Root cause 1 (primary, ~83% of errors): 504  backend_timeout  \u2014 30s default backend timeout on streaming endpoints All 504s are on  GET /api/v1/stream/application...  \u2014 ArgoCD's long-lived watch/SSE streams used by the UI for live updates. Failures recur on a fixed ~30s cadence (e.g. 18:17:37 \u2192 18:17:07 \u2192 18:16:36), indicating a 30-second timeout cutting each stream; the UI then reconnects and is cut again. The  argocd-webservices-server  GCPBackendPolicy has  no  timeoutSec  set, so the backend service uses GCP's default of  30s . Long-lived ArgoCD streams cannot survive that. Proposed fix:  set a long backend timeout on the server GCPBackendPolicy: spec:\n  default:\n    timeoutSec: 1800   # or 3600; keep existing iap + logging config Root cause 2 (secondary): 503  backend_connection_closed_before_data_sent_to_client  \u2014 pod termination without NEG drain These arrived as a burst (17:48:20\u201327), the signature of a pod being terminated while the LB still routes to it. The  argocd-webservices-server  HPA is thrashing between 2 and 3 replicas (over 32 days: 75 scale-ups on CPU-above-target, 107 scale-downs on memory-below-target; CPU 19%/50%, memory 37%/50% \u2014 the two metrics fight at the threshold). Each scale-down churns the GKE NEG membership (repeated Attach/Detach observed on the Service). Server pods have an  empty  preStop  hook and  terminationGracePeriodSeconds: 30 , so there is no drain delay for the LB/NEG to deregister the endpoint before the process exits. Proposed fix:  add a  preStop  drain ( sleep 30\u201360 ), bump  terminationGracePeriodSeconds  to ~60, and add an HPA  scaleDown  stabilization window (and/or raise the memory target / minReplicas) to stop the 2\u21943 flapping. Impact Users of the ArgoCD web UI experience intermittent failures \u2014 most visibly the live application view repeatedly dropping (504 on the watch stream every ~30s), plus occasional 503s during autoscaler scale-down. Next steps Apply GCPBackendPolicy  timeoutSec  fix (addresses the majority of errors). Address HPA thrash + graceful NEG drain for the 503s. Chart/config source to be located (likely outside the  wikimo  dir, in the argocd-webservices deployment config). Investigated 2026-07-24 via kubectl + GCP Log Explorer.",
      "comment_count": 1,
      "comments": [
        {
          "author": "Automation for Jira",
          "body": "Reduced Support for CloudEng work week: Expect delays for customer support requests. Non-urgent requests may get pushed off until after July 24th, 2026. If this is an incident, please follow the Incident Process.  https://mozilla-hub.atlassian.net/wiki/spaces/MIR/overview?homepageId=20512772 If this is a security incident, please follow the Security Incident Response process.  https://mozilla-hub.atlassian.net/wiki/spaces/SECURITY/pages/27165574/Security+Incident+Response If your request can't wait until after July 24th, please let us know when you need it by and why in a comment in this Jira issue."
        }
      ]
    }
  }
]

log