run_35a13496f5013ee2

AWAITING_APPROVAL retrigger →

jira-triage v1 · jira-triage-k8s · owner ingest:jira-new-ticket (api) · via jira-new-ticket · cloud-engineering

input

{
  "request": "Triage Jira ticket MZCLD-3682: fix flapping \"namespaces with unschedulable pods - gcp-v2-prod platform webservices-high-prod prod high\"",
  "references": [
    "https://mozilla-hub.atlassian.net/browse/MZCLD-3682"
  ],
  "sourceContext": {
    "issue": "MZCLD-3682"
  }
}

summary

MZCLD-3682 tracks a flapping PagerDuty alert ("namespaces with unschedulable pods") for the gcp-v2-prod / webservices-high-prod cluster, high severity. The ticket is still in Backlog with no comments or assignee, and no runbook/design doc is linked in Jira (the only reference is a Grafana/Yardstick dashboard URL, not a Confluence page). The description notes the alert self-resolves within 5-10 minutes, indicating a likely transient scheduling condition (e.g., autoscaler lag, pod churn during deploys, or resource pressure) rather than a persistent failure. Recommend triaging by reviewing the linked Grafana panel for timing correlation with deploys/HPA scaling events and considering either tuning the alert's for/threshold duration or investigating scheduler/autoscaler latency in that node pool.

findings

Alert is self-healing/flapping, not a sustained outage
Ticket lacks triage progress: no comments, no assignee info returned, status still Backlog
Only external context provided is a Grafana/Yardstick dashboard link, not a Confluence runbook; no confluence.page.read lookup was possible since no Confluence page ID/URL was referenced
Alert scope is specific: gcp-v2-prod platform, webservices-high-prod namespace/cluster, high severity page

proposed actions

jira.comment.postrisk low - informational triage comment only, no infrastructure changes proposed

MZCLD-3682

Triage notes: This alert appears to be a transient/flapping condition rather than a persistent scheduling failure — it self-resolves within 5-10 minutes each time it pages. Suggested next steps: (1) Pull up the linked Grafana dashboard (yardstick.mozilla.org panel ffojrsr5wy4n4f) and overlay pod-scheduling-failure timestamps against deploy events, HPA scaling activity, and cluster-autoscaler node-add latency for webservices-high-prod in gcp-v2-prod to identify the root cause (e.g., brief resource pressure during rollouts vs. autoscaler catching up). (2) If the pattern correlates with routine deploy/scale events and always self-resolves quickly, consider adjusting the alert's 'for' duration/threshold (e.g., require unschedulable pods to persist >10-15 min before paging) to reduce noise while still catching genuine stuck-scheduling incidents. (3) If it does not correlate with expected scaling activity, check node pool capacity/quota and pending pod events (kubectl describe) during the next occurrence for FailedScheduling reasons. Recommend assigning to the platform/webservices on-call owner and moving out of Backlog given repeated paging.

tool calls

[
  {
    "capability": "jira.issue.read",
    "args": {
      "key": "MZCLD-3682"
    },
    "result": {
      "key": "MZCLD-3682",
      "summary": "fix flapping \"namespaces with unschedulable pods - gcp-v2-prod platform webservices-high-prod prod high\"",
      "status": "Backlog",
      "description_text": "This alert has paged a couple of times already and usually resolves itself within 5 to 10 minutes.  Pagerduty: Yardstick:  https://yardstick.mozilla.org/alerting/grafana/ffojrsr5wy4n4f/view?orgId=1",
      "comment_count": 0,
      "comments": []
    }
  }
]

usage

{
  "model": "claude-sonnet-5",
  "input_tokens": 3185,
  "output_tokens": 1403,
  "estimated_cost_usd": 0.030600000000000002
}

approvals

[]

log

launched k8s job run-run-35a13496f5013ee2 in ns/mzcld-demo-dev image=us-docker.pkg.dev/moz-fx-mzcld-demo-prod/mzcld-demo-prod/agentplat:latest

runner started

live reasoning model=claude-sonnet-5

awaiting approval for 1 action(s)