run_c5208d41959a156e

input

{
  "request": "Triage Jira ticket MZCLD-3837: Write the ArgoCD incident runbook and wire it into the ArgoCD alerts",
  "references": [
    "https://mozilla-hub.atlassian.net/browse/MZCLD-3837"
  ],
  "sourceContext": {
    "issue": "MZCLD-3837"
  }
}

summary

MZCLD-3837 is a well-specified, unstarted (Backlog) engineering-writing task: author a Confluence incident runbook for the six ArgoCD alerting rules in the "MozCloud / ArgoCD Alerting" Yardstick group, then attach runbook_url annotations to those rules so pages carry actionable next steps instead of a bare one-line summary. The ticket description is thorough and largely self-contained (triage question, LB log queries, dashboard guidance, per-alert sections, known-noise caveats, escalation path), but it depends on two things outside this ticket's scope: the "Runbook: OTel Collector Resource Utilization Alerts" page as a style template and MZCLD-3140 which tracks the underlying dashboard/alert-quality fixes. I could not fetch either MZCLD-3140 (blocked by policy — not in run subject) or any Confluence page (no page ID/URL was resolvable from the ticket text, only a plaintext description), so those dependencies are unverified from my side. No comments exist yet, so this needs a first triage note that: confirms priority/sizing, flags the two follow-on artifacts required (new Confluence page + PR/config change to the 6 alert rules), and reiterates the ticket's own guidance to spin off any alert found to be non-actionable into MZCLD-3140 rather than fixing scope creep in the runbook prose.

findings

• none

proposed actions

tool calls

[
  {
    "capability": "jira.issue.read",
    "args": {
      "key": "MZCLD-3837"
    },
    "result": {
      "key": "MZCLD-3837",
      "summary": "Write the ArgoCD incident runbook and wire it into the ArgoCD alerts",
      "status": "Backlog",
      "description_text": "Problem The  MozCloud / ArgoCD Alerting  group in Yardstick has six rules and  none of them carries a  runbook  or  runbook_url  annotation: alert severity ArgoCD Webservices: Container Restarts high ArgoCD Dataservices: Container Restarts high ArgoCD: Sync Status (unknown) high ArgoCD Webservices: 5XX Errors high Sync Activity over the last hour low Production Applications not using \"main\" as Target Revision low So a page arrives with a one-line summary and no next step. The Admin Guide's only guidance is that we respond on a best-effort basis. Responders are rediscovering the same triage path each time, and finding the right log query alone eats meaningful minutes at 1:30 AM. Proposed work Write the runbook first, then wire it into the alerts. 1. Runbook page  (Confluence, following the pattern of  Runbook: OTel Collector Resource Utilization Alerts ): Is this ArgoCD or a tenant?  The Admin Guide already notes that a large share of \"ArgoCD is broken\" reports turn out to be CI failures or Helm rendering problems. Put that triage question first, with how to tell the difference quickly. Load balancer logs.  Paste-ready  gcloud beta logging tail  and Logs Explorer queries for each instance, plus how to look up the current forwarding rule and backend target names rather than trusting a copied string (they carry generated suffixes, e.g.  gkegw1-guzw-argocd-webser-argocd-webservices-serve-7ods8bprjr3i ). Confirm and document whether the filter supports regex so one query can cover all three instances, since that question came up directly. Dashboard.  Where to start on the  ArgoCD dashboard  and how to narrow by namespace, linking to the component reference for what each row means. Per-component troubleshooting.  Once a component is implicated, treat it like any other Kubernetes app in  argocd-<domain> : pods, restarts, logs, resource pressure. Link out rather than restating the component reference. A section per alert above:  what it means, what it does not mean, first checks, and likely causes. For sync-status-unknown, carry over the existing annotation's insight (GKE auth failures, expired credentials, or a jump in onboarded applications). Known noise.  Availability troughs on panel 126 are low-traffic artifacts, and roughly 98.5% of load balancer traffic is 4xx, so that panel largely measures whether the load balancer is answering. Cite  Wil's analysis  so responders do not chase a dip that means nothing. Escalation.  Who to pull in, and the CLI access path (the Admin Guide's IAP token steps) for when the UI is unavailable. 2. Wire it in.  Add a  runbook_url  annotation to all six rules pointing at the runbook, with deep links to the relevant section where the anchor is stable. The  wiki uptime check  alert already does this and is a good in-house model. Notes The runbook should be honest about signal quality rather than waiting on the dashboard and alert fixes tracked in  MZCLD-3140 . If writing it surfaces an alert that cannot be made actionable as currently defined, file that against MZCLD-3140 rather than papering over it in prose.",
      "comment_count": 0,
      "comments": []
    }
  },
  {
    "capability": "jira.issue.read",
    "args": {
      "key": "MZCLD-3140"
    },
    "result": {
      "error": "read blocked by policy: resource not in run subject",
      "args": {
        "key": "MZCLD-3140"
      }
    },
    "blocked": true
  }
]

log