run_c527262a968e2d02

input

{
  "request": "Triage Jira ticket MZCLD-3669: Inventory the 230 Yardstick alert rules that bypass the notification policy",
  "references": [
    "https://mozilla-hub.atlassian.net/browse/MZCLD-3669"
  ],
  "sourceContext": {
    "issue": "MZCLD-3669"
  }
}

summary

MZCLD-3669 is a well-scoped, inventory-only ticket asking for a full accounting of the 230 Yardstick alert rules (of 706 total) that hard-code a notification-policy-bypassing contact point via notification_settings.receiver, 69 of which page PagerDuty directly. It was surfaced as a side-finding while closing OPST-2460 (PagerDuty API key rotation) and explicitly does not change any alert rule or re-scope epic MZCLD-3140. The ticket is currently in Backlog with no comments. Triage recommendation: confirm scope/acceptance criteria are actionable, request the deliverable format and storage location, and flag the deferred cross-references to OPST-2460 and MZCLD-3140 (not accessible in this run) for the assignee to review before work starts.

findings

• Ticket is inventory/documentation-only by explicit scope note; no alert rule changes and no re-scoping of epic MZCLD-3140.
• Quantitative baseline is already established: 230 of 706 Yardstick alert rules (33%) hard-code a receiver via notification_settings.receiver, bypassing the notification policy; 69 of those page PagerDuty directly. A top-receiver breakdown (LLM-Proxy-prod PagerDuty=17, SRE Green: Low=10, Taskcluster Firefox-CI=7, etc.) is already included in the description.
• Acceptance criteria are clear and specific: full inventory listing rule, contact point, destination type (PagerDuty/Slack/email), and owning team, plus a documented/re-runnable query or script.
• The ticket does not yet specify where the inventory artifact will live (attachment, Confluence page, repo) beyond 'attach it to this ticket', nor does it name an owner/assignee or target Grafana Yardstick instance details beyond the Grafana provisioning API.
• Ticket originated as a byproduct of OPST-2460 (PagerDuty API key rotation) and is a precursor to future cleanup potentially tied to MZCLD-3140, but both linked tickets were not accessible in this triage run (out of run subject scope), so cross-ticket coordination context could not be independently verified.

proposed actions

Triage: This is a clear, well-scoped inventory/documentation task (no rule changes, no re-scoping of MZCLD-3140). Baseline numbers and top hard-coded PagerDuty receivers are already captured in the description, which is a good start. Before pulling this into active work, please clarify: (1) where the final inventory artifact should live — a Jira attachment (CSV/spreadsheet) is fine per the ask, but consider also linking a Confluence page if this will be referenced repeatedly by multiple teams; (2) who will own producing/re-running the Grafana provisioning API query going forward (SRE/Cloud Ops?); (3) whether 'owning team' will be derived from Yardstick folder/tag metadata or requires manual mapping, since that's likely the highest-effort part of the acceptance criteria. Recommend assigning to whoever owns Yardstick/Grafana provisioning tooling and linking this ticket to OPST-2460 and MZCLD-3140 for traceability once inventory is ready to hand off for cleanup planning.

tool calls

[
  {
    "capability": "jira.issue.read",
    "args": {
      "key": "MZCLD-3669"
    },
    "result": {
      "key": "MZCLD-3669",
      "summary": "Inventory the 230 Yardstick alert rules that bypass the notification policy",
      "status": "Backlog",
      "description_text": "Context:  Surfaced while closing OPST-2460 (PagerDuty API key rotation). Problem.  Measured 2026-08-05 via the Grafana provisioning API: of  706 alert rules  in Yardstick,  230 (33%)  hard-code a contact point via  notification_settings.receiver  and bypass the notification policy.  69 of those page PagerDuty directly. There is currently no inventory of these, so nobody can say how large the cleanup actually is, which teams own it, or whether a given rotation or routing change is safe. Largest hard-coded PagerDuty receivers by rule count: Rules Contact point 17 LLM-Proxy-prod PagerDuty 10 SRE Green: Low 7 Taskcluster Firefox-CI 4 Release SRE - PD - Android Hardware TC Queues - Low Priority 4 SRE Green: High 4 merino 3 Release SRE - PD - Meltdown Alerts 3 FxA - PagerDuty 2 SRE Green: Medium 2 Data SRE - STMO 2 IAM Team - PagerDuty 2 Taskcluster Community 2 SUMO - PagerDuty Ask.  Produce and record a full inventory of every alert rule with a hard-coded receiver, grouped by receiver and owning team, and attach it to this ticket as the reference for any later cleanup work. Acceptance criteria. An inventory exists listing every rule with a hard-coded receiver, its contact point, its destination type (PagerDuty/Slack/email) and its owning team. The query or script that produces it is written down so it can be re-run. Scope note: this ticket is inventory and documentation only. It deliberately does  not  change any alert rule and does  not  re-scope MZCLD-3140 \u2014 that epic stays as it is.",
      "comment_count": 0,
      "comments": []
    }
  },
  {
    "capability": "jira.issue.read",
    "args": {
      "key": "MZCLD-3140"
    },
    "result": {
      "error": "read blocked by policy: resource not in run subject",
      "args": {
        "key": "MZCLD-3140"
      }
    },
    "blocked": true
  },
  {
    "capability": "jira.issue.read",
    "args": {
      "key": "OPST-2460"
    },
    "result": {
      "error": "read blocked by policy: resource not in run subject",
      "args": {
        "key": "OPST-2460"
      }
    },
    "blocked": true
  }
]

log