run_d69d3dccb389cd01

input

{
  "request": "Triage Jira ticket MZCLD-3677: cloudops-deployment: Datadog notification addresses point at the shut-down moz-svc-ops PagerDuty account",
  "references": [
    "https://mozilla-hub.atlassian.net/browse/MZCLD-3677"
  ],
  "sourceContext": {
    "issue": "MZCLD-3677"
  }
}

summary

MZCLD-3677 is a well-scoped sub-task (currently in Backlog, no comments yet) to remove/replace dead moz-svc-ops.pagerduty.com Datadog notification targets from cloudops-deployment, excluding mozphab/Phabricator files which are explicitly deferred to SREIN-1528. The ticket already contains clear file-level pointers and acceptance criteria, so triage mainly needs confirming the databricks/servicebook services are still live and assigning an owner to do the investigation + fix.

findings

• Ticket scope is explicit and self-contained: fix dead moz-svc-ops.pagerduty.com email addresses in Datadog notification config for databricks and servicebook, but explicitly NOT mozphab/Phabricator files.
• Four concrete hit locations are identified for in-scope work: one in a failure_emails list (databricks-prod.yml#L3, has two live mozilla.com addresses as fallback) and three that directly @-mention the dead address in alert bodies (servicebook stage.yml#L3, api-monitoring.yml#L17, ui-monitoring.yml#L17).
• The ticket depends on an open question (an 'Ask') that must be resolved before code changes: whether databricks and servicebook services/monitors are still live at all, which determines whether to replace the address or remove the config entirely.
• The related Phabricator alert cleanup work is tracked separately under SREIN-1528 ('fix phabricator and lando alerts', reported as In Progress at ticket-write time), so no duplicate work should be done here for the three devsvc*.yml hits.
• Ticket currently has zero comments and is sitting in Backlog with no assignee activity noted, so no owner has yet started the liveness investigation the acceptance criteria require.

proposed actions

Triage note: Scope and acceptance criteria here are clear and actionable — replace/remove dead moz-svc-ops.pagerduty.com Datadog notification targets in databricks (ansible/envs/databricks-prod.yml#L3) and servicebook (ansible/envs/stage.yml#L3, api-monitoring.yml#L17, ui-monitoring.yml#L17), while leaving the mozphab/devsvc* files untouched for SREIN-1528 to handle.

Before code changes land, we need to resolve the open 'Ask': confirm whether the databricks and servicebook services and their Datadog monitors are still live.
- If live: swap in a working notification target (e.g. an existing mozilla.com address or a functioning PagerDuty/Datadog integration) or simply drop the dead address if remaining recipients (databricks already has two live mozilla.com addresses in failure_emails) are sufficient.
- If dead: remove the associated monitor/notification config entirely rather than patching the address.

Recommend: (1) assign an owner to do the liveness check for both services, (2) once confirmed, open a PR updating the four listed locations, (3) verify each remaining notification target actually resolves/delivers before closing, per the acceptance criteria, and (4) double check with the SREIN-1528 owner that the three devsvc*.yml (mozphab) hits are indeed being tracked there so nothing falls through the cracks. No action needed on the mozphab files from this ticket.

tool calls

[
  {
    "capability": "jira.issue.read",
    "args": {
      "key": "MZCLD-3677"
    },
    "result": {
      "key": "MZCLD-3677",
      "summary": "cloudops-deployment: Datadog notification addresses point at the shut-down moz-svc-ops PagerDuty account",
      "status": "Backlog",
      "description_text": "Different mechanism from the Terraform side: here the dead account is reached by email, from Datadog. cloudops-deployment  configures Datadog notification recipients as  \u2026@moz-svc-ops.pagerduty.com  addresses. That domain belonged to the closed PagerDuty account, so every one of these alerts mails into a void. In scope for this sub-task: databricks \u2014  ansible/envs/databricks-prod.yml#L3 :  data-pipeline@moz-svc-ops\u2026  sits in  failure_emails  alongside two live mozilla.com addresses, so failures still reach someone. servicebook \u2014  ansible/envs/stage.yml#L3 , and two monitor definitions that  @ -mention the dead address directly in the alert message:  api-monitoring.yml#L17 ,  ui-monitoring.yml#L17 Explicitly NOT in scope \u2014 mozphab.  Three further hits belong to Phabricator: devsvcprod.yml#L6 ,  devsvcstage.yml#L6 ,  devsvcdev.yml#L6 These most likely fall under  SREIN-1528  (\"fix phabricator and lando alerts\", In Progress). Flagged to its owner rather than actioned here, so we do not work over someone else's ticket. Ask. Establish for databricks and servicebook whether the services and their Datadog monitors are still live at all. Where live: replace the dead address with a working target, or drop it if the remaining recipients are sufficient. Where dead: remove the configuration with the service. Acceptance criteria. No  moz-svc-ops.pagerduty.com  address remains in  cloudops-deployment  outside the mozphab files. For each remaining monitor, the notification target is confirmed reachable. The mozphab findings are recorded with their owner, not fixed here.",
      "comment_count": 0,
      "comments": []
    }
  },
  {
    "capability": "jira.issue.read",
    "args": {
      "key": "SREIN-1528"
    },
    "result": {
      "error": "read blocked by policy: resource not in run subject",
      "args": {
        "key": "SREIN-1528"
      }
    },
    "blocked": true
  }
]

log