run_35ff3ecf2bdb6bf7

input

{
  "request": "Triage Jira ticket MZCLD-3672: Resolve the two hardcoded PagerDuty service keys in cloudops-infra",
  "references": [
    "https://mozilla-hub.atlassian.net/browse/MZCLD-3672"
  ],
  "sourceContext": {
    "issue": "MZCLD-3672"
  }
}

summary

MZCLD-3672 tracks remediation of two hardcoded PagerDuty service keys (32-hex, plaintext) in cloudops-infra Terraform, feeding google_monitoring_notification_channel resources for the Coverity and Fraud Pipeline stacks. The ticket is well-scoped and already contains most of the investigation needed; it's currently sitting in Backlog with no comments. The main open item is verifying the keys' live/dead status on the moz-svc-ops.pagerduty.com account (not reachable with the current token) and confirming whether removal should happen standalone or fold into the referenced GCPv1 decommission tickets (OPST-2171 for coverity, OPST-2198 for productdelivery). I was unable to pull those OPST tickets directly due to run-scope policy restrictions, so triage recommendations below note that as a dependency for whoever picks this up. No Confluence runbook/design doc was referenced in the ticket text, so none was consulted.

findings

• Two hardcoded PagerDuty service keys are committed in plaintext in cloudops-infra Terraform, used as 'service_key' in google_monitoring_notification_channel resources.
• Neither key was found among the 182 live integration keys checked on mozilla.pagerduty.com, but this does not prove they are dead since both configs actually reference a different account, moz-svc-ops.pagerduty.com.
• Resolution may overlap with in-flight GCPv1 decommission work for the same infrastructure (coverity and productdelivery), which could make standalone remediation unnecessary if the surrounding infra is already being torn down.
• Ticket has clear, actionable acceptance criteria but no progress recorded yet (Backlog status, zero comments), and cross-referenced OPST tickets could not be read in this session due to policy scoping, so their current status is unknown to this triage pass.

proposed actions

Triage note: This ticket is well-scoped and ready to be picked up. Suggested next steps in order:

1. Get access/credentials for moz-svc-ops.pagerduty.com (separate account from mozilla.pagerduty.com) and check both service keys there. If live, revoke them immediately since they're exposed in plaintext in version control.
2. Before doing standalone remediation, check current status of OPST-2171 (coverity GCPv1 decommission) and OPST-2198 (productdelivery decommission). If either stack is already slated for teardown in the near term, this ticket's fix may simply be 'delete the Terraform module' as part of that decom rather than a targeted key-removal PR — please link/cross-reference outcomes here.
3. Regardless of live/dead status, remove both hardcoded keys from the cloudops-infra repo (coverity/tf/modules/coverity/main.tf and fraudpipeline/tf/prod/envs/prod/monitoring/main.tf) and replace with a reference to a secret manager / TF variable rather than a literal, per acceptance criteria ('a plaintext integration key does not belong in version control').
4. Record final disposition (dead/revoked, and whether infra was already decommissioned) directly on this ticket to satisfy acceptance criteria.

Note: I could not pull OPST-2171/2198/2460 directly in this triage pass (out of scope for this run), so please confirm their current state manually before closing this out.

tool calls

[
  {
    "capability": "jira.issue.read",
    "args": {
      "key": "MZCLD-3672"
    },
    "result": {
      "key": "MZCLD-3672",
      "summary": "Resolve the two hardcoded PagerDuty service keys in cloudops-infra",
      "status": "Backlog",
      "description_text": "Context:  Surfaced while closing OPST-2460 (PagerDuty API key rotation). Problem.  Two 32-hex PagerDuty service keys are committed in plaintext Terraform in  cloudops-infra  and fed into a  google_monitoring_notification_channel : projects/coverity/tf/modules/coverity/main.tf  (local  pagerduty_key , used at  service_key ) projects/fraudpipeline/tf/prod/envs/prod/monitoring/main.tf  (same pattern) Neither key appears among the 182 live integration keys in  mozilla.pagerduty.com  (checked 2026-08-05, read-only).  But that does not prove they are dead  \u2014 both configs reference  moz-svc-ops.pagerduty.com , which is a separate PagerDuty account that could not be reached with the token available. Ask. Check both keys against  moz-svc-ops.pagerduty.com . Revoke them if they are live. Remove them from the repository either way \u2014 a plaintext integration key does not belong in version control. Coordinate with the GCPv1 decommission: coverity is tracked in OPST-2171 and productdelivery in OPST-2198, so this may fold into that work rather than needing separate action. Acceptance criteria. Both keys are confirmed dead or have been revoked. Neither key appears in the repository. If the surrounding infrastructure is already gone, that is recorded and the decom tickets are referenced.",
      "comment_count": 0,
      "comments": []
    }
  },
  {
    "capability": "jira.issue.read",
    "args": {
      "key": "OPST-2171"
    },
    "result": {
      "error": "read blocked by policy: resource not in run subject",
      "args": {
        "key": "OPST-2171"
      }
    },
    "blocked": true
  },
  {
    "capability": "jira.issue.read",
    "args": {
      "key": "OPST-2198"
    },
    "result": {
      "error": "read blocked by policy: resource not in run subject",
      "args": {
        "key": "OPST-2198"
      }
    },
    "blocked": true
  },
  {
    "capability": "jira.issue.read",
    "args": {
      "key": "OPST-2460"
    },
    "result": {
      "error": "read blocked by policy: resource not in run subject",
      "args": {
        "key": "OPST-2460"
      }
    },
    "blocked": true
  }
]

log