run_b3541c4563d4b602

input

{
  "request": "Triage Jira ticket MZCLD-3654: Investigate autopush's impact on Cloud NAT port allocation",
  "references": [
    "https://mozilla-hub.atlassian.net/browse/MZCLD-3654"
  ],
  "sourceContext": {
    "issue": "MZCLD-3654"
  }
}

summary

MZCLD-3654 asks the team to confirm whether autopush's rollout actually caused the Cloud NAT port-allocation spike (~94%) that triggered a flapping alert/PagerDuty page, or whether that was coincidental timing. The ticket is still in Backlog with no comments and no linked runbook/Confluence doc, so investigation has not yet started; a temporary mitigation (adding NAT IPs 14→20, tracked in global-platform-admin#6900) has already been applied to relieve pressure. The core open question raised by Jason is whether autopush's VMs even traverse Cloud NAT, since each VM reportedly has its own external IP and runs on a separate cluster — if true, autopush's egress would bypass Cloud NAT entirely and could not be the root cause.

findings

• Ticket is unassigned/untriaged: status is Backlog with zero comments, meaning no one has yet done the investigation work described.
• A stopgap mitigation was already applied (increasing Cloud NAT IPs from 14 to 20) to relieve port pressure, but this only buys time and does not answer the causal question.
• There is a credible architectural reason to doubt autopush is the cause: autopush runs on its own cluster where each VM has its own external IP, which would let its egress traffic bypass Cloud NAT altogether.
• The ticket itself defines two concrete, sequential investigation steps that haven't been executed yet: (1) verify autopush's traffic path (Cloud NAT vs. direct external IP egress), and (2) if autopush is ruled out, identify what else was consuming NAT ports during the spike window.
• No linked runbook or Confluence design doc is referenced in the ticket description or comments, so no additional context could be pulled beyond the Jira description and the linked GCP admin change (global-platform-admin#6900).

proposed actions

Triage notes: This is still unassigned/untriaged (Backlog, no comments) despite the mitigation already merged in global-platform-admin#6900 (NAT IPs 14→20). Recommend prioritizing the root-cause check before closing the loop:

1. Confirm autopush's egress path: check whether autopush's GKE/cluster nodes are configured with `--no-nat` / per-node external IPs (or a Cloud NAT config that excludes autopush's subnet/cluster) in the GCP project config. If each autopush VM has its own external IP and NAT is not configured for that subnet, its egress traffic will not consume Cloud NAT ports — this can be verified quickly via `gcloud compute instances describe` on an autopush VM (look for an external IP under networkInterfaces.accessConfigs) and by checking the Cloud NAT gateway's associated subnets/ranges in the VPC config.
2. Pull VPC Flow Logs or Cloud NAT logging (if enabled) for the alert window to see which source IPs/subnets were driving port allocation right before the ~94% spike. Cross-reference the top consumers against known services (batch jobs, cron jobs, other GKE workloads sharing the default egress path) rather than autopush.
3. If autopush is confirmed to bypass Cloud NAT, close out the 'autopush caused it' theory explicitly in this ticket and open a follow-up (or repurpose this ticket) to identify the actual top NAT port consumer, so the IP increase in global-platform-admin#6900 isn't masking a recurring/growing issue.
4. Consider enabling Cloud NAT logging (ERRORS_ONLY or ALL) proactively if not already on, to make the next port-pressure event easier to diagnose without guesswork.

Suggest assigning an owner and moving this out of Backlog given it's tied to a PagerDuty page and an already-applied stopgap fix.

tool calls

[
  {
    "capability": "jira.issue.read",
    "args": {
      "key": "MZCLD-3654"
    },
    "result": {
      "key": "MZCLD-3654",
      "summary": "Investigate autopush's impact on Cloud NAT port allocation",
      "status": "Backlog",
      "description_text": "Cloud NAT port allocation on the default egress instance recently reached ~94%, which set off a flapping alert and a PagerDuty page. This happened around the same time as an autopush rollout, so autopush was our first suspect. We added more IP addresses (14 \u2192 20) to relieve the immediate pressure ( global-platform-admin#6900 ), but we don't yet know whether autopush was actually the cause. Jason pointed out that autopush may not use Cloud NAT at all: it runs on its own cluster, and each of its VMs has its own external IP address, so its traffic to the internet shouldn't go through Cloud NAT. If that's right, autopush wouldn't explain the port usage \u2014 the timing may just have been a coincidence. We want to confirm one way or the other: was autopush responsible for the Cloud NAT port pressure, or not? To look into Check whether autopush's traffic actually goes through Cloud NAT, or bypasses it via the VMs' own external IPs. If it bypasses Cloud NAT, figure out what else was using the ports during that time.",
      "comment_count": 0,
      "comments": []
    }
  }
]

log