run_7eda1595e0d8a42c

input

{
  "request": "Triage Jira ticket SREIN-1677: Autoconnect 5xx regression from node affinity change increasing pod density",
  "references": [
    "https://mozilla-hub.atlassian.net/browse/SREIN-1677"
  ],
  "sourceContext": {
    "issue": "SREIN-1677"
  }
}

summary

SREIN-1677 documents a well-investigated regression: autoconnect's 5xx rate jumped 8x (0.23%→1.84%, ~39.5M failed WebSocket handshakes/day) starting 2026-08-03, driven by pods failing their own liveness/readiness probes under high pod density (14 pods per c2d-standard-16 node) rather than by the code rollout in the same window or by CPU/memory/FD/lock-contention issues, all of which the reporter has already ruled out with solid evidence. Root cause is traced to webservices-infra#11429 (merged 2026-06-17, live in prod 07/29), which replaced a nodeSelector pinning autoconnect to small nodepools with a nodeAffinity rule; this let the autoscaler consolidate pods onto larger c2d-standard-16 nodes, incidentally removing the density cap that had been keeping the node network stack (softirq/conntrack/socket memory — costs invisible to pod-level cgroup metrics) under control. The reporter proposes reducing pods/node (e.g., raising cpu request to 1500m to go from 14→10 pods/node) and is soliciting alternative approaches. The ticket is thorough, actionable, and currently sitting in Backlog with zero comments — it needs triage prioritization and engineering input on the proposed mitigation before it can move to execution.

findings

• Clear, quantified regression correlated to a specific infra change, not the concurrent code rollout
• Reporter systematically excluded common alternative root causes with concrete evidence
• Root cause isolated to pod density on a node, not code/config/image differences
• Plausible causal mechanism proposed for why density causes probe failures despite low CPU/memory
• Concrete but unvalidated mitigation proposed; ticket asks for reviewer input and is not yet actioned

proposed actions

tool calls

[
  {
    "capability": "jira.issue.read",
    "args": {
      "key": "SREIN-1677"
    },
    "result": {
      "key": "SREIN-1677",
      "summary": "Autoconnect 5xx regression from node affinity change increasing pod density",
      "status": "Backlog",
      "description_text": "Problem autoconnect pods are packed 14-per-node onto c2d-standard-16 nodes. At that density I believe the node's kernel network stack saturates, leading to pods being unable to answer any HTTP requests including their own liveness probe.Since 2026-08-03 this has driven autoconnect's 5xx rate from  0.23% baseline (07/30-08/02) to 1.84%  (8x), sustained, at roughly 39.5M failed WebSocket handshakes per day. Request volume was flat throughout, so this is entirely a failure-rate change, not load. Investigation In addition to the increased 5xx rate I noticed significantly elevated pod churn and restarts; pods would become unhealthy due to timeout on   /__lbheartbeat__  (a no-dependency check which returned immediately) and be restarted. One pod logged 1,451 liveness and 2,239 readiness failures in 23h. There was a code rollout in the same window, which we investigated first. We specifically checked and excluded: excess logging cpu exhaustion:  log lines per request are flat across the boundary (0.99, 0.99, 0.99, 0.98, 0.94, 0.97). No change in log amounts. CPU is at low utilization for affected pods (1.7-3.4%) bigtable channel consolidation:  its failure modes are Hello-time and would shorten session lifetimes, raising reconnect volume. Volume was flat. Blocked threads:  per-minute log output is continuous  through  the probe failures (1,000-5,000 lines/min, never zero except at restarts), indicating process is not wedged. Lock contention:  the client registry is a DashMap with sharded locks and non-blocking  try_send . The reliability path uses  deadpool_redis  with  redis::aio  \u2014 fully async, no blocking client on worker threads. Worker sizing:  actix-server 2.6.0 uses  available_parallelism() , which is cgroup-aware and returns 3, matching the CPU limit. CPU is at low utilization for affected pods (1.7-3.4%), memory is at 40% which does not suggest memory pressure. File-descriptor exhaustion:  each WebSocket connection holds an FD   (~83k per pod), and hitting the limit would stop  accept()  with the same  \"connections not serviced, no CPU used\" signature. actix logs   error accepting connection  and backs off 500ms when this happens, but this never was observed. I noticed that the unhealthy events seemed to specifically affect some pods, so I started investigating pod-specific causes. I discovered that the unhealthy events clustered for pods on a specific node. Same instance group  d71fb13a , all c2d-standard-16, same image, same config, comparable age (2026-08-04/05): node pods Unhealthy  events Started vnns 14 4,170 70 b9rs 2 0 4 The other six 14-pod nodes in that group: 3,875-4,172 each. The 2-pod node has zero, and its  Started  records confirm its pods existed and its kubelet was reporting. Fleet-wide on 08/05, 17 of 103 nodes had any probe failures:  all 12  of the 14-pod nodes, and 5 of ~91 sparse nodes (three with a single record). placement pods/vCPU outcome c2d-standard-16 @ 14 0.875 fails (12/12 nodes) c2d-standard-4  @ 2 0.50 healthy (~86/91 nodes) c2d-standard-16 @ 2 0.125 healthy (b9rs) Hypothesis Networking is a hidden resource cost and drives the errors. Each pod holds ~83k long-lived WebSocket connections, so a 14-pod node carries ~1.16M concurrent TCP connections. Softirq, conntrack and socket memory are node-level costs not charged to the container cgroup ,  so they appear in no pod metric and cannot be expressed as a resource request. webservices-infra#11429 replaced  nodeSelector: cloud.google.com/gke-nodepool  with  nodeAffinity  on  tenant: autopush  (merged 2026-06-17, live in prod by 07/29). The nodepool pin had incidentally enforced a safe density by confining autoconnect to small machines; once removed, the autoscaler consolidated the now-optional small pool and successive rollouts moved pods onto c2d-standard-16 nodes, completing with the 08/03 rollout. We aren\u2019t fully ruling out the possibility of code change impact, but the   decisive point is that the same binary and config on the same machine type fails at 14 pods/node and not at 2. Proposal The solution is to reduce the number of pods per node. We don\u2019t yet know what the optimum number is, just that 14 is too much and 2 is OK. We could consider bumping resources.requests.cpu  to  1500m , which would reduce the number of nodes on c2d-standard-16 from 14 to 10. This isn\u2019t guaranteed to be the right number, but it could work. Are there any other suggestions you have? Verification Once pods per node are reduced, per-node probe failures should go to zero on the previously-dense nodes, and  the LB 5xx rate return toward ~0.23%. SELECT jsonPayload.source.host AS node,\n       COUNT(DISTINCT jsonPayload.involvedObject.name) AS pods,\n       COUNTIF(jsonPayload.reason = 'Unhealthy') AS unhealthy\nFROM `moz-fx-autopush-prod.gke_autopush_prod_log.events`\nWHERE jsonPayload.involvedObject.name LIKE 'autoconnectrs-%'\n  AND _PARTITIONTIME = TIMESTAMP(CURRENT_DATE())\nGROUP BY node ORDER BY unhealthy DESC",
      "comment_count": 0,
      "comments": []
    }
  }
]

log