run_111ba543120abfb8

input

{
  "request": "Triage Jira ticket SREIN-1698: ichabod-stage limits.cpu ResourceQuota at 97%, next Firefox version deploy will be blocked",
  "references": [
    "https://mozilla-hub.atlassian.net/browse/SREIN-1698"
  ],
  "sourceContext": {
    "issue": "SREIN-1698"
  }
}

summary

SREIN-1698 is a well-documented, non-urgent capacity/config issue: the ichabod-stage limits.cpu ResourceQuota is pinned at 97% because three Firefox versions (151/152/153) are simultaneously resident in the namespace instead of the normal single-version steady state, not because of actual CPU saturation (real usage is <6% of declared limits). The ticket already contains root-cause analysis, quota math, and three well-scoped remediation options. The immediate risk is that the next Firefox version deploy to stage will be rejected at admission (Deployment stuck Progressing with 0 pods, no visible error), but no outage or current blocking is reported. The key open question that needs an answer before choosing a fix is whether three-version residency is tabstack's new intended steady state or an accidental accumulation (version 140 failed to reap).

findings

• Alert is a reservation problem, not a saturation problem
• Root cause is Firefox version fan-out, not organic load growth
• Immediate blast radius is limited but a future deploy will fail at admission
• Reporter (Central SRE/MozCloud) proposed three concrete, non-mutually-exclusive remediation options with quantified impact
• Decision is blocked on input from the tabstack engineering team
• Similar precedent exists for this alert family

proposed actions

tool calls

[
  {
    "capability": "jira.issue.read",
    "args": {
      "key": "SREIN-1698"
    },
    "result": {
      "key": "SREIN-1698",
      "summary": "ichabod-stage limits.cpu ResourceQuota at 97%, next Firefox version deploy will be blocked",
      "status": "Backlog",
      "description_text": "Service:  ichabod (stage). Namespace  ichabod-stage , project  moz-fx-webservices-high-nonpro , cluster  webservices-high-nonprod , us-west1. Engineering Team:  tabstack Urgency:  No outage. Alert is firing now and the next Firefox version deploy into stage will be rejected at admission. Not blocking anyone today as far as I can tell, so treat as normal priority unless tabstack has a version rollout queued.  Details and context The  limits.cpu  ResourceQuota for  ichabod-stage  jumped from about 40% to 97% at 15:04 PDT on 2026-08-13 and has stayed there. The Central-SRE per-namespace quota alert ( limits.cpu - gcp-v2-nonprod , fires on  used / hard > 0.9  for 5m) has been in Alerting since 22:46Z. This is a reservation problem, not CPU saturation.  Actual CPU use in the namespace is under 6% of the declared limits. Quota state ( ichabod-stage-compute ) Resource Used Hard Utilization limits.cpu 9.7 10 97% requests.cpu 4.25 5 85% limits.memory 15.97G 53.69G 29.7% requests.memory 7.99G 26.84G 29.8% Only  limits.cpu  breaches the 90% threshold.  requests.cpu  at 85% is close behind. Root cause At 15:04 the firefox pod count went from 2 to 6. The namespace moved from a single resident Firefox version (140) to three concurrent versions (151, 152, 153). before 15:04 now Firefox versions resident 140 151, 152, 153 containers in namespace 11 28 limits.cpu used 3.9 to 4.1 9.7 Each version costs 2 firefox pods plus 2 geckodriver pods, and every one of those pods carries an app container (0.5 CPU limit) plus a  proxy  sidecar (0.2 CPU limit): limits.cpu: 2.8 cores per version requests.cpu: 1.2 cores per version Arithmetic check: 3 versions x 2.8 = 8.4, plus api 2 x 0.5 = 1.0, console 0.1, reaper CronJob pod 0.2 = 9.7. The prior pattern was single-version rollover: 2 steady-state firefox pods with brief 4 to 5 pod overlaps during deploys. The 15:04 change was different. Version 140 went away and three new versions stayed resident. Risk Headroom is 0.3 cores on  limits.cpu  and 0.75 on  requests.cpu . A fourth Firefox version needs 2.8 and 1.2 respectively.  The next version deploy gets rejected at admission with  exceeded quota: ichabod-stage-compute . The Deployment stays Progressing with zero pods, no crash and no application-level error, which is easy to miss. Firefox pod count already blipped 6 to 7 to 6 between 15:14 and 15:24, so there is churn at the ceiling.  Request Decide between (or combine) the two fixes below. Which one is right depends on a question for tabstack:  is three-version residency the new intended steady state, or did version 140 fail to reap and 151/152/153 accumulate by accident? Option 1: raise the quota (needs MozCloud) limits.cpu  10 to 20 and  requests.cpu  5 to 10 on  ichabod-stage-compute . This accommodates three resident versions plus rollover overlap. Correct fix if three-version residency is intended. Option 2: right-size container limits (needs tabstack) 30-day peak CPU utilization per container, measured with  max by (container_name)(max_over_time({\"kubernetes.io/container/cpu/limit_utilization\", namespace_name=\"ichabod-stage\"}[10m]))  range-queried at 600s step. Limits were constant across the whole window, so the percentages are comparable. Container limit p50 p99 30d peak peak cores proposed limit new headroom firefox 0.50 6.6% 28.1% 77.1% 0.385 keep 0.50 1.3x geckodriver 0.50 0.0% 0.2% 26.5% 0.133 0.25 1.9x api 0.50 0.1% 0.5% 6.0% 0.030 0.25 8.3x proxy 0.20 0.1% 1.0% 5.2% 0.010 0.05 5x console 0.10 0.4% 1.9% 50.2% 0.050 keep 0.10 2.0x reaper 0.20 1.3% 2.5% 2.8% 0.006 0.05 8x firefox and console cannot be trimmed.  firefox peaked at 0.385 cores against its 0.5 limit on 08-04, and console peaked at 50% of its 0.1 limit. The proposed limits for the others are floored by their existing CPU requests (Kubernetes requires limit >= request), which is why geckodriver and api stop at 0.25 and proxy at 0.05. Result at three versions: firefox 6 x 0.5 = 3.0, geckodriver 6 x 0.25 = 1.5, proxy 12 x 0.05 = 0.6, api 2 x 0.25 = 0.5, console 0.1, reaper 0.05 =  5.75 of 10 (58%) . Per-version limits cost drops from 2.8 to 1.7. Important:  option 2 alone does not unblock a fourth version. It leaves  requests.cpu  untouched at 4.25 of 5, and a fourth version adds 1.2 requests, reaching 5.45 and exceeding the 5-core cap. Requests are actually the more over-provisioned side: geckodriver requests 0.25 against a p50 of 0.0 and a 30-day peak of 0.133. So either requests come down too, or the  requests.cpu  quota rises regardless. Option 3: cap resident versions (needs tabstack) If three concurrent Firefox versions was not intended, reap the older ones and the quota pressure disappears with no config change.  Caveats on the measurement GKE  cpu/limit_utilization  is averaged over its sample period, so sub-second bursts are invisible. CPU limits throttle rather than kill, and the GKE metric set has no throttled-time counter, so existing throttling cannot be confirmed from Grafana. Anything above roughly 50% average peak should be treated as possibly already throttling at burst. That is the reason firefox and console are left alone above. A plain range query at 600s step reports the firefox 30-day peak as 54.3%. That is aliasing from point-sampling the gauge. The  max_over_time(...[10m])  wrapper recovers the true 77.1%. kube_pod_container_resource_limits  and  kube_pod_info  are not scraped on yardstick. Only  kube_resourcequota  is available from kube-state-metrics, so per-container limits and requests come from  kubernetes.io/container/cpu/{limit,request}_cores  instead. Reproducing the numbers Datasource  gcp-v2-nonprod  (uid  edq6thuke248we ): Quota:  kube_resourcequota{namespace=\"ichabod-stage\"} Per-container limits:  sum by (pod_name,container_name)({\"kubernetes.io/container/cpu/limit_cores\", namespace_name=\"ichabod-stage\"}) Peak utilization:  max by (container_name)(max_over_time({\"kubernetes.io/container/cpu/limit_utilization\", namespace_name=\"ichabod-stage\"}[10m])) Note that kube-state-metrics uses the  namespace  label while GMP container metrics use  namespace_name . Related Same alert rule family and same reservation-vs-consumption pattern as the earlier experimenter-prod ResourceQuota investigation (there the driver was a misconfigured HPA memory target rather than version fan-out).",
      "comment_count": 0,
      "comments": []
    }
  }
]

log