run_7d45317a5a72a183

input

{
  "request": "Triage Jira ticket MZCLD-3775: Investigate BigQuery Physical Storage Metering with New Contract",
  "references": [
    "https://mozilla-hub.atlassian.net/browse/MZCLD-3775"
  ],
  "sourceContext": {
    "issue": "MZCLD-3775"
  }
}

summary

MZCLD-3775 is a FinOps investigation ticket into a BigQuery billing metering change (physical storage → active + fail-safe SKUs) that coincided with the 2026-06-19 GCP contract go-live. The ticket already contains a complete, well-evidenced investigation (root cause, quantified cost impact, contract analysis, mitigation levers, and a drafted support-case/account-team action plan) but is still sitting in "Backlog" with zero comments and no assignee/owner action recorded. This is effectively "investigation complete, action not yet initiated" — triage should focus on getting it moved to an actionable status and routed to the right owner(s) to execute the outlined next steps, not on further root-cause work.

findings

• Root cause is fully identified and reconciled: on 2026-06-19 Google retired SKU 434D (Physical Storage) and replaced it with 901F (Active Physical Storage) + 2FAA (Long-Term Physical Storage). The new SKUs meter total physical bytes PLUS fail-safe bytes, whereas the old private 2020 flat-rate SKU (434D, $0.0196/GiB/mo) never billed fail-safe. Storage consumption itself is flat (7.04–7.05 PB for moz-fx-data-shared-prod across the transition, staying 6.89–7.08 PB through 8/13).
• Cost impact is quantified and reconciled to within 0.3% for moz-fx-data-shared-prod, and extrapolated account-wide.
• Contract analysis concludes this is a change-management/modeling gap, not a billing error or contractual violation — the 50% discount rate is correctly applied per the addendum, but fail-safe billing itself is a metering-scope change with no advance-notice clause.
• A concrete, ready-to-execute action plan already exists: a drafted Cloud Billing Support case (finops/bq_physical_storage_support_case.md) and an account-team ask list (SKU/config inventory, advance-notice commitment, discount increase from 50% to 58% to neutralize fail-safe cost — worth ~$1.47M over the 60-month term — and support for a time-travel reduction review). None of this appears to have been executed yet since the ticket has no comments.
• There is a known correction needed to two other artifacts (outside this ticket) that currently contain a stale/incorrect claim about BigQuery billing at net $0.
• Ticket is stale relative to its content: status is Backlog, zero comments, and no confluence runbook/design doc was referenced in the description (only local repo file paths: finops/bq_physical_storage_0619.md/.sql/_support_case.md/_reconciliation.csv), so no Confluence lookup was applicable.

proposed actions

tool calls

[
  {
    "capability": "jira.issue.read",
    "args": {
      "key": "MZCLD-3775"
    },
    "result": {
      "key": "MZCLD-3775",
      "summary": "Investigate BigQuery Physical Storage Metering with New Contract",
      "status": "Backlog",
      "description_text": "Ben Wu flagged that the BigQuery GiB-month units billed per day jumped for every project at the new GCP contract go-live, with moz-fx-data-shared-prod up about 18% and no increase in stored data. Analysis confirms a metering change rather than consumption growth. Billing account 01E7D5-97288E-E2EBA0. h2. What changed on 2026-06-19 Three things happened at once. Metering. Fail-safe storage is now billed. The retired SKU billed total physical bytes. The replacement SKUs bill total physical bytes plus fail-safe physical bytes. Fail-safe is a fixed 7-day retention of deleted and changed data sitting behind the time travel window. It is standard behavior under physical storage billing and was never billed under Mozilla's 2020 private flat-rate SKU. SKUs. 434D-04F2-758E Physical Storage was retired and replaced by 901F-36D9-3005 Active Physical Storage (US) and 2FAA-620A-0686 Long-Term Physical Storage (US), plus small us-west1 equivalents. Rate. The old contract priced physical storage at a fixed $0.0196/GiB/month for everything (Discount 7 of the 2020 addendum). The new addendum gives 50% off list, which is $0.02 for active and $0.01 for long-term. Active is 2% more expensive, long-term is 49% cheaper. h2. Evidence Consumption is flat. Total physical bytes for moz-fx-data-shared-prod were 7.04 PB on 6/18 and 7.05 PB on 6/20, and have stayed between 6.89 PB and 7.08 PB through 8/13. Billed quantity times days in the calendar month matches a different definition on each side of the change: 6/8 to 6/17: 0.988 to 0.993 of total_physical_bytes 6/20 to 8/9: 0.992 to 1.003 of total_physical_bytes plus fail_safe_physical_bytes Average daily GiB-month, 6/5 to 6/17 against 6/20 to 6/30: || Project || Pre || Post || Change ||\n| moz-fx-data-shared-prod | 230,695 | 271,986 | +17.9% |\n| moz-fx-data-shredder | 4,089 | 11,764 | +187.7% |\n| moz-fx-glam-prod | 2,768 | 4,630 | +67.3% |\n| moz-fx-fxa-prod | 683 | 745 | +9.1% |\n| moz-fx-sync-prod | 868 | 937 | +8.0% | 210 projects are affected and all changed on the same date, so this was an account-level action rather than anything configured per project. Shredder's 188% is the signature of the mechanism: it deletes constantly, so nearly all of its physical footprint is deleted data in time travel and fail-safe. Fail-safe is genuinely additive, not already counted inside total physical bytes, so Google is not double-counting. On 8/13, 897 of 12,095 tables in shared-prod had fail_safe_physical_bytes greater than total_physical_bytes, and 743 had fail-safe bytes with zero total physical bytes. For every row total equalled active plus long-term, and current equalled total minus time travel. No SKU with fail-safe or time travel in its description has ever been billed to this account, checked across every SKU billed since 2026-05-01. h2. Cost effect moz-fx-data-shared-prod, July 2026, the first clean full month after the change: || Effect || Monthly || Annual ||\n| Fail-safe bytes now billed | +$24,474 | +$294K |\n| Active rate $0.0196 to $0.02 | +$2,522 | +$30K |\n| Long-term rate $0.0196 to $0.01 | -$6,576 | -$79K |\n| Net | +$20,420 | +$245K | Modeled new regime is $157,428/month against actual billed net of $156,931, so the decomposition reconciles to 0.3%. Across all projects and all physical storage SKUs, July net was $171,669/month against a pre-change run rate of $143,590/month, roughly +$337K/year. h2. Contract position The rate is in the contract. Section 5.2 of the 6/8/2026 pricing addendum (v2.0, Quote Q-393443) lists \"BigQuery Physical Storage 50.00%\". The string 0196 appears nowhere in that addendum, so the 2020 fixed-price arrangement is gone. Section 7 lets Google apply addendum discounts to successor SKUs in the same SKU Group, which is what happened, and the 50% was applied correctly to both new SKUs. The fail-safe charge is not a contract term and does not appear in the addendum. There is no clause covering metered quantity and no general change-notice obligation. Both sides priced the renewal off historical billed quantities that excluded fail-safe, because Google had Mozilla on a private physical storage SKU since 2020. This is a change-management and modeling gap, not a billing error and not a contractual claim. h2. Levers Time travel window is configurable from 2 to 7 days per dataset. shared-prod carries 1.21 PB of time travel bytes, about $24K/month at the new rate. Shortening it on high-churn derived and staging tables is the largest available reduction. Fail-safe cannot be shortened or disabled. Only reducing deletes and rewrites reduces it. h2. Action Reach out to Cloud Billing Support and the GCP account team. Billing support case: ask them to confirm the SKU mapping and effective date, confirm the quantity definition on 901F including fail-safe and time travel, confirm fail-safe was never billed on 434D, explain the 6/18 partial day and 6/19 zero-quantity day, confirm the logical storage SKUs stopped metering as expected, list any other private SKUs or account-level billing configurations that changed on 6/19 or remain in place, provide per-project fail-safe and time travel byte totals as Google meters them, and confirm the project-level time travel default. Case text is drafted and ready to paste. Account team: raise it alongside the other post-go-live variances rather than standalone. Asks are an inventory of remaining non-standard account configuration, a commitment to give advance notice of account-level changes affecting metered quantity and not only price, an increase to the BigQuery Physical Storage discount from 50% to 58% which neutralizes the increase (about $1.47M over the 60-month term for the fail-safe component alone), and account team support for a time travel reduction review. h2. Notes Fail-safe attribution is verified directly for moz-fx-data-shared-prod only. monitoring_derived.table_storage_v1 covers that one project, and it is a straight passthrough of region-US.INFORMATION_SCHEMA.TABLE_STORAGE. Other projects are inferred from the shape of the billed-unit jump. 6/18 billed at 29% of a normal day and 6/19 billed nothing, about 1.7 days unbilled during the cutover, roughly $8K one-time in Mozilla's favor. Annualized figures use monthly times 12. Storage is priced per calendar month, so daily times 365 distorts it. Correction to prior work: the 7/10 finding that physical storage bills 100% off at net $0 since 6/20 does not hold. The export has since been restated and 901F bills at gross $0.02 against $0.04 list, exactly the contracted 50%, every day from 6/20 through 8/9. The stale line appears in the Chris deal brief and in the 6/19 discount deviation summary and needs fixing there. h2. Local artifacts finops/bq_physical_storage_0619.md, findings and method finops/bq_physical_storage_0619.sql, all queries finops/bq_physical_storage_support_case.md, support case text finops/bq_physical_storage_0619_reconciliation.csv, daily billed quantity against metered bytes",
      "comment_count": 0,
      "comments": []
    }
  }
]

log