run_23f40b892a421d38

input

{
  "request": "Triage Jira ticket MZCLD-3723: ArgoCD not updating moderator image tags anymore",
  "references": [
    "https://mozilla-hub.atlassian.net/browse/MZCLD-3723"
  ],
  "sourceContext": {
    "issue": "MZCLD-3723"
  }
}

summary

MZCLD-3723 reports that ArgoCD Image Updater detects new moderator images but the running workload keeps the old tag. Root cause has already been identified in the ticket description: the git write-back file for the moderator-stage-us-west1 app (.argocd-source-moderator-stage-us-west1-moderator.yaml in mozilla/webservices-infra) contains duplicate `global.mozcloud.image.tag` helm parameter entries — the Image Updater writes the new tag first, but a stale duplicate entry later in the list wins because ArgoCD/Helm apply --set-string params in list order and the last one takes precedence. This causes the Image Updater to loop endlessly re-detecting a "new" image that never actually gets rendered. The ticket has a clear remediation checklist but no comments yet and no linked runbook/design doc was found to cross-reference. Ticket is already In Progress; recommend confirming remediation status and next steps.

findings

• Root cause is already diagnosed and documented in the ticket: duplicate 'global.mozcloud.image.tag' helm parameter entries in the ArgoCD write-back source file cause Helm to apply the stale tag last, overriding the new tag written by Image Updater.
• The duplicate 'global.mozcloud.image.repository' parameter is also present but harmless since both values are identical.
• Ticket includes a concrete remediation/verification checklist but there is no indication yet (via comments) that any of these steps have been executed.
• Ticket status is 'In Progress' but with zero comments, so it's unclear how far along the fix/investigation is or whether other moderator environments have been checked for the same duplicate-parameter issue.
• No linked Confluence runbook or design doc was found in the ticket description/comments to cross-reference for broader ArgoCD write-back conventions or prior incidents.

proposed actions

Triage note: Root cause looks solid and well-documented — duplicate `global.mozcloud.image.tag` (and harmless duplicate `global.mozcloud.image.repository`) entries in `.argocd-source-moderator-stage-us-west1-moderator.yaml` cause Helm to apply the stale tag last, so Image Updater's new tag never actually renders. Recommended next steps to close this out:
1. Confirm current state with `argocd app manifests moderator-stage-us-west1-moderator | grep image:` to verify which tag is actually rendered right now.
2. Fix the file: remove the duplicate parameter entries, keeping a single `global.mozcloud.image.tag` / `global.mozcloud.image.repository` pair with the correct (latest) values.
3. Sweep other moderator environments for the same defect: `grep -c 'global.mozcloud.image.tag' moderator/k8s/moderator/.argocd-source-*.yaml` — any file with count > 1 needs the same fix.
4. Run `git log -p --follow` on the affected file to determine how the duplicate got introduced (bad merge resolution vs. manual pin), and note the finding here for the record.
5. Once fixed, consider adding a CI/pre-commit check that rejects duplicate parameter names in `.argocd-source-*.yaml` files to prevent recurrence — this could be tracked as a follow-up ticket if it's non-trivial to add now.
Please post an update here once the rendered-tag check and the multi-environment grep have been run so we can confirm scope before closing.

tool calls

[
  {
    "capability": "jira.issue.read",
    "args": {
      "key": "MZCLD-3723"
    },
    "result": {
      "key": "MZCLD-3723",
      "summary": "ArgoCD not updating moderator image tags anymore",
      "status": "In Progress",
      "description_text": "h2. Symptom ArgoCD Image Updater picks up new images for moderator stage, but the new tag never reaches the running workload. The old tag stays deployed. h2. Root cause (identified) The git write-back file contains  duplicate  helm parameter entries: {{moderator/k8s/moderator/.argocd-source-moderator-stage-us-west1-moderator.yaml}} (repo: {{mozilla/webservices-infra}}) {code:yaml} name: global.mozcloud.image.tag\nvalue: v2.2.3-10-ga713830   # new, written by Image Updater name: global.mozcloud.image.tag\nvalue: v2.2.2-4-g3fe2b5c    # stale\n{code} {{global.mozcloud.image.repository}} is duplicated as well, but with identical values, so it is harmless. ArgoCD translates {{helm.parameters}} into {{--set-string}} arguments in list order, and Helm applies them in order \u2014 the  last  one wins. The effective rendered tag is therefore the stale {{v2.2.2-4-g3fe2b5c}}. The Image Updater keeps re-detecting the newer image every cycle because the effective tag never advances. h2. To do Verify the rendered tag: {{argocd app manifests moderator-stage-us-west1-moderator | grep image:}} Remove the duplicate entries, keep a single {{global.mozcloud.image.tag}} / {{global.mozcloud.image.repository}} pair Check the other moderator environments for the same problem: {{grep -c 'global.mozcloud.image.tag' moderator/k8s/moderator/.argocd-source-*.yaml}} \u2014 anything > 1 is affected Determine how the duplicates got in (badly resolved merge conflict vs. manual pin) via {{git log -p --follow}} on the file, so it does not recur Consider a guard (CI check / pre-commit) that rejects duplicate parameter names in {{.argocd-source-*.yaml}} files",
      "comment_count": 0,
      "comments": []
    }
  }
]

log