run_22c293bfc2ca5c6f

input

{
  "request": "Triage Jira ticket SREIN-1716: Lando in GCPv1 is failing to deploy, trees are closed",
  "references": [
    "https://mozilla-hub.atlassian.net/browse/SREIN-1716"
  ],
  "sourceContext": {
    "issue": "SREIN-1716"
  }
}

summary

SREIN-1716 tracks a failed production deploy of Lando in GCPv1 (Jenkins build #257, tag release-89) that caused trees to close. The failure occurs during the pre-deploy sequence step, where `kubectl exec` into the `api-prod-lando-landingworker-1` statefulset pod fails because the target container `landingworker` cannot be found ("container not found"). This looks like a pod/container naming or scheduling issue (e.g., pod not yet running, wrong container name in the exec target, or a stale/renamed container within the statefulset) rather than an application-level bug in Lando itself. The ticket is still in Backlog status despite trees being closed, suggesting it needs to be picked up/prioritized urgently, and once resolved the team intends to decommission Lando in GCPv1 entirely.

findings

• Deploy failure identified: Jenkins job landoapi-production build #257 failed during the 'Pre-deploy Sequence' stage in prod.
• Root cause narrowed down by Jon Buckley to a kubectl exec command targeting a specific statefulset pod/container that fails because the container isn't found.
• This is impacting tree state (trees closed) implying it's release-blocking / high urgency, yet the Jira ticket status is still 'Backlog', indicating a possible triage/prioritization gap.
• Ticket notes this deploy is a step toward decommissioning Lando in GCPv1, so resolving it unblocks that migration/cleanup effort as well.
• No linked runbook or Confluence design doc was found in the ticket description or comments to consult for additional context.

proposed actions

Triage notes: This is a prod-impacting deploy failure (trees closed) and should likely be re-prioritized out of Backlog into active work given the urgency.

Root cause so far (per Jon Buckley's comments): the pre-deploy sequence step runs `kubectl -n prod-lando exec statefulset/api-prod-lando-landingworker-1 -- lando-cli run-pre-deploy-sequence`, which fails with `error: unable to upgrade connection: container not found ("landingworker")`. This points to a mismatch between the expected container name `landingworker` and what's actually running in the `api-prod-lando-landingworker-1` pod (or the pod/statefulset isn't in a ready/running state at the time of exec).

Suggested next steps:
1. Check `kubectl -n prod-lando describe pod <api-prod-lando-landingworker-1-pod>` and `kubectl -n prod-lando get pod <pod> -o jsonpath='{.spec.containers[*].name}'` to confirm the actual container name(s) in the pod spec — it may have been renamed or removed in a recent chart/manifest change.
2. Compare against the pre-deploy sequence script/pipeline config to see if the hardcoded container name `landingworker` is stale relative to the current statefulset definition.
3. Check recent changes to the Lando Helm chart / k8s manifests around release-89 that might have altered the container name or statefulset pod template.
4. Once identified, either fix the container name reference in the pre-deploy script or restore the expected container name in the manifest, then re-run the deploy to reopen trees.
5. Given this deploy is also required to decommission Lando in GCPv1, consider escalating priority since it's currently blocking both prod deploys and the planned decommission.

tool calls

[
  {
    "capability": "jira.issue.read",
    "args": {
      "key": "SREIN-1716"
    },
    "result": {
      "key": "SREIN-1716",
      "summary": "Lando in GCPv1 is failing to deploy, trees are closed",
      "status": "Backlog",
      "description_text": "lando build #257: tag release-89: environment prod: failed:  https://ops-master.jenkinsv2.prod.mozaws.net/job/gcp-pipelines/job/lando/job/landoapi-production/257/  - prod Pre-deploy Sequence Failure\n\nOnce this is deployed, we can finally decommission lando in GCPv1 Issue created in Slack from a   message .",
      "comment_count": 4,
      "comments": [
        {
          "author": "Slack",
          "body": "lando build #257: tag release-89: environment prod: failed:  https://ops-master.jenkinsv2.prod.mozaws.net/job/gcp-pipelines/job/lando/job/landoapi-production/257/  - prod Pre-deploy Sequence Failure"
        },
        {
          "author": "Slack",
          "body": "Jon Buckley  synced this conversation thread with the Slack thread   . Replies in synced threads will automatically update across both Slack and Jira."
        },
        {
          "author": "Jon Buckley",
          "body": "From the Jenkins output, it's failing to run the command  kubectl -n prod-lando exec statefulset/api-prod-lando-landingworker-1 -- lando-cli run-pre-deploy-sequence"
        },
        {
          "author": "Jon Buckley",
          "body": "which is failing with  error: unable to upgrade connection: container not found (\"landingworker\")"
        }
      ]
    }
  }
]

log