run_ccdc688862ade712

input

{
  "request": "Triage Jira ticket MZCLD-3571: Investigate Argo errors/performance issues",
  "references": [
    "https://mozilla-hub.atlassian.net/browse/MZCLD-3571"
  ],
  "sourceContext": {
    "issue": "MZCLD-3571"
  }
}

summary

MZCLD-3571 reports an uptick in Argo CD UI slowness accompanied by ComparisonError/DeadlineExceeded gRPC errors when comparing target state for tenant applications. The ticket is unassigned in Backlog status with no comments or linked runbook. The hypothesis is resource contention (CPU/memory) on repo-server or application-controller components, or excessive concurrent reconciliation/comparison load. Recommend triage comment requesting resource metrics, error frequency/scope, and pointers to relevant dashboards/runbooks before prioritizing.

findings

• Core symptom is a gRPC DeadlineExceeded error during Argo CD's application comparison/reconciliation step, specifically 'failed to compare revisions for source 1 of 1'.
• This error signature is characteristic of the argocd-repo-server (or app-controller) timing out, commonly caused by CPU/memory resource limits being too low, too many concurrent repo/manifest generation requests, or slow git/helm/kustomize operations under load.
• Ticket lacks supporting diagnostic data: no metrics (CPU/memory/pod restarts), no timeframe/frequency of occurrence, no list of affected tenant applications, and no linked runbook or dashboard for Argo CD health.
• Ticket is unprioritized and unassigned (status: Backlog), suggesting it needs triage before engineering work begins.

proposed actions

Thanks for filing this. To triage effectively, could we gather a bit more data:

1. Resource usage: Please check CPU/memory utilization and any throttling/OOM events for argocd-repo-server and argocd-application-controller pods over the period when slowness/errors were observed (Grafana/Prometheus dashboards if available).
2. Scope & frequency: How many tenant applications are affected, and how often does the ComparisonError/DeadlineExceeded occur (e.g., % of syncs, specific times of day, correlated with load spikes)?
3. Repo-server queue depth: Check `argocd-repo-server` logs/metrics for manifest generation queue length and average generation time — this is the most common root cause of comparison timeouts under resource pressure.
4. Current sizing: What are the current resource requests/limits and replica counts for repo-server and application-controller? Have these been tuned since initial deployment, or do they reflect default Helm chart values?
5. Recent changes: Any recent increase in number of Applications/tenants, repo size, or Argo CD version upgrade that correlates with the onset of slowness?

Given the error pattern (gRPC DeadlineExceeded on repo comparison), this strongly points to repo-server being under-resourced or overloaded (increase repo-server replicas/CPU, and/or increase `argocd-repo-server` gRPC timeout via `server.repo.server.timeout.seconds`, and consider enabling repo-server caching). Once we have the metrics above, we can decide whether to scale resources, tune timeouts, or investigate a specific slow repo/source.

Suggest prioritizing this out of Backlog if the issue is currently impacting tenant deploys.

tool calls

[
  {
    "capability": "jira.issue.read",
    "args": {
      "key": "MZCLD-3571"
    },
    "result": {
      "key": "MZCLD-3571",
      "summary": "Investigate Argo errors/performance issues",
      "status": "Backlog",
      "description_text": "We have been seeing an uptick of slowness in the Argo UI, usually coupled with the following error in tenant applications: ComparisonError: Failed to load target state: failed to compare revisions for source 1 of 1: rpc error: code = DeadlineExceeded desc = context deadline exceeded We need to see if this is a matter of Argo being under resourced.",
      "comment_count": 0,
      "comments": []
    }
  }
]

log