run_4932ca8145e648e3

jira-triage v1 · jira-triage-k8s · owner ingest:jira-new-ticket (api) · via jira-new-ticket · cloud-engineering

input

{
  "request": "Triage Jira ticket MZCLD-3842: Add Cloud Armor rate-based bans to the ArgoCD external backends as an interim mitigation",
  "references": [
    "https://mozilla-hub.atlassian.net/browse/MZCLD-3842"
  ],
  "sourceContext": {
    "issue": "MZCLD-3842"
  }
}

summary

MZCLD-3842 is a well-specified, backlog interim mitigation ticket to add a Cloud Armor rate-based ban policy to the ArgoCD external `server` backend across all three realms (sandbox, dataservices, webservices), as defense-in-depth ahead of the (gated/spiked) Fastly work in MZCLD-3841. The ticket already contains concrete traffic baselines, sequencing guidance, and explicit constraints (don't touch internal gateway backends, use rate-based IP ban, consider preview mode, ensure diagnosability). There are no comments yet and status is still Backlog, so this needs prioritization/assignment and a few implementation details confirmed before work starts.

findings

[
{
"
d
e
s
c
r
i
p
t
i
o
n
"
:
"
T
i
c
k
e
t
i
s
s
e
l
f
-
c
o
n
t
a
i
n
e
d
a
n
d
t
e
c
h
n
i
c
a
l
l
y
w
e
l
l
-
s
c
o
p
e
d
:
i
t
n
a
m
e
s
t
h
e
e
x
a
c
t
f
i
l
e
/
p
a
t
t
e
r
n
t
o
f
o
l
l
o
w
(
a
r
g
o
/
k
8
s
/
a
r
g
o
c
d
-
b
o
o
t
s
t
r
a
p
/
t
e
m
p
l
a
t
e
s
/
b
a
c
k
e
n
d
-
p
o
l
i
c
i
e
s
.
y
a
m
l
G
C
P
B
a
c
k
e
n
d
P
o
l
i
c
y
.
s
e
c
u
r
i
t
y
P
o
l
i
c
y
f
i
e
l
d
,
a
n
d
a
r
g
o
/
t
f
a
l
o
n
g
s
i
d
e
e
x
i
s
t
i
n
g
a
r
g
o
c
d
-
w
e
b
h
o
o
k
-
p
o
l
i
c
y
a
n
d
p
e
r
-
r
e
a
l
m
i
n
t
e
r
n
a
l
p
o
l
i
c
i
e
s
)
,
a
n
d
c
l
a
r
i
f
i
e
s
o
n
l
y
t
h
e
e
x
t
e
r
n
a
l
`
s
e
r
v
e
r
`
b
a
c
k
e
n
d
n
e
e
d
s
t
h
e
n
e
w
p
o
l
i
c
y
s
i
n
c
e
`
/
a
p
i
/
w
e
b
h
o
o
k
`
a
l
r
e
a
d
y
h
a
s
o
n
e
.
"
,
"
e
v
i
d
e
n
c
e
"
:
[
"
'
G
C
P
B
a
c
k
e
n
d
P
o
l
i
c
y
a
l
r
e
a
d
y
e
x
p
o
s
e
s
a
s
e
c
u
r
i
t
y
P
o
l
i
c
y
f
i
e
l
d
f
o
r
e
x
t
e
r
n
a
l
s
e
r
v
i
c
e
s
(
a
r
g
o
/
k
8
s
/
a
r
g
o
c
d
-
b
o
o
t
s
t
r
a
p
/
t
e
m
p
l
a
t
e
s
/
b
a
c
k
e
n
d
-
p
o
l
i
c
i
e
s
.
y
a
m
l
)
.
.
.
t
h
e
s
e
r
v
e
r
s
e
r
v
i
c
e
s
i
m
p
l
y
h
a
s
n
o
p
o
l
i
c
y
a
t
t
a
c
h
e
d
.
'
"
,
"
'
D
e
f
i
n
e
t
h
e
p
o
l
i
c
y
i
n
a
r
g
o
/
t
f
a
l
o
n
g
s
i
d
e
t
h
e
e
x
i
s
t
i
n
g
a
r
g
o
c
d
-
w
e
b
h
o
o
k
-
p
o
l
i
c
y
a
n
d
t
h
e
p
e
r
-
r
e
a
l
m
i
n
t
e
r
n
a
l
p
o
l
i
c
i
e
s
,
f
o
l
l
o
w
i
n
g
t
h
o
s
e
c
o
n
v
e
n
t
i
o
n
s
.
'
"
]
}
,
{
"
d
e
s
c
r
i
p
t
i
o
n
"
:
"
R
o
o
t
c
a
u
s
e
c
o
n
t
e
x
t
i
s
d
o
c
u
m
e
n
t
e
d
:
a
s
t
a
l
e
I
A
P
-
a
u
t
h
e
n
t
i
c
a
t
e
d
A
r
g
o
C
D
U
I
s
e
s
s
i
o
n
r
e
t
r
i
e
d
w
i
t
h
o
u
t
b
a
c
k
o
f
f
,
h
a
m
m
e
r
i
n
g
t
h
e
b
a
c
k
e
n
d
.
T
h
i
s
t
i
c
k
e
t
i
s
e
x
p
l
i
c
i
t
l
y
a
n
i
n
t
e
r
i
m
/
d
e
f
e
n
s
e
-
i
n
-
d
e
p
t
h
c
o
n
t
r
o
l
,
n
o
t
a
f
i
x
f
o
r
t
h
e
c
l
i
e
n
t
-
s
i
d
e
r
e
t
r
y
b
e
h
a
v
i
o
r
(
w
h
i
c
h
i
s
t
o
b
e
r
e
p
o
r
t
e
d
u
p
s
t
r
e
a
m
s
e
p
a
r
a
t
e
l
y
)
.
"
,
"
e
v
i
d
e
n
c
e
"
:
"
'
T
h
e
u
n
d
e
r
l
y
i
n
g
t
r
i
g
g
e
r
w
a
s
a
n
A
r
g
o
C
D
U
I
s
e
s
s
i
o
n
w
h
o
s
e
I
A
P
a
u
t
h
h
a
d
g
o
n
e
s
t
a
l
e
a
n
d
w
h
i
c
h
t
h
e
n
r
e
t
r
i
e
d
w
i
t
h
o
u
t
b
a
c
k
o
f
f
.
T
h
i
s
t
i
c
k
e
t
d
a
m
p
s
t
h
e
b
l
a
s
t
r
a
d
i
u
s
;
t
h
e
c
l
i
e
n
t
-
s
i
d
e
b
e
h
a
v
i
o
u
r
i
s
w
o
r
t
h
r
e
p
o
r
t
i
n
g
u
p
s
t
r
e
a
m
s
e
p
a
r
a
t
e
l
y
.
'
"
}
,
{
"
d
e
s
c
r
i
p
t
i
o
n
"
:
"
C
o
n
c
r
e
t
e
t
r
a
f
f
i
c
b
a
s
e
l
i
n
e
i
s
g
i
v
e
n
t
o
s
i
z
e
t
h
e
r
a
t
e
-
b
a
s
e
d
b
a
n
t
h
r
e
s
h
o
l
d
s
,
r
e
d
u
c
i
n
g
r
i
s
k
o
f
g
u
e
s
s
w
o
r
k
:
w
e
b
s
e
r
v
i
c
e
s
r
e
a
l
m
m
e
d
i
a
n
~
1
.
7
r
e
q
/
s
,
p
9
0
~
3
1
r
e
q
/
s
,
w
i
t
h
g
u
i
d
a
n
c
e
t
o
l
e
a
v
e
h
e
a
d
r
o
o
m
f
o
r
l
e
g
i
t
i
m
a
t
e
m
u
l
t
i
-
s
t
r
e
a
m
s
e
s
s
i
o
n
s
.
"
,
"
e
v
i
d
e
n
c
e
"
:
"
'
t
h
e
w
h
o
l
e
w
e
b
s
e
r
v
i
c
e
s
r
e
a
l
m
r
u
n
s
a
m
e
d
i
a
n
o
f
r
o
u
g
h
l
y
1
.
7
r
e
q
/
s
w
i
t
h
a
p
9
0
n
e
a
r
3
1
r
e
q
/
s
,
s
o
a
s
i
n
g
l
e
c
l
i
e
n
t
s
u
s
t
a
i
n
i
n
g
h
u
n
d
r
e
d
s
o
f
r
e
q
/
s
i
s
u
n
a
m
b
i
g
u
o
u
s
.
L
e
a
v
e
r
e
a
l
h
e
a
d
r
o
o
m
a
b
o
v
e
a
b
u
s
y
l
e
g
i
t
i
m
a
t
e
s
e
s
s
i
o
n
,
w
h
i
c
h
o
p
e
n
s
s
e
v
e
r
a
l
c
o
n
c
u
r
r
e
n
t
s
t
r
e
a
m
s
.
'
"
}
,
{
"
d
e
s
c
r
i
p
t
i
o
n
"
:
"
E
x
p
l
i
c
i
t
r
o
l
l
o
u
t
s
e
q
u
e
n
c
i
n
g
i
s
s
p
e
c
i
f
i
e
d
p
e
r
a
n
'
A
d
m
i
n
G
u
i
d
e
'
c
o
n
v
e
n
t
i
o
n
:
s
a
n
d
b
o
x
f
i
r
s
t
,
t
h
e
n
d
a
t
a
s
e
r
v
i
c
e
s
,
t
h
e
n
w
e
b
s
e
r
v
i
c
e
s
,
v
a
l
i
d
a
t
i
n
g
U
I
/
C
L
I
a
t
e
a
c
h
s
t
e
p
.
T
h
e
r
e
f
e
r
e
n
c
e
d
A
d
m
i
n
G
u
i
d
e
i
t
s
e
l
f
w
a
s
n
o
t
l
i
n
k
e
d
i
n
t
h
e
t
i
c
k
e
t
,
s
o
i
t
s
h
o
u
l
d
b
e
l
o
c
a
t
e
d
/
c
o
n
f
i
r
m
e
d
b
e
f
o
r
e
i
m
p
l
e
m
e
n
t
a
t
i
o
n
.
"
,
"
e
v
i
d
e
n
c
e
"
:
"
'
S
e
q
u
e
n
c
e
:
l
a
n
d
i
n
s
a
n
d
b
o
x
f
i
r
s
t
p
e
r
t
h
e
A
d
m
i
n
G
u
i
d
e
'
s
r
u
l
e
,
c
o
n
f
i
r
m
t
h
e
U
I
a
n
d
C
L
I
a
r
e
u
n
a
f
f
e
c
t
e
d
,
t
h
e
n
d
a
t
a
s
e
r
v
i
c
e
s
a
n
d
w
e
b
s
e
r
v
i
c
e
s
.
'
"
}
,
{
"
d
e
s
c
r
i
p
t
i
o
n
"
:
"
T
i
c
k
e
t
e
x
p
l
i
c
i
t
l
y
s
c
o
p
e
s
o
u
t
t
h
e
i
n
t
e
r
n
a
l
g
a
t
e
w
a
y
b
a
c
k
e
n
d
s
(
a
l
r
e
a
d
y
s
u
b
n
e
t
-
r
e
s
t
r
i
c
t
e
d
w
i
t
h
d
i
f
f
e
r
e
n
t
t
r
a
f
f
i
c
p
a
t
t
e
r
n
s
)
,
a
n
d
a
s
k
s
f
o
r
o
b
s
e
r
v
a
b
i
l
i
t
y
/
d
i
a
g
n
o
s
a
b
i
l
i
t
y
o
f
b
a
n
s
(
c
l
e
a
r
b
r
o
w
s
e
r
-
f
a
c
i
n
g
s
t
a
t
u
s
c
o
d
e
,
r
e
s
p
o
n
d
e
r
-
v
i
s
i
b
l
e
b
a
n
s
i
g
n
a
l
)
,
p
l
u
s
c
o
n
s
i
d
e
r
a
t
i
o
n
o
f
C
l
o
u
d
A
r
m
o
r
p
r
e
v
i
e
w
m
o
d
e
b
e
f
o
r
e
e
n
f
o
r
c
i
n
g
.
"
,
"
e
v
i
d
e
n
c
e
"
:
"
'
D
o
n
o
t
r
a
t
e
l
i
m
i
t
t
h
e
i
n
t
e
r
n
a
l
g
a
t
e
w
a
y
b
a
c
k
e
n
d
s
;
t
h
o
s
e
a
l
r
e
a
d
y
h
a
v
e
s
u
b
n
e
t
-
r
e
s
t
r
i
c
t
e
d
p
o
l
i
c
i
e
s
a
n
d
d
i
f
f
e
r
e
n
t
t
r
a
f
f
i
c
c
h
a
r
a
c
t
e
r
i
s
t
i
c
s
.
'
"
,
"
C
o
n
s
i
d
e
r
s
t
a
r
t
i
n
g
i
n
p
r
e
v
i
e
w
m
o
d
e
i
f
C
l
o
u
d
A
r
m
o
r
s
u
p
p
o
r
t
s
i
t
f
o
r
t
h
e
c
h
o
s
e
n
r
u
l
e
s
h
a
p
e
,
s
o
w
e
c
a
n
s
e
e
w
h
a
t
w
o
u
l
d
h
a
v
e
b
e
e
n
b
a
n
n
e
d
b
e
f
o
r
e
e
n
f
o
r
c
i
n
g
.
"
,
"
'
M
a
k
e
s
u
r
e
a
b
a
n
p
r
o
d
u
c
e
s
a
d
i
a
g
n
o
s
a
b
l
e
s
i
g
n
a
l
.
.
.
p
i
c
k
a
s
t
a
t
u
s
c
o
d
e
t
h
e
U
I
h
a
n
d
l
e
s
s
a
n
e
l
y
.
'
"
]
}
,
{
"
d
e
s
c
r
i
p
t
i
o
n
"
:
"
T
h
i
s
t
i
c
k
e
t
i
s
c
o
u
p
l
e
d
t
o
t
w
o
s
i
b
l
i
n
g
t
i
c
k
e
t
s
:
M
Z
C
L
D
-
3
8
4
1
(
F
a
s
t
l
y
w
o
r
k
,
g
a
t
e
d
o
n
a
s
p
i
k
e
)
a
n
d
M
Z
C
L
D
-
3
8
4
0
(
t
h
e
s
p
i
k
e
i
t
s
e
l
f
)
.
I
f
M
Z
C
L
D
-
3
8
4
0
c
o
n
c
l
u
d
e
s
F
a
s
t
l
y
c
a
n
n
o
t
f
r
o
n
t
I
A
P
,
t
h
i
s
C
l
o
u
d
A
r
m
o
r
p
o
l
i
c
y
b
e
c
o
m
e
s
t
h
e
p
r
i
m
a
r
y
(
n
o
t
i
n
t
e
r
i
m
)
c
o
n
t
r
o
l
a
n
d
t
h
r
e
s
h
o
l
d
s
s
h
o
u
l
d
b
e
r
e
v
i
s
i
t
e
d
.
"
,
"
e
v
i
d
e
n
c
e
"
:
"
'
I
f
t
h
e
s
p
i
k
e
(
M
Z
C
L
D
-
3
8
4
0
)
c
o
n
c
l
u
d
e
s
F
a
s
t
l
y
c
a
n
n
o
t
f
r
o
n
t
I
A
P
,
t
h
i
s
b
e
c
o
m
e
s
t
h
e
p
r
i
m
a
r
y
c
o
n
t
r
o
l
r
a
t
h
e
r
t
h
a
n
t
h
e
i
n
t
e
r
i
m
o
n
e
,
a
n
d
t
h
e
t
h
r
e
s
h
o
l
d
s
d
e
s
e
r
v
e
a
s
e
c
o
n
d
l
o
o
k
i
n
t
h
a
t
l
i
g
h
t
.
'
"
}
,
{
"
d
e
s
c
r
i
p
t
i
o
n
"
:
"
N
o
c
o
m
m
e
n
t
s
e
x
i
s
t
o
n
t
h
e
t
i
c
k
e
t
y
e
t
a
n
d
i
t
r
e
m
a
i
n
s
i
n
B
a
c
k
l
o
g
s
t
a
t
u
s
,
m
e
a
n
i
n
g
i
t
h
a
s
n
o
t
b
e
e
n
t
r
i
a
g
e
d
/
p
r
i
o
r
i
t
i
z
e
d
o
r
a
s
s
i
g
n
e
d
;
n
o
C
o
n
f
l
u
e
n
c
e
r
u
n
b
o
o
k
l
i
n
k
w
a
s
e
m
b
e
d
d
e
d
i
n
t
h
e
t
i
c
k
e
t
t
e
x
t
t
o
v
e
r
i
f
y
t
h
e
'
A
d
m
i
n
G
u
i
d
e
'
r
o
l
l
o
u
t
r
u
l
e
d
i
r
e
c
t
l
y
.
"
,
"
e
v
i
d
e
n
c
e
"
:
"
c
o
m
m
e
n
t
_
c
o
u
n
t
:
0
,
s
t
a
t
u
s
:
B
a
c
k
l
o
g
"
}
]

proposed actions

none

tool calls

[
{
"capability": "jira.issue.read",
"args": {
"key": "MZCLD-3842"
},
"result": {
"key": "MZCLD-3842",
"summary": "Add Cloud Armor rate-based bans to the ArgoCD external backends as an interim mitigation",
"status": "Backlog",
"description_text": "Why this is worth doing on its own The Fastly work ( MZCLD-3841 ) is gated on a spike and will take a while. This closes the specific gap from the stale-session incident in days, and it does not become wasted work if Fastly lands later: a rate limit at the load balancer is still useful defense in depth behind an edge. Two things make it cheap right now: GCPBackendPolicy already exposes a securityPolicy field for external services ( argo/k8s/argocd-bootstrap/templates/backend-policies.yaml ), and the values schema already carries it per service. The /api/webhook service uses it today for argocd-webhook-policy , so the wiring is proven; the server service simply has no policy attached. Nothing fronts the load balancers, so Cloud Armor still sees real client IPs . Per-IP rate limiting works correctly today in a way it will not once Fastly is in the path. This is the window in which the simple fix is also the effective one. Work Add a Cloud Armor security policy with a rate-based ban rule and attach it to the external server backend on all three realms ( sandbox , webservices , dataservices ). Define the policy in argo/tf alongside the existing argocd-webhook-policy and the per-realm internal policies, following those conventions. Use a rate-based ban keyed on client IP: a request threshold over an interval, then a ban for a fixed duration. Pick numbers from observed traffic rather than intuition. For scale, the whole webservices realm runs a median of roughly 1.7 req/s with a p90 near 31 req/s, so a single client sustaining hundreds of req/s is unambiguous. Leave real headroom above a busy legitimate session, which opens several concurrent streams. Do not rate limit the internal gateway backends; those already have subnet-restricted policies and different traffic characteristics. Consider starting in preview mode if Cloud Armor supports it for the chosen rule shape, so we can see what would have been banned before enforcing. Make sure a ban produces a diagnosable signal. A user who trips this should be able to find out why without guessing, and a responder should be able to see bans happening. Note the throttled response is what the browser sees, so pick a status code the UI handles sanely. Notes Sequence: land in sandbox first per the Admin Guide's rule, confirm the UI and CLI are unaffected, then dataservices and webservices . The underlying trigger was an ArgoCD UI session whose IAP auth had gone stale and which then retried without backoff. This ticket damps the blast radius; the client-side behaviour is worth reporting upstream separately. If the spike ( MZCLD-3840 ) concludes Fastly cannot front IAP, this becomes the primary control rather than the interim one, and the thresholds deserve a second look in that light.",
"comment_count": 0,
"comments": []
}
}
]

usage

{
  "model": "claude-sonnet-5",
  "input_tokens": 4005,
  "output_tokens": 2121,
  "estimated_cost_usd": 0.04383
}

approvals

[]

log

launched k8s job run-run-4932ca8145e648e3 in ns/mzcld-demo-dev image=us-docker.pkg.dev/moz-fx-mzcld-demo-prod/mzcld-demo-prod/agentplat:latest

runner started

live reasoning model=claude-sonnet-5