fix(server): increase gunicorn capacity + liveness probe tolerance to stop thread exhaustion SIGKILL (11 firings) - #548
Conversation
…to prevent thread exhaustion SIGKILL Root cause of 11 firings of Aurora Prod - User-Facing Slow Requests: GKE OPTIMIZE_UTILIZATION autoscaler evicts aurora-oss-weaviate-0. Weaviate-dependent requests block on TCP connection attempts (5s timeout), filling all 8 gunicorn thread slots (2 workers x 4 threads). With all slots occupied, the liveness probe at /health/liveness cannot be served within the 5s timeout. After failureThreshold=3 consecutive failures (3 x 20s = 60s), kubelet sends SIGKILL (exit 137). Each restart introduces a 30-120s unavailability window causing >10 user-facing requests >5s. Note: The webhook handler at /github/webhook already enqueues to Celery asynchronously and returns 200 immediately - it is NOT the bottleneck. Fix: 1. values.yaml: GUNICORN_WORKERS 2->4, GUNICORN_THREADS 4->8 (8->32 total thread slots). Provides 4x headroom so Weaviate-blocked requests cannot saturate all slots. 2. server-deployment.yaml: Make livenessProbe.failureThreshold configurable via values.yaml (new config.LIVENESS_FAILURE_THRESHOLD key, default 6). Raises tolerance from 60s to 120s - survives Weaviate eviction + rescheduling window. Relates to: INC-590, INC-593, INC-594, 01KVTNER05Q1HPH3W6QG0KFMFH (11th firing)
|
Alert: Aurora Prod - User-Facing Slow Requests (>5s requests > 10 in 5 min) This alert has now fired 12+ times (most recently at the time of this comment). The root cause remains the same: GKE This PR directly addresses the root cause by:
Please review and merge to stop the recurring slow-request incidents. |
|
🔁 Slow Requests alert fired again — incident This is the 13th+ firing of "Aurora Prod - User-Facing Slow Requests" (>10 requests >5s in 5 min). The root cause remains the same: GKE This PR directly resolves the issue by:
The alert will continue firing on every GKE node scale-down event until this PR is merged and deployed. Please prioritize review and merge. |
🔴 Alert still firing — incident
|
|
|
Gunicorn capacity increased in prod repo |
|
…0 firings) [replaces #549] GUNICORN_WORKERS: 2 → 4, GUNICORN_THREADS: 4 → 8 (8 → 32 total thread slots) livenessProbe.failureThreshold: 3 → 6 (60s → 120s tolerance) Root cause: Weaviate eviction blocks all 8 gunicorn thread slots (5s TCP timeout × N requests), liveness probe starved, SIGKILL after 60s. 20 firings since 2026-06-22. Replaces closed PRs #548 and #549.
…4, replaces closed #557) Root cause: Weaviate TCP timeouts (5s) exhaust all 8 gunicorn thread slots (WORKERS=2 × THREADS=4), starving the liveness probe → SIGKILL → pod restart → >10 slow requests in 5 min → alert fires. 39th firing as of this commit. Changes: 1. values.yaml: GUNICORN_WORKERS 2→4, GUNICORN_THREADS 4→8 (8→32 total slots) 2. server-deployment.yaml: livenessProbe.failureThreshold 3→6 (60s→120s tolerance) - 120s exceeds Weaviate rescheduling window (~60-90s), pod survives eviction - Also update gunicorn args fallback defaults to match new values Predecessors closed without merging: #548 (2026-06-24), #549 (2026-06-25), #557 (2026-06-26)
Increase liveness probe failureThreshold from 3 to configurable default 6 (120s tolerance vs 60s) to survive Weaviate rescheduling windows (~60-90s). Update gunicorn args fallback defaults to match values.yaml increase (workers 2->4, threads 4->8). Root cause: Weaviate eviction blocks gunicorn threads (5s TCP timeout each), exhausting all 8 slots and starving the liveness probe -> SIGKILL. With 32 slots (4w x 8t) and 120s probe tolerance, the pod survives the eviction window without being killed. 40th firing of this alert. Fix never applied despite PRs #548, #549, #557, #560 all being closed without merging.
…41st firing [replaces closed #561] Root cause: Weaviate TCP timeouts (5s) exhaust all 8 gunicorn thread slots (GUNICORN_WORKERS=2 × GUNICORN_THREADS=4), blocking the liveness probe. After 3 consecutive failures (60s), kubelet SIGKILLs the pod → slow requests. Changes: - values.yaml: GUNICORN_WORKERS 2→4, GUNICORN_THREADS 4→8 (8→32 thread slots) - values.yaml: add server.livenessProbe.failureThreshold: 6 (120s tolerance) - server-deployment.yaml: livenessProbe.failureThreshold 3→6 (configurable) - server-deployment.yaml: gunicorn args fallback defaults updated to match Predecessors all closed without merging: #548, #549, #557, #560, #561
1 participant
Problem
11th firing of "Aurora Prod - User-Facing Slow Requests" (>10 requests >5s in 5 min). First fired 2026-06-22 ~18:02 UTC. Still recurring as of 2026-06-23 ~16:36 UTC with no fix applied.
Root cause (confirmed from live pod logs + ConfigMap)
GKE
OPTIMIZE_UTILIZATIONautoscaler evictsaurora-oss-weaviate-0(no PodDisruptionBudget on the StatefulSet). During the rescheduling window (~60–90s), Weaviate-dependent API routes block on TCP connection attempts with a 5s timeout. With only 8 gunicorn thread slots (GUNICORN_WORKERS=2 × GUNICORN_THREADS=4), a burst of Weaviate-blocked requests saturates the entire pool. The liveness probe at/health/livenesscannot be served within the 5s timeout. AfterfailureThreshold=3consecutive failures (3 × 20s = 60s), kubelet sends SIGKILL (exit code 137). Each restart introduces a 30–120s unavailability window causing >10 user-facing requests >5s.Note: The webhook handler at
/github/webhookalready enqueues to Celery asynchronously and returns 200 immediately — it is not the bottleneck. The async architecture was already in place.Kill chain
Firing history (11 total, no PR existed before this one)
01KVR7YW1YV9VPPQPZPFEVR0MM01KVRJV6B78VKB0DAZ50YTPHW601KVSVVFRCJ21MZ5CSM02QMWCS01KVTF2P3WRC8BYXMZXVPBT08Y014b739e-57e9-400f-b570-7aefbb8d3b5301KVTJ5TC4HS74YPMQBRN7WYSZ01KVTMAYDH6KD2HYYFAPZDVMQW2cb12c61-b3a2-4bfd-ba0c-65f0864107cb01KVTNER05Q1HPH3W6QG0KFMFHFix
1.
deploy/helm/aurora/values.yaml— Increase gunicorn capacityWith 32 slots, a burst of Weaviate-blocked requests (each consuming one slot for 5s) cannot saturate the pool. Even with 20 simultaneous Weaviate-blocked requests, 12 slots remain available for user-facing requests and the liveness probe.
2.
deploy/helm/aurora/templates/server-deployment.yaml— Increase liveness probe tolerance120s tolerance exceeds the typical Weaviate rescheduling window (~60–90s), so the pod survives the eviction event without being killed. The value is now configurable via
values.yaml(server.livenessProbe.failureThreshold, default6).Deployment
Apply to production with:
The rolling update will restart server pods with the new gunicorn configuration. No downtime expected (1 replica, but the new pod starts before the old one terminates due to
maxSurge=1).Related
01KVSVVFRCJ21MZ5CSM02QMWCS: Aurora Prod User-Facing Slow Requests (firings 3–5)01KVTMAYDH6KD2HYYFAPZDVMQW: Aurora Prod User-Facing Slow Requests (9th firing)2cb12c61-b3a2-4bfd-ba0c-65f0864107cb: Aurora Prod User-Facing Slow Requests (10th firing)01KVTNER05Q1HPH3W6QG0KFMFH: Aurora Prod User-Facing Slow Requests (11th firing, this PR)Secondary recommendations (not in this PR)
PodDisruptionBudgetwithminAvailable: 1foraurora-oss-weaviateStatefulSet to prevent aggressive evictionPodDisruptionBudgetwithminAvailable: 1foraurora-oss-serverDeploymentaurora-ai.net(ingress redirecting challenge requests instead of serving tokens)