fix(helm): raise resource defaults for CPU-throttled and OOM-prone services - #648
Conversation
…rvices Follow-up to #639/#641, which only covered the services our production Terraform explicitly overrides. Memgraph, and the CPU side of Celery beat/worker and the frontend, were left at the original defaults. Sized from cgroup counters (memory.peak, memory.events, cpu.stat) read from live production pods: memgraph peak 951Mi / 1Gi limit (93%), 508,254 throttled periods celery-worker 9 oom_kill events, 20k-24k throttled periods celery-beat 57,188 throttled periods at a 200m limit frontend peak 787Mi / 1Gi (77%), 2 prior OOMKills, 40,310 throttled Memgraph was the worst container in the cluster on both axes. Its --memory-limit also stayed at 512MB while the container peaked at 951Mi, so the graph store cap is raised to 1536MB, below the new 2Gi container limit, so an oversized query is rejected instead of the pod being killed. Raises requests by 550m CPU / 768Mi memory total. No value is lowered. Docker Compose keeps --memory-limit=512 since it targets single hosts.
![arvo-ai-staging[bot]](https://avatars.githubusercontent.com/in/4088260?s=60&v=4){kind=small}
|
Understand this PR’s impact Explore downstream dependencies and potential security impact with Blast Radius. No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Repository: Arvo-AI/aurora/.coderabbit.yaml Review profile: ASSERTIVE Plan: Advanced Run ID: 📒 Files selected for processing (2)
Included review availability: Your plan provides up to 2 included reviews per hour; 1 remains after this review. WalkthroughThe Helm chart increases default CPU and memory resources for several services. It also raises the default Memgraph memory limit from 512 MB to 1536 MB. ChangesResource defaults
Priority: ⬇️ Low Estimated code review effort: 1 (Trivial) | ~5 minutes Change: Bug fix Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
2 participants
Why
#639 and #641 reconciled Helm defaults with production, but only covered the five services our production Terraform explicitly overrides (
server,frontend,celeryWorker,celeryBeat,chatbot). Services prod doesn't override silently run chart defaults — and those defaults were never validated against real usage.I read cgroup counters (
memory.peak,memory.events,cpu.stat) from every live production pod. Four containers are measurably starved:OOMKilledmemgraphwas never touched by either prior PR and is the worst container in the cluster on both axes.celeryBeatgot a memory bump in #639 but its CPU was left at50m/200m.celeryWorkermemory is already correct from #641; only its CPU was still low.Verified fine, left alone:
server(24% of limit),chatbot(54%),mcp(6%),searxng(35%).redis/vault/postgres/minioare disabled in prod, so there's no usage evidence to size them from.Changes
Also
MEMGRAPH_MEMORY_LIMIT512->1536(MB). This is Memgraph's internal--memory-limit, which was capping the graph store at 512MB while the container itself peaked at 951Mi. Keeping it below the new 2Gi container limit means an oversized query gets rejected rather than the pod being OOM-killed.Notes
cpu/memory/ephemeral-storagefield againstorigin/mainin normalized units.--memory-limit=512— it targets single hosts/laptops, where the prod sizing isn't appropriate. Say the word if you'd rather they match.celeryWorkermemory down to2Giand it's actively OOM-killing (this chart's4Giis correct), andserveris atrestartCount 33with reasonErrorplus liveness failures — not OOM, so a different root cause.Testing
helm lintpasses.helm templaterenders the expected requests/limits on all containers and--memory-limit=1536in the Memgraph args.Summary by CodeRabbit