Unbounded concurrent Monitoring API requests exhaust memory and OOM-kill the container #537

[ntmspavan]

Summary

Fixes #537.

Problem

reportMonitoringMetrics starts one goroutine per metric descriptor per project, with no cap. When google.projects.filter (or a long google.project-ids list) resolves to many projects, or a project has many metric descriptors, the number of in-flight TimeSeries.List requests is unbounded. Each one holds a fully decoded JSON response in memory. In a memory-constrained pod the heap can pass the container's memory limit and the pod is OOM-killed.

Fix

1. Cap concurrent TimeSeries.List requests process-wide at a fixed internal limit (maxConcurrentTimeSeriesRequests = 20 in collectors/monitoring_collector.go), shared by every project's collector. This bounds how many decoded responses a scrape holds in memory at once, however many projects or descriptors it covers. It is an internal constant, not a new flag. 20 is a conservative starting value, and I'm happy to tune it based on feedback or profiling.

2. At startup, read the container's cgroup memory limit and set GOMEMLIMIT to 90% of it via automemlimit (https://github.com/KimMachineGun/automemlimit), so the garbage collector reclaims more aggressively as usage nears the real limit. It applies only when a container memory limit exists (for example resources.limits.memory). Otherwise it changes nothing. The GOMEMLIMIT and AUTOMEMLIMIT environment variables still override it.

Trade-offs

• The cap bounds in-flight API responses, not the total size of the scrape output, which the Prometheus registry still builds in memory.
• Scrapes that cover many descriptors take longer, because requests are now processed at most 20 at a time. Narrowing monitoring.metrics-prefixes or raising scrape_timeout helps in large setups.

Testing

• go build ./..., go vet ./..., gofmt -l . and golangci-lint run ./... are clean.
• go test -race ./... passes.
• Added TestTimeSeriesRequestLimiterBoundsConcurrency, which fails if concurrency exceeds the limit or never reaches it.