Skip to content

fix(monitoring): improve Grafana alert quality, message format, and env scoping - #221

Open
NikhilMM89 wants to merge 5 commits into
mainfrom
fix/grafana-alert-quality
Open

fix(monitoring): improve Grafana alert quality, message format, and env scoping#221
NikhilMM89 wants to merge 5 commits into
mainfrom
fix/grafana-alert-quality

Conversation

@NikhilMM89

@NikhilMM89 NikhilMM89 commented Sep 8, 2026 •
edited by github-actions Bot
Loading

Copy link
Copy Markdown
Collaborator

Description

Reworks Grafana/Loki alerting to cut false-positive noise, make notifications scannable, and scope alerts to the environments that matter. Addresses the alerting ticket (Grafana alerts were hard to triage — e.g. critical fired when a healthy-but-idle service produced no logs for 10 min).

What changed

Notification message format (n8n webhook template — values-tools-sso.yaml)

  • One line per alert: Status | Severity | Alert name — short description
    • 🔴 FIRING | CRITICAL | Notify: <alert> — <short>
    • 🟠 FIRING | WARNING | Notify: <alert> — <short>
    • 🟢 RESOLVED | Notify: <alert> — <short>
  • Icon derived from severity (no per-alert icons); removed the old icon-heavy multi-line block.
  • Every rule gained a short annotation for the one-line headline.

Signal quality

  • Removed NoLogsReceived (critical on 10-min log absence) — fired on healthy but idle services. Liveness now comes from pod state.
  • Removed log-grep HighAPI500ErrorRate (matched any "500" in a log line) in favor of an accurate metric-based 5xx rule.
  • Promoted PodNotReady to critical.

New metric-based rules (infra-alert-rules.yaml)

  • ApiHigh5xxRate (crit), ApiHighLatency (warn, traffic-gated so idle never alerts), PVCUsageHigh/Critical (85/95%), PostgresUnavailable, RedisUnavailable.
  • DB/Redis pod-name regexes verified against the live dev/test/prod cluster — they match only the primary Crunchy DB pod and the Redis master, and exclude the crunchy backup jobs (repo1-full/repo1-incr/repo-host) which finish Completed/Error and would otherwise fire a permanent false-positive. Also confirmed kube_pod_status_ready is present in the tools Prometheus.

New log-based stopgap rules (common-notify-delivery)

CHES / ACS / SMS / webhook failures, queue job failures, queue-not-draining. These providers/queues expose no Prometheus metrics today, so thresholds are log-line rates — flagged inline for follow-up backend instrumentation.

Environment scoping

  • All alerts scoped to test + prod only — dev is deployed to constantly, so its restarts/churn are expected, not actionable.
  • Only exception: PodCrashLoopBackOff stays dev+test+prod (a crash-loop is a real breakage even mid-deploy).

Testing

  • All three YAML files validated (yaml.safe_load_all).
  • DB/Redis regexes and the kube_pod_status_ready metric verified against live Prometheus in f6bc3f-tools across dev/test/prod.

Notes / follow-ups

  • Delivery/queue alerts are log-pattern stopgaps until the backend exposes real metrics (queue depth, job outcomes, provider delivery) — recommend a separate dev ticket.
  • PVC/memory-% rules depend on kubelet/cAdvisor metrics that a namespace-scoped Prometheus may not scrape; documented inline (they no-op rather than error if absent).

🤖 Generated with Claude Code

Thanks for the PR!

Deployments, as required, will be available below:

Please create PRs in draft mode. Mark as ready to enable:

After merge, new images are deployed in:

@NikhilMM89
fix(monitoring): improve Grafana alert signal-to-noise and message fo…
7db44b3 
…rmat

Reworks alerting so criticals reflect real service impact and the
notification message is consistent and scannable.

Message format (n8n webhook template):
- One line per alert: Status | Severity | Alert name — short description
- Icon derived from severity (critical/warning/resolved); removes the
  old icon-heavy multi-line block
- Every rule gains a `short` annotation for the headline

Signal quality:
- Remove NoLogsReceived (critical on 10m log-absence) — fired on healthy
  but idle services. Liveness now comes from pod state.
- Remove log-grep HighAPI500ErrorRate (matched any "500" in logs) in
  favor of a metric-based 5xx-rate rule.
- Promote PodNotReady to critical.

New metric-based rules (infra-alert-rules.yaml):
- ApiHigh5xxRate (crit), ApiHighLatency (warn, traffic-gated so idle
  never alerts), PVCUsageHigh/Critical (85/95%), PostgresUnavailable,
  RedisUnavailable. DB/Redis pod-name regexes verified against the live
  dev/test/prod cluster to match only the primary DB / Redis master and
  exclude crunchy backup jobs (which finish Completed/Error).

New log-based stopgap rules (delivery/queue) until backend exposes
metrics: CHES/ACS/SMS/webhook failures, queue job failures, queue-not-
draining.
@NikhilMM89
fix(monitoring): scope alerts to test+prod, keep only CrashLoopBackOf…
879c0bb 
…f in dev

Dev is deployed to constantly, so most alerts there are deploy churn
(transient restarts, brief unavailability, redeploy log errors) rather
than actionable incidents. Narrow every rule to f6bc3f-(test|prod).

The sole exception is PodCrashLoopBackOff, which stays dev|test|prod — a
crash-loop is a real breakage worth surfacing even mid-deploy.

Updated the SCOPE header comments in both rule files to document the
new scoping and the single dev exception.
@NikhilMM89
ci(monitoring): deploy alert rules + dashboards via pipeline
2af2b49 
The Deploy Monitoring Stack pipeline installed Loki/Grafana but never
provisioned the alert rules or dashboards — those were applied by hand
(monitoring/deploy-dashboards-alerts.sh) and drifted from git. Notably:
- loki/values-tools.yaml mounts a `loki-alert-rules` ConfigMap that
  nothing created, so the log-based rules never loaded;
- grafana/infra-alert-rules.yaml (metric rules) had no deploy path at all.

Wire both into deploy-monitoring.yml so git is the single source of truth:
- Loki job: create the `loki-alert-rules` ConfigMap from
  loki/alert-rules.yaml before the helm install (populates the ruler
  mount), and restart Loki on redeploy so the ruler reloads.
- Grafana job: create `cn-infra-alert-rules` (grafana_alert=1) from
  infra-alert-rules.yaml and one `cn-dashboard-*` (grafana_dashboard=1)
  per dashboards/*.json — the sidecar hot-loads both.
@NikhilMM89
fix(monitoring): plain-text alert message, drop emoji/icons
2dc54af 
The notification wording (e.g. "🚨 Grafana Alert: No Logs Received")
read as noisy and inconsistent. Switch the Grafana contactPoint message
to plain text, one line per alert, no emoji:

  FIRING <SEVERITY> | <Alert name> | <detail>
  RESOLVED | <Alert name> | <detail>

detail = the `short` annotation, falling back to `summary`.

Note: if the downstream n8n webhook re-wraps this text (adds its own
"🚨 Grafana Alert:" prefix), that wrapper must be removed in the n8n
workflow too — the repo can only control what Grafana sends.
@NikhilMM89
NikhilMM89 force-pushed the fix/grafana-alert-quality branch from fb841e3 to 93822ab Compare September 8, 2026 18:15
@NikhilMM89
NikhilMM89 force-pushed the fix/grafana-alert-quality branch from 2829c94 to 2decfc6 Compare September 10, 2026 05:05
@NikhilMM89
fix(monitoring): app scrape, NoData storm, ticket-format message
e7dc0ce 
Fixes found after deploying rules to f6bc3f-tools and iterating on the
alert format with the team.

Prometheus scrape:
- Add fallback_scrape_protocol: PrometheusText0.0.4 to the
  common-notify-app job. Prometheus 3.x rejected the backend
  /api/metrics target (blank Content-Type), so the whole app scrape was
  down and the API 5xx/latency rules read NoData. Now all 3 targets up.

Alert NoData storm:
- Set noDataState: OK and execErrState: OK on all 14 metric rules so a
  transient no-data on Grafana restart no longer pages
  (DatasourceNoData). Root cause of the storm was also two stale
  duplicate rule ConfigMaps in-cluster (grafana-alert-rules,
  grafana-infra-alert-rules) that pinned 8 rules to noDataState=NoData;
  those were removed live and are not in git.

Message format (per updated ticket):
- Rework the contactPoint message to the ticket format with icons and
  environment:
    🔴 FIRING | CRITICAL | <env> | <Alert> — <detail>
    🟠 FIRING | WARNING  | <env> | <Alert> — <detail>
    🟢 RESOLVED | <env> | <Alert> — <detail>
  env is derived from .Labels.namespace (f6bc3f-dev→dev).

n8n forwarding:
- The 🚨/State/Severity:INFO wrapper is added by the n8n webhook node,
  which rebuilds its own body instead of forwarding Grafana's message.
  That node lives in n8n and has to be corrected there — it cannot be
  applied from this repo, so nothing in this change addresses it.
@NikhilMM89
NikhilMM89 force-pushed the fix/grafana-alert-quality branch from 2decfc6 to e7dc0ce Compare September 10, 2026 05:14

This branch was successfully deployed

5 active (outdated) deployments
TOOLS — 2829c944 Deployed Sep 8, 2026 by NikhilMM89 via Deploy Grafana (Centralized - Tools) #55
dev — 2829c944 Deployed Sep 8, 2026 by NikhilMM89 via Deploy Grafana (Individual) #55
DEV — 93822abc Deployed Sep 8, 2026 by NikhilMM89 via Deploy kube-state-metrics (Dev) #54
TEST — 93822abc Deployed Sep 8, 2026 by NikhilMM89 via Deploy kube-state-metrics (Test) #54
PROD — 93822abc Deployed Sep 8, 2026 by NikhilMM89 via Deploy kube-state-metrics (Prod) #54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@NikhilMM89