Skip to content

dispatcher: add suppressed alerts metric by reason#5248

Open
ashleyprimo wants to merge 2 commits into
prometheus:mainfrom
ashleyprimo:alerts_suppressed
Open

dispatcher: add suppressed alerts metric by reason#5248
ashleyprimo wants to merge 2 commits into
prometheus:mainfrom
ashleyprimo:alerts_suppressed

Conversation

@ashleyprimo
Copy link
Copy Markdown

@ashleyprimo ashleyprimo commented May 20, 2026

Add alertmanager_alerts_suppressed, a gauge for active non-resolved alerts suppressed by reason (silence or inhibition); intention being able to ascertain how many alerts are currently not firing due to a silence (useful for anomaly detection, for example a silence that covers large blast radius)

Pull Request Checklist

Please check all the applicable boxes.

  • Is this a new feature?
    • I have added tests that test the new feature's functionality
  • Is this a breaking change?
    • My changes do not break the existing cluster messages
    • My changes do not break the existing api
  • I have added/updated the required documentation
  • I have signed-off my commits
  • I will follow best practices for contributing to this project

Which user-facing changes does this PR introduce?

  [FEATURE] dispatcher: Add `alertmanager_alerts_suppressed` metric for active alerts suppressed by silence or inhibition.

Summary by CodeRabbit

  • New Features

    • Added alertmanager_alerts_suppressed Prometheus metric to count alerts suppressed by silence or inhibition
    • Metrics can include an optional group_key label when group-key metrics are enabled
    • Optional alertname label support for relevant metrics when enabled
  • Tests

    • Added tests covering suppressed-alerts metric emission with and without group_key
  • Documentation

    • Updated metrics docs to describe the new metric and label behavior

Review Change Stack

Signed-off-by: Ashley Primo <ashley.primo@transferwise.com>
@ashleyprimo ashleyprimo requested a review from a team as a code owner May 20, 2026 01:00
@coderabbitai
Copy link
Copy Markdown

coderabbitai Bot commented May 20, 2026

📝 Walkthrough

Walkthrough

Adds a new Prometheus metric alertmanager_alerts_suppressed counting alerts suppressed by reason (silence or inhibition), implements collector descriptor/storage and dual collection paths for group-key-enabled and disabled modes, updates emission helpers and counting model, and adds tests and documentation.

Changes

Suppressed Alert Metrics

Layer / File(s) Summary
Metric descriptor and collector setup
dispatch/metric.go
New alertmanager_alerts_suppressed descriptor is created with state and reason labels, optionally including group_key. Collector stores and describes both the existing state metric and the new suppressed metric.
Counting model and helper methods
dispatch/metric.go
New types alertCounts and alertStatusSnapshot model state and suppression-reason tracking. Methods countAlerts, addState, addSuppressedReasons accumulate counts; emitCounts emits per-state and suppressed-by-reason metrics. emit now accepts a descriptor parameter.
Collection with group-key metrics enabled
dispatch/metric.go
When group-key metrics are enabled, Collect calls emitCounts per aggregation group with the group-key label, emitting both state and suppressed-by-reason counts grouped by aggregation key.
Collection with group-key metrics disabled
dispatch/metric.go
When group-key metrics are disabled, deduplication is rewritten to track per-fingerprint highest-priority state plus silenced/inhibited flags via snapshots, then totals are computed and emitted without group labels.
Suppressed metrics test suite
dispatch/metric_test.go
Two tests validate the suppressed-by-reason metric: one without group-key labels and one with group-key labels enabled. Helper constructors build test dispatchers, routes, aggregation groups, and alerts with silenced/inhibited states.
Metric documentation updates
docs/alertmanager.md
Documentation for alertmanager_alerts_suppressed describes suppression reasons and group_key label support. Notes added about alertname label when alert-names-in-metrics is enabled.

🎯 3 (Moderate) | ⏱️ ~25 minutes

Suggested reviewers:

  • SoloJacobs
  • siavashs
🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 14.29% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description check ✅ Passed The PR description is complete: it explains the feature, includes checked items for tests and documentation, documents no breaking changes, and provides release notes with the [FEATURE] prefix.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title accurately summarizes the main change: adding a new suppressed alerts metric that breaks down counts by reason (silence vs inhibition).

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Signed-off-by: Ashley Primo <ashley.primo@transferwise.com>
@ashleyprimo ashleyprimo changed the title dispatcher(metrics): add suppressed alerts metric by reason dispatcher: add suppressed alerts metric by reason May 20, 2026
@ashleyprimo
Copy link
Copy Markdown
Author

@coderabbitai help

@coderabbitai
Copy link
Copy Markdown

coderabbitai Bot commented May 20, 2026

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>, please review it.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

CodeRabbit commands

These commands are invoked using PR/Issue comments.

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai full review to do a full review from scratch and review all the files again.
  • @coderabbitai rate limit to show your current review rate limit status.
  • @coderabbitai summary to regenerate the summary of the PR.
  • @coderabbitai evaluate custom pre-merge check --instructions <custom-checks-instructions> --name <custom-checks-title> [--mode <error|warning>] to validate the custom pre-merge checks instructions. Defaults to error when --mode is omitted.
  • @coderabbitai ignore pre-merge checks to override pre-merge checks and get an approval on PR.
  • @coderabbitai run pre-merge checks to run pre-merge checks on the pull request.
  • @coderabbitai generate docstrings to generate docstrings for this PR.
  • @coderabbitai generate sequence diagram to generate a sequence diagram of the changes in this PR.
  • @coderabbitai generate unit tests to generate unit tests for this PR.
  • @coderabbitai resolve merge conflict to automatically resolve merge conflicts.
  • @coderabbitai autofix to automatically fix issues identified in unresolved review comments.
  • @coderabbitai resolve to resolve all the CodeRabbit review comments.
  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
  • @coderabbitai generate configuration to create a PR that adds the current resolved configuration as .coderabbit.yaml (or show it if already present).
  • @coderabbitai help to get help.

Other keywords and placeholders

  • Add @coderabbitai ignore or @coderabbit ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai anywhere in the PR title to generate the title automatically.

Status, support, documentation and community

  • Visit our status page to check the current availability of CodeRabbit.
  • Create a ticket on our support page for assistance with any issues or questions.
  • Visit our documentation site for detailed information on how to use CodeRabbit.
  • Join our Discord community to connect with other users and get help from the community.
  • Follow us on X/Twitter for updates and announcements.

@ashleyprimo
Copy link
Copy Markdown
Author

@coderabbitai review

@coderabbitai
Copy link
Copy Markdown

coderabbitai Bot commented May 20, 2026

✅ Actions performed

Review triggered.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant