What did you do?
We recently upgraded alertmanager from version 0.28.0 to 0.31.0.
Since upgrading, our inhibition rules for alerts have failed, and we're seeing firing alerts matching the target_matcher, despite source_matchers also firing.
What did you expect to see?
No response
What did you see instead? Under which circumstances?
Alerts were not inhibited. This was confirmed by curling alertmanager directly when the alert fired. I could see that both the target and source alerts were firing, but the target had no inhibitors.
This was resolved by separating our source_matchers:
previously
source_matchers:
- alertname =~ "(alert_one|alert_two|alert_three)"
into three separate rules, each with one alert as the source matcher. It is often the case that multiple of these alerts are firing at once.
We also removed some regex from our target_matchers:
- target_matchers:
- alertname =~ "(alert_four|alert_five|alert_six_.*)"
We removed the final alert_six._* from the target_matchers.
System information
Linux 5.15.0-1084-aws x86_64
Alertmanager version
alertmanager-0.31.0.linux-amd64
Alertmanager configuration file
Prometheus version
Prometheus configuration file
Logs
What did you do?
We recently upgraded alertmanager from version 0.28.0 to 0.31.0.
Since upgrading, our inhibition rules for alerts have failed, and we're seeing firing alerts matching the target_matcher, despite source_matchers also firing.
What did you expect to see?
No response
What did you see instead? Under which circumstances?
Alerts were not inhibited. This was confirmed by curling alertmanager directly when the alert fired. I could see that both the target and source alerts were firing, but the target had no inhibitors.
This was resolved by separating our source_matchers:
previously
into three separate rules, each with one alert as the source matcher. It is often the case that multiple of these alerts are firing at once.
We also removed some regex from our target_matchers:
We removed the final
alert_six._*from the target_matchers.System information
Linux 5.15.0-1084-aws x86_64
Alertmanager version
Alertmanager configuration file
Prometheus version
Prometheus configuration file
Logs