Rendering notification templates should not silently swallow errors

[multani]

Proposal

Notifications templates are configured to silently swallow the errors and passing the final output, including the rendering error, to the underlying notification handler.

Depending on the notification handler, this may or may not work and errors may be reported in a very difficult way to investigate.

For instance, the Slack notifier accepts several configuration fields as Go templates. If one of the template fails to render, the error is ignored and the result sent to the Slack API, which may fail to process the message and fails with:

time=2026-06-23T15:14:49.930Z level=ERROR source=dispatch.go:611 msg="Notify for alerts failed" component=dispatcher aggrGroup="..." num_alerts=1 err="alerts/slack[0]: notify retry canceled due to unrecoverable error after 1 attempts: channel "": error response from Slack: invalid_arguments"

You then have to go through all the templates and find out which one was failing.

It would be much more resilient instead to:

1. Log errors when templates are failing to render
2. Not even try to call the notification handler if a related template fails to render

In my case, amtool template render correctly reports the problem so it could have been identified earlier, but as formatting can also be highly dependent on the alerts themselves, it may be difficult to achieve 100% test coverage ahead of time and not being able the actual error makes it really difficult to find out what the problem is.

Related:

• Rendering notification templates should not silently swallow errors #5322

[multani]

It's a bit similar to #5322, but I think it covers something slightly different:

1. Templating errors should be reported, and not silently ignored
2. Templating errors should not be passed down to the notification handlers
3. I would argue that notification handlers should not be called when templating didn't work, but Rendering notification templates should not silently swallow errors #5322 argues differently :)

[grobinson-grafana]

FWIW, the notification handlers are where the templating happens. There no errors getting "passed down" because templating doesn't happen until the notification handler runs.

[multani]

FWIW, the notification handlers are where the templating happens. There no errors getting "passed down" because templating doesn't happen until the notification handler runs.

Thanks, makes sense. I'll reformulate my message a bit, what I meant was that the template fails to render, the error message is directly into the final templating output, which is then used directly by the notification handler.

In my example (Slack handler), Slack refuses the request when it contained the templating error message (for some reasons), so:

• the templating error never show up in the Slack message (as the call was rejected)
• the templating error was also not showing up in Alertmanager's logs