[BUG] Doc-level monitor action templates cannot reliably access the original matched document for notifications on routed indices

[carmeroa]

What is the bug?
Doc-level monitor action templates cannot reliably access the original matched document when generating notifications for routed indices.

In particular, when a doc-level monitor is used to watch an alerts index and send one notification per matched document, the notification template should be able to access the original matched document in a stable and intuitive way.

This is especially relevant for Security Analytics hidden alert indices, where documents are stored with required routing. In this scenario, query-level monitors can access the original document through ctx.results[*].hits.hits[*]._source, but doc-level monitors do not reliably expose an equivalent original-document context for notifications.

As a result, it is difficult or impossible to generate per-document notifications containing the actual matched alert details such as:

• trigger_name
• severity
• state
• monitor_name
• start_time
• aggregation bucket information

How can one reproduce the bug?
Steps to reproduce the behavior:

1. Create a doc-level monitor over an index that contains alert-like documents
2. Configure an action template that tries to render fields from the original matched document
3. Use a source index where documents are written with required routing (for example a Security Analytics hidden alerts index)
4. Trigger the monitor with matching documents
5. Observe that the notification template cannot reliably access the original matched document fields

Example monitored source document:

{
  "version": -1,
  "monitor_id": "qINV3p0BxAvHzvX9I94C",
  "workflow_id": "yINV3p0BxAvHzvX9I96C",
  "workflow_name": "",
  "associated_alert_ids": [],
  "schema_version": 5,
  "monitor_version": 1,
  "monitor_name": "Doc-Level Monitor",
  "execution_id": "yINV3p0BxAvHzvX9I96C_2026-04-30T12:32:35.947658843_4762cbfd-0492-43f5-bcbb-1be361a0c38e",
  "monitor_user": {
    "name": "",
    "backend_roles": [],
    "roles": [],
    "user_requested_tenant": null,
    "user_requested_tenant_access": null,
    "custom_attribute_names": []
  },
  "trigger_id": "UoP73Z0BxAxHzvX9h6fl",
  "trigger_name": "My Trigger",
  "finding_ids": [
    "27f71637-5b44-4582-91c4-b571287a49d9"
  ],
  "related_doc_ids": [],
  "state": "ACTIVE",
  "error_message": null,
  "alert_history": [],
  "severity": "medium",
  "action_execution_results": [],
  "start_time": 1779174052000,
  "last_notification_time": 1779174052000,
  "end_time": null,
  "acknowledged_time": null,
  "agg_alert_content": {
    "parent_bucket_path": "result_agg",
    "bucket_keys": [
      "my_key"
    ],
    "bucket": {
      "doc_count": 25,
      "key": "my_key"
    }
  }
}

Example doc-level notification template that should be supported:

{{#ctx.alerts}}{{#sample_documents}}
rule={{_source.trigger_name}}
severity={{_source.severity}}
state={{_source.state}}
monitor={{_source.monitor_name}}
started={{_source.start_time}}
bucket={{#_source.agg_alert_content}}{{#bucket}}{{key}}{{/bucket}}{{/_source.agg_alert_content}}
{{/sample_documents}}{{/ctx.alerts}}

What is the expected behavior?
A doc-level monitor notification should be able to access the original matched document in a stable, documented shape, for example:

{
  "_id": "...",
  "_index": "...",
  "_source": {
    "trigger_name": "...",
    "severity": "...",
    "state": "...",
    "monitor_name": "...",
    "start_time": 1779174052000
  }
}

This would allow action templates to reliably render the original matched document fields using:

{{#ctx.alerts}}{{#sample_documents}}
{{_source.trigger_name}}
{{_source.severity}}
{{_source.state}}
{{/sample_documents}}{{/ctx.alerts}}

In other words, doc-level monitor notification templates should have stable access to the matched original document, similar in spirit to how query-level monitors expose _source through ctx.results[*].hits.hits[*].

What is your host/environment?

• OS: macOS Tahoe
• Version: OpenSearch 3.6.0-based environment
• Plugins:
  • opensearch-alerting
  • opensearch-security-analytics

Additional context for reproduction:

• monitored source was a Security Analytics hidden alert index
• documents in that index were stored with required routing
• notifications were generated through doc-level monitor actions

Do you have any screenshots?
No screenshots included. The issue is reproducible from the monitor configuration and resulting notification output.

Do you have any additional context?
The behavior appears to be caused by the doc-level notification path not preserving a stable, clean copy of the original matched document for Mustache rendering.

The relevant characteristics of the problem are:

• the original matched document is initially available during shard search
• doc-level percolation later transforms field names for internal matching
• the notification rendering path does not guarantee a stable original-document payload for templates
• in routed-index scenarios, any later re-fetch path is especially fragile

This is not just a convenience issue. It blocks a common workflow where doc-level monitors are used to:

• match one document
• generate one notification per matched document
• include the original alert context in that notification for triage, escalation, or forwarding

A robust fix would be to preserve the original matched document before any internal field-name transformation and expose it to action templates via a stable sample_documents[*] structure containing _id, _index, and _source.

A code-level fix has already been validated locally and can be provided in a follow-up pull request.

[andrross]

Catch All Triage - 1