diff --git a/docset.yml b/docset.yml index a6a20b4619..f06250f976 100644 --- a/docset.yml +++ b/docset.yml @@ -123,8 +123,8 @@ subs: ls-pipelines-app: "Logstash Pipelines" maint-windows-app: "Maintenance Windows" maint-windows-cap: "Maintenance windows" - alerting-v2: "experimental alerting features" - alerting-v2-cap: "Experimental alerting features" + alerting-v2-system: "experimental alerting system" + alerting-v2-system-cap: "Experimental alerting system" custom-roles-app: "Custom Roles" data-source: "data view" data-sources: "data views" diff --git a/explore-analyze/alerting/kibana-alerting-experimental/rules.md b/explore-analyze/alerting/kibana-alerting-experimental/rules.md new file mode 100644 index 0000000000..921cbb53c0 --- /dev/null +++ b/explore-analyze/alerting/kibana-alerting-experimental/rules.md @@ -0,0 +1,34 @@ +--- +navigation_title: Rules +applies_to: + stack: experimental 9.5+ + serverless: experimental +products: + - id: kibana +description: "Rules in Kibana's experimental alerting system define what to detect using ES|QL. Evaluation runs on a schedule; alerts, action policies, and notifications flow from rule detections." +--- + +# Rules in the {{alerting-v2-system}} + +A rule is where the {{alerting-v2-system}} starts. It points {{kib}} at the data you care about, describes what counts as a problem in {{esql}}, and says how often to check. Alerts, action policies, and notifications all flow from what a rule detects. + +This page explains what rules do, what they don't control, and how to choose a creation path. + +## What rules do [detection-and-notification] + +On each run, a rule executes an {{esql}} query against your data. If the query finds a match and the rule is in Signal mode, it writes a _signal_, a point-in-time record that the condition was met. In Alert mode, it also maintains an _alert episode_ for each matched series, tracking state from first breach through recovery. + +When creating a rule, choose Signal mode to record and query results without alerting anyone, or Alert mode when you want to track issues and route notifications. + +## What rules don't do + +Rules only define *what* to detect. They don't control notifications, who gets notified, or when. That's the job of action policies, which are global objects scoped to your space that match alert episodes from any rule. A rule has no say in which policies pick it up. + +This separation means you can build and test a rule without anyone getting paged, update notification routing without touching the rule, and have multiple action policies respond to the same rule independently. + +## Next steps + +- **[Create a rule](rules/create-a-rule.md):** Compare creation paths and choose the one that fits your workflow. +- **[Configure a rule](rules/configure-a-rule.md):** Set the schedule, grouping, activation thresholds, recovery conditions, and no-data behavior. +- **[View and manage rules](rules/view-manage-rules.md):** Enable, disable, clone, delete, and bulk-manage rules from the rules list. +- **[ES|QL query patterns](rules/esql-query-patterns.md):** Browse query patterns ordered by complexity, from a basic event filter to SLO burn rate and persistent breach detection. \ No newline at end of file diff --git a/explore-analyze/alerting/kibana-alerting-experimental/rules/configure-a-rule.md b/explore-analyze/alerting/kibana-alerting-experimental/rules/configure-a-rule.md new file mode 100644 index 0000000000..9ebce4c2b9 --- /dev/null +++ b/explore-analyze/alerting/kibana-alerting-experimental/rules/configure-a-rule.md @@ -0,0 +1,27 @@ +--- +navigation_title: Configure a rule +applies_to: + stack: experimental 9.5+ + serverless: experimental +products: + - id: kibana +description: "Configure rules in the experimental alerting system: mode, ES|QL query, grouping, schedule, activation and recovery thresholds, no-data handling, tags, and evaluation." +--- + +# Configure a rule in the {{alerting-v2-system}} [rule-settings] + +Rules in the {{alerting-v2-system}} have three required settings and several optional ones. The table below lists each setting, what it controls, and whether it is required to save a rule. + + +| Setting | Description | Required | +| --- | --- | --- | +| [Rule mode](configure-rule-mode.md) | Signal or Alert. Controls whether matching rows generate signal documents or tracked alert episodes. | Required | +| [ES\|QL query](configure-rule-query.md) | The detection logic and the parameters available in query expressions. | Required | +| [Schedule and lookback](configure-rule-schedule.md) | How often the rule evaluates and how far back the query looks. | Required | +| [Severity](configure-rule-severity.md) | Assign severity levels to alert episodes using a `severity` column in query output. | Optional | +| [Grouping](configure-rule-grouping.md) | Track multiple subjects (hosts, services, users) as independent alert series in one rule. | Optional | +| [Activation and recovery thresholds](configure-rule-thresholds.md) | Reduce noise with delay modes for opening and closing alert episodes. (Alert mode only) | Optional | +| [No-data handling](configure-no-data-handling.md) | What the rule records when the base query returns no results. | Optional | +| [Tags and runbooks](configure-rule-tags.md) | Free-form labels and investigation guides attached to the rule. (Alert mode only) | Optional | diff --git a/explore-analyze/alerting/kibana-alerting-experimental/rules/configure-no-data-handling.md b/explore-analyze/alerting/kibana-alerting-experimental/rules/configure-no-data-handling.md new file mode 100644 index 0000000000..3e47fc0622 --- /dev/null +++ b/explore-analyze/alerting/kibana-alerting-experimental/rules/configure-no-data-handling.md @@ -0,0 +1,40 @@ +--- +navigation_title: No-data handling +applies_to: + stack: experimental 9.5+ + serverless: experimental +products: + - id: kibana +description: "Configure no-data handling for rules in Kibana's experimental alerting system to control what happens when a query returns no results." +--- + +# No-data handling in the {{alerting-v2-system}} [no-data-handling] + +No-data handling is an optional setting for rules in the {{alerting-v2-system}}. Use `no_data_strategy` to control what the rule records when the base query returns no results. Setting this correctly prevents false recoveries and misleading `no_data` events when data sources stop reporting. + +Set `no_data_strategy` to one of the following values: + +| Value | Description | +| --- | --- | +| `emit` | Record a no-data event. | +| `last_known_status` | Hold the last known lifecycle state. An active breach stays active; a recovered episode stays recovered. | +| `recover` | Treat absence as recovery. | +| `none` | Turn off no-data detection | + +:::{note} +`no_data_strategy` does not detect when a specific host or data source stops reporting while others continue. Refer to [No-data detection](esql-no-data-detection.md) for an {{esql}} pattern that surfaces silent sources as alert rows. +::: + +## Examples + +### Maintain alert state during a metrics collection outage + +This rule monitors infrastructure CPU. If the metrics collection agent stops sending data, you don't want an active CPU breach to auto-recover just because the query returned nothing. Set `no_data_strategy` to `last_known_status`. The rule holds the alert in its current state until data resumes. + +Use this when an empty query result most likely means a pipeline problem rather than a genuine recovery. + +### Surface a broken data pipeline as an alert + +This rule monitors for login events from an identity provider. If no events appear in the lookback window, it's unusual enough to warrant attention: either the pipeline is broken or something has suppressed activity. Set `no_data_strategy` to `emit`. The absence is recorded as a `no_data` event in `.rule-events`, making it visible alongside other rule activity. + +Use this when receiving no data is itself a signal worth investigating. diff --git a/explore-analyze/alerting/kibana-alerting-experimental/rules/configure-rule-grouping.md b/explore-analyze/alerting/kibana-alerting-experimental/rules/configure-rule-grouping.md new file mode 100644 index 0000000000..50fd62d759 --- /dev/null +++ b/explore-analyze/alerting/kibana-alerting-experimental/rules/configure-rule-grouping.md @@ -0,0 +1,61 @@ +--- +navigation_title: Grouping +applies_to: + stack: experimental 9.5+ + serverless: experimental +products: + - id: kibana +description: "Configure rule grouping in Kibana's experimental alerting system to track multiple subjects as independent alert series." +--- + +# Rule grouping in the {{alerting-v2-system}} [rule-grouping] + +Rule grouping is an optional setting in the {{alerting-v2-system}} that lets a single rule track multiple things independently. For example, a rule monitoring CPU usage across hosts can produce a separate alert series for each host, rather than one alert for everything combined. + + + +:::{note} +Rule grouping controls how alert series are created. Notification grouping (configured on an action policy) controls how those alert episodes are batched into messages. These are separate settings. +::: + +## Aligning `BY` fields with your rule's query + +The `BY` fields you specify for grouping must match the columns in the `BY` clause of your {{esql}} `STATS` command. If they don't match, the grouping configuration won't work as expected. + +:::{tip} +Write the query first, then set the group fields. That way the `BY` columns are already defined and you can select them directly. If you later add or remove a `BY` field in the query, update the group fields to match. +::: + + + +## Examples + +### Track error rates per service + +This rule counts HTTP errors per service and opens a separate alert series for each service that exceeds the threshold. Each service gets its own lifecycle: if the checkout service recovers but the payments service stays critical, those are tracked independently. + +```esql +FROM logs-* +| WHERE @timestamp >= ?_tstart AND @timestamp < ?_tend +| STATS error_count = COUNT_IF(http.response.status_code >= 500) BY service.name // Group field: one series per service +| WHERE error_count > 10 +| KEEP service.name, error_count +``` + +Without a matching `grouping.fields` entry, the rule treats all services as a single combined series. A spike in one service would activate the alert for everything, and recovery requires all services to drop below the threshold at the same time. + +### Track CPU usage per host and region + +When the query groups by multiple fields, include all of them in `grouping.fields` to create one alert series per unique combination. + +```esql +FROM metrics-* +| WHERE @timestamp >= ?_tstart AND @timestamp < ?_tend +| STATS avg_cpu = AVG(system.cpu.total.pct) BY host.name, cloud.region // Group fields: one series per host+region pair +| WHERE avg_cpu > 0.90 +| KEEP host.name, cloud.region, avg_cpu +``` diff --git a/explore-analyze/alerting/kibana-alerting-experimental/rules/configure-rule-mode.md b/explore-analyze/alerting/kibana-alerting-experimental/rules/configure-rule-mode.md new file mode 100644 index 0000000000..495a65c905 --- /dev/null +++ b/explore-analyze/alerting/kibana-alerting-experimental/rules/configure-rule-mode.md @@ -0,0 +1,28 @@ +--- +navigation_title: Rule mode +applies_to: + stack: experimental 9.5+ + serverless: experimental +products: + - id: kibana +description: "Choose between Signal and Alert mode for rules in Kibana's experimental alerting system." +--- + +# Rule mode in the {{alerting-v2-system}} [rule-mode] + +Rule mode is a required setting for rules in the {{alerting-v2-system}}. Rule mode set by the rule creation method and some creation paths only support one mode. Refer to [Create a rule](create-a-rule.md) for available options. + +| Mode | Behavior | Best for | +| --- | --- | --- | +| Signal | Records each matching row as a signal document. No alert episodes, no notifications. | Testing a new query, building detection history, or observing matches without notifying anyone. | +| Alert | Creates an alert episode for each matching row. Episodes are tracked through lifecycle states, appear on the **Alerts** page, and can be routed to notifications by action policies. | Production rules where breaches should be tracked, escalated, or routed to a notification channel. | + +## Examples + +### Build detection history before enabling alerts + +You're writing a new detection query and want to verify it produces the results you expect before anyone gets paged. Create the rule in Signal mode so matches are recorded in `.rule-events` and you can inspect them in Discover without opening any alert episodes or triggering notifications. Once the matches look correct, edit the rule and switch it to Alert mode. + +### Route critical episodes to an on-call workflow + +You have a checkout service error rate rule and want on-call engineers notified when it fires. Create the rule in Alert mode so each breach opens a tracked episode that action policies can route to a notification channel. The rule's episodes appear on the **Alerts** page and are visible to any action policy whose KQL matcher matches the episode fields. diff --git a/explore-analyze/alerting/kibana-alerting-experimental/rules/configure-rule-query.md b/explore-analyze/alerting/kibana-alerting-experimental/rules/configure-rule-query.md new file mode 100644 index 0000000000..5263a262a5 --- /dev/null +++ b/explore-analyze/alerting/kibana-alerting-experimental/rules/configure-rule-query.md @@ -0,0 +1,96 @@ +--- +navigation_title: ES|QL query +applies_to: + stack: experimental 9.5+ + serverless: experimental +products: + - id: kibana +description: "Configure the ES|QL query and query parameters for rules in Kibana's experimental alerting system." +--- + +# {{esql}} query in the {{alerting-v2-system}} [esql-query-rule] + +Every rule in the {{alerting-v2-system}} uses an {{esql}} query to define what to evaluate. The query has two parts: a base query that shapes and filters the data, and an optional alert condition that determines which rows become alert events. For more advanced use cases, the query also supports [dynamic values](#dynamic-query-values) for filtering by the evaluation window or setting configurable thresholds through the rule form. + +## Base query [query-base] + +The base query is the main {{esql}} expression. Use `FROM` to point the rule at the indices or data streams to read. Shape results with `STATS`, `WHERE`, and `EVAL`, and control which fields are stored with `KEEP`. The base query runs on every evaluation, even when no match occurs, which is what enables no-data detection and recovery. The [{{esql}} reference](elasticsearch://reference/query-languages/esql.md) covers all available commands and processing functions. + +This query counts HTTP 5xx errors per service over the lookback window and stores only the fields needed for triage: + +```esql +FROM logs-* +| WHERE @timestamp >= ?_tstart AND @timestamp < ?_tend +| STATS error_count = COUNT_IF(http.response.status_code >= 500) BY service.name +| KEEP service.name, error_count +``` + +Without an alert condition, every row returned by this query is treated as a breach, so the rule fires for every service that logged at least one 5xx error. + +## Alert condition [query-alert-condition] + +The alert condition is an optional `WHERE` clause appended after the base query. Only rows that pass the condition are treated as breaches. Use an alert condition when the base query returns aggregate results and you only want to alert when a value crosses a threshold. + +This example extends the base query above to only fire when a service exceeds 10 errors: + +```esql +FROM logs-* +| WHERE @timestamp >= ?_tstart AND @timestamp < ?_tend +| STATS error_count = COUNT_IF(http.response.status_code >= 500) BY service.name +| KEEP service.name, error_count +// Alert condition: only services with more than 10 errors become breaches +| WHERE error_count > 10 +``` + +The `KEEP` command controls which fields appear on each stored alert event. Only the fields in `KEEP` are available for action policy matchers, grouping keys, and triage. + +## Use dynamic values in your rule query [dynamic-query-values] + +{{esql}} rule queries support two kinds of parameters that make queries more dynamic: time bounds that the executor injects automatically, and form variables you define when creating a rule. You don't need either to write a working rule, but they're useful for scoping queries precisely to the evaluation window or making thresholds configurable without editing the query. + +### Filter your query to the evaluation window (`?_tstart` and `?_tend`) [time-bound-parameters] + +`?_tstart` and `?_tend` are reserved parameter names that the rule executor binds automatically on every evaluation. They hold the start and end timestamps of the lookback window, so you can scope a query to exactly the period the rule is evaluating: + +```esql +FROM logs-* +| WHERE @timestamp >= ?_tstart AND @timestamp < ?_tend +| STATS error_count = COUNT(*) BY service.name +| WHERE error_count > 10 +``` + +These parameters work across all rule creation methods. + +### Set configurable values in the rule form (`?param`) [form-variables] + +When creating a rule through the form, you can use `?param` placeholders, such as `?threshold`, as {{esql}} Control variables. The form resolves these variables and inlines their values into the query before saving. The stored rule and the YAML representation of it contains the resolved values, not the placeholder tokens. + +## Examples + +### Scoping a query to the evaluation window + +This query counts HTTP errors per service over the rule's lookback window. `?_tstart` and `?_tend` are bound automatically at runtime, so the query always covers exactly the configured window regardless of when the rule runs. + +```esql +FROM logs-* +| WHERE @timestamp >= ?_tstart AND @timestamp < ?_tend +| STATS error_count = COUNT_IF(http.response.status_code >= 500) BY service.name +| WHERE error_count > 0 +| KEEP service.name, error_count +``` + +If you omit the time filter, the query scans the full index on every evaluation, which increases query cost and can return stale matches from earlier runs. + +### Using a form variable for a configurable threshold + +This query uses `?threshold` as a form variable so the threshold can be set in the rule form UI without editing the query directly. When the rule is saved, the form resolves `?threshold` to its configured value and inlines it. The stored query contains the literal number, not the placeholder. + +```esql +FROM logs-* +| WHERE @timestamp >= ?_tstart AND @timestamp < ?_tend +| STATS p99_latency = PERCENTILE(http.response_time, 99) BY service.name +| WHERE p99_latency > ?threshold +| KEEP service.name, p99_latency +``` + +Because `?threshold` is resolved before saving, YAML and API representations of this rule always show the resolved value. diff --git a/explore-analyze/alerting/kibana-alerting-experimental/rules/configure-rule-schedule.md b/explore-analyze/alerting/kibana-alerting-experimental/rules/configure-rule-schedule.md new file mode 100644 index 0000000000..9bbc393fc6 --- /dev/null +++ b/explore-analyze/alerting/kibana-alerting-experimental/rules/configure-rule-schedule.md @@ -0,0 +1,53 @@ +--- +navigation_title: Schedule and lookback +applies_to: + stack: experimental 9.5+ + serverless: experimental +products: + - id: kibana +description: "Configure the execution schedule and lookback window for rules in Kibana's experimental alerting system." +--- + +# Schedule and lookback in the {{alerting-v2-system}} [schedule-lookback] + +Schedule and lookback are required settings for rules in the {{alerting-v2-system}}. They control how often a rule runs and how far back it looks when evaluating data. + +Both fields accept duration strings such as `30s`, `5m`, `2h`, or `7d`. Refer to [Duration format](yaml-rule-schema-reference.md#duration-format) for supported units. + +## Execution interval + +The execution interval (`schedule.every`) determines how frequently the rule evaluates. The minimum is `5s` and the maximum is `365d`. Values outside that range are rejected. + +## Lookback window + +The lookback window (`schedule.lookback`) determines the time range that the {{esql}} query covers. The minimum is `5s` and the maximum is `365d`. + +If the lookback is shorter than the execution interval, evaluations can miss data between runs. Use a lookback at least as long as the execution interval unless you have a deliberate reason not to. + +## Examples + +### High-frequency security rule + +This rule detects a burst of failed login attempts. Because the threat can develop quickly and needs fast detection, the interval is set to **1 minute** and the lookback to **5 minutes**. The 5-minute lookback is five times the interval, so a burst that straddles two evaluation windows is never missed. + +```esql +FROM logs-* +| WHERE @timestamp >= ?_tstart AND @timestamp < ?_tend // Covers the 5-minute lookback on each evaluation +| STATS failed_logins = COUNT_IF(event.outcome == "failure") BY user.name +| WHERE failed_logins > 10 +| KEEP user.name, failed_logins +``` + +The `?_tstart` and `?_tend` parameters are automatically bound to the lookback window, so the query always covers exactly the configured 5-minute range. + +### Cost-optimized infrastructure rule + +This rule monitors disk utilization for capacity planning. Fast response isn't critical, so the interval is set to **15 minutes** and the lookback to **30 minutes**. The wider window smooths out brief spikes that don't indicate a real capacity problem, reducing evaluation cost without sacrificing coverage. + +```esql +FROM metrics-* +| WHERE @timestamp >= ?_tstart AND @timestamp < ?_tend // Covers the 30-minute lookback on each evaluation +| STATS max_disk_pct = MAX(system.filesystem.used.pct) BY host.name, system.filesystem.mount_point +| WHERE max_disk_pct > 0.90 +| KEEP host.name, system.filesystem.mount_point, max_disk_pct +``` diff --git a/explore-analyze/alerting/kibana-alerting-experimental/rules/configure-rule-severity.md b/explore-analyze/alerting/kibana-alerting-experimental/rules/configure-rule-severity.md new file mode 100644 index 0000000000..0c5dfa27c8 --- /dev/null +++ b/explore-analyze/alerting/kibana-alerting-experimental/rules/configure-rule-severity.md @@ -0,0 +1,85 @@ +--- +navigation_title: Severity +applies_to: + stack: experimental 9.5+ + serverless: experimental +products: + - id: kibana +description: "Assign severity levels to alert episodes in Kibana's experimental alerting system using a severity column in ES|QL query output." +--- + +# Severity in the {{alerting-v2-system}} [rule-severity] + +Severity is an optional setting for rules in the {{alerting-v2-system}}. To set it, include a column named `severity` in your {{esql}} query output and add it to your `KEEP` list. The framework reads that column after each evaluation and maps it to one of five fixed levels: + +| Value | Description | Urgency | +| --- | --- | --- | +| `info` | Informational event worth recording. | No action required. | +| `low` | Minor condition that may need monitoring. | Review when convenient. | +| `medium` | Notable condition that warrants investigation. | Investigate soon. | +| `high` | Serious condition requiring prompt attention. | Address promptly. | +| `critical` | Severe condition requiring immediate action. | Act immediately. | + +## How the {{alerting-v2-system}} maps severity values + +The {{alerting-v2-system}} maps the `severity` column to an internal level after each evaluation using the following rules: + +- Matching is case-insensitive. +- Values that don't match one of the five levels are silently ignored. The alert episode is still created, but `severity` isn't set. +- Severity is only set on `breached` events. `recovered` and `no_data` events don't carry a severity value. + +## Stored fields + +When severity is set, the {{alerting-v2-system}} stores the following field on the alert episode, available to action policy matchers: + +| Field | Description | +| --- | --- | +| `severity` | The severity value from the most recent breached event. | + +Refer to [Rule event and field reference](rule-event-field-reference.md#episode-fields) for more information about this field. + +## Examples + +### Static severity for a simple threshold rule + +If every breach of a rule is equally urgent, assign a fixed severity rather than computing it dynamically. The `EVAL` command adds a constant `severity` column to every row the query returns. + +```esql +FROM logs-* +| WHERE @timestamp >= ?_tstart AND @timestamp < ?_tend +| STATS error_count = COUNT_IF(http.response.status_code >= 500) BY service.name +| WHERE error_count > 100 +| EVAL severity = "critical" +| KEEP service.name, error_count, severity +``` + +Every breach from this rule produces a `critical` episode. Use this when the threshold itself represents a critical condition and intermediate severity levels don't apply. + +### Dynamic severity based on burn rate + +Use `CASE` to map a computed metric to different severity levels. This query grades each service's error rate: services consuming error budget at 14.4× baseline or above are `critical`; those between 6× and 14.4× are `high`; and so on. Only services above 1× are returned, so below-threshold services don't generate alert rows. + +```esql +FROM metrics-* +| WHERE @timestamp >= ?_tstart AND @timestamp < ?_tend // Bind to the rule's configured lookback window +| STATS + errors = COUNT_IF(outcome == "failure"), + total = COUNT(*) + BY service.name +| EVAL burn_rate = errors / total +| EVAL severity = CASE( + burn_rate > 14.4, "critical", + burn_rate > 6.0, "high", + burn_rate > 1.0, "medium", + "low" + ) +| WHERE burn_rate > 1.0 +| KEEP service.name, burn_rate, severity +``` + +- **`WHERE`** (time filter): Scopes the query to the rule's configured lookback window using the reserved `?_tstart` and `?_tend` parameters. +- **`STATS`**: Counts failures and total requests, grouped by service. +- **`EVAL burn_rate`**: Computes the error rate as a fraction of failures to total requests. +- **`EVAL severity`**: Maps the burn rate to a severity level. +- **`WHERE burn_rate`**: Only services above the minimum threshold count as breaches. +- **`KEEP`**: Includes `severity` in the output so the {{alerting-v2-system}} reads and stores it. diff --git a/explore-analyze/alerting/kibana-alerting-experimental/rules/configure-rule-tags.md b/explore-analyze/alerting/kibana-alerting-experimental/rules/configure-rule-tags.md new file mode 100644 index 0000000000..75bcd88429 --- /dev/null +++ b/explore-analyze/alerting/kibana-alerting-experimental/rules/configure-rule-tags.md @@ -0,0 +1,42 @@ +--- +navigation_title: Tags and runbooks (Alert mode only) +applies_to: + stack: experimental 9.5+ + serverless: experimental +products: + - id: kibana +description: "Add tags and runbooks to Alert-mode rules in Kibana's experimental alerting system for filtering and investigation context." +--- + +# Tags and runbooks in the {{alerting-v2-system}} (Alert mode only) [tags-investigation] + +Tags and runbooks are optional metadata fields for Alert-mode rules in the {{alerting-v2-system}}. + +- **Tags**: Free-form labels for filtering and organization. Maximum 20 tags per rule; each tag can be up to 128 characters. +- **Runbooks**: An investigation guide stored with the rule so responders have context when alerts are generated. + +## Examples + +### Tag a rule for team ownership and severity + +Tags make it easy to filter alerts by team, environment, or severity tier. For a checkout service rule, you might add tags like: + +- `team:payments` +- `env:production` +- `sev:p1` + +On-call engineers can then narrow the alerts view to rules their team owns without scanning every active episode. + +### Add a runbook with triage steps + +A runbook gives responders immediate context when an alert fires. Write it as plain text in the rule's description field. Include enough detail that an engineer unfamiliar with the service can triage without asking for help: + +``` +Fires when checkout error rate exceeds 10% for 3 consecutive evaluations. + +Triage steps: +1. Check the checkout service deployment history in the last 30 minutes. +2. Review the error breakdown dashboard: https://kibana.example.com/dashboards/checkout-errors +3. If errors are concentrated in one region, escalate to the infra team. +4. If errors are global, page the payments on-call lead. +``` diff --git a/explore-analyze/alerting/kibana-alerting-experimental/rules/configure-rule-thresholds.md b/explore-analyze/alerting/kibana-alerting-experimental/rules/configure-rule-thresholds.md new file mode 100644 index 0000000000..b0e2865a9a --- /dev/null +++ b/explore-analyze/alerting/kibana-alerting-experimental/rules/configure-rule-thresholds.md @@ -0,0 +1,67 @@ +--- +navigation_title: Activation and recovery thresholds (Alert mode only) +applies_to: + stack: experimental 9.5+ + serverless: experimental +products: + - id: kibana +description: "Configure activation and recovery thresholds for Alert-mode rules in Kibana's experimental alerting system to reduce noise from spikes and flapping." +--- + +# Activation and recovery thresholds in the {{alerting-v2-system}} (Alert mode only) [activation-recovery-thresholds] + +Activation and recovery thresholds are optional settings for Alert-mode rules in the {{alerting-v2-system}}. They control when alerts transition between lifecycle states, reducing noise from short spikes and rapid flapping between active and recovered. + +## Activation thresholds + +Activation thresholds control when a breached rule transitions from pending to active. Three delay modes are available: + +| Mode | Behavior | When to use | +| --- | --- | --- | +| Immediate | Opens an alert episode as soon as the threshold is breached on the first evaluation. | Use when any single breach warrants attention and latency matters. | +| Breaches | Opens an alert episode after the threshold is breached a set number of times in a row. | Use when a single breach isn't enough reason to act, for example when brief spikes are normal and you only care if the condition keeps firing. | +| Duration | Opens an alert episode after the threshold has been continuously breached for a set time. | Use when duration of the problem matters more than how many evaluations caught it, for example sustained high CPU rather than a momentary spike. | + +### Activation fields + +Use the following fields to configure the Breaches and Duration modes. Timeframe fields accept duration strings between `5s` and `365d`. Refer to [Duration format](yaml-rule-schema-reference.md#duration-format) for supported units. + +| Field | Type | Description | +| --- | --- | --- | +| `pending_count` | Positive integer | Number of consecutive breach evaluations required before the alert episode opens. | +| `pending_timeframe` | Duration string | How long the condition must remain breached before the alert episode opens. | +| `pending_operator` | `AND` or `OR` | When both `pending_count` and `pending_timeframe` are set, controls whether both must be satisfied (`AND`) or either one is enough (`OR`). | + +You can combine Breaches and Duration. For example, require the threshold to be breached five times in a row _and_ persist for at least two minutes before an alert episode opens. Set `pending_operator` to `AND` to require both, or `OR` if either is enough. + +## Recovery thresholds + +Recovery thresholds control when an active alert episode transitions back to inactive. The same delay modes available for activation apply here: consecutive recoveries required, minimum recovery duration, or both. + +| Field | Type | Description | +| --- | --- | --- | +| `recovering_count` | Positive integer | Number of consecutive non-breaching evaluations required before the alert episode closes. | +| `recovering_timeframe` | Duration string | How long the condition must remain non-breaching before the alert episode closes. | +| `recovering_operator` | `AND` or `OR` | When both `recovering_count` and `recovering_timeframe` are set, controls whether both must be satisfied (`AND`) or either one is enough (`OR`). | + +Timeframe fields accept the same `5s` to `365d` bounds as activation timeframes. Refer to [Duration format](yaml-rule-schema-reference.md#duration-format) for supported units. + +:::{note} +The `recovery_strategy` field controls how recovery is detected, separately from how many recoveries are required. When creating a rule through the UI, `recovery_strategy` defaults to `no_breach`, which recovers the alert episode when its active group no longer appears in the breach batch. + +When creating a rule through Agent Builder, you can omit `recovery_strategy` entirely to suppress recovery events and keep alert episodes active until closed manually. For the full field reference, go to [YAML rule schema reference](yaml-rule-schema-reference.md#recovery-strategy). +::: + +## Examples + +### Ignore brief CPU spikes + +This rule monitors CPU usage and runs every minute. A single high reading is often just a process starting up. Set `pending_count` to `3` so the rule requires 3 consecutive breaches before opening an episode, meaning the condition has been true for at least 3 minutes. This filters out noise without losing real signals. + +### Require sustained breach before escalating + +This rule monitors a payment error rate. Brief spikes happen during deployments and are expected. Set `pending_count` to `5`, `pending_timeframe` to `2m`, and `pending_operator` to `AND`. The rule only fires when the error rate has breached on 5 consecutive evaluations and has been continuously elevated for at least 2 minutes. Either condition alone isn't enough. + +### Prevent flapping on recovery + +This rule monitors database connection pool saturation. After the condition clears, set `recovering_count` to `3` to require 3 consecutive non-breaching evaluations before closing the episode. Without this, a rule that alternates between breaching and recovering on consecutive evaluations generates a constant stream of open and closed notifications. diff --git a/explore-analyze/alerting/kibana-alerting-experimental/rules/create-a-rule.md b/explore-analyze/alerting/kibana-alerting-experimental/rules/create-a-rule.md new file mode 100644 index 0000000000..4bf9217191 --- /dev/null +++ b/explore-analyze/alerting/kibana-alerting-experimental/rules/create-a-rule.md @@ -0,0 +1,24 @@ +--- +navigation_title: Create a rule +applies_to: + stack: experimental 9.5+ + serverless: experimental +products: + - id: kibana +description: "Create rules in Kibana's experimental alerting system using the ES|QL editor, AI Agent, rule builder, or directly from a Discover session." +--- + +# Create a rule in the {{alerting-v2-system}} [create-a-rule] + +The {{alerting-v2-system}} in {{kib}} provides several ways to create rules. For descriptions of configurable rule setting, refer to [Configure a rule](configure-a-rule.md). + +| Option | Best for | +| --- | --- | +| [Create an ES\|QL rule](create-esql-rule.md) | Full control over the query. Supports both a step-by-step form and a YAML editor. | +| [Create with AI Agent](create-rule-ai-agent.md) | When you know what you want to detect but aren't sure how to write the ES\|QL. | +| [Use the rule builder](use-rule-builder.md) | Rules that follow a guided, form-based setup. | +| [Create from Discover](create-rule-from-discover.md) | When you already have an ES\|QL query working in Discover and want to convert it into a rule. | + +:::{note} +For query examples ranging from a basic event filter to SLO burn rate and persistent breach detection, refer to [ES|QL query patterns](esql-query-patterns.md). +::: \ No newline at end of file diff --git a/explore-analyze/alerting/kibana-alerting-experimental/rules/create-esql-rule.md b/explore-analyze/alerting/kibana-alerting-experimental/rules/create-esql-rule.md new file mode 100644 index 0000000000..49f78bcce2 --- /dev/null +++ b/explore-analyze/alerting/kibana-alerting-experimental/rules/create-esql-rule.md @@ -0,0 +1,32 @@ +--- +navigation_title: Create an ES|QL rule +applies_to: + stack: experimental 9.5+ + serverless: experimental +products: + - id: kibana +description: "Write ES|QL detection queries for rules in Kibana's experimental alerting system using the rule form or YAML editor, with a live query sandbox for previewing results." +--- + +# Create an {{esql}} rule in the {{alerting-v2-system}} [create-esql-rule] + +The {{esql}} rule path lets you write the detection query directly. There are two ways to define the rule: + +- **Rule form** - Fill in the step-by-step form with a live preview of results. +- **YAML mode** - Switch to YAML and edit the raw rule definition. You can switch between form and YAML at any point; edits are preserved. + +For descriptions of each rule setting, refer to [Configure a rule](configure-a-rule.md). For a list of supported YAML fields, refer to [YAML rule schema reference](yaml-rule-schema-reference.md). + +## Preview query results in the sandbox [rule-builder-query-sandbox] + +The query sandbox lets you run your {{esql}} query against current data and preview the results before applying them to the rule form. Use the time field selector and date picker to control the time range, then select **Search** (or press ⌘↵) to execute. When the results look correct, select **Apply changes** to populate the form. + +Use the sandbox to: + +- **Confirm grouping** - Check that your `BY` clause produces the series you intend, for example, one distinct series per host or per service, not a single undifferentiated result. +- **Catch unexpected output** - Verify that the query returns data in the right shape for the alert condition you plan to set. A query that returns zero rows or an unexpected field name won't behave as expected once the rule runs on a schedule. +- **Refine before committing** - Edit the query and re-run it as many times as needed without leaving the rule creation form. + +## Using the YAML editor [yaml-editor] + +Use the YAML editor when you want to copy or adapt a rule quickly without re-entering settings by hand, or provision many rules at once. The YAML editor isn't available within the Threshold Alert builder. diff --git a/explore-analyze/alerting/kibana-alerting-experimental/rules/create-rule-ai-agent.md b/explore-analyze/alerting/kibana-alerting-experimental/rules/create-rule-ai-agent.md new file mode 100644 index 0000000000..cc0b010742 --- /dev/null +++ b/explore-analyze/alerting/kibana-alerting-experimental/rules/create-rule-ai-agent.md @@ -0,0 +1,34 @@ +--- +navigation_title: Create with AI Agent +applies_to: + stack: experimental 9.5+ + serverless: experimental +products: + - id: kibana +description: "Use the Elastic AI Agent to generate ES|QL rules from plain-language descriptions in Kibana's experimental alerting system." +--- + +# Create a rule with AI Agent in the {{alerting-v2-system}} [create-rule-ai-agent] + +The AI Agent option opens the Elastic AI agent pre-loaded with rule management knowledge. Describe what you want to monitor in plain language and the agent resolves the relevant data source and builds a rule proposal. + +The proposal appears as an inline attachment card in the conversation showing the rule name, type, schedule, and tags. Select the card to open a flyout with three tabs: + +- **Conditions** - The full rule configuration, including query, thresholds, grouping, and schedule. +- **Query preview** - Runs the {{esql}} query and shows results inline so you can verify the detection logic without leaving the conversation. +- **Runbook** - A free-text runbook associated with the rule. + +The agent doesn't save the rule automatically. When the proposal looks correct, select **Save as rule** from the flyout header to persist it. After saving, you can ask the agent to configure notifications, which creates an action policy scoped to that rule. + +:::{note} +Signal-mode rules don't support notifications. If you ask the agent to set up notifications on a signal rule, the agent will explain the limitation and offer to convert the rule to Alert mode or create a new Alert-mode rule. +::: + +## Example prompts [ai-agent-sample-prompts] + +Use these prompts as a starting point, then adjust them to your data and thresholds: + +- Create an error threshold rule on my checkout service data. Alert when there are more than 3 HTTP 5xx errors in the past 5 minutes, grouped by URL path. +- Monitor average CPU usage across all hosts. Alert when any host exceeds 90% for more than 10 minutes. +- Alert me when log volume from the payments service drops below 100 events in a 5-minute window. This likely means data has stopped flowing. +- Set up a rule that tracks error rate by service. Alert at medium severity when the rate exceeds 1%, and critical when it exceeds 5%. diff --git a/explore-analyze/alerting/kibana-alerting-experimental/rules/create-rule-from-discover.md b/explore-analyze/alerting/kibana-alerting-experimental/rules/create-rule-from-discover.md new file mode 100644 index 0000000000..131b0e9e53 --- /dev/null +++ b/explore-analyze/alerting/kibana-alerting-experimental/rules/create-rule-from-discover.md @@ -0,0 +1,31 @@ +--- +navigation_title: Create from Discover +applies_to: + stack: experimental 9.5+ + serverless: experimental +products: + - id: kibana +description: "Convert an ES|QL query from a Discover session into a rule in Kibana's experimental alerting system, with the query pre-filled and a preview panel for verifying grouping." +--- + +# Create a rule from Discover in the {{alerting-v2-system}} [create-from-discover] + +When you build an {{esql}} query in Discover that surfaces interesting patterns, you can convert it into a rule without rewriting the query. Starting from Discover means your query is already tested and returns the shape you expect before the rule is ever saved. + +## Entry points [discover-rule-entry-points] + +Two paths lead to Discover-based rule creation: + +- **Discover Alerts menu**: When you're in Discover with an active {{esql}} query and the {{alerting-v2-system}} is enabled, the Alerts menu includes a **Create ES|QL rule** option. The rule creation flyout opens pre-populated with the current query. This path is only available in ES|QL mode. +- **Rules list**: Selecting the option to create a rule from Discover opens a flyout that embeds a live Discover session so you can compose and test the query before saving the rule. + +## How it works [discover-rule-flow] + +When you trigger rule creation from Discover, your {{esql}} query pre-fills the **Create ES|QL rule** form. The rule creation form also shows a preview panel that reflects how your query partitions results into alert series. If your query uses a `BY` clause, the preview shows the series that would be evaluated on each run, letting you verify grouping logic against live data before committing to a schedule. + +The rule creation flyout supports both a step-by-step form and a YAML editing mode. You can switch between them at any point; edits in YAML mode are preserved when you return to the form view. The YAML editor includes {{esql}} autocomplete in the query field. + +For descriptions of what each rule setting does, refer to [Configure a rule](configure-a-rule.md). + + diff --git a/explore-analyze/alerting/kibana-alerting-experimental/rules/esql-first-rule-query.md b/explore-analyze/alerting/kibana-alerting-experimental/rules/esql-first-rule-query.md new file mode 100644 index 0000000000..0d7f3fa700 --- /dev/null +++ b/explore-analyze/alerting/kibana-alerting-experimental/rules/esql-first-rule-query.md @@ -0,0 +1,47 @@ +--- +navigation_title: Your first rule query +applies_to: + stack: experimental 9.5+ + serverless: experimental +products: + - id: kibana +description: "Write your first ES|QL rule query in Kibana's experimental alerting system. Covers the minimum query structure and how to adapt it to your data." +--- + +# Your first rule query in the {{alerting-v2-system}} [esql-first-rule-query] + +If you're new to {{esql}} or to writing rules, this page shows the simplest query structure that a rule needs. It requires only a basic familiarity with your data. + +## The simplest rule query + +This query returns one row for every log entry at `ERROR` level in the lookback window. No aggregation happens; each matching event becomes its own alert event. Use this pattern when any single occurrence of a condition is worth alerting on: any critical error log, any failed authentication attempt, any payment rejection. + +```esql +FROM logs-* +| WHERE @timestamp >= ?_tstart AND @timestamp < ?_tend // Scope to the rule's configured lookback window +| WHERE level == "ERROR" // The alert condition: every match is a breach +| KEEP service.name, message, @timestamp // Fields stored on each alert event +``` + +Every row the query returns is treated as a breach. If the query returns nothing, no alert fires. + +### What each part does + +| Part | Purpose in a rule | +| --- | --- | +| `FROM logs-*` | The index or data stream to query. | +| `WHERE @timestamp >= ?_tstart AND @timestamp < ?_tend` | Scopes the query to the rule's lookback window. `?_tstart` and `?_tend` are bound automatically at runtime. Always include this filter to avoid scanning your entire index on every evaluation. | +| `WHERE level == "ERROR"` | The alert condition. Only rows that pass this filter generate an alert event. | +| `KEEP service.name, message, @timestamp` | The fields stored on each alert event. Only fields in `KEEP` are available for routing, grouping, and triage. | + +### How to adapt this query + +To use this query with your own data, change three things: + +1. **Index**: Replace `logs-*` with the index or data stream that holds your data. +2. **Alert condition**: Replace `level == "ERROR"` with the field name and value that defines a breach in your data. For example: `event.outcome == "failure"`, `http.response.status_code >= 500`, or `process.name == "malware.exe"`. +3. **KEEP fields**: Replace the field list with whatever you need for triage. Include any fields you plan to use for grouping, routing, or displaying in the alert detail view. + +## Next steps + +This pattern alerts on individual events. If you want to alert only when a count or rate crosses a threshold rather than on individual occurrences, the [threshold query pattern](esql-threshold-queries.md) is the next step. diff --git a/explore-analyze/alerting/kibana-alerting-experimental/rules/esql-no-data-detection.md b/explore-analyze/alerting/kibana-alerting-experimental/rules/esql-no-data-detection.md new file mode 100644 index 0000000000..7ff031fae9 --- /dev/null +++ b/explore-analyze/alerting/kibana-alerting-experimental/rules/esql-no-data-detection.md @@ -0,0 +1,32 @@ +--- +navigation_title: No-data detection +applies_to: + stack: experimental 9.5+ + serverless: experimental +products: + - id: kibana +description: "Detect silent hosts and stopped data sources in Kibana's experimental alerting system using ES|QL last-seen queries." +--- + +# No-data detection in the {{alerting-v2-system}} [no-data-esql-query] + +No-data detection solves a different problem from threshold alerting. A threshold query evaluates data that's present. No-data detection identifies sources that have gone silent. Threshold queries can't do this. A silent host produces no rows at all, so there's nothing to evaluate a condition against. + +The pattern inverts the normal approach: use a broad lookback to find all known hosts, then surface only those that have not reported recently. + +```esql +FROM metrics-* +| WHERE @timestamp >= NOW() - 12 hours // Cover the longest expected reporting gap for your hosts. + // Too short: silent hosts fall outside the window and are never checked. + // Too long: increases query cost on high-frequency data streams. +| STATS last_seen = MAX(@timestamp) BY host.name // Find the most recent event timestamp per host +| WHERE last_seen < NOW() - 15 minutes // Silence threshold: set slightly above your expected maximum reporting interval + // to avoid false positives from brief gaps +| KEEP host.name, last_seen // Each returned row is a silent host +``` + +Every row returned is a host that has gone silent. The query result itself drives the alert; no separate alert condition is needed. + +:::{note} +This pattern detects specific silent sources. For controlling what the rule records when the entire base query returns zero rows, for example when an index is completely empty, refer to [No-data handling](configure-no-data-handling.md). +::: diff --git a/explore-analyze/alerting/kibana-alerting-experimental/rules/esql-persistent-breach.md b/explore-analyze/alerting/kibana-alerting-experimental/rules/esql-persistent-breach.md new file mode 100644 index 0000000000..d6471397ed --- /dev/null +++ b/explore-analyze/alerting/kibana-alerting-experimental/rules/esql-persistent-breach.md @@ -0,0 +1,45 @@ +--- +navigation_title: Persistent breach detection +applies_to: + stack: experimental 9.5+ + serverless: experimental +products: + - id: kibana +description: "Detect conditions that persist across consecutive time buckets in Kibana's experimental alerting system using ES|QL bucket counting." +--- + +# Persistent breach detection in the {{alerting-v2-system}} [persistent-breach] + +A persistent breach condition detects a metric that stays above a threshold across several consecutive time buckets, for example CPU above 90% in all 10 of the last 10 five-minute windows. This filters out transient spikes and fires only when a problem has been sustained. + +{{esql}} can express this with bucket counting: + +```esql +FROM metrics-* +| WHERE @timestamp >= NOW() - 50 minutes // Lookback must cover all 10 buckets (10 × 5 min = 50 min) +| EVAL bucket = BUCKET(@timestamp, 5 minutes) // Assign each event to its 5-minute time bucket +| STATS + total_buckets = COUNT_DISTINCT(bucket), // How many distinct buckets exist in the window + exceeding_buckets = COUNT_DISTINCT( + CASE(system.cpu.total.pct > 0.90, bucket, null) // Count only buckets where CPU exceeded threshold; + ) // null values are excluded by COUNT_DISTINCT + BY host.name +| WHERE total_buckets >= 10 // Require a full window of data before firing; + AND exceeding_buckets == total_buckets // every bucket must have breached +| KEEP host.name, total_buckets, exceeding_buckets +``` + +The rule's lookback window must cover all the buckets you want to check. In this example, 10 five-minute buckets requires at least 50 minutes of lookback. + +## Handling gaps in data + +If any bucket is missing because the host stopped reporting briefly mid-window, `total_buckets` drops below 10 and the condition doesn't fire. This is a deliberate safety check: a host that went silent for one bucket is treated as "we don't have enough data to confirm persistence" rather than "breach." + +If you want to allow some gaps, replace `exceeding_buckets == total_buckets` with a ratio or a minimum count: + +```esql +| WHERE total_buckets >= 8 // Tolerate up to 2 missing buckets + AND exceeding_buckets >= total_buckets * 0.9 // 90% of present buckets must have breached +``` + +Design the query so that gaps in reporting produce the behavior you want before deploying it. diff --git a/explore-analyze/alerting/kibana-alerting-experimental/rules/esql-query-patterns.md b/explore-analyze/alerting/kibana-alerting-experimental/rules/esql-query-patterns.md new file mode 100644 index 0000000000..604ff57bd9 --- /dev/null +++ b/explore-analyze/alerting/kibana-alerting-experimental/rules/esql-query-patterns.md @@ -0,0 +1,21 @@ +--- +navigation_title: ES|QL query patterns +applies_to: + stack: experimental 9.5+ + serverless: experimental +products: + - id: kibana +description: "ES|QL query patterns for rules in Kibana's experimental alerting system, from basic event filters to SLO burn rates, silent source detection, and persistent breach checks." +--- + +# {{esql}} query patterns for rules in the {{alerting-v2-system}} [esql-query-patterns] + +The following pages cover {{esql}} query patterns for rules in the {{alerting-v2-system}}, ordered from the simplest starting point to advanced use cases. Start with [Your first rule query](esql-first-rule-query.md) if you're new, or jump to the pattern you need. + +| Pattern | What it solves | Complexity | +| --- | --- | --- | +| [Your first rule query](esql-first-rule-query.md) | Alert on any individual event that matches a condition. No aggregation needed. | Beginner | +| [Threshold queries](esql-threshold-queries.md) | Alert when a metric crosses a limit. Covers single-series and grouped (per-host, per-service) rules. | Beginner | +| [No-data detection](esql-no-data-detection.md) | Alert when a specific host or data source stops reporting. | Intermediate | +| [SLO burn rate](esql-slo-burn-rate.md) | Alert when error budget consumption exceeds safe rates across multiple time windows. | Advanced | +| [Persistent breach detection](esql-persistent-breach.md) | Alert when a condition has been continuously true across consecutive time buckets. | Advanced | diff --git a/explore-analyze/alerting/kibana-alerting-experimental/rules/esql-slo-burn-rate.md b/explore-analyze/alerting/kibana-alerting-experimental/rules/esql-slo-burn-rate.md new file mode 100644 index 0000000000..98a53f0e47 --- /dev/null +++ b/explore-analyze/alerting/kibana-alerting-experimental/rules/esql-slo-burn-rate.md @@ -0,0 +1,75 @@ +--- +navigation_title: SLO burn rate +applies_to: + stack: experimental 9.5+ + serverless: experimental +products: + - id: kibana +description: "Detect SLO error budget burn across multiple time windows in Kibana's experimental alerting system using ES|QL conditional aggregation." +--- + +# SLO burn rate detection in the {{alerting-v2-system}} [slo-burn-rate-query] + +An SLO burn rate query asks a different question than a basic threshold: are you consuming your error budget faster than you can afford to? Rather than checking a single metric at a fixed limit, it calculates error rates across multiple time windows simultaneously and assigns a severity level based on how fast the budget is being consumed. + +## Single-tier query + +This version checks one window pair and fires only at `critical` severity. It shows the core two-window pattern with the minimum number of moving parts. + +```esql +FROM metrics-* +| WHERE @timestamp >= NOW() - 1 hour // Cover the longest window in the pair +| STATS + errors_5m = COUNT_IF(outcome == "failure" AND @timestamp >= NOW() - 5 minutes), + total_5m = COUNT_IF(@timestamp >= NOW() - 5 minutes), + errors_1h = COUNT_IF(outcome == "failure"), + total_1h = COUNT(*) + BY slo.id // Each SLO is evaluated independently +| EVAL + burn_5m = errors_5m / total_5m, + burn_1h = errors_1h / total_1h +| WHERE burn_5m > 14.4 AND burn_1h > 14.4 // Both windows must exceed the threshold to fire; + // the short window detects the spike, the long window confirms it's sustained +| EVAL severity = "critical" +| KEEP slo.id, severity, burn_5m, burn_1h +``` + +The 14.4× multiplier reflects the rate at which a service would exhaust a 99.9% monthly error budget in one hour. Adjust it to match your SLO target. + +## Multi-tier query + +This version extends the single-tier pattern to detect both `critical` and `high` severity in one pass. Each severity level uses its own window pair with different time scales and thresholds. + +```esql +FROM metrics-* +| WHERE @timestamp >= NOW() - 3 days // Cover the longest window pair used below. + // Keep this in sync with the rule's lookback setting. +| STATS + // CRITICAL window pair: 5 min fast signal, 1 hour sustained confirmation + errors_5m = COUNT_IF(outcome == "failure" AND @timestamp >= NOW() - 5 minutes), + total_5m = COUNT_IF(@timestamp >= NOW() - 5 minutes), + errors_1h = COUNT_IF(outcome == "failure" AND @timestamp >= NOW() - 1 hour), + total_1h = COUNT_IF(@timestamp >= NOW() - 1 hour), + // HIGH window pair: 30 min fast signal, 6 hours sustained confirmation + errors_30m = COUNT_IF(outcome == "failure" AND @timestamp >= NOW() - 30 minutes), + total_30m = COUNT_IF(@timestamp >= NOW() - 30 minutes), + errors_6h = COUNT_IF(outcome == "failure" AND @timestamp >= NOW() - 6 hours), + total_6h = COUNT_IF(@timestamp >= NOW() - 6 hours) + BY slo.id +| EVAL + burn_5m = errors_5m / total_5m, + burn_1h = errors_1h / total_1h, + burn_30m = errors_30m / total_30m, + burn_6h = errors_6h / total_6h +| EVAL severity = CASE( + burn_5m > 14.4 AND burn_1h > 14.4, "critical", + burn_30m > 6.0 AND burn_6h > 6.0, "high", + "none" + ) +| WHERE severity != "none" +| KEEP slo.id, severity, burn_5m, burn_1h, burn_30m, burn_6h +``` + +The rule's lookback window must cover the longest window in the query. In this example that's 3 days, driven by the 6-hour window pair. + +The `severity` column in `KEEP` maps directly to the `severity` field on each resulting alert episode. For accepted values and matching rules, refer to [Severity](configure-rule-severity.md). diff --git a/explore-analyze/alerting/kibana-alerting-experimental/rules/esql-threshold-queries.md b/explore-analyze/alerting/kibana-alerting-experimental/rules/esql-threshold-queries.md new file mode 100644 index 0000000000..9ef349ed7c --- /dev/null +++ b/explore-analyze/alerting/kibana-alerting-experimental/rules/esql-threshold-queries.md @@ -0,0 +1,59 @@ +--- +navigation_title: Threshold queries +applies_to: + stack: experimental 9.5+ + serverless: experimental +products: + - id: kibana +description: "Write ES|QL threshold queries for rules in Kibana's experimental alerting system. Covers single-series and grouped rules using STATS aggregation." +--- + +# Threshold queries in the {{alerting-v2-system}} [esql-threshold-queries] + +A threshold query aggregates your data first, then applies an alert condition to the result. Use this pattern when a single matching event isn't enough to warrant an alert. You want to know whether a metric (count, rate, average) has crossed a limit over a time window. + +This page covers two variants: a single-series threshold that produces one result for all your data, and a grouped threshold that tracks each subject (host, service, user) independently. + +## Single-series threshold + +This query counts HTTP 5xx responses and error rates over the lookback window and alerts when the error rate for any service exceeds 10%. + +```esql +FROM logs-* +| WHERE @timestamp >= ?_tstart AND @timestamp < ?_tend // Scope to the rule's configured lookback window +| STATS + error_count = COUNT_IF(http.response.status_code >= 500), // Count only error responses + total_count = COUNT(*) // Count all requests for the denominator + BY service.name +| EVAL error_rate = error_count / total_count // Compute error rate as a fraction (0–1) +| WHERE error_rate > 0.10 // Alert condition: services above 10% error rate are breaches +| KEEP service.name, error_rate, error_count, total_count +``` + +One window, one aggregate per service, one threshold check. The result is either a breach or no breach for each service. + +## Grouped threshold + +Adding `BY` to the `STATS` command splits the result into one row per unique field value. Each row is evaluated independently against the alert condition, and each breach creates its own alert series. + +This query tracks average CPU usage per host and alerts when any host exceeds 90%: + +```esql +FROM metrics-* +| WHERE @timestamp >= ?_tstart AND @timestamp < ?_tend +| STATS avg_cpu = AVG(system.cpu.total.pct) BY host.name // One result row per host +| WHERE avg_cpu > 0.90 // Each host above the threshold is a breach +| KEEP host.name, avg_cpu +``` + +Without the `BY host.name` clause, the query would produce a single average across all hosts combined. A cluster-wide spike might push the average above the threshold even if no single host is the problem. Grouping by host gives each host its own alert episode with its own lifecycle. If host A recovers but host B stays critical, those are tracked separately. + +The `BY` fields you use here must match the grouping configuration in the rule. Refer to [Rule grouping](configure-rule-grouping.md) for how to align them. + +## Next steps + +If both of these patterns are too simple for your use case, the next pages cover more specific detection problems: + +- [No-data detection](esql-no-data-detection.md): when a host or source stops reporting entirely +- [SLO burn rate](esql-slo-burn-rate.md): when you need to check error budget consumption across multiple time windows +- [Persistent breach detection](esql-persistent-breach.md): when you only want to alert if a condition has been continuously true diff --git a/explore-analyze/alerting/kibana-alerting-experimental/rules/rule-event-field-reference.md b/explore-analyze/alerting/kibana-alerting-experimental/rules/rule-event-field-reference.md new file mode 100644 index 0000000000..e955317ed5 --- /dev/null +++ b/explore-analyze/alerting/kibana-alerting-experimental/rules/rule-event-field-reference.md @@ -0,0 +1,70 @@ +--- +navigation_title: Rule and event fields +applies_to: + stack: experimental 9.5+ + serverless: experimental +products: + - id: kibana +description: "Field reference for .rule-events documents in Kibana's experimental alerting system. Covers signal and alert base fields, episode fields, and the append-only data stream behavior." +--- + +# Rule event and field reference in the {{alerting-v2-system}} [rule-reference] + +This page is a field reference for `.rule-events` documents written by the {{alerting-v2-system}}. For rule configuration settings, refer to [Configure a rule](configure-a-rule.md). + + +:::{important} +The `.rule-events` and `.alert-actions` data streams are [system indices](/reference/glossary/index.md#glossary-system-index). {{kib}} manages their versioning, retention, and lifecycle through ILM. Older backing indices are deleted automatically when the retention window expires. Do not change mappings or index settings for these streams yourself. +::: + +## Rule event documents + +Each time a rule evaluates, {{kib}} writes one document per matched series to `.rule-events`. The `type` field determines the document kind: + +- **signal:** A point-in-time record that the query matched. Useful for querying history or chaining into follow-on rules. Signal documents don't include `episode.*` fields. +- **alert:** A lifecycle-tracked episode visible in the alert inbox, episode details, and triage views. Alert documents include `episode.*` fields and represent a breach that stays open until the condition clears. + +Both kinds share the base fields below. Only `alert` documents add the [Episode fields](#episode-fields) listed further down. + +:::{note} +`.rule-events` is a data stream, so it is append-only. A new document is written on every rule evaluation; existing documents are never updated. Each document is a snapshot of that moment: the `episode.status` field records the lifecycle stage the episode was in at that evaluation. To view the full history of an episode, query all documents that share the same `episode.id`. + +::: + +### Signal and alert fields + +These fields appear on all `.rule-events` documents, regardless of whether the rule is in Signal or Alert mode. + +| Field | Type | Required | Description | +|---|---|---|---| +| `@timestamp` | date | Yes | When this document was written to `.rule-events`. | +| `scheduled_timestamp` | date | No | Scheduled execution time for this rule run. | +| `rule.id` | keyword | Yes | Rule identifier. | +| `rule.version` | long | Yes | Rule version at the time this event was emitted. | +| `group_hash` | keyword | Yes | Series identity key for grouped evaluations. | +| `data` | flattened | Yes | Payload from the {{esql}} query output. Shape depends on your rule. | +| `status` | keyword | Yes | One of: `breached`, `recovered`, `no_data`. | +| `source` | keyword | Yes | Origin of this event. Product-specific identifier. | +| `type` | keyword | Yes | `signal` or `alert`. Application field on each rule event document written by {{kib}}. | + +:::{admonition} Fields not stored as a dedicated column +There's no top-level or nested `duration` field on `.rule-events` documents. For triage or reporting, derive duration from the alert UI or your own queries over timestamps and episode identifiers. + +::: + +### Episode fields [episode-fields] + +These fields are stored in `.rule-events`, on the same document as the base fields, when the rule runs in Alert mode. They only appear on documents with `type: alert` and carry the lifecycle state for the episode associated with the matched series. + +| Field | Type | Description | +|---|---|---| +| `episode.id` | keyword | Episode identifier for this series. | +| `episode.status` | keyword | One of: `inactive`, `pending`, `active`, `recovering`. | +| `episode.status_count` | long | Count of consecutive evaluations in the current `episode.status`. Only set when `episode.status` is `pending` or `recovering`. | +| `severity` | keyword | Severity level from the most recent breached event. One of: `info`, `low`, `medium`, `high`, `critical`. Not set when the query output does not include a `severity` column, or when the value does not match a recognized level. Never set on `recovered` or `no_data` events. | diff --git a/explore-analyze/alerting/kibana-alerting-experimental/rules/use-rule-builder.md b/explore-analyze/alerting/kibana-alerting-experimental/rules/use-rule-builder.md new file mode 100644 index 0000000000..216c6c4898 --- /dev/null +++ b/explore-analyze/alerting/kibana-alerting-experimental/rules/use-rule-builder.md @@ -0,0 +1,17 @@ +--- +navigation_title: Use the rule builder +applies_to: + stack: experimental 9.5+ + serverless: experimental +products: + - id: kibana +description: "Create Threshold Alert rules in Kibana's experimental alerting system using a guided form that generates ES|QL automatically." +--- + +# Use the rule builder in the {{alerting-v2-system}} [use-rule-builder] + +The rule builder provides a guided form for creating rules without writing {{esql}} by hand. The builder generates the {{esql}} query automatically from structured inputs for the data source, aggregation, filters, and alert conditions. + +## Threshold Alert [use-threshold-alert-builder] + +Threshold Alert is the only rule type available in the rule builder. Use it to monitor one or more metrics and alert when they cross a threshold, with multi-condition support and custom aggregations. diff --git a/explore-analyze/alerting/kibana-alerting-experimental/rules/view-manage-rules.md b/explore-analyze/alerting/kibana-alerting-experimental/rules/view-manage-rules.md new file mode 100644 index 0000000000..fda179e400 --- /dev/null +++ b/explore-analyze/alerting/kibana-alerting-experimental/rules/view-manage-rules.md @@ -0,0 +1,52 @@ +--- +navigation_title: View and manage rules +applies_to: + stack: experimental 9.5+ + serverless: experimental +products: + - id: kibana +description: "Search, filter, sort, and bulk-manage rules from the rules list in Kibana's experimental alerting system. Inline editing and a summary flyout let you update settings without a full page reload." +--- + +# View and manage rules in the {{alerting-v2-system}} [manage-rules] + +Rule management is part of the {{alerting-v2-system}} in {{kib}}. This page covers how to find and filter rules in the rules list, quick-edit settings without leaving the list, use the rule summary flyout, and navigate the rule details page including the alert activity timeline. + +## Find and filter rules [find-filter-rules] + +Use the search bar to find rules by name or description. Each space-separated term is matched independently using prefix matching. Tags and grouping fields appear in results but aren't searchable. + +Combine text search with filter controls to narrow by rule type, status, or tags. Select any column header to sort, or use bulk actions to enable, disable, or delete multiple rules at once. + +## Edit a rule inline [quick-edit-rule] + +To update common rule settings without opening the full rule details page, use the inline edit option on any row in the rules list. The inline editor also opens from the rule summary flyout header. + +Use inline edit when you need to adjust metadata or scheduling settings quickly without navigating away from the list. + +## Inspect a rule with the summary flyout [rule-summary-flyout] + +To inspect a rule without navigating away from the rules list, select the expand icon on any row. The rule summary flyout opens alongside the list and shows a snapshot of the rule: its status, last run time, recent alert episode activity, and quick actions such as enable, disable, and snooze. + +Use the flyout when you want to confirm a rule is healthy or take a quick action without committing to a full page load. To open the complete rule configuration with all settings and edit controls, select the rule name in the table row or in the flyout header. + +## Review rule configuration and activity [rule-details-page] + +The rule details page is organized into tabs that let you review a rule's configuration and activity history. + +- **Overview** (Alert mode only): Shows an alert activity timeline for each series the rule tracks. The timeline displays a color-coded history of alert episode state transitions per series, along with summary statistics: alert episodes started, recovered, still open, and median duration. A link to view all matching alert episodes is available, filtered to the current rule and time range. Lane labels appear only for grouped rules. The overview tab is not shown for Signal-mode rules, which don't open alert episodes. +- **Conditions**: The full {{esql}} base query, alert condition, schedule, lookback, grouping, and recovery settings. +- **Runbook**: If the rule has an investigation guide, it appears here. + +Use **Edit** to modify the rule, or the actions menu to enable, disable, clone, or delete it. + +## Disable or snooze a rule [disable-snooze-rule] + +Use **Disable** when you want the rule to stop running entirely until you re-enable it. This is different from snoozing, which suppresses notifications or quiets a series or policy without stopping rule evaluation. + + + diff --git a/explore-analyze/alerting/kibana-alerting-experimental/rules/yaml-rule-schema-reference.md b/explore-analyze/alerting/kibana-alerting-experimental/rules/yaml-rule-schema-reference.md new file mode 100644 index 0000000000..b8af5ea4e5 --- /dev/null +++ b/explore-analyze/alerting/kibana-alerting-experimental/rules/yaml-rule-schema-reference.md @@ -0,0 +1,130 @@ +--- +navigation_title: YAML rule schema reference +applies_to: + stack: experimental 9.5+ + serverless: experimental +products: + - id: kibana +description: "YAML rule definitions in Kibana's experimental alerting system support fields for detection mode, schedule, query, grouping, and recovery. Reference tables list all valid field values." +--- + +# YAML rule schema reference for the {{alerting-v2-system}} [yaml-rule-schema-reference] + + +YAML rule schema is part of the {{alerting-v2-system}} in {{kib}}. This page lists valid fields for YAML rule definitions. For examples and authoring guidance, refer to [Create an ES|QL rule](create-esql-rule.md#yaml-editor). + +## Base rule fields + +These four fields are required on every rule, regardless of format or mode. The value of `query.format` determines which additional query fields are required. + +| Field | Type | Accepted values | Description | +|---|---|---|---| +| `kind` | string | `alert` or `signal` | Whether the rule tracks ongoing episodes (`alert`) or records point-in-time observations (`signal`). | +| `metadata.name` | string | Any string | The name of the rule. Max 256 characters. | +| `schedule.every` | duration | Any duration string | How often the rule runs. For example: `5s`, `1m`, `5m`. Minimum interval applies. | +| `query.format` | string | `composed` or `standalone` | The query structure the rule uses. `standalone` means each condition (breach, recovery, no-data) is a separate, self-contained ES\|QL query. `composed` means you write one base query and each condition is a pipe segment appended to it. The UI always creates `standalone` rules. | + +### Fields for `query.format: composed` + +Use `composed` when breach, recovery, and no-data conditions all start from the same data shape. Define that shape once in the base query and each condition adds only what differs. + +| Field | Type | Description | +|---|---|---| +| `query.base` | ES\|QL string | Base query that runs on every evaluation. Time filters are applied automatically using the lookback window. Required. | +| `query.breach.segment` | ES\|QL segment string | Appendable ES\|QL segment for breach detection. Written as a pipe command, for example `\| WHERE count > 5`. Required. | +| `query.recovery.segment` | ES\|QL segment string | Appendable ES\|QL segment for recovery detection. Required when `recovery_strategy` is `query`. | + +### Fields for `query.format: standalone` + +Use `standalone` when conditions need full independence. Each query can target different indices, apply different filters, or return a completely different shape. + +| Field | Type | Description | +|---|---|---| +| `query.breach.query` | Full ES\|QL string | Full ES\|QL query for breach detection. Required. | +| `query.recovery.query` | Full ES\|QL string | Full ES\|QL query for recovery detection. Required when `recovery_strategy` is `query`. | +| `query.no_data.query` | Full ES\|QL string | Full ES\|QL query that detects presence of data. Required when `no_data_strategy` is not `none`. Only supported on `standalone` format. | + +## Metadata fields + +These optional fields add descriptive information to a rule for identification, ownership, and filtering. None affect rule evaluation behavior. + +| Field | Type | Accepted values | Description | +|---|---|---|---| +| `metadata.description` | string | Any string | Optional description of what the rule monitors. Max 1,024 characters. | +| `metadata.owner` | string | Any string | Team or person responsible for the rule. Max 256 characters. | +| `metadata.tags` | array of strings | Array of strings | Labels for filtering and organization. Max 20 tags, each max 128 characters. | + +## Schedule fields + +These fields control how far back each evaluation looks and which timestamp field is used for the time range filter. Both are optional, but omitting `schedule.lookback` means the query runs without a time bound. + +| Field | Type | Accepted values | Description | +|---|---|---|---| +| `schedule.lookback` | duration | Any duration string | How far back in time the query searches on each run. For example: `5m`, `24h`. | +| `time_field` | string | Any field name | The timestamp field used for the lookback window filter. Max 128 characters. Defaults to `@timestamp`. | + +## Recovery strategy [recovery-strategy] + +The `recovery_strategy` field is optional. When omitted, the rule emits no recovery events and active alert episodes don't close automatically. + +| Field | Type | Accepted values | Description | +|---|---|---|---| +| `recovery_strategy` | string | `no_breach`, `query`, or `none` | How recovery is detected.

-`no_breach`: Recovers an episode when its active group no longer appears in the breach results.
- `query`: Evaluates a separate recovery query defined in `query.recovery.segment` (composed) or `query.recovery.query` (standalone)
- `none`: Turns off recovery. | + +:::{note} +Signal-mode rules (`kind: signal`) must omit `recovery_strategy` or set it to `none`. Any other value fails validation. +::: + +## State transition fields + +Only valid when `kind: alert`. Controls how many consecutive detections are required before an episode becomes active or recovers. + +| Field | Type | Accepted values | Description | +|---|---|---|---| +| `state_transition.pending_operator` | string | `AND` or `OR` | Whether both the count and timeframe must be met (`AND`) or either one (`OR`) before becoming active. | +| `state_transition.pending_count` | integer | Integer ≥ 0 | Number of consecutive breaches required before the episode becomes active. | +| `state_transition.pending_timeframe` | duration | Any duration string | Time window within which the breach count must be met. For example: `5m`. | +| `state_transition.recovering_operator` | string | `AND` or `OR` | Whether both the count and timeframe must be met (`AND`) or either one (`OR`) before recovering. | +| `state_transition.recovering_count` | integer | Integer ≥ 0 | Number of consecutive clear evaluations required before the episode recovers. | +| `state_transition.recovering_timeframe` | duration | Any duration string | Time window within which the recovery count must be met. For example: `5m`. | + +## Grouping fields + +Use grouping to split a rule's detections into independent series, one per unique combination of field values. This lets a single rule track multiple subjects without creating a separate rule for each, for example, tracking CPU usage per host. Each series maintains its own alert episode lifecycle. + +| Field | Type | Accepted values | Description | +|---|---|---|---| +| `grouping.fields` | array of strings | Array of field names | Fields to group results by. Each unique combination becomes its own series. Max 16 fields, each max 256 characters. | + +## No-data strategy + +Use `no_data_strategy` to control what the rule does when an evaluation returns no results. This matters when data sources can go silent: without a no-data strategy, a quiet data source and a healthy one look identical to the rule. + +| Field | Type | Accepted values | Description | +|---|---|---|---| +| `no_data_strategy` | string | `emit`, `last_known_status`, `recover`, or `none` | Optional. What happens when the rule evaluates and returns no results. `emit` records a no-data event. `last_known_status` holds the last known status. `recover` forces recovery. `none` disables no-data detection. | + +:::{note} +No-data detection is only supported with `query.format: standalone`. Setting `no_data_strategy` to any active value on a `composed` rule has no effect because `query.no_data.query` can only be defined on a standalone query. Signal-mode rules (`kind: signal`) must omit `no_data_strategy` or set it to `none`. +::: + +## Artifact fields + +Artifacts let you attach reference material directly to a rule, such as a runbook. The content is stored with the rule and displayed in the rule detail view so responders have context when an alert fires. All artifact fields are optional. + +| Field | Type | Accepted values | Description | +|---|---|---|---| +| `artifacts[].id` | string | Any string | Artifact identifier. Required. Max 256 characters. | +| `artifacts[].type` | string | Any string | The type of artifact being attached. For example: `runbook`. | +| `artifacts[].value` | string | Any string | The content of the artifact. Accepts markdown. Runbooks are rendered as markdown in the rule detail view. | + +## Duration format [duration-format] + +All duration fields accept the following units: + +| Unit | Example | Meaning | +|---|---|---| +| `s` | `30s` | Seconds | +| `m` | `5m` | Minutes | +| `h` | `1h` | Hours | +| `d` | `7d` | Days | diff --git a/explore-analyze/toc.yml b/explore-analyze/toc.yml index 46aad620e1..50531ec8bd 100644 --- a/explore-analyze/toc.yml +++ b/explore-analyze/toc.yml @@ -382,6 +382,35 @@ toc: - file: report-and-share/reporting-troubleshooting-pdf.md - file: alerting.md children: + - file: alerting/kibana-alerting-experimental/rules.md + children: + - file: alerting/kibana-alerting-experimental/rules/create-a-rule.md + children: + - file: alerting/kibana-alerting-experimental/rules/create-esql-rule.md + children: + - file: alerting/kibana-alerting-experimental/rules/yaml-rule-schema-reference.md + - file: alerting/kibana-alerting-experimental/rules/create-rule-ai-agent.md + - file: alerting/kibana-alerting-experimental/rules/use-rule-builder.md + - file: alerting/kibana-alerting-experimental/rules/create-rule-from-discover.md + - file: alerting/kibana-alerting-experimental/rules/configure-a-rule.md + children: + - file: alerting/kibana-alerting-experimental/rules/configure-rule-mode.md + - file: alerting/kibana-alerting-experimental/rules/configure-rule-query.md + - file: alerting/kibana-alerting-experimental/rules/configure-rule-severity.md + - file: alerting/kibana-alerting-experimental/rules/configure-rule-grouping.md + - file: alerting/kibana-alerting-experimental/rules/configure-rule-schedule.md + - file: alerting/kibana-alerting-experimental/rules/configure-rule-thresholds.md + - file: alerting/kibana-alerting-experimental/rules/configure-no-data-handling.md + - file: alerting/kibana-alerting-experimental/rules/configure-rule-tags.md + - file: alerting/kibana-alerting-experimental/rules/view-manage-rules.md + - file: alerting/kibana-alerting-experimental/rules/esql-query-patterns.md + children: + - file: alerting/kibana-alerting-experimental/rules/esql-first-rule-query.md + - file: alerting/kibana-alerting-experimental/rules/esql-threshold-queries.md + - file: alerting/kibana-alerting-experimental/rules/esql-no-data-detection.md + - file: alerting/kibana-alerting-experimental/rules/esql-slo-burn-rate.md + - file: alerting/kibana-alerting-experimental/rules/esql-persistent-breach.md + - file: alerting/kibana-alerting-experimental/rules/rule-event-field-reference.md - file: alerting/alerts.md children: - file: alerting/alerts/alerting-getting-started.md