feat: multi-vantage confirmation via the peers

The headline of 0.6.0: when a monitor confirms down locally, the peers
probe the same target from their side before the alert goes out, and
the alert carries the verdict - "confirmed down from 3/3 vantage
points" (real outage) vs "seen UP by hora-b - network issue near this
node?". Two Raspberry Pi at two homes become a distributed Pingdom.

Built to be boring under failure:

- Strictly fail-open: peers being slow, broken, unreachable or
  misconfigured never block, delay past a hard 10s concurrent
  deadline, or suppress the alert. The worst outcome is an alert
  without the annotation - exactly what Hora sent before. The incident
  record is written before the peers are consulted.

- Never a proxy: POST /api/peer/probe only probes targets present in
  the responder's OWN config (matched on kind + target, probed with
  its own timeout/assertions/proxy and no retries), so a leaked token
  cannot turn a peer into an SSRF relay. The requesting peer's
  listen_token is strictly required (the id alone never authorizes),
  unknown peers answer exactly like wrong tokens, and response bodies
  are read bounded (16 KiB) like the witness reports.

- A disputed down still alerts: a peer seeing the target up softens
  the message, never silences it - geo-partial outages are real.

- Plumbing: Event::Down gains a vantage field rendered by all ten
  backends; DownAlert carries it through the coalescer; peer requests
  ride the shared plain client, never a monitor's proxy client.

Enabled with [health] confirm_with_peers = true (per-monitor
override), validated at load (needs [health] + a peer with ping_url;
push monitors can't opt in). Tested by a two-real-nodes round trip
over live HTTP sockets covering agreement, dispute, wrong-token and
dead-peer modes - plus a manual two-daemon smoke test end to end.

Author: uplg

CHANGELOG.md

@@ -5,10 +5,34 @@ All notable changes to this project are documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
-## [Unreleased]
+## [0.6.0] - 2026-06-12
 
 ### Added
 
+- **Multi-vantage confirmation** (`confirm_with_peers`, the headline of
+  0.6.0): when a monitor confirms down locally, the peers probe the same
+  target from their side before the alert goes out, and the alert carries
+  the verdict - *"confirmed down from 3/3 vantage points"* (a real outage)
+  vs *"seen UP by hora-b - network issue near this node?"*. Two Raspberry Pi
+  at two homes become a distributed Pingdom. Built to be boring under
+  failure:
+  - **Strictly fail-open**: peers being slow, broken, unreachable or
+    misconfigured never block, delay past a hard 10s deadline (probes run
+    concurrently), or suppress the alert - the worst outcome is an alert
+    without the annotation, exactly what Hora sent before. The incident
+    record is written *before* the peers are consulted.
+  - **Never a proxy**: the new `POST /api/peer/probe` only probes targets
+    present in the responder's *own* configuration (matched on kind +
+    target, probed with its own settings), so a leaked token cannot turn a
+    peer into an SSRF relay. It strictly requires the requesting peer's
+    `listen_token` - the id alone never authorizes - and unknown peers are
+    indistinguishable from wrong tokens.
+  - **A disputed down still alerts**: a peer seeing the target up softens
+    the message, never silences it - geo-partial outages are real outages.
+  - Enabled globally with `[health] confirm_with_peers = true`, overridden
+    per monitor; peer probe requests never ride a monitor's proxy; verified
+    end-to-end by a two-real-nodes test over live HTTP sockets.
+
 - **Per-group status pages** (`/status/{group}`): one display group's
   monitors, nothing else - lightweight multi-tenancy for an operator hosting
   several clients on one Hora. A per-group token (`server.group_tokens`)

Cargo.lock

@@ -8,7 +8,7 @@ members = [
 ]
 
 [workspace.package]
-version = "0.5.1"
+version = "0.6.0"
 edition = "2024"
 license = "MIT"
 publish = false

Cargo.toml

@@ -44,6 +44,12 @@ Full guides for everything below live in the
 **Alerting that never cries wolf**
 
 - **Down only after N consecutive failures**; degraded alerts opt-in.
+- **Multi-vantage confirmation** (`confirm_with_peers`) - before alerting, your
+  peers probe the same target from *their* side: _"confirmed down from 3/3
+  vantage points"_ (real outage) vs _"seen UP by hora-b"_ (network issue near
+  this node). Two Raspberry Pi at two homes become a distributed Pingdom -
+  and a peer only ever probes targets in its **own** config, never arbitrary
+  ones.
 - **Root-cause grouping** - a database taking ten services down sends **one**
   notification, annotated _"caused by X"_ / _"impacts: A, B, C"_ via `depends_on`.
 - **SLOs & error budgets** - `slo_uptime = 99.9` shows the budget left and arms
@@ -221,6 +227,7 @@ rate-limit settings are read once at startup and still require a restart.
 | `GET /api/monitors/{id}/latency?hours=24` | Latency samples `[{ "t", "latency_ms" }]` (404 if unknown). |
 | `POST /api/push/{id}` | Record a heartbeat for a push monitor. Send the token as an `X-Push-Token` header (preferred - it stays out of proxy access logs) or as `?token=…`. Optional `status=up\|down\|degraded`, `msg`, `ping`. 401 on a wrong token, 404 if not a push monitor. |
 | `POST /api/silence?monitors=api,web&duration=10m` | Mute alerts ad hoc (deploy hook): `monitors` is a comma-separated id list or `all`, `duration` like `10m`/`1h30m` (max 7d), optional `reason`. **Requires `server.auth_token`** (as `Authorization: Bearer` or `?token=`); without one configured the endpoint is closed. |
+| `POST /api/peer/probe` | Multi-vantage confirmation between Hora nodes: probe a target *present in this node's own config* and answer with the verdict. Requires the requesting peer's `listen_token` (`X-Push-Token`). Never probes arbitrary targets. |
 | `GET /api/monitors/{id}/heatmap.svg` | 28-day hours-by-days latency heatmap (SVG), colour relative to the monitor's median. |
 | `GET /api/badge/{id}/status` | Embeddable SVG status badge for a monitor. |
 | `GET /api/badge/{id}/uptime` | Embeddable SVG 24h-uptime badge for a monitor. |

README.md

@@ -4,6 +4,25 @@ Version-specific notes when moving between Hora releases. The general
 procedure (pull the new image, recreate the container, history lives on the
 `hora-data` volume) is in the [README](README.md#upgrade).
 
+## 0.5.1 → 0.6.0
+
+No behavioural changes: everything is opt-in. Notes:
+
+- **Multi-vantage confirmation is off by default.** To enable it, set
+  `[health] confirm_with_peers = true` on the node(s) that should ask, make
+  sure each `[[peers]]` entry has a `ping_url` (its origin is where probe
+  requests go) and that the *responding* node sets a `listen_token` for the
+  requester - `POST /api/peer/probe` strictly requires it. Both nodes must
+  know the monitor (same kind + target); a peer never probes a target outside
+  its own config.
+- **One new endpoint** (`POST /api/peer/probe`) exists even with the feature
+  off; without a matching authenticated peer it answers 401 to everyone.
+- New config keys (`confirm_with_peers`, plus 0.6.0's `[digest]`,
+  `server.group_tokens`) are rejected by older binaries
+  (`deny_unknown_fields`): **deploy the binary before the config**, on every
+  node of the mesh. One schema migration (the digest's `meta` table) applies
+  automatically.
+
 ## 0.5.0 → 0.5.1
 
 Two things to know:

UPGRADES.md

@@ -309,6 +309,16 @@ interval_secs = 90000        # max gap before "down" (~25h for a daily job)
 # quorum = true             # before alerting a peer down, ask the OTHER peers'
 #                           # /healthz: if any still sees it up, it's a partition
 #                           # (PeerLinkDegraded), not an outage. No-op with < 3 nodes.
+# confirm_with_peers = true # multi-vantage confirmation: when a MONITOR confirms
+#                           # down, ask the peers to probe the same target before
+#                           # alerting. The alert says "confirmed down from 3/3
+#                           # vantage points" (real outage) or "seen UP by hora-b"
+#                           # (network issue near this node). A peer only probes
+#                           # targets in its OWN config (never an arbitrary one),
+#                           # and its [[peers]] entry for this node must set a
+#                           # listen_token. Strictly fail-open: unreachable peers
+#                           # never block or suppress the alert. Override per
+#                           # monitor with confirm_with_peers = true/false.
 # heartbeat_url = "${HC_PING_URL}"   # optional extra dead-man target (e.g. healthchecks.io)
 
 # Each [[peers]] entry has two independent halves; set either or both:

config.example.toml

@@ -50,6 +50,8 @@ pub struct DownAlert {
     pub cause_name: Option<String>,
     pub impacted: Vec<String>,
     pub notify: Option<Vec<String>>,
+    /// Multi-vantage verdict, when peers were asked before this alert.
+    pub vantage: Option<String>,
 }
 
 /// How long a sent (or folded) down alert keeps absorbing late symptom
@@ -260,6 +262,7 @@ async fn send_down(notifier: &Notifiers, alert: &DownAlert) {
                 error: alert.error.as_deref(),
                 cause: alert.cause_name.as_deref(),
                 impacted: &impacted,
+                vantage: alert.vantage.as_deref(),
             },
             alert.notify.as_deref(),
         )
@@ -279,6 +282,7 @@ mod tests {
             cause_name: upstreams.first().map(|u| u.to_uppercase()),
             impacted: Vec::new(),
             notify: None,
+            vantage: None,
         }
     }
 

crates/hora-core/src/coalesce.rs

@@ -492,6 +492,14 @@ pub struct Health {
     /// `interval_secs` when locally healthy, alongside the peers' `ping_url`.
     #[serde(default)]
     pub heartbeat_url: Option<Secret>,
+    /// Multi-vantage confirmation: when a monitor confirms down locally, ask
+    /// the peers to probe the same target from their side before alerting,
+    /// and annotate the alert with the verdict ("confirmed from 3/3 vantage
+    /// points" vs "seen UP by hora-b"). Per-monitor `confirm_with_peers`
+    /// overrides this default either way. Strictly fail-open: peers being
+    /// slow, broken or unreachable never blocks or suppresses the alert.
+    #[serde(default)]
+    pub confirm_with_peers: bool,
 }
 
 impl Health {
@@ -568,16 +576,30 @@ impl Peer {
         if let Some(url) = &self.witness_url {
             return Some(url.clone());
         }
+        self.api_origin().map(|origin| format!("{origin}/healthz"))
+    }
+
+    /// The peer's `/api/peer/probe` URL for multi-vantage confirmation, derived
+    /// from the origin of `ping_url`. `None` when there is no `ping_url` (an
+    /// IN-only or external peer cannot be asked to probe).
+    #[must_use]
+    pub fn probe_url(&self) -> Option<String> {
+        self.api_origin()
+            .map(|origin| format!("{origin}/api/peer/probe"))
+    }
+
+    /// The `scheme://host[:port]` origin of the peer's API, from its `ping_url`.
+    fn api_origin(&self) -> Option<String> {
         let ping = self.ping_url.as_ref()?;
         let url = reqwest::Url::parse(ping.as_ref()).ok()?;
         let origin = url.origin();
-        origin
-            .is_tuple()
-            .then(|| format!("{}/healthz", origin.ascii_serialization()))
+        origin.is_tuple().then(|| origin.ascii_serialization())
     }
 }
 
-#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, Deserialize)]
+#[derive(
+    Debug, Clone, Copy, Default, PartialEq, Eq, Deserialize, serde::Serialize, utoipa::ToSchema,
+)]
 #[serde(rename_all = "lowercase")]
 pub enum Kind {
     #[default]
@@ -736,6 +758,10 @@ pub struct Monitor {
     /// public-suffix list, and the operator already knows the answer.
     #[serde(default)]
     pub domain_expiry: Option<String>,
+    /// Multi-vantage confirmation override for this monitor; unset = the
+    /// `[health].confirm_with_peers` default. See that field for semantics.
+    #[serde(default)]
+    pub confirm_with_peers: Option<bool>,
 }
 
 // Manual `Debug` (rather than derived) so credentials never leak: `target` and
@@ -780,6 +806,7 @@ impl std::fmt::Debug for Monitor {
             .field("dns_resolver", &self.dns_resolver)
             .field("cert_pin", &self.cert_pin)
             .field("domain_expiry", &self.domain_expiry)
+            .field("confirm_with_peers", &self.confirm_with_peers)
             .finish()
     }
 }
@@ -1182,8 +1209,52 @@ fn validate(config: &Config) -> anyhow::Result<()> {
 
     validate_peers(config, &seen, &channel_names)?;
 
-    // Digest: the cron must parse at load (not at the first missed send), and
-    // routes must name real channels, like a monitor's `notify`.
+    validate_confirm(config)?;
+    validate_digest(config, &channel_names)?;
+    Ok(())
+}
+
+/// Multi-vantage confirmation needs peers that can actually be asked: a
+/// `[health]` section and at least one peer with a `ping_url` (its API origin).
+/// A flag that can never confirm anything is a config mistake, not a plan.
+fn validate_confirm(config: &Config) -> anyhow::Result<()> {
+    let confirmable_peers = config.peers.iter().any(|peer| peer.probe_url().is_some());
+    let confirm_requested = config
+        .health
+        .as_ref()
+        .is_some_and(|health| health.confirm_with_peers)
+        || config
+            .monitors
+            .iter()
+            .any(|monitor| monitor.confirm_with_peers == Some(true));
+    if confirm_requested {
+        anyhow::ensure!(
+            config.health.is_some(),
+            "confirm_with_peers requires a [health] section"
+        );
+        anyhow::ensure!(
+            confirmable_peers,
+            "confirm_with_peers requires at least one peer with a ping_url \
+             (its API origin is where probe requests go)"
+        );
+    }
+    for monitor in &config.monitors {
+        anyhow::ensure!(
+            !(monitor.confirm_with_peers == Some(true) && monitor.kind == Kind::Push),
+            "monitor {}: confirm_with_peers needs an active probe (push monitors \
+             have no target a peer could probe)",
+            monitor.id
+        );
+    }
+    Ok(())
+}
+
+/// Digest: the cron must parse at load (not at the first missed send), and
+/// routes must name real channels, like a monitor's `notify`.
+fn validate_digest(
+    config: &Config,
+    channel_names: &std::collections::HashSet<&str>,
+) -> anyhow::Result<()> {
     if let Some(digest) = &config.digest {
         digest.schedule.parse::<croner::Cron>().map_err(|err| {
             anyhow::anyhow!("digest: invalid schedule {:?}: {err}", digest.schedule)
@@ -2219,6 +2290,92 @@ mod tests {
         assert!(error.contains("duplicate channel name"), "got: {error}");
     }
 
+    #[test]
+    fn confirm_with_peers_requires_health_and_a_probeable_peer() {
+        // Global flag without [health]: rejected.
+        let err = super::parse(
+            r#"
+            [page]
+            [server]
+            [[monitors]]
+            id = "web"
+            name = "Web"
+            target = "https://example.com"
+            interval_secs = 60
+            confirm_with_peers = true
+        "#,
+        )
+        .unwrap_err()
+        .to_string();
+        assert!(err.contains("[health]"), "{err}");
+
+        // [health] flag but no peer with a ping_url: rejected.
+        let err = super::parse(
+            r#"
+            [page]
+            [server]
+            [health]
+            id = "hora-a"
+            confirm_with_peers = true
+            [[peers]]
+            id = "hora-b"
+            name = "B"
+            expect_every_secs = 60
+        "#,
+        )
+        .unwrap_err()
+        .to_string();
+        assert!(err.contains("ping_url"), "{err}");
+
+        // A push monitor cannot opt in (nothing to probe).
+        let err = super::parse(
+            r#"
+            [page]
+            [server]
+            [health]
+            id = "hora-a"
+            [[peers]]
+            id = "hora-b"
+            name = "B"
+            ping_url = "https://b.example/api/push/hora-a"
+            [[monitors]]
+            id = "beat"
+            name = "Beat"
+            kind = "push"
+            interval_secs = 60
+            confirm_with_peers = true
+        "#,
+        )
+        .unwrap_err()
+        .to_string();
+        assert!(err.contains("push"), "{err}");
+
+        // The healthy shape parses, and probe_url derives from ping_url's origin.
+        let config = super::parse(
+            r#"
+            [page]
+            [server]
+            [health]
+            id = "hora-a"
+            confirm_with_peers = true
+            [[peers]]
+            id = "hora-b"
+            name = "B"
+            ping_url = "https://b.example:8443/api/push/hora-a"
+            [[monitors]]
+            id = "web"
+            name = "Web"
+            target = "https://example.com"
+            interval_secs = 60
+        "#,
+        )
+        .expect("valid");
+        assert_eq!(
+            config.peers[0].probe_url().as_deref(),
+            Some("https://b.example:8443/api/peer/probe")
+        );
+    }
+
     #[test]
     fn digest_validates_schedule_and_routes() {
         let base = r#"