fix(edge): 看门狗错误分类——网络不可达不退出，卡死类维持阈值

[anrenlx2025]

Problem

After a successful registration, the edge heartbeat watchdog treats every heartbeat / re-register failure as evidence of a stuck tunnel. During a public-DNS outage (or any network path failure to the cloud), that misclassification makes the edge process exit(1) after 5 failed heartbeats — an unnecessary restart: restarting cannot fix a network outage, event logs gain noise, and in-process state is lost. After the restart, the initial dial keeps backing off (or registration keeps retrying at edge_id=0), so an outage is not necessarily a continuous restart loop — but every threshold crossing is still a pointless exit.

This contradicts the initial-dial path, which retries unreachable clouds with unlimited backoff and never exits. The same process held two opposite policies for the same failure.

Root cause

heartbeatLoop counted all heartbeat RPC failures toward tunnelStuckThreshold without distinguishing "cannot reach the cloud" from "cloud reachable but tunnel unresponsive" (the watchdog's original target). Classifying raw net error shapes at the biz layer also misses the blackholed-connection case (context deadline exceeded), and stale stuck counts survived across unreachable gaps.

Fix

Two layers, so the tunnel Call exit is the single classification point.

tunnel layer — error response ladder (internal/pkg/tunnel/probe.go):

• Dial/DNS-stage failures map directly to the new sentinel ErrCloudUnreachable (fast path kept, %w chain preserved).
• Timeout-shaped failures (context deadline, including retry-abort wraps; an actively cancelled context is excluded even when a deadline error is embedded in the same chain) trigger one throttled reachability probe that uses the same transport semantics as the tunnel dialer: same resolved address, shared 10s dial budget, shared CA material and certificate verification, and an explicit handshake deadline so a SYN-answering middlebox cannot fake a green conclusion. The probe deliberately bypasses trackConnection so it cannot disturb the connection/generation state machine.
  • Probe red → ErrCloudUnreachable (whether to stay alive is the caller's decision).
  • Probe green → the transport is half-open rather than down: recycle it (generation-guarded) so the underlying RetryEnd redial recovers on the next tick, then return ErrRPCTimeout so the caller's accounting stays in charge.
• Concurrent probes are merged (singleflight) and a fresh conclusion is reused for 30s, preventing a probe storm during a partition.

biz layer — watchdog classification switch (internal/edgeagent/biz/agent.go):

• isUnreachableError now matches errors.Is(tunnel.ErrCloudUnreachable) instead of inspecting raw net error shapes — the biz layer no longer depends on the transport library's error forms.
• All three failure points (register retry, heartbeat, re-register) treat unreachable the same way: reset the stuck streak, log a WARN including the cloud address, stay alive, retry next tick. Stale counts no longer survive an unreachable gap (no more "4 stuck → outage → 1 stuck → exit").
• Stuck class (cloud reachable but the tunnel keeps failing) is unchanged: 5 consecutive failures + failed re-register → errTunnelStuck → exit(1).
• Accepted trade-off, documented in code: a flapping network can postpone the exit indefinitely; manager-side device_offline alerting covers that residual.

Config.CloudAddr is added purely for the unreachable WARN log, so operators can tell "network down, process alive" from "process dead" without digging through config files.

Testing

• Table-driven classification tests pin the full ladder: context cancel wins over embedded deadlines, dial-stage vs write-stage OpError, retry-abort wraps, DNS errors.
• Probe unit tests via an injection seam cover every classification row, throttle behavior (singleflight + TTL), and recycle-on-green with stale-generation rejection.
• Integration tests against a real three-layer geminio server: healthy round-trip; blackholed handler (RPC goes silent → probe green → transport recycled → redial recovers); port refused (direct unreachable, no probe).
• Watchdog tests: unreachable-gap streak reset sequence, fresh counting after recovery, unreachable register-retry staying alive.
• -race, -count=2 clean on both packages (matches CI).

Behavior summary

Failure	Before	After
DNS resolution failure	counted → exit(1)	WARN, stay alive, retry
TCP dial failure	counted → exit(1)	WARN, stay alive, retry
Blackholed connection (RPC timeout)	counted → exit(1)	probe red: stay alive; probe green: recycle transport + count as RPC timeout
Cloud reachable, heartbeat + re-register failing	exit(1) after 5	unchanged: exit(1) after 5
Auth-type remote errors	exit path preserved	unchanged

Notes

• GOOS=windows go build ./... currently fails in internal/edgeagent/cmdpolicy (syscall.SysProcAttr.Setpgid is Unix-only). The same failure exists on upstream/main (pre-existing) and is unrelated to the files touched here.
• Runtime fault-injection evidence for the review's acceptance scenarios will be attached in a follow-up comment.

Author confirmation

• I have read and agree to the project contribution guide. Guide: https://github.com/ongridio/ongrid/blob/main/CONTRIBUTING.md

[singchia]

感谢定位并处理这个问题。网络不可达不应被简单等同于 Tunnel 卡死，避免无意义的进程退出，这个方向是合理的。

不过，这次改动位于通用 heartbeatLoop，会改变 Linux systemd、Windows Service/supervisor、Kubernetes 以及直接运行的 Edge 在运行期的退出与自恢复决策，属于比较核心的跨平台运行时逻辑。因此本轮先记录阶段性 review，暂不作最终合并判断；我们会在真实运行时故障注入测试完成后继续评估。

当前代码 review

1. “不可达”分类目前仍偏窄

isUnreachableError 只识别 *net.DNSError 和 *net.OpError{Op:"dial"}，可以覆盖 DNS 失败、连接拒绝和 RetryEnd 重新拨号失败，但无法覆盖已有连接进入网络黑洞时常见的 context deadline exceeded。

可能仍然出现：

heartbeat 超时
  → 不属于 unreachable
  → consecutiveFail +1
  → re-register 再次超时
  → 连续 5 次后 errTunnelStuck
  → Edge 退出

不能简单把所有 timeout 都视为不可达，因为 Manager/Tunnel 卡死同样可能超时。更稳妥的是由 tunnel 层提供明确的 typed error/连接状态，或者在 RPC 超时后通过受控的 DNS/TCP reachability 检查辅助判断，避免 biz 层长期依赖 geminio 当前透传的底层错误形态。

2. 不可达期间没有清除旧的 stuck 计数

当前不可达分支只 continue，不会重置 consecutiveFail。因此存在：

4 次 stuck-class 失败
  → 一段时间网络不可达
  → 恢复后再出现 1 次 stuck-class 失败
  → 立即达到阈值并退出

这已经不再是“连续 5 次 Tunnel 卡死”。建议明确状态语义：不可达既然不构成卡死证据，就应重置旧计数，或单独维护连续 stuck-class 失败窗口。

3. 当前单元测试没有经过真实 Tunnel 错误链

测试通过 fake client 直接返回人工构造的 net.OpError，尚未覆盖：

• io.EOF → RetryEnd.reinit → dial error；
• tunnel.geminioClient.Call 的 %w 包装；
• 已建立连接后的黑洞/超时；
• 4 次 stuck → unreachable → 1 次 stuck；
• 故障恢复后 Edge 是否重新注册并继续上报。

单元测试对分类函数有价值，但不足以证明实际运行时行为。

运行时验收场景

后续评估至少会关注：

1. DNS 解析失败：Edge PID 不变化，恢复 DNS 后自动重连、注册和心跳；
2. TCP 连接拒绝/目标不可达：Edge 不退出，恢复后自动上线；
3. 已建立连接后的网络黑洞：确认最终错误类型、退出行为和恢复时间；
4. Frontier/Manager 重启：能够重连并重新注册，不误判为永久不可达；
5. 云端可达但 RPC/Tunnel 确实卡死：达到阈值后仍能正确退出并由服务管理器拉起；
6. 混合故障序列：旧失败计数不会跨越不可达阶段造成误退出；
7. Linux 与 Windows 托管环境分别观察 PID、重启次数、插件子进程状态和日志量。

另外，PR 描述中的“整个故障期间持续 restart storm”建议在运行时证据出来后再确认。按当前启动路径，重启后的 Dial() 会持续退避，或者在 EdgeID == 0 时持续重试注册，更可能表现为一次不必要的重启，而不一定是持续重启循环。

整体上认可问题方向，但由于这是 Edge 核心自愈策略，最终是否达到合并条件将在上述运行时测试之后继续评估。

[anrenlx2025]

Thanks for the thorough staged review. The branch has been reworked as a single squashed commit (5fdd173) addressing the three points directly:

1. Narrow classification — the tunnel Call exit is now the single classification point. It returns typed errors (ErrCloudUnreachable / ErrRPCTimeout): dial/DNS failures map straight to unreachable, while timeout-shaped failures (context deadline, including retry-abort wraps; active cancellation excluded) trigger one throttled reachability probe that reuses the tunnel's own transport semantics — same resolved address, shared 10s dial budget, shared CA material and certificate verification, explicit handshake deadline — before deciding. Probe red → unreachable (stay alive); probe green → the transport is half-open, so it is recycled for a redial and the caller gets ErrRPCTimeout so its own accounting stays in charge. Concurrent probes are singleflight-merged with a 30s conclusion TTL.

2. Stale stuck counts — all three failure points (register retry / heartbeat / re-register) now reset the streak on unreachable-class failures; the "4 stuck → outage → 1 stuck → exit" shape is covered by a dedicated test sequence.

3. Real error chains — integration tests now run against a real three-layer geminio server: healthy round-trip; blackholed handler (RPC goes silent → probe green → transport recycled → redial recovers); port refused (direct unreachable, no probe). The watchdog tests inject errors shaped exactly like the tunnel's %w exit contract.

The PR description's "restart storm" wording has also been corrected per your analysis: after a restart the initial dial keeps backing off (or registration keeps retrying at edge_id=0), so an outage is not necessarily a continuous restart loop — each threshold crossing is an unnecessary exit rather than a storm.

On direction: long-term it may be cleaner for geminio to expose typed errors / connection-state verdicts natively — happy to follow your preference there. Short-term, this PR keeps the classification self-contained in the ongrid wrapper so it is not blocked on the transport's release cadence.

Runtime fault-injection evidence for the acceptance scenarios (DNS failure, TCP connection refusal, established-connection blackhole, frontier/manager restart, reachable-but-wedged cloud, mixed failure sequences, Linux + Windows managed environments) is being collected and will be attached in follow-up comments before requesting final evaluation.

[anrenlx2025]

As promised, below is the runtime fault-injection evidence for the acceptance
scenarios. The branch has also been rebased onto current main (aa30f43).
The rebase overlapped with #229's TLS hardening in buildDialer; the
resolution extracts the ServerName derivation into applyTLSServerName,
shared by the tunnel dialer and the probe, so the probe's certificate
verification can never diverge from the live transport. The
TLSRequired fail-closed behavior is preserved in loadTLSConfig.

Runtime verification — 7 fault scenarios, two managed environments

Both test beds ran the same build of this branch: a Linux edge under
systemd and a Windows edge as an nssm-managed service. The broker endpoint
resolves via dynamic DNS. "Tick" = heartbeat interval (~30 s healthy,
inflating to ~131 s when each call burns its full timeout budget).

Scenario 1 — DNS resolution failure (Windows). Interface DNS pointed at
an unroutable resolver; the broker was restarted once to force a redial
(DNS failure does not disturb an established tunnel). PID constant across
the window; heartbeat logged cloud unreachable; staying alive with the
underlying lookup <broker>: no such host; metrics/scrape channels
classified identically. After DNS restore: re-registered in 3.9 s,
heartbeats resumed within 2 ticks.

Scenario 2 — TCP refused / RST (Windows). Broker name pinned to
127.0.0.1 (closed port → guaranteed RST). The dial OpError
(connectex: ... actively refused) took the direct unreachable path — no
probe — with the same classification and keep-alive behavior as DNS.
Recovery after restore: re-register in 5 s. Both dial-stage failure shapes
collapse into one classification.

Scenario 3 — Blackhole on the established connection (Windows). A
methodology note first: a Windows Firewall block rule does not cover
already-established flows — heartbeats continued for 4 minutes behind it.
Covering established traffic required a WFP transport-layer filter
(outbound transport layer, remote-port condition). With it: both layers
logged the expected evidence (rpc timed out and probe says cloud unreachable at the tunnel layer — the probe dial was dropped too — and
cloud unreachable; staying alive at the watchdog layer), the stuck
counter was reset by the unreachable classification, and removing the
filter produced redial + re-register within 1 tick.

Scenario 4 — Broker restart (both). No exit on either bed. Reconnect +
re-register in ~9 s / ~8 s. Linux shows the ladder end to end: probe-green
→ route is stale; recycling transport → re-registered after reconnect.
The first re-register hit register_edge: not found while the broker
route was warming up; the retry 4 s later succeeded — the intended
reconnect-and-reregister behavior.

Scenario 5 — Broker SIGSTOP: reachable cloud, silent tunnel (both).
The kernel keeps completing handshakes into the listen backlog, so the
probe correctly concludes reachable — exactly the "handshake alive, RPC
silent" shape the probe exists to distinguish. Zero cloud unreachable
warnings; failures were all rpc timeout; the stuck counter climbed 1→5
on both beds, then tunnel stuck after 5 heartbeat failures → exit(1).
systemd/nssm revived the process in 5–6 s. After CONT, both beds
re-registered. The watchdog exit path is preserved for the stuck case.

Scenario 6 — Mixed sequence (Linux; Windows witness). STOP → streak
reaches 5 → port blackhole (nftables) lands exactly at the exit-evaluation
edge → the 5th re-register is classified unreachable instead of
hard-failing → streak reset, process alive; blackhole lifted (STOP kept) →
counter resumes from 1 and keeps counting (2, 3, ...) — live, not frozen.
No exit over 25 minutes, PID constant. The Windows bed, sharing the
STOPped broker without a blackhole, accumulated its own streak and exited
at 5 (revived in 6 s) — scenario 5 replaying naturally inside the sequence.
Side observation tracked separately: after the final CONT the Linux edge
hit a distinct all-goroutines-parked wedge (zero logs, no reconnect) that
STOP alone does not reproduce; we are reducing it to a reproducible case
and will report it separately once isolated.

Scenario 7 — Cross-OS comparison, same blackhole parameters:

dimension	Linux (systemd)	Windows (nssm)
injection tool	nftables port drop	WFP transport filter
PID during window	unchanged	unchanged
restarts during window	0	0
first probe-red latency	~30 s	~26 s
recovery → re-register	5 s	30 s (~1 tick)
semantics	probe red → unreachable → streak reset → no exit → redial on recovery	identical

Summary

• Unreachable failures (DNS / RST / blackhole): classified, process kept
  alive, streak reset, recovery within 1–2 ticks of the fault clearing.
• Reachable-but-stuck tunnel (SIGSTOP): the 5-failure watchdog exit still
  fires; the service manager revives the process.
• The behaviors compose correctly under mixed sequences.

With this evidence attached, the branch is ready for the next stage of
your review whenever you have time.