Skip to content

fix(discovery): survive an offline node instead of throwing 595 - #25

Merged
ChrisonSimtian merged 2 commits into
mainfrom
fix/discovery-offline-node
Oct 1, 2026
Merged

ChrisonSimtian merged 2 commits into
mainfrom
fix/discovery-offline-node

Conversation

@ChrisonSimtian

@ChrisonSimtian ChrisonSimtian commented Oct 1, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #24.

Problem: when any cluster node is offline, PVE answers that node's per-node calls (/nodes/<n>/lxc etc.) with HTTP 595. Kiota throws, and DiscoverAsync returns nothing at all. In the homelab, desktop-01 sleeps most of the time, so every converge plan fell back to intent-only.

Fix:

  • Nodes that /nodes lists as offline or unknown are not queried per-node.
  • A node listed online that still answers 595 (it went to sleep mid-discovery) is treated the same.
  • Such a node is returned with Reachable = false, empty Storage/Network (not read, rather than none), and its guests taken from /cluster/resources?type=vm (status unknown; memory, cores and tags still come from the shared config). That costs one extra call, made only if some node is unreachable.
  • Why not just skip the node: a consumer would then read its guests as absent. The Homelab engine's differ maps "absent" to Create.

New API: NodeSnapshot.Reachable, defaulting to true, so a snapshot JSON written before this change still deserialises as before.

Tests: DiscoveryOfflineNodeTests covers three cases:

  • a node listed offline is never queried;
  • a node listed online that answers 595;
  • with all nodes online, no /cluster/resources call is made.

The 595 test fails on the old code with the exact production exception. Suite: 40 passed, 3 skipped.

Live check (2026-10-02):

  • nuc-01: 6 CTs, discovered in full.
  • hpe-01: 28 CTs and 1 VM, discovered in full.
  • desktop-01: Offline, with 8 CTs and 4 VMs, all unknown.

Version bumped to 0.2.2.

Second commit, a build fix CI needed: the Fallout 2026.1.0-preview.5 pin no longer exists on the Fallout-build feed, which now carries only 10.5.0-preview.*, so restore failed before compiling anything. Fixed the same way as Homelab#390:

  • Fallout 10.4.0 from nuget.org;
  • the feed, its credentials and the PACKAGES_PAT env are dropped from both workflows.

Test and Pack verified locally without the PAT.

🤖 Generated with Claude Code

https://claude.ai/code/session_01V2afcTcowT1YVBwLzrJFtH

ChrisonSimtian and others added 2 commits October 2, 2026 00:17
A node that is offline (the homelab's Wake-on-LAN desktop-01, asleep most of the
time) answers every per-node call with HTTP 595, which threw out of DiscoverAsync
and left consumers with no snapshot at all.

Nodes listed offline/unknown are no longer queried; a node listed online that
still answers 595 is treated the same. Such a node comes back with
Reachable = false, empty storage/network, and its guests read from
/cluster/resources?type=vm, which lists them from the shared config. Dropping
those guests instead would make a consumer read them as absent and plan to
create them.

Verified live: nuc-01 and hpe-01 discovered in full, desktop-01 offline with its
8 CTs and 4 VMs present (status "unknown"). Bump to 0.2.2.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V2afcTcowT1YVBwLzrJFtH
CI failed at restore: Fallout.* 2026.1.0-preview.5.gd84fad5cd7 no longer exists on
the Fallout-build GitHub Packages feed (it now carries only 10.5.0-preview.*), so
the build/ project could not restore. Local builds hid it via the package cache.

Same move as Homelab#390: pin the 10.4.0 stable line, restore everything from
nuget.org, and drop the Fallout feed, its credentials and the PACKAGES_PAT env
from both workflows. Test and Pack both verified locally without the PAT.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V2afcTcowT1YVBwLzrJFtH
@ChrisonSimtian
ChrisonSimtian merged commit 032d204 into main Oct 1, 2026
1 check passed
@ChrisonSimtian
ChrisonSimtian deleted the fix/discovery-offline-node branch October 1, 2026 11:23
ChrisonSimtian added a commit that referenced this pull request Oct 3, 2026
#25 moved Fallout to nuget.org and removed the PAT from the build path, but
missed these two lines in README.md and build.ps1.


Claude-Session: https://claude.ai/code/session_01V2afcTcowT1YVBwLzrJFtH

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Discovery crashes with HTTP 595 when any cluster node is offline

1 participant