Stale sandbox routing record after ungraceful node loss causes persistent 502 until TTL (max_length_in_hours) #3695

Summary

When a sandbox node is lost ungracefully (force-drain, docker kill, crash, infra-level stop), the orchestrator never runs MarkStopping, so the orchestrator-owned routing record sandbox:routing:{sandboxID} is never deleted. It keeps pointing at the dead node until its TTL — which is sandbox_max_length_in_hours from write time (can be many hours).

During that window, client-proxy resolves the stale record, dials the dead node's :5007, the connection fails, and every request to those sandboxes returns 502 — surfaced to users as "Failed to route request to sandbox". There is no reconcile that detects "this routing record points to a node no longer in discovery" and cleans it up; only the TTL does.

Why the existing self-heal does not cover this

• Miss path works: if the record is absent, client-proxy correctly falls through to the auto-resume path (ResumeSandbox gRPC to the API).
• Stale path does not: if the record is present but points to a dead node, client-proxy does not miss — it resolves a node IP, tries to connect, and fails. Nothing re-checks node liveness against the record.

On the API side, losing a node only calls deregisterNode (packages/api/internal/orchestrator/client.go), which does o.nodes.Remove(...) — it removes the node from the in-memory map and touches no routing records. The periodic node Reconcile (nodemanager/sync.go) only reaps orchestrator-reported orphan instances, which requires the node to still answer GetOrphanCandidates — impossible once it is gone.

Impact

• Sandboxes on a hard-lost node keep returning 502 "Failed to route request to sandbox" for up to max_length_in_hours.
• Easy to mistake for a routing/proxy bug (the message says "route"), when it is a stale orchestrator-owned record pointing at a dead node.

Repro

1. Create a sandbox on node A (record sandbox:routing:{id} points to A).
2. Hard-stop node A's orchestrator so MarkStopping never runs (or nomad node drain -force).
3. Request the sandbox through client-proxy → 502, repeatedly, until TTL.

Suggested direction (design-level, discussion welcome)

The routing record is orchestrator-owned and its delete is execution_id-guarded by design (so no other component removes a newer execution's record), so the API must not blindly delete it. Options:

1. Proxy-layer self-heal: on an upstream dial/connection failure (not an envd 5xx), treat it like a miss and fall through to the existing auto-resume path, which can re-place/kill and correct the stale record.
2. Liveness-aware sweeper: a reconcile that, for routing records whose orchestrator_id is no longer in the discovered node set, invokes the same execution_id-guarded cleanup the orchestrator would, or drives the sandbox through the auto-resume/kill path.
3. Shorter heartbeat TTL + refresh: write the record with a short TTL refreshed by a live-node heartbeat, so a dead node's records expire in seconds rather than max_length_in_hours (bigger change; affects write cadence).

Related

The messaging for this exact failure is improved by #3692 (distinct 502 "Failed to reach sandbox (upstream connection error)" vs the 500 routing-failure message), which makes stale-record 502s distinguishable from genuine routing failures in logs. This issue is the underlying cause behind those 502s on node loss.