fix(ssh): persist sandbox host identities

[quocanh261997]

Summary

Sandbox SSH host keys currently change when the supervisor restarts, so clients cannot reliably recognize the same sandbox. Give each sandbox ID one Ed25519 host key and expose its public SHA256 fingerprint. Keep the private key in the configured gateway credential store and deliver it only to the supervisor.

Related Issue

Closes #3835

Replaces #4027, which the vouch check closed before contributor approval. The contributor is now vouched.

Changes

• Wait for confirmed sandbox deletion before reusing its name in the SSH identity E2E test. Check the delete command succeeds and retain a bounded failure if cleanup stalls.
• Merge current main into the branch, resolve the compute and schema conflicts, and update the SSH handshake test for russh 0.63. Preserve provisioning ownership and cancellation-safe failed-create cleanup. Use the database record version when deleting sandbox-owned credentials and the parent record.
• Route failed SSH-key preparation and rejected sandbox creation through the shared cleanup path. Remove the sandbox and key when cleanup succeeds; retain a durable Deleting record when cleanup needs a retry. Keep cleanup running if the caller disconnects, including while waiting for the cleanup lock.
• Persist a sandbox-owned credential handle and fingerprint. Keep identity across stop/start, automatic restarts, and gateway recovery; remove credentials during deletion. Retain cleanup ownership when credential deletion fails and finish staging if a request is interrupted.
• Require the managed supervisor to use its assigned host key. Reject missing, invalid, or mismatched key material instead of replacing an established identity.
• Return the fingerprint on Sandbox and SSH-session responses, sandbox JSON output, and Rust, Python, Go, and TypeScript sandbox references.
• Document identity lifetime, authenticated fingerprint lookup, workload isolation, and matching runtime releases. CLI/TUI automatic verification remains the optional follow-on from the issue.

Testing

Latest SSH E2E test update:

• Rootless Podman CI caught a test race: recreation began while deletion still reserved the sandbox name. The test now checks deletion succeeds and polls for the old sandbox to disappear before recreating it.
• Podman-feature lint of ssh_host_identity — passed.
• mise run pre-commit — passed.
• Live Docker conformance — all six scenarios passed.
• Live SSH identity E2E — one test passed, no failures or skips, including managed gateway restart and delete/recreate with a different fingerprint.
• Rootless Podman execution of this update is pending GitHub CI; it was not run locally on macOS.

GitHub Branch Checks passed on the preceding commit 6ec9c0a97. The only failed E2E job was the rootless Podman SSH identity test addressed above.

Earlier local verification of 6ec9c0a97 after merging main at 71c3cd957:

• mise run pre-commit — passed, including formatting, lint, and license checks.
• mise run ci — ran; Python (190 tests), Go checks, TypeScript (142 tests), Rust lint and compile checks passed. The workspace run stopped at one unchanged network test: proxy::tests::mediated_connect_keeps_workload_bytes_read_with_the_synthesized_header, which timed out waiting for a local TCP request. The same failure reproduces when that test runs alone. The full CI command is not green locally.
• cargo test -p openshell-server --features test-support — passed: 1,949 unit tests, 8 ignored, and all integration tests. Includes the failed-create, ownership, deadline, cancellation, and watch/delete tests.
• cargo test -p openshell-supervisor-process -p openshell-tui -p openshell-vfio — passed: 102, 92, and 59 tests respectively. The SSH tests exercise the updated russh callback and pinned keys.
• Content-guard example — all 16 tests passed.
• Docker CLI conformance — all six scenarios passed.
• Docker SSH identity E2E — one test passed, with no failures or skips. Verified Get/List against the presented SSH fingerprint, a pinned OpenSSH connection after stop/start and managed gateway restart, workload read restrictions, and a new identity after delete/recreate.

The imported gateway timeout assertion now uses its existing diagnostic-normalization helper so macOS line wrapping does not cause a false failure; all eight preflight tests passed. The failed-create fixture explicitly reports an absent compute resource, preserving the separate test for asynchronous deletion.

Rust checks used LIBRARY_PATH=/opt/homebrew/lib for Homebrew Z3 on macOS. The live Docker checks use an ignored copy of the launcher with host.docker.internal as the supervisor endpoint and the temporary mTLS gateway bound on all interfaces. This local launcher adjustment is excluded from the PR. Kubernetes pod rescheduling and other compute-driver lanes have not been exercised locally.

Checklist

• Follows Conventional Commits.
• Commits are signed off (DCO).
• Architecture docs updated where applicable; public behavior and related operating skills are documented.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[quocanh261997]

Local verification evidence

The saved local run passed all six Docker CLI conformance scenarios and the SSH identity E2E test. This report covers the working-tree implementation subsequently committed as e5de0a892813408bf08a36d93ab33b66a9929d18, the PR’s initial implementation commit. Updated verification for the cleanup fix is recorded in this later comment.

Environment: macOS on ARM64, Docker Desktop. Live test run: October 1, 2026. The excerpts below come from the saved test output.

What the live SSH test verified

Check	Result
Sandbox Get and List expose the same public fingerprint	Passed
ssh-keygen reports the same fingerprint for the key actually presented over SSH	Passed
The workload cannot read /.openshell/supervisor/auth.json	Passed
Stop/start preserves the fingerprint and an OpenSSH connection with StrictHostKeyChecking=yes succeeds using the previously saved host key	Passed
Managed gateway restart preserves the fingerprint and the same pinned SSH connection succeeds	Passed
Delete/recreate under the same name produces a different fingerprint	Passed

The test source at this commit contains these assertions. The gateway restart branch ran; its skip message is absent from the saved output.

Actual SSH test output:

running 1 test
test ssh_host_identity_survives_restarts_and_changes_after_recreation ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 22.25s

Supporting checks

• Docker CLI conformance: smoke, sandbox-lifecycle, file-transfer, mechanistic-proposal, new-hostname-proposal, and policy-local each returned "passed": true; the overall report also returned "passed": true.
• mise run ci completed successfully. Its gateway test summary was 1882 passed; 0 failed; 8 ignored. The seven new SSH credential tests below all passed.
• mise run pre-commit completed successfully before publishing the PR.

Actual SSH credential test output

test ssh_identity::tests::cancelled_prepare_finishes_before_deletion_and_cannot_recreate_identity ... ok
test ssh_identity::tests::concurrent_candidates_keep_one_resolvable_identity ... ok
test ssh_identity::tests::deletion_retries_before_releasing_key_ownership ... ok
test ssh_identity::tests::identity_survives_reload_and_is_never_public ... ok
test ssh_identity::tests::losing_candidate_cleanup_failure_remains_owned_for_deletion ... ok
test ssh_identity::tests::default_credential_store_persists_key_across_runtime_reconstruction ... ok
test ssh_identity::tests::published_identity_is_never_replaced_when_storage_is_inconsistent ... ok

Credential test source

Reproduction and scope

The standard focused Docker command is:

OPENSHELL_E2E_DOCKER_TEST=ssh_host_identity mise run e2e:docker

On this Mac, the stock launcher's container-side 127.0.0.1 endpoint could not reach the host gateway. The successful run used an ignored copy of e2e/with-docker-gateway.sh, with just this Darwin-specific addition after the existing container-network check:

if [ "$(uname -s)" = "Darwin" ]; then
  GATEWAY_BIND_IP="0.0.0.0"
  SUPERVISOR_GATEWAY_HOST="host.docker.internal"
fi

The gateway still used mTLS. The successful local invocation was:

LIBRARY_PATH=/opt/homebrew/lib \
OPENSHELL_E2E_DOCKER_TEST=ssh_host_identity \
OPENSHELL_CONFORMANCE_BIN="$PWD/target/debug/openshell-conformance" \
mise exec -- bash plans/3835-with-docker-gateway-macos.sh \
  bash e2e/rust/e2e-docker.sh __openshell_run_docker_e2e

The launcher adjustment is excluded from the PR. This evidence covers local Docker verification; Kubernetes pod rescheduling and other driver E2E lanes were not exercised. GitHub runner validation is still pending.

[drew]

gator-agent

PR Review Status

Thanks @quocanh261997 for documenting the current-head Docker and SSH verification. I checked the full patch and the creation/cleanup paths those tests exercise; one PR-owned failure path can leave a sandbox record and its host-key credential behind after create reports an error.

Action required: make every post-commit identity failure either fully remove the sandbox and key or durably leave the sandbox in a retryable deletion state.

Blocking findings:

• GATOR-e5de0a89-01: see the inline Warning on the post-commit identity preparation call.

Carried findings:

• None

Gator metadata

• Validation: project-valid through linked, validated issue #3835
• Docs: sandbox identity behavior is documented under docs/; related operating skills are updated
• Checks: DCO and vouch gates are green; required branch workflows have not been dispatched for this head
• E2E: test:e2e is required for sandbox lifecycle and credential-flow changes, but dispatch waits until blocking review feedback is resolved
• Head SHA: e5de0a892813408bf08a36d93ab33b66a9929d18
• Base SHA: 76cfd0e31d5e1633db7ccd86ad9023ef7a2461b2
• Merge base SHA: 021400be8af471f8669369e679de3e18cf0bd672
• Patch ID: accf65c51eed8be52b1848efb2eb9bfbfe089f99
• Gator payload: 10
• Review mode: initial
• Previous reviewed SHA: none
• Review budget exhausted: no
• Maintainer decision required: no
• Next state: gator:in-review

[quocanh261997]

🏗️ build-from-issue-agent

Local verification after the failed-create fix

Tested the working tree now committed as a645d9fd5d61993ed8c8f3d848955e38f0e81be5 on October 2, 2026, using macOS ARM64 and Docker Desktop. This covers the fix for the failed-create cleanup finding.

The six new regression tests all passed. They inject failures to check that:

• A failed SSH credential save leaves no sandbox or key, and the name can be reused.
• If key deletion fails after the driver rejects creation (AlreadyExists, FailedPrecondition, or another error), the durable sandbox stays Deleting. Reconciliation then removes the row and key.
• Failed fingerprint publication removes the staged key and sandbox.
• A backend cleanup error does not restore the failed sandbox to Provisioning.
• Cleanup finishes after the caller is canceled, both during driver deletion and while waiting for its lock.

test compute::tests::ssh_credential_delete_failure_keeps_failed_create_deleting_until_reconciliation ... ok
test compute::tests::ssh_credential_store_failure_compensates_create_and_releases_name ... ok
test compute::tests::ssh_failed_create_backend_cleanup_error_does_not_restore_provisioning ... ok
test compute::tests::ssh_failed_create_compensation_survives_cancellation_while_waiting_for_lock ... ok
test compute::tests::ssh_failed_create_compensation_survives_request_cancellation ... ok
test compute::tests::ssh_fingerprint_persistence_failure_compensates_staged_identity ... ok

Full LIBRARY_PATH=/opt/homebrew/lib mise run ci passed, including workspace Rust tests, gateway tests (1888 passed; 0 failed; 8 ignored), Python (258 passed), TypeScript (142 passed), Go checks, and formatting/lint/compile checks. mise run pre-commit passed. The eight ignored gateway tests are existing suite exclusions; all six added regressions ran.

E2E Test Attestation

Gateway mode: Docker. Every live test/scenario executed by the focused launcher passed:

Test / scenario	Result
CLI conformance smoke	Passed
CLI conformance sandbox-lifecycle	Passed
CLI conformance file-transfer	Passed
CLI conformance mechanistic-proposal	Passed
CLI conformance new-hostname-proposal	Passed
CLI conformance policy-local	Passed
ssh_host_identity_survives_restarts_and_changes_after_recreation	Passed

test ssh_host_identity_survives_restarts_and_changes_after_recreation ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 22.33s

The SSH test checks the published fingerprint against the key actually presented, workload read restrictions, a pinned OpenSSH connection after stop/start and a managed gateway restart, and a different key after delete/recreate. The gateway restart branch ran; no SSH test was skipped.

Command used:

LIBRARY_PATH=/opt/homebrew/lib \
OPENSHELL_E2E_DOCKER_TEST=ssh_host_identity \
OPENSHELL_CONFORMANCE_BIN="$PWD/target/debug/openshell-conformance" \
mise exec -- bash plans/3835-with-docker-gateway-macos.sh \
  bash e2e/rust/e2e-docker.sh __openshell_run_docker_e2e

As in the earlier evidence, the ignored launcher copy uses host.docker.internal for the supervisor endpoint and binds the temporary mTLS gateway on all interfaces so Docker Desktop containers can reach the macOS host. This launcher adjustment is excluded from the PR. The launcher's optional static-linkage check was skipped because readelf is unavailable on this Mac. Kubernetes rescheduling and other driver E2E lanes were not run locally.

[github-actions]

Label test:e2e applied, but pull-request/4094 does not exist yet. A maintainer needs to comment /ok to test a645d9fd5d61993ed8c8f3d848955e38f0e81be5 to mirror this PR. Once the mirror exists, re-apply the label or re-run Branch E2E Checks from the Actions tab.

[drew]

/ok to test a645d9f

[drew]

gator-agent

PR Review Status

Thanks @quocanh261997. I checked your cleanup update against the prior failed-create finding: post-commit identity failures and driver-create rejections now share compensation that removes the sandbox and key when possible, retains a durable Deleting record when cleanup must retry, and continues cleanup after caller cancellation. The added regressions cover the requested credential-store and key-deletion failures. No blocking code-review findings remain.

Blocking findings:

• No blocking findings remain

Carried findings:

• GATOR-e5de0a89-01: resolved by this head; the Gator-owned thread has been closed

Required test dispatch is not complete yet. Gator applied test:e2e and posted /ok to test for the current head; the contributor mirror must be created before the E2E label can be re-applied and the required workflows confirmed queued.

Gator metadata

• Validation: project-valid through linked, validated issue #3835
• Docs: sandbox identity behavior is documented under docs/; no additional docs change is required for this cleanup-only delta
• Checks: DCO is green; required current-head branch workflows are awaiting contributor-mirror dispatch
• E2E: test:e2e is required; mirror creation was requested with /ok to test a645d9fd5d61993ed8c8f3d848955e38f0e81be5 and workflow queue confirmation remains pending
• Head SHA: a645d9fd5d61993ed8c8f3d848955e38f0e81be5
• Base SHA: 76cfd0e31d5e1633db7ccd86ad9023ef7a2461b2
• Merge base SHA: 021400be8af471f8669369e679de3e18cf0bd672
• Patch ID: 3b6a5b4b21488f22e5a5a68154f7db415e0e635c
• Gator payload: 10
• Review mode: follow_up
• Previous reviewed SHA: e5de0a892813408bf08a36d93ab33b66a9929d18
• Review budget exhausted: no
• Maintainer decision required: no
• Next state: gator:in-review

[drew]

/ok to test a645d9f

[quocanh261997]

Updated the branch in commit 6ec9c0a975e3334bc97e646b1ee6df69bb8fd9c2 by merging main at 71c3cd957abef062eb7f37010056717cd49f2ed3. GitHub now reports the PR as mergeable, with the conflicts cleared.

The failed merge-queue run hit a russh callback signature change. The SSH pinning test now accepts PublicKeyOrCertificate and checks its public-key fingerprint. The merge also preserves the new provisioning ownership/deadline checks alongside failed-create SSH cleanup.

Live verification caught and fixed a cleanup integration problem: the payload can contain an older record version than the database row. Cleanup now uses the authoritative version, so a sandbox does not remain listed after its key is removed. The gateway timeout test uses its existing normalization helper to handle wrapped macOS error text.

Verification of this commit:

• mise run pre-commit: passed, including the commit hook.
• Server tests with test-support: 1,949 unit tests passed, 8 ignored; integration tests passed.
• Supervisor process, TUI, and VFIO tests: 102, 92, and 59 passed; content-guard example: 16 passed.
• Python: 190 tests passed; TypeScript: 142 passed; Go checks passed. Rust lint and compile checks passed.
• Live Docker conformance: all six scenarios passed.
• Live SSH identity E2E: one passed, no failures or skips. Get/List matched the presented fingerprint; pinned OpenSSH connections worked after stop/start and a managed gateway restart; the workload could not read supervisor authentication; delete/recreate produced a new identity.

mise run ci is not green locally: the unchanged test proxy::tests::mediated_connect_keeps_workload_bytes_read_with_the_synthesized_header times out after five seconds waiting for a local TCP request. It also fails when run alone. Its root cause remains unresolved; no network-runtime changes are included in this update.

Live checks used the Docker driver on macOS with the previously documented, ignored launcher adjustment (host.docker.internal). Other compute-driver lanes were not exercised locally. GitHub checks and maintainer review must run again for the new commit.

[drew]

gator-agent

PR Review Status

Thanks @quocanh261997. I checked your October 4 current-main integration update, including the conflict resolutions around provisioning ownership, cancellation-safe failed-create cleanup, authoritative resource versions, and the updated russh host-key callback. The prior cleanup obligation remains resolved, and the merge-resolution delta introduces no new blocking findings.

Blocking findings:

• No blocking findings remain

Carried findings:

• GATOR-e5de0a89-01: remains resolved; this integration preserves the compensation and retryable deletion behavior

Required test dispatch is not complete for this head. test:e2e remains required, and the contributor mirror still points at the previous head; Gator will request a current-head mirror and wait for the required workflows to queue before entering pipeline watch.

Gator metadata

• Validation: project-valid through linked, validated issue #3835
• Docs: the stable host-identity contract is documented under docs/how-it-works/sandboxes/overview.mdx; no additional docs change is required for this integration-only delta
• Checks: DCO is green; current-head branch, Helm, Trivy, and E2E gates have not completed
• E2E: test:e2e is required; the contributor mirror is stale and current-head dispatch is pending
• Head SHA: 6ec9c0a975e3334bc97e646b1ee6df69bb8fd9c2
• Base SHA: 71c3cd957abef062eb7f37010056717cd49f2ed3
• Merge base SHA: 71c3cd957abef062eb7f37010056717cd49f2ed3
• Patch ID: c893ec58b812d4be42e12fd6381b58c83cfb6134
• Gator payload: 10
• Review mode: follow_up
• Previous reviewed SHA: a645d9fd5d61993ed8c8f3d848955e38f0e81be5
• Review budget exhausted: no
• Maintainer decision required: no
• Next state: gator:in-review

[drew]

/ok to test 6ec9c0a

[quocanh261997]

Fixed the rootless Podman SSH test in 7800f2f3fd34deccbd7dab62533e48779acbcd19.

The failed job reached delete/recreate, then failed with sandbox already exists. Deletion can be accepted before removal finishes, while the original name remains reserved. The test now checks the delete command succeeds and waits until the gateway no longer lists the sandbox before reusing its name. The wait has a two-minute limit and fails if cleanup does not complete. Host-key comparisons remain intact.

Verification:

• Podman-feature lint of ssh_host_identity: passed.
• mise run pre-commit, including the commit hook: passed.
• Live Docker conformance: all six scenarios passed.
• Live SSH identity E2E: one passed, no failures or skips. Covered fingerprint matching, pinned SSH after stop/start and managed gateway restart, workload access restrictions, and a different fingerprint after same-name recreation.

This update changes only the E2E test. Rootless Podman execution of the updated test still needs GitHub CI; it was not run locally on macOS. GitHub Branch Checks passed on the preceding commit. The PR description now separates the latest verification from the earlier results.

GitHub currently reports Waiting for /ok to test mirror for this head. A maintainer needs to approve the new commit with:

/ok to test 7800f2f3fd34deccbd7dab62533e48779acbcd19

[drew]

gator-agent

PR Review Status

Thanks @quocanh261997. I checked your October 4 deletion-polling update against the prior cleanup obligation and the test harness behavior. Waiting for the original sandbox to disappear before same-name recreation fixes the observed rootless Podman race, the earlier failed-create finding remains resolved, and this test-only delta introduces no new Critical defect.

Blocking findings:

• No blocking findings remain

Carried findings:

• GATOR-e5de0a89-01: remains resolved; this delta does not alter the compensation or retryable deletion implementation

Required test dispatch is not complete for this head. test:e2e remains required, and Gator will request the current-head contributor mirror before entering pipeline watch.

Gator metadata

• Validation: project-valid through linked, validated issue #3835
• Docs: the stable host-identity contract is documented under docs/how-it-works/sandboxes/overview.mdx; no additional docs change is required for this test-only delta
• Checks: DCO is green; current-head Branch Checks, Helm, Trivy, and E2E workflows are awaiting contributor-mirror dispatch
• E2E: test:e2e is required; current-head mirror creation and workflow queue confirmation are pending
• Head SHA: 7800f2f3fd34deccbd7dab62533e48779acbcd19
• Base SHA: 71c3cd957abef062eb7f37010056717cd49f2ed3
• Merge base SHA: 71c3cd957abef062eb7f37010056717cd49f2ed3
• Patch ID: 2cc204fdaa30f3122895ef858cb7524e04f99c26
• Gator payload: 10
• Review mode: critical_only
• Previous reviewed SHA: 6ec9c0a975e3334bc97e646b1ee6df69bb8fd9c2
• Review budget exhausted: yes
• Maintainer decision required: no
• Next state: gator:in-review

[drew]

/ok to test 7800f2f