Repository navigation
feat(sandbox): support stop requests conditional on sandbox execution identity #4009
Description
User Story
As an operator building an event-driven controller, I want to stop only the sandbox execution that produced an observation, so delayed events cannot stop a newer execution.
Problem Statement
The public StopSandboxRequest accepts workspace_scope, name, and request_id. It cannot express an expected sandbox incarnation or execution. The request ID provides durable at-most-once admission; it is not a target precondition.
The live public Go SDK reproduction demonstrated this scenario against an unpatched Gateway at 374c035 with SQLite, for both stop/start and same-name delete/recreate:
- Execution A produces an observation that is queued for processing.
- The sandbox is stopped and started, producing execution B under the same sandbox identity and name.
- The controller processes A's observation and requests a stop by workspace and name. It cannot require the Gateway to reject the request if B is now current.
Deleting and recreating a sandbox with the same name presents a similar stale-target problem.
Impact / Why This Matters
Controllers must either disable automatic stop or risk stopping a replacement execution. Reading current state before stopping leaves a race between the read and the mutation. Direct Kubernetes Pod UID checks/deletes are backend-specific and bypass the Gateway lifecycle contract.
This blocks portable automation that must bind an action to the execution observed. The finding is a missing API capability; this investigation has not established an authorization bypass or sandbox-isolation vulnerability.
Proposed Design
Retain the public workspace/name addressing convention and support a conditional stop workflow:
- A controller receives trustworthy, Gateway/supervisor-attested execution identity with the relevant extension observation. Define its lifecycle semantics, including stop/start and runtime replacement.
- The controller supplies an expected-target precondition when requesting stop. An opaque token is one possible public representation; internal identifiers need not become public entity references.
- The Gateway checks that precondition as part of the serialized lifecycle operation. A stale or unverifiable target produces a distinguishable result without stopping the current replacement.
- Execution identity changes on explicit start, automatic runtime replacement, and recovery launch attempts that may reconstruct compute. Credential refresh, supervisor reconnection, and replica handoff preserve it when compute is unchanged. Persistent SSH host identity is separate and survives runtime replacement.
- Successful stop receipts and Gateway-confirmed, effect-free stale-target rejections replay their original results for 24 hours when a request ID is supplied. Reusing the request ID with another execution identity is a payload conflict. Driver errors or interrupted operations remain unresolved; they must not be recorded as proven no-effect rejections.
- The initial implementation supports one Gateway process with SQLite. PostgreSQL, including single-replica PostgreSQL, and HA deployments explicitly reject conditional stopping until distributed lifecycle fencing is established. Supervisor-session ownership alone does not provide that fencing.
- Upgrade handling must fail closed for runtime records without an authenticated bootstrap execution binding. Operators establish a fresh binding through a Gateway-controlled stop/start with matching runtime releases and verify new attested observations before enabling automatic responses.
The exact field names, token representation, and internal implementation are open for maintainers to choose.
Acceptance Criteria
- Supported extension observations expose authenticated metadata sufficient to bind a stop to the observed sandbox incarnation and execution.
- A matching conditional request stops the intended execution through the normal authorized Gateway lifecycle.
- A condition from before stop/start or same-name delete/recreate cannot stop the replacement, including when replacement races with the request.
- Missing, stale, or unverifiable conditions in the conditional workflow never silently fall back to an unconditional stop.
- Successful and stale-target retries preserve the original target and terminal result without affecting a replacement; changed payloads cannot reuse a request ID.
- Uncertain driver failures remain unresolved and cannot impersonate a replayable stale-target rejection.
- Reconnection and session handoff preserve execution identity; runtime replacement changes execution identity while retaining the sandbox SSH identity.
- Unsupported persistence deployments reject conditional stop; legacy-runtime upgrade handling is documented and tested.
- The contract documents identity lifetime, result semantics, and compatibility with existing unconditional stop, and has regression coverage plus at least one real-driver end-to-end test.
Alternatives Considered
- Client-side read/check before stop: cannot close the read-to-mutation race.
- Use only the persistent sandbox ID: distinguishes same-name recreation, but not stop/start of the same sandbox.
- Use backend-native deletion: sacrifices portability and Gateway lifecycle semantics.
- Use a durable stop hold: addresses subsequent activation, but does not establish which execution a delayed stop should target.
Agent Investigation
Original source investigation reviewed main at 912a077bd641272016fb8b2fd58209f6c7c6f194:
- Public stop request has no expected-target precondition.
- Middleware request context carries sandbox identity, but no explicit execution-incarnation field.
- The handler authorizes the request, and the compute implementation locks and rereads the selected sandbox. These protections do not compare the selected execution against the controller's earlier observation.
Related: #3556 concerns driver-facing sandbox references; #3153 concerns durable activation holds. Neither defines this conditional execution-targeting workflow.
Checklist
- I've reviewed existing issues and the published docs.
- This is a design proposal, not a "please build this" request.
Upstream Refresh (2026-10-08)
Reviewed upstream fe3942f2d. The public stop API still has no execution precondition. PR #4235 revalidates delayed internal driver exit observations; it does not fence controller-issued stop requests. PR #4094 introduces persistent SSH identity, while #3661 and #4321 introduce supervisor placement and handoff. PR #4217 must preserve those behaviors alongside execution targeting. The independent policy-version change remains in #4219.
Activity
aetherxeg-source commentedon Oct 2, 2026
One additional invariant that may help tighten this design is:
"target identity != target generation"
A stable sandbox identity/name can tell us which logical sandbox is being addressed, but it does not prove that the execution being acted on is the same execution that produced the triggering observation.
For an event-driven controller, I would treat the stop authorization as a compare-and-act operation over an immutable execution generation:
observation
-> conditional lifecycle effect
The important property is that the execution-generation value must come from an authority that can actually establish the lifecycle transition, not merely from a client-supplied correlation field.
That also suggests keeping two replay dimensions separate:
Those answer different questions.
A particularly useful adversarial test would be:
Every retry should deterministically report a stale-target result and B must remain running.
I would also test delete/recreate separately from stop/start, because a persistent sandbox identifier may fence one transition but not necessarily both.
A compact evidence chain would be:
"observed execution -> attested execution generation -> serialized lifecycle precondition -> effect"
rather than allowing:
"observed sandbox name -> current sandbox by name -> effect"
The latter preserves addressing, but not causal binding between the observation and the execution being stopped.
danehans commentedon Oct 5, 2026
I added a live SDK repro of the scenario behind this feature request: https://github.com/danehans/openshell-4009-repro
Use case: an event-driven controller that stops a sandbox in response to a delayed observation. The observation is bound to one execution, but the stop request is addressed only by workspace and name.
What it does (public Go SDK only, unpatched gateway
374c035, SQLite): observe execution A, replace it, then issueSandboxes().Stop(workspace, name)for A.Status.MainProcessStartedAtMsdiffers. The stale stop returned no error and stopped the replacement.This isn't a bug in unconditional
Stop. It shows the API has no way to express "stop only if this is still the execution I observed", and that the sandbox ID alone fences delete/recreate but not stop/start. I can extend the repo to exercise a conditional-stop API once the design settles.aetherxeg-source commentedon Oct 6, 2026
Thanks for publishing the live SDK repro. It closes the main uncertainty cleanly: this is not an unconditional "Stop" correctness issue; it is a missing compare-and-act primitive at the execution boundary.
The two scenarios also show that the public precondition should not be defined as “sandbox ID equality”:
I would therefore make the public contract an opaque execution-incarnation token (name is unimportant) issued by the Gateway/supervisor authority and changed on every transition that creates a new execution. "MainProcessStartedAtMs" is useful evidence for the repro, but I would avoid making a timestamp the durable concurrency token unless its uniqueness/lifetime semantics are explicitly guaranteed.
The critical implementation property is:
conditional_stop(expected_execution = E)
== compare current_execution with E
and perform Stop
within the same serialized lifecycle operation
A client-side "Get -> compare -> Stop" sequence must not be considered equivalent.
I would also make the idempotency semantics explicit because "request_id" and the execution precondition protect different races:
A useful invariant is:
request_id -> {expected_execution, terminal_result}
Once a request ID is admitted, retries should replay that same terminal result rather than re-evaluate the request against a newer execution. Reusing the same "request_id" with a different "expected_execution" should fail as a request conflict, not silently retarget.
That prevents an important case:
For the stale path, I would prefer a typed result such as "STALE_TARGET" / "FAILED_PRECONDITION", carrying the current execution identity only to the extent the API already authorizes its disclosure. Missing or unverifiable preconditions in the conditional API should fail closed and never degrade to unconditional "Stop".
The repro repo is already a strong basis for the conformance suite. I would extend it with at least these cases:
If those properties hold, the API establishes the stronger guarantee the controller actually needs:
an observation-bound lifecycle action cannot cross an execution-generation boundary.
Happy to review the eventual token/result semantics or help turn the current repro into that conformance matrix once the API shape is proposed.