Skip to content

fix(gateway): give image preparation its own deadline - #4038

Open
shiju-nv wants to merge 4 commits into
mainfrom
feat/3952-image-preparation-deadline-accepted
Open

shiju-nv wants to merge 4 commits into
mainfrom
feat/3952-image-preparation-deadline-accepted

Conversation

@shiju-nv

@shiju-nv shiju-nv commented Oct 1, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

An uncached image can use the gateway's entire five-minute policy repair window before the supervisor starts. Give image preparation a persisted deadline, then start the existing repair window when the current supervisor sends its first authenticated configuration report. Preparation timeouts direct users to image preparation and supervisor startup diagnostics and the preparation budget.

Related Issue

Closes #3952, accepted as part of #3955. The maintainer's accepted triage identifies both premature cold-image timeout and incorrect configuration-repair guidance.

The gateway invokes StopSandbox after timeout. The VM worker and staging cleanup implementation belongs to #3953.

Changes

  • Add image_preparation_timeout_seconds, defaulting to 1800 and restricted to 1 through 86400. Persist each attempt's absolute deadline so progress, configuration writes and restarts cannot extend preparation.
  • Start admission timing on authenticated supervisor registration. Preserve renewal for committed configuration changes and the first rejected load; duplicate reports and no-op writes do not renew it.
  • Release a recovered start operation when its persisted deadline expires, so cleanup can acquire the lifecycle lock. Retain timeout diagnostics and cleanup status for inspection and explicit retry.
  • Report ImagePreparationTimedOut separately from admission timeout. Show the phase in CLI JSON and TUI notes, give each timeout the appropriate recovery guidance, and update bindings and lifecycle documentation.

Testing

  • Checks appropriate to the affected code and behavior pass.
  • Unit tests added/updated.
  • E2E tests added/updated (if applicable). A successful physical-Mac cold preparation exceeding five minutes remains unverified for this candidate.

The resolved merge passed the server, CLI and TUI package check, formatting, documentation checks, and six exact library tests covering cold preparation, registration timing, stalled-start cleanup, and phase-specific CLI output. Independent review found no unresolved findings. Branch Checks passed for signed commit 87755ffe78f, including Rust lint, tests and build modes on both Linux architectures and macOS, plus protobuf compatibility.

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO). The published implementation and correction commits include Signed-off-by: Shiju <shiju@nvidia.com> and have verified SSH signatures.
  • Architecture docs updated (if applicable)

Give sandbox image preparation its own deadline and start the admission
deadline after preparation finishes. Persist both phases across gateway
restarts and show recovery guidance for the phase that expired.

Preserve the current service authorization schema and regenerate the Go
bindings with the preparation timestamps.

Fixes #3952
Related to #3955

Signed-off-by: Shiju <shiju@nvidia.com>
Use lazy Option fallbacks while preserving timeout messages and retained
sandbox behavior.

Signed-off-by: Shiju <shiju@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Oct 1, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@github-actions

github-actions Bot commented Oct 1, 2026

Copy link
Copy Markdown

Signed-off-by: Shiju <shiju@nvidia.com>
@shiju-nv
shiju-nv marked this pull request as ready for review October 1, 2026 19:20
Merge current main while preserving its startup-write retry guidance and
the separate image-preparation and admission-repair deadlines.

Keep effective policy changes renewing the repair window and unchanged
writes leaving it intact.

Signed-off-by: Shiju <shiju@nvidia.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Give first-time image preparation its own bounded deadline

1 participant