feat(provider-sync): restart the agents attached to a rotated provider

[mattj-monad]

Summary

When a credential rotates, provider-sync now restarts only the agents attached to that provider, so they start fresh on the new values. This replaces the connection-level approach in fix(supervisor): close only the rotated provider's tunnels on credential rotation (OpenShell#14), which is closed unmerged. Agents are expected to be stateless or to resume from external state, so a restart is acceptable.

Related Issue

Dropped Connections on Credential Change (ENG-11222): https://linear.app/monad-v3/issue/ENG-11222

Changes

• Restart set: after a successful UpdateProvider, the controller restarts the sandboxes named in the response's target_receipts. That's the gateway's own list of sandboxes attached to the provider, so no extra lookup is needed. Each one gets StopSandbox then StartSandbox, with request ids.
  • Other providers' sandboxes aren't touched.
  • A sandbox restarts once even when several of its providers rotate in the same reconcile.
  • A sandbox that is stopped, stopping, completed or deleting is left alone.
  • The start is retried for up to 10 minutes on FailedPrecondition, Unavailable or Aborted, so a stopped agent isn't left down.
• Rotations vs baselines: only a value that replaces one already applied restarts agents. The first values seen for a provider are a baseline.
• --state-secret: the salt and the per-value fingerprints are persisted in a Secret in the controller's namespace via server-side apply. Values are never stored. A provider-sync restart then neither re-sends unchanged values nor restarts every agent. A failed save is logged; it only costs an extra restart later.
• --no-restart: turns restarts off.
• Events on the Secret: AgentRestarted, AgentRestartSkipped and AgentRestartFailed.
• Docs: the crate README (behaviour, flags, RBAC: sandbox:read/sandbox:write for the principal, the state Secret for the ServiceAccount) and the debug-openshell-cluster skill.

Testing

• 37 provider-sync unit tests pass. New ones:
  • a rotation restarts only the rotated provider's agents, and a baseline or unchanged value restarts nothing;
  • one restart per agent when several of its providers rotate;
  • persisted state keeps a controller restart from re-sending or restarting, and a later rotation still restarts;
  • state round-trips without values, and unreadable state starts fresh;
  • receipts become a deduplicated sandbox list;
  • stopped phases are left alone.
• mise run pre-commit passes.
• Not yet run on kind. That's next, after the image builds and the fleet chart sets --state-secret and RBAC.

Checklist

• Conventional commit, signed off
• Tests added
• Docs and debug skill updated

🤖 Generated with Claude Code

https://claude.ai/code/session_011cAJ1CcusffKn8nYhwhUqu