feat(provider-sync): sync provider credentials from Kubernetes Secrets

.github/workflows/monad-images.yml

@@ -1,8 +1,8 @@
 # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
 # SPDX-License-Identifier: Apache-2.0
 
-# Builds Monad's OpenShell images (gateway, supervisor, sandbox) and pushes
-# them to ghcr.io/monad-inc/openshell. See MONAD.md.
+# Builds Monad's OpenShell images (gateway, supervisor, sandbox, CLI and
+# provider sync) and pushes them to ghcr.io/monad-inc/openshell. See MONAD.md.
 
 name: Monad Images
 
@@ -82,6 +82,8 @@ jobs:
           - { package: openshell-sandbox, binary: openshell-sandbox, triple: aarch64-unknown-linux-musl }
           - { package: openshell-cli, binary: openshell, triple: x86_64-unknown-linux-musl }
           - { package: openshell-cli, binary: openshell, triple: aarch64-unknown-linux-musl }
+          - { package: openshell-provider-sync, binary: openshell-provider-sync, triple: x86_64-unknown-linux-gnu }
+          - { package: openshell-provider-sync, binary: openshell-provider-sync, triple: aarch64-unknown-linux-gnu }
     # Repository variables MONAD_RUNNER_X64 / MONAD_RUNNER_ARM64 select larger runners
     # (e.g. ubuntu-24.04-8core / ubuntu-24.04-arm-8core); unset uses the standard ones.
     runs-on: ${{ startsWith(matrix.triple, 'aarch64') && (vars.MONAD_RUNNER_ARM64 || 'ubuntu-24.04-arm') || (vars.MONAD_RUNNER_X64 || 'ubuntu-24.04') }}
@@ -117,6 +119,7 @@ jobs:
           - { component: supervisor, binary: openshell-supervisor, target-suffix: unknown-linux-gnu }
           - { component: sandbox, binary: openshell-sandbox, target-suffix: unknown-linux-musl }
           - { component: cli, binary: openshell, target-suffix: unknown-linux-musl }
+          - { component: provider-sync, binary: openshell-provider-sync, target-suffix: unknown-linux-gnu }
     runs-on: ubuntu-24.04
     timeout-minutes: 30
     permissions:

AGENTS.md

@@ -41,6 +41,7 @@ Do not rely on this file for a full inventory. The detailed public and contribut
 | `crates/openshell-gateway/` | Gateway binary composition | Links selected first-party compute drivers into the backend-agnostic server registry |
 | `crates/openshell-sdk/` | Shared client SDK | Async Rust gateway client (gRPC transport, TLS, OIDC refresh, edge tunnel); consumed by CLI, TUI, and `@openshell/sdk` |
 | `crates/openshell-providers/` | Provider management | Credential provider backends |
+| `crates/openshell-provider-sync/` | Provider sync controller | Keeps provider credentials current with labelled Kubernetes Secrets |
 | `crates/openshell-tui/` | Terminal UI | Ratatui-based dashboard for monitoring |
 | `crates/openshell-driver-kubernetes-secrets/` | Kubernetes Secrets credential driver | In-process `CredentialDriver` backend for OpenShell-managed K8s Secret storage |
 | `crates/openshell-driver-vault/` | Vault credential driver | In-process `CredentialDriver` backend for Vault-compatible KV storage |

MONAD.md

@@ -17,7 +17,7 @@ This fork carries Monad's changes on top of upstream OpenShell releases: OCSF Au
 
 ## Images
 
-`.github/workflows/monad-images.yml` builds the gateway, supervisor, sandbox runtime, and CLI for amd64 and arm64 and pushes them to `ghcr.io/monad-inc/openshell/{gateway,supervisor,sandbox,cli}`. The packages are private.
+`.github/workflows/monad-images.yml` builds the gateway, supervisor, sandbox runtime, CLI, and provider sync controller for amd64 and arm64 and pushes them to `ghcr.io/monad-inc/openshell/{gateway,supervisor,sandbox,cli,provider-sync}`. The packages are private.
 
 The `cli` image only carries the static `openshell` binary. Copy it into images that need a client matching the gateway build:
 
@@ -34,6 +34,8 @@ COPY --from=ghcr.io/monad-inc/openshell/cli:<tag> /usr/local/bin/openshell /usr/
 
 The gateway binary is compiled to launch Monad's supervisor and sandbox runtime images at the same `sha-` tag, so the gateway and supervisor always match.
 
+The `provider-sync` image runs `openshell-provider-sync`, which pushes credential values from labelled Kubernetes Secrets to gateway providers. It is a separate Deployment with the same tag as the gateway; see `crates/openshell-provider-sync/README.md`.
+
 Binaries build on the standard `ubuntu-24.04` and `ubuntu-24.04-arm` runners. To use larger org runners, set the repository variables `MONAD_RUNNER_X64` and `MONAD_RUNNER_ARM64` (for example `ubuntu-24.04-8core` and `ubuntu-24.04-arm-8core`); the runner group must allow this repository.
 
 To cut a release, tag `monad/main` with `monad-v<semver>`, for example `git tag monad-v0.1.3-monad.1 && git push origin monad-v0.1.3-monad.1`.

crates/openshell-provider-sync/Cargo.toml

@@ -0,0 +1,38 @@
+# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+[package]
+name = "openshell-provider-sync"
+description = "Keeps OpenShell provider credentials in sync with Kubernetes Secrets"
+version.workspace = true
+edition.workspace = true
+rust-version.workspace = true
+license.workspace = true
+repository.workspace = true
+
+[[bin]]
+name = "openshell-provider-sync"
+path = "src/main.rs"
+
+[dependencies]
+openshell-core = { path = "../openshell-core", default-features = false }
+openshell-sdk = { path = "../openshell-sdk" }
+async-trait = "0.1"
+clap = { workspace = true }
+futures = { workspace = true }
+k8s-openapi = { workspace = true }
+kube = { workspace = true }
+miette = { workspace = true }
+rand = { workspace = true }
+sha2 = { workspace = true }
+thiserror = { workspace = true }
+tokio = { workspace = true }
+tonic = { workspace = true }
+tracing = { workspace = true }
+tracing-subscriber = { workspace = true }
+
+[dev-dependencies]
+tokio = { workspace = true, features = ["macros", "rt-multi-thread"] }
+
+[lints]
+workspace = true

crates/openshell-provider-sync/README.md

@@ -0,0 +1,124 @@
+# openshell-provider-sync
+
+Keeps provider credentials current with Kubernetes Secrets that something else
+manages, such as External Secrets Operator syncing from Vault or 1Password.
+
+The controller watches Secrets labelled `openshell.nvidia.com/provider-sync=true`.
+Each Secret names the providers it feeds and the credential keys it backs for
+each one:
+
+```yaml
+apiVersion: v1
+kind: Secret
+metadata:
+  name: pm
+  labels:
+    openshell.nvidia.com/provider-sync: "true"
+  annotations:
+    openshell.nvidia.com/provider-workspace: pm   # optional
+    openshell.nvidia.com/provider-credentials: >-
+      slack-pm:SLACK_BOT_TOKEN, slack-pm:SLACK_APP_TOKEN,
+      linear:LINEAR_API_KEY,
+      claude-code-apikey:ANTHROPIC_API_KEY,
+      claude-code-oauth:CLAUDE_CODE_OAUTH_TOKEN
+```
+
+An entry is `[provider:]CREDENTIAL_KEY[=secret-key]`, separated by commas or
+newlines:
+
+- `slack-pm:SLACK_BOT_TOKEN` reads the Secret key `SLACK_BOT_TOKEN` into the
+  credential `SLACK_BOT_TOKEN` of provider `slack-pm`.
+- `linear:LINEAR_API_KEY=linear-key` reads the Secret key `linear-key` instead.
+- An entry without a `provider:` prefix goes to the provider named by the
+  `openshell.nvidia.com/provider` annotation, which is otherwise optional.
+
+All providers of one Secret are in one workspace, from
+`openshell.nvidia.com/provider-workspace` or the controller's default. Only listed
+keys are read, so a key added to the Secret later never becomes a credential by
+accident, and one Secret key may be listed only once.
+
+## Behavior
+
+- **What it sends.** When a mapped value changes, the controller sends only the
+  changed keys of that provider as a value-only `UpdateProvider`. The gateway
+  merges them into the provider, and attached sandboxes refetch on their next
+  provider poll, every 10s by default.
+- **Running processes.** A process that is already running gets the new value
+  only when the credential's profile sets `stable_placeholder: true`. Without it,
+  only processes started after the update see the new value.
+- **What it leaves alone.** It never creates or deletes providers, never clears
+  a credential when its Secret is deleted or unlabelled, and never touches
+  credential keys that no Secret lists. Providers are still created by whatever
+  creates them today.
+- **Each provider on its own.** A provider that fails does not hold back the
+  other providers of the same Secret.
+- **Empty means absent.** A Secret key that is missing and one whose value is
+  empty are treated the same, so a Secret that writes every known key and leaves
+  unused ones empty works. An empty value is never sent.
+- **All or nothing per provider.** If some but not all of a provider's keys are
+  set, or any is not UTF-8, nothing is sent to that provider, so a half-written
+  Secret cannot produce a partial rotation.
+- **Absent providers.** A provider with none of its keys set in the Secret is
+  skipped. This lets a Secret list alternative credentials, such as an API key
+  and an OAuth token, and feed whichever one it holds. When it switches, the
+  newly present provider is sent in full.
+- **One Secret per credential.** A second live Secret that maps the same
+  workspace, provider and credential key is refused with a
+  `ProviderSyncConflict` event.
+- **Startup and restarts.** The watch starts with a full list, so every labelled
+  Secret is synced at startup. State is in memory: a salted fingerprint of each
+  value sent, never the value. After a restart every Secret is sent once more.
+  That is a value-only update and does not disturb running sandboxes.
+- **Retries.**
+  - A missing provider is retried every 30s.
+  - Transient gateway failures are retried every 10s.
+  - Invalid annotations, conflicts, partly set keys and rejected updates
+    are retried every 5m, or as soon as the Secret changes.
+  - An absent provider is not retried; the next change to the Secret is seen.
+- **Audit.** Each outcome is published as a Kubernetes Event on the Secret
+  (`ProviderSynced`, `ProviderSyncSkipped`, `ProviderNotFound`,
+  `ProviderSyncConflict`, `InvalidProviderSync`, `ProviderSyncFailed`). The
+  gateway separately audits each `UpdateProvider` with the controller's
+  principal. Neither carries values.
+
+## Running
+
+The Monad fork publishes the controller as `ghcr.io/monad-inc/openshell/provider-sync`,
+with the same tags as the gateway. Run it as its own Deployment.
+
+```shell
+openshell-provider-sync \
+  --gateway http://openshell.openshell.svc:8080 \
+  --namespace agents \
+  --default-workspace default
+```
+
+| Flag | Environment | Purpose |
+|---|---|---|
+| `--gateway` | `OPENSHELL_GATEWAY_ENDPOINT` | Gateway URL |
+| `--gateway-ca-file` | `OPENSHELL_GATEWAY_CA_FILE` | CA for a private-CA gateway |
+| `--gateway-token-file` | `OPENSHELL_GATEWAY_TOKEN_FILE` | Bearer token, re-read when the gateway rejects it. Omit for a gateway that allows unauthenticated users |
+| `--namespace` | `OPENSHELL_PROVIDER_SYNC_NAMESPACE` | Namespace to watch; all namespaces when omitted |
+| `--default-workspace` | `OPENSHELL_PROVIDER_SYNC_DEFAULT_WORKSPACE` | Workspace when a Secret has no workspace annotation |
+| `--reporter`, `--reporter-instance` | `OPENSHELL_PROVIDER_SYNC_REPORTER`, `POD_NAME` | Event reporter |
+| `--no-events` | | Disable Kubernetes Events |
+| `--log-json` | `OPENSHELL_LOG_JSON` | Log as JSON lines |
+
+On a gateway that requires authentication, the principal needs `provider:write`
+with the workspace `admin` role in every workspace it updates. The token file
+suits a projected ServiceAccount token or a Secret-mounted token that something
+else rotates on disk.
+
+The ServiceAccount needs `get`, `list` and `watch` on Secrets in the watched
+namespace, and `create` and `patch` on `events.k8s.io` Events.
+
+The gateway client is the Rust SDK, which speaks bearer-token auth only. A
+gateway that requires mTLS client certificates is not supported yet.
+
+## Limits
+
+- **One replica.** There is no leader election. Two replicas would send the same
+  idempotent updates twice; that is wasteful but not wrong.
+- **No activation wait.** The controller does not wait for supervisors to report
+  that they activated the update. The gateway's mutation id is logged for
+  correlation.