From 439a456d5c040255fac22f73b21427830bd27693 Mon Sep 17 00:00:00 2001 From: Drew Newberry Date: Thu, 24 Sep 2026 15:15:14 -0700 Subject: [PATCH] fix(kubernetes): remove NetworkPolicy acknowledgement Signed-off-by: Drew Newberry --- .agents/skills/test-release-canary/SKILL.md | 1 - .github/workflows/release-canary.yml | 1 - README.md | 6 +++-- architecture/compute-runtimes.md | 2 +- crates/openshell-driver-kubernetes/README.md | 6 ++--- .../openshell-driver-kubernetes/src/config.rs | 23 ------------------- .../openshell-driver-kubernetes/src/main.rs | 8 ------- deploy/helm/openshell/README.md | 12 ++++++---- deploy/helm/openshell/README.md.gotmpl | 11 +++++---- deploy/helm/openshell/ci/values-keycloak.yaml | 3 +-- .../openshell/ci/values-openshift-scc.yaml | 3 +-- deploy/helm/openshell/ci/values-skaffold.yaml | 1 - .../openshell/templates/gateway-config.yaml | 1 - .../templates/network-policy-ack.yaml | 5 ---- .../tests/network_policy_ack_test.yaml | 20 ---------------- deploy/helm/openshell/values.yaml | 2 -- deploy/helm/test-split-ownership.sh | 2 -- docs/kubernetes/high-availability.mdx | 2 -- docs/kubernetes/ingress.mdx | 2 -- docs/kubernetes/managing-certificates.mdx | 2 -- docs/kubernetes/openshift.mdx | 12 +++++++--- docs/kubernetes/sandbox-runtime.mdx | 7 +++--- docs/kubernetes/setup.mdx | 20 ++++++++-------- docs/reference/gateway-config.mdx | 8 ++++--- docs/reference/sandbox-compute-drivers.mdx | 6 ++++- .../Dockerfile.external-kubernetes-gateway | 3 +-- examples/gateway-deploy-connect.md | 2 +- skills/debug-openshell-cluster/SKILL.md | 5 ++-- tasks/helm.toml | 4 ++-- 29 files changed, 64 insertions(+), 116 deletions(-) delete mode 100644 deploy/helm/openshell/templates/network-policy-ack.yaml delete mode 100644 deploy/helm/openshell/tests/network_policy_ack_test.yaml diff --git a/.agents/skills/test-release-canary/SKILL.md b/.agents/skills/test-release-canary/SKILL.md index 2a996bf0b5..4b95b805f8 100644 --- a/.agents/skills/test-release-canary/SKILL.md +++ b/.agents/skills/test-release-canary/SKILL.md @@ -109,7 +109,6 @@ helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart \ --namespace openshell --create-namespace \ --set server.disableTls=true \ --set server.telemetryEnabled=false \ - --set supervisor.sandboxRuntime.networkPolicyEnforced=true \ --wait --timeout 5m kubectl wait --namespace openshell \ diff --git a/.github/workflows/release-canary.yml b/.github/workflows/release-canary.yml index 723bb4b005..239f49eced 100644 --- a/.github/workflows/release-canary.yml +++ b/.github/workflows/release-canary.yml @@ -359,7 +359,6 @@ jobs: --set server.disableTls=true \ --set server.auth.allowUnauthenticatedUsers=true \ --set "server.telemetryEnabled=${OPENSHELL_TELEMETRY_ENABLED}" \ - --set supervisor.sandboxRuntime.networkPolicyEnforced=true \ --wait --timeout 5m - name: Verify gateway pod is Ready diff --git a/README.md b/README.md index 6049b609a3..4972acad1d 100644 --- a/README.md +++ b/README.md @@ -41,12 +41,14 @@ The installer installs the latest stable release by default. See [Prerelease and **Kubernetes installation:** > **Experimental** — the Kubernetes deployment path is under active development. Expect rough edges and breaking changes. +> **Required:** Your cluster CNI MUST enforce Kubernetes `NetworkPolicy` for +> ingress and egress in every sandbox namespace. OpenShell creates the policies, +> but Kubernetes does not verify that the CNI applies them. Deploy the OpenShell gateway into a Kubernetes cluster from the OCI chart published to GHCR: ```bash -helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart \ - --set supervisor.sandboxRuntime.networkPolicyEnforced=true +helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart ``` See [`deploy/helm/openshell/README.md`](deploy/helm/openshell/README.md) for available versions, dev tag conventions, and configuration. diff --git a/architecture/compute-runtimes.md b/architecture/compute-runtimes.md index f4745b3ac7..71a43bce2a 100644 --- a/architecture/compute-runtimes.md +++ b/architecture/compute-runtimes.md @@ -301,7 +301,7 @@ delete, reconciliation removes the row; otherwise it can remain `Deleting`. |---|---|---|---| | Docker | Local development with Docker available. | Capability-free workload container. | Uses `network_mode=none`; a separate capability-free supervisor container mediates egress and access over a private daemon-local Unix socket volume. | | Podman | Existing rootless driver. | Container. | Not converted by this isolation stack. | -| Kubernetes | Cluster deployment through Helm. | Capability-free sandbox Pod. | Uses one namespace-wide empty-egress workload NetworkPolicy and a separate capability-free supervisor Pod over mutually authenticated TLS. It requires an enforcing CNI and trusted sandbox namespace. | +| Kubernetes | Cluster deployment through Helm. | Capability-free sandbox Pod. | Always creates a namespace-wide empty-egress workload NetworkPolicy and a separate capability-free supervisor Pod over mutually authenticated TLS. It requires an enforcing CNI and trusted sandbox namespace; the Kubernetes API does not attest policy enforcement. | | VM | Experimental microVM isolation. | Per-sandbox libkrun or QEMU VM. | The NIC-less guest runs `openshell-sandbox` as PID 1; host `openshell-supervisor` owns gateway networking and reaches the guest over vsock. | | Extension | Out-of-tree drivers operated alongside the gateway. | Whatever boundary the driver implements. | Selected by a custom `compute_drivers = [""]` entry with `[openshell.drivers.].socket_path`, or at launch time by pairing `--drivers ` with `--compute-driver-socket=`. A launch-time endpoint may use a canonical built-in name to preserve its driver-config key while replacing in-process construction. The gateway connects to an operator-provisioned UDS, snapshots `GetCapabilities`, and dispatches all sandbox lifecycle calls through `compute_driver.proto`. The driver process and socket lifecycle are operator-owned; the gateway does not spawn, supervise, or remove unmanaged extension drivers. The trust boundary is the socket's filesystem permissions: the operator must ensure only the gateway uid can read/write it. | diff --git a/crates/openshell-driver-kubernetes/README.md b/crates/openshell-driver-kubernetes/README.md index 084f197fb6..a71fa55d0a 100644 --- a/crates/openshell-driver-kubernetes/README.md +++ b/crates/openshell-driver-kubernetes/README.md @@ -81,9 +81,9 @@ and permits OpenShell supervisor Pods to reach the sandbox TLS port. The authenticated Sandbox Protocol binds each connection to the exact sandbox and supervisor Pod identities. Supervisors have normal egress for gateway, DNS, and policy-approved upstream connections unless an operator policy restricts -them. Set -`sandbox_runtime.network_policy_enforced = true` only after verifying that the cluster -CNI enforces ingress and egress `NetworkPolicy` for sandbox namespaces. +them. The cluster CNI must enforce ingress and egress `NetworkPolicy` for every +sandbox namespace. Kubernetes accepts policy objects without confirming +enforcement, so operators must verify CNI support before running sandboxes. Each sandbox generation uses two immutable bootstrap Secrets. A trusted init container stages the sandbox bootstrap into memory, and the sandbox removes it diff --git a/crates/openshell-driver-kubernetes/src/config.rs b/crates/openshell-driver-kubernetes/src/config.rs index c9697021ae..40bcdc8422 100644 --- a/crates/openshell-driver-kubernetes/src/config.rs +++ b/crates/openshell-driver-kubernetes/src/config.rs @@ -78,9 +78,6 @@ pub const DEFAULT_WORKSPACE_STORAGE_SIZE: &str = "2Gi"; #[derive(Debug, Clone, Serialize, Deserialize)] #[serde(default, deny_unknown_fields)] pub struct KubernetesSandboxRuntimeConfig { - /// Explicit operator assertion that the cluster CNI enforces - /// `networking.k8s.io/v1` `NetworkPolicy` for the sandbox namespaces. - pub network_policy_enforced: bool, /// TCP port exposed by the workload boundary to its paired control pod. pub boundary_port: u16, } @@ -88,7 +85,6 @@ pub struct KubernetesSandboxRuntimeConfig { impl Default for KubernetesSandboxRuntimeConfig { fn default() -> Self { Self { - network_policy_enforced: false, boundary_port: 5500, } } @@ -96,12 +92,6 @@ impl Default for KubernetesSandboxRuntimeConfig { impl KubernetesSandboxRuntimeConfig { pub fn validate(&self) -> Result<(), String> { - if !self.network_policy_enforced { - return Err( - "sandbox_runtime.network_policy_enforced must be true after the operator has verified CNI NetworkPolicy enforcement" - .to_string(), - ); - } if self.boundary_port < 1024 { return Err("sandbox_runtime.boundary_port must be at least 1024".to_string()); } @@ -887,19 +877,6 @@ mod tests { assert!(cfg.workspace_storage_class.is_empty()); } - #[test] - fn sandbox_runtime_requires_network_policy_enforcement_acknowledgement() { - let mut cfg = KubernetesComputeConfig::default(); - assert!( - cfg.validate_proxy_uid() - .unwrap_err() - .contains("network_policy_enforced") - ); - - cfg.sandbox_runtime.network_policy_enforced = true; - cfg.validate_proxy_uid().unwrap(); - } - #[test] fn serde_rejects_sidecar_binary_identity_field() { let json = serde_json::json!({ diff --git a/crates/openshell-driver-kubernetes/src/main.rs b/crates/openshell-driver-kubernetes/src/main.rs index fcb8faa1e6..7fbce18fee 100644 --- a/crates/openshell-driver-kubernetes/src/main.rs +++ b/crates/openshell-driver-kubernetes/src/main.rs @@ -128,13 +128,6 @@ struct Args { #[arg(long, env = "OPENSHELL_SUPERVISOR_IMAGE_PULL_POLICY")] supervisor_image_pull_policy: Option, - #[arg( - long, - env = "OPENSHELL_K8S_SANDBOX_RUNTIME_NETWORK_POLICY_ENFORCED", - default_value_t = false - )] - sandbox_runtime_network_policy_enforced: bool, - #[arg( long, env = "OPENSHELL_K8S_SANDBOX_RUNTIME_BOUNDARY_PORT", @@ -270,7 +263,6 @@ async fn main() -> Result<()> { .unwrap_or_else(openshell_core::config::default_supervisor_image), supervisor_image_pull_policy: args.supervisor_image_pull_policy, sandbox_runtime: KubernetesSandboxRuntimeConfig { - network_policy_enforced: args.sandbox_runtime_network_policy_enforced, boundary_port: args.sandbox_runtime_boundary_port, }, https_proxy: args.https_proxy, diff --git a/deploy/helm/openshell/README.md b/deploy/helm/openshell/README.md index f90359832f..4ab3517b27 100644 --- a/deploy/helm/openshell/README.md +++ b/deploy/helm/openshell/README.md @@ -19,6 +19,12 @@ multiple pre-provisioned workspace namespaces. ## Prerequisites +> **Required:** Your cluster CNI MUST enforce Kubernetes `NetworkPolicy` for +> ingress and egress in every sandbox namespace. OpenShell creates the policies, +> but Kubernetes accepts them even if no CNI enforces them. Without enforcement, +> sandbox workloads may connect directly and bypass supervisor network policy. +> Verify CNI support before installing OpenShell. + The Kubernetes Agent Sandbox CRDs and controller must be installed on the cluster before deploying OpenShell. Install them with: ```shell @@ -35,8 +41,7 @@ where Helm cannot discover cluster APIs. ## Install on Kubernetes ```shell -helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart --version \ - --set supervisor.sandboxRuntime.networkPolicyEnforced=true +helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart --version ``` ## Install on OpenShift @@ -49,7 +54,6 @@ oc create ns openshell # Deploy openshell with overrides to allow SCC assignment of fsGroup and runAsUser for the gateway helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart --version -n openshell \ - --set supervisor.sandboxRuntime.networkPolicyEnforced=true \ --set server.disableTls=true \ --set podSecurityContext.fsGroup=null \ --set securityContext.runAsUser=null @@ -110,7 +114,6 @@ Then install the chart pointing at that Secret: ```bash helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart --version \ -n openshell \ - --set supervisor.sandboxRuntime.networkPolicyEnforced=true \ --set workload.kind=deployment \ --set server.externalDbSecret=my-pg-credentials ``` @@ -352,7 +355,6 @@ discovery endpoint or its TLS CA. | supervisor.image.repository | string | `"openshell/supervisor"` | Supervisor image repository. | | supervisor.image.tag | string | `""` | Supervisor image tag. Defaults to the chart appVersion when empty. | | supervisor.sandboxRuntime.boundaryPort | int | `5500` | Workload boundary TLS listener port. | -| supervisor.sandboxRuntime.networkPolicyEnforced | bool | `false` | Required operator acknowledgement that the cluster CNI enforces NetworkPolicy. | | tolerations | list | `[]` | Tolerations for the gateway pod. | | upstreamProxy | object | `{"authAllowInsecure":false,"authSecret":{"key":"","name":""},"caBundle":{"configMapName":"","key":"ca.crt"},"connectByHostname":false,"noProxy":"","url":""}` | Operator-owned corporate forward proxy for policy-approved TLS egress from Kubernetes sandboxes. The workload cannot select or override it. | | upstreamProxy.authAllowInsecure | bool | `false` | Required when authSecret is configured because Basic auth to an HTTP proxy is cleartext. | diff --git a/deploy/helm/openshell/README.md.gotmpl b/deploy/helm/openshell/README.md.gotmpl index 59e8118f3b..6e49260c20 100644 --- a/deploy/helm/openshell/README.md.gotmpl +++ b/deploy/helm/openshell/README.md.gotmpl @@ -19,6 +19,12 @@ multiple pre-provisioned workspace namespaces. ## Prerequisites +> **Required:** Your cluster CNI MUST enforce Kubernetes `NetworkPolicy` for +> ingress and egress in every sandbox namespace. OpenShell creates the policies, +> but Kubernetes accepts them even if no CNI enforces them. Without enforcement, +> sandbox workloads may connect directly and bypass supervisor network policy. +> Verify CNI support before installing OpenShell. + The Kubernetes Agent Sandbox CRDs and controller must be installed on the cluster before deploying OpenShell. Install them with: ```shell @@ -35,8 +41,7 @@ where Helm cannot discover cluster APIs. ## Install on Kubernetes ```shell -helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart --version \ - --set supervisor.sandboxRuntime.networkPolicyEnforced=true +helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart --version ``` ## Install on OpenShift @@ -49,7 +54,6 @@ oc create ns openshell # Deploy openshell with overrides to allow SCC assignment of fsGroup and runAsUser for the gateway helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart --version -n openshell \ - --set supervisor.sandboxRuntime.networkPolicyEnforced=true \ --set server.disableTls=true \ --set podSecurityContext.fsGroup=null \ --set securityContext.runAsUser=null @@ -110,7 +114,6 @@ Then install the chart pointing at that Secret: ```bash helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart --version \ -n openshell \ - --set supervisor.sandboxRuntime.networkPolicyEnforced=true \ --set workload.kind=deployment \ --set server.externalDbSecret=my-pg-credentials ``` diff --git a/deploy/helm/openshell/ci/values-keycloak.yaml b/deploy/helm/openshell/ci/values-keycloak.yaml index ae1810f414..6df74e3258 100644 --- a/deploy/helm/openshell/ci/values-keycloak.yaml +++ b/deploy/helm/openshell/ci/values-keycloak.yaml @@ -8,8 +8,7 @@ # # Then layer this file on top of values.yaml when deploying: # helm upgrade --install openshell . \ -# -f values.yaml -f ci/values-skaffold.yaml -f ci/values-keycloak.yaml \ -# --set supervisor.sandboxRuntime.networkPolicyEnforced=true +# -f values.yaml -f ci/values-skaffold.yaml -f ci/values-keycloak.yaml # # Or add this file to skaffold.yaml valuesFiles for iterative dev. # diff --git a/deploy/helm/openshell/ci/values-openshift-scc.yaml b/deploy/helm/openshell/ci/values-openshift-scc.yaml index 8f1a8d07d6..b7f37be6e0 100644 --- a/deploy/helm/openshell/ci/values-openshift-scc.yaml +++ b/deploy/helm/openshell/ci/values-openshift-scc.yaml @@ -4,8 +4,7 @@ # OpenShift SCC compatibility overlay. Removes the hardcoded runAsUser and # fsGroup so that OpenShift's restricted-v2 SCC can inject the namespace- # assigned UID/GID range. Layer after values.yaml: -# helm install openshell deploy/helm/openshell -f ci/values-openshift-scc.yaml \ -# --set supervisor.sandboxRuntime.networkPolicyEnforced=true +# helm install openshell deploy/helm/openshell -f ci/values-openshift-scc.yaml # # The e2e Kubernetes harness applies this automatically when it detects an # OpenShift cluster (route.openshift.io API present). diff --git a/deploy/helm/openshell/ci/values-skaffold.yaml b/deploy/helm/openshell/ci/values-skaffold.yaml index c3c1a84a4e..8def2796a4 100644 --- a/deploy/helm/openshell/ci/values-skaffold.yaml +++ b/deploy/helm/openshell/ci/values-skaffold.yaml @@ -20,4 +20,3 @@ supervisor: # The local k3s cluster created by `mise run helm:k3s:create` enables its # built-in NetworkPolicy controller for sandbox namespaces. sandboxRuntime: - networkPolicyEnforced: true diff --git a/deploy/helm/openshell/templates/gateway-config.yaml b/deploy/helm/openshell/templates/gateway-config.yaml index 13b453041b..b1ec38bd0a 100644 --- a/deploy/helm/openshell/templates/gateway-config.yaml +++ b/deploy/helm/openshell/templates/gateway-config.yaml @@ -232,7 +232,6 @@ data: gateway_pod_selector = { "app.kubernetes.io/name" = {{ include "openshell.name" . | quote }}, "app.kubernetes.io/instance" = {{ .Release.Name | quote }} } [openshell.drivers.kubernetes.sandbox_runtime] - network_policy_enforced = {{ .Values.supervisor.sandboxRuntime.networkPolicyEnforced }} boundary_port = {{ .Values.supervisor.sandboxRuntime.boundaryPort | default 5500 }} {{- if not $credentialDrivers }} diff --git a/deploy/helm/openshell/templates/network-policy-ack.yaml b/deploy/helm/openshell/templates/network-policy-ack.yaml deleted file mode 100644 index 780d4314c9..0000000000 --- a/deploy/helm/openshell/templates/network-policy-ack.yaml +++ /dev/null @@ -1,5 +0,0 @@ -# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -# SPDX-License-Identifier: Apache-2.0 -{{- if not .Values.supervisor.sandboxRuntime.networkPolicyEnforced }} -{{- fail "supervisor.sandboxRuntime.networkPolicyEnforced must be true after you verify that the cluster CNI enforces ingress and egress NetworkPolicy in every sandbox namespace" }} -{{- end }} diff --git a/deploy/helm/openshell/tests/network_policy_ack_test.yaml b/deploy/helm/openshell/tests/network_policy_ack_test.yaml deleted file mode 100644 index 12ca399e73..0000000000 --- a/deploy/helm/openshell/tests/network_policy_ack_test.yaml +++ /dev/null @@ -1,20 +0,0 @@ -# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -# SPDX-License-Identifier: Apache-2.0 - -suite: NetworkPolicy enforcement acknowledgement -templates: - - templates/network-policy-ack.yaml -tests: - - it: rejects an install without operator acknowledgement - set: - supervisor.sandboxRuntime.networkPolicyEnforced: false - asserts: - - failedTemplate: - errorMessage: supervisor.sandboxRuntime.networkPolicyEnforced must be true after you verify that the cluster CNI enforces ingress and egress NetworkPolicy in every sandbox namespace - - - it: accepts an acknowledged CNI - set: - supervisor.sandboxRuntime.networkPolicyEnforced: true - asserts: - - hasDocuments: - count: 0 diff --git a/deploy/helm/openshell/values.yaml b/deploy/helm/openshell/values.yaml index 34067eb31d..b48a7c269f 100644 --- a/deploy/helm/openshell/values.yaml +++ b/deploy/helm/openshell/values.yaml @@ -70,8 +70,6 @@ supervisor: # -- Supervisor image digest. When set, this takes precedence over tag. digest: "" sandboxRuntime: - # -- Required operator acknowledgement that the cluster CNI enforces NetworkPolicy. - networkPolicyEnforced: false # -- Workload boundary TLS listener port. boundaryPort: 5500 diff --git a/deploy/helm/test-split-ownership.sh b/deploy/helm/test-split-ownership.sh index a7a865363b..30fd7360d4 100755 --- a/deploy/helm/test-split-ownership.sh +++ b/deploy/helm/test-split-ownership.sh @@ -11,7 +11,6 @@ trap 'rm -rf "${work_dir}"' EXIT helm template openshell "${repo_root}/deploy/helm/openshell" \ --namespace openshell \ --set agentSandbox.preflight.enabled=false \ - --set supervisor.sandboxRuntime.networkPolicyEnforced=true \ --set workspaceResources.enabled=false \ >"${work_dir}/gateway.yaml" @@ -41,7 +40,6 @@ fi helm template openshell "${repo_root}/deploy/helm/openshell" \ --namespace openshell \ --set agentSandbox.preflight.enabled=false \ - --set supervisor.sandboxRuntime.networkPolicyEnforced=true \ --set-json workspaceResources=null \ >"${work_dir}/legacy-reuse-values.yaml" diff --git a/docs/kubernetes/high-availability.mdx b/docs/kubernetes/high-availability.mdx index c45ce9a1f6..f83b181462 100644 --- a/docs/kubernetes/high-availability.mdx +++ b/docs/kubernetes/high-availability.mdx @@ -90,7 +90,6 @@ helm upgrade --install openshell \ oci://ghcr.io/nvidia/openshell/helm-chart \ --version \ --namespace openshell \ - --set supervisor.sandboxRuntime.networkPolicyEnforced=true \ --values values-ha.yaml \ --wait ``` @@ -196,7 +195,6 @@ helm upgrade openshell \ oci://ghcr.io/nvidia/openshell/helm-chart \ --version \ --namespace openshell \ - --set supervisor.sandboxRuntime.networkPolicyEnforced=true \ --values values-ha.yaml \ --wait ``` diff --git a/docs/kubernetes/ingress.mdx b/docs/kubernetes/ingress.mdx index fd6be78fbd..c9e081065b 100644 --- a/docs/kubernetes/ingress.mdx +++ b/docs/kubernetes/ingress.mdx @@ -57,7 +57,6 @@ helm upgrade --install openshell \ oci://ghcr.io/nvidia/openshell/helm-chart \ --version \ --namespace openshell \ - --set supervisor.sandboxRuntime.networkPolicyEnforced=true \ --set grpcRoute.enabled=true \ --set grpcRoute.gateway.create=true \ --set grpcRoute.gateway.className=eg @@ -114,7 +113,6 @@ helm upgrade --install openshell \ oci://ghcr.io/nvidia/openshell/helm-chart \ --version \ --namespace openshell \ - --set supervisor.sandboxRuntime.networkPolicyEnforced=true \ --set grpcRoute.enabled=true \ --set grpcRoute.gateway.create=true \ --set grpcRoute.gateway.className=eg \ diff --git a/docs/kubernetes/managing-certificates.mdx b/docs/kubernetes/managing-certificates.mdx index 13815e5cf9..322815cbf1 100644 --- a/docs/kubernetes/managing-certificates.mdx +++ b/docs/kubernetes/managing-certificates.mdx @@ -52,7 +52,6 @@ helm upgrade --install openshell \ oci://ghcr.io/nvidia/openshell/helm-chart \ --version \ --namespace openshell \ - --set supervisor.sandboxRuntime.networkPolicyEnforced=true \ --set certManager.enabled=true ``` @@ -74,7 +73,6 @@ helm upgrade --install openshell \ oci://ghcr.io/nvidia/openshell/helm-chart \ --version \ --namespace openshell \ - --set supervisor.sandboxRuntime.networkPolicyEnforced=true \ --set certManager.enabled=true \ --set certManager.serverIssuerRef.name=letsencrypt-prod \ --set certManager.serverIssuerRef.kind=ClusterIssuer \ diff --git a/docs/kubernetes/openshift.mdx b/docs/kubernetes/openshift.mdx index e510fc44c4..ecfddefd0e 100644 --- a/docs/kubernetes/openshift.mdx +++ b/docs/kubernetes/openshift.mdx @@ -41,17 +41,23 @@ in the sandbox qualification output. - [Agent Sandbox](/kubernetes/setup#install-agent-sandbox) controller and CRDs. - A CNI that enforces ingress and egress `NetworkPolicy` in sandbox namespaces. + +Your cluster MUST enforce ingress and egress `NetworkPolicy` in every sandbox +namespace. OpenShell creates the policies, but Kubernetes does not verify that +the CNI applies them. Without enforcement, sandbox workloads may bypass +supervisor network policy through direct connections. + + ## Install OpenShell Pre-create the namespace, then install the chart. Keep the default restricted -security posture and acknowledge NetworkPolicy only after validating the CNI. +security posture. ```shell oc create ns openshell helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart \ --version \ - --namespace openshell \ - --set supervisor.sandboxRuntime.networkPolicyEnforced=true + --namespace openshell ``` The driver reads the namespace's `openshift.io/sa.scc.uid-range` annotation and diff --git a/docs/kubernetes/sandbox-runtime.mdx b/docs/kubernetes/sandbox-runtime.mdx index 1376916b44..ac22288af4 100644 --- a/docs/kubernetes/sandbox-runtime.mdx +++ b/docs/kubernetes/sandbox-runtime.mdx @@ -81,9 +81,10 @@ restricts them. Kubernetes policies are additive. Keep sandbox namespaces under administrative control so another principal cannot add permissive policies, create Pods with -OpenShell labels, or read bootstrap Secrets. Set -`supervisor.sandboxRuntime.networkPolicyEnforced: true` only after you verify that the -cluster CNI enforces both ingress and egress policies for these namespaces. +OpenShell labels, or read bootstrap Secrets. The cluster CNI MUST enforce both +ingress and egress `NetworkPolicy` in every sandbox namespace. Kubernetes accepts +policy objects even when no CNI enforces them; OpenShell does not test traffic +to verify enforcement. ## Bootstrap a Sandbox diff --git a/docs/kubernetes/setup.mdx b/docs/kubernetes/setup.mdx index 95c3e6c620..dc9a138207 100644 --- a/docs/kubernetes/setup.mdx +++ b/docs/kubernetes/setup.mdx @@ -12,6 +12,14 @@ position: 1 The OpenShell Helm chart is experimental and under active development. Templates, values, and defaults can change between releases. Do not use it in production. + +Your cluster MUST use a CNI that enforces Kubernetes `NetworkPolicy` for both +ingress and egress in every sandbox namespace. OpenShell creates the policies, +but Kubernetes accepts them even when no CNI enforces them. Without enforcement, +sandbox workloads may reach the network directly and bypass supervisor policy. +Verify CNI support before installing OpenShell. + + Use the Kubernetes deployment when the gateway should run on a shared cluster, in a cloud environment, or as part of team infrastructure. The Helm chart handles PKI bootstrap, RBAC, sandbox namespace setup, and the gateway workload. It uses a StatefulSet by default for the SQLite database, and can render a Deployment when `server.externalDbSecret` points at an external database. ## Prerequisites @@ -21,6 +29,7 @@ Make sure the following are in place before you install. | Prerequisite | Required | Notes | |---|---|---| | Kubernetes 1.29+ with RBAC enabled | Yes | No additional notes. | +| CNI that enforces ingress and egress `NetworkPolicy` in sandbox namespaces | Yes | Verify enforcement on your cluster. | | Helm 3.x | Yes | No additional notes. | | Agent Sandbox controller and CRDs | Yes | Install before the OpenShell chart. Refer to [Install Agent Sandbox](#install-agent-sandbox). | | cert-manager | No | Refer to [Managing Certificates](/kubernetes/managing-certificates). Use cert-manager only if you prefer it over the built-in PKI job. | @@ -85,8 +94,7 @@ Install from the OCI registry on GHCR. Replace `` with the chart versio helm upgrade --install openshell \ oci://ghcr.io/nvidia/openshell/helm-chart \ --version \ - --namespace openshell \ - --set supervisor.sandboxRuntime.networkPolicyEnforced=true + --namespace openshell ``` To use the latest development build instead of a stable release: @@ -95,8 +103,7 @@ To use the latest development build instead of a stable release: helm upgrade --install openshell \ oci://ghcr.io/nvidia/openshell/helm-chart \ --version 0.0.0-dev \ - --namespace openshell \ - --set supervisor.sandboxRuntime.networkPolicyEnforced=true + --namespace openshell ``` The chart automatically generates PKI secrets on first install using pre-install Helm hooks. No manual secret creation is required. @@ -111,7 +118,6 @@ helm upgrade --install openshell \ oci://ghcr.io/nvidia/openshell/helm-chart \ --version \ --namespace openshell \ - --set supervisor.sandboxRuntime.networkPolicyEnforced=true \ --set workspaceResources.enabled=false \ --set server.sandboxNamespace=app-a @@ -149,7 +155,6 @@ helm upgrade --install openshell \ oci://ghcr.io/nvidia/openshell/helm-chart \ --version \ --namespace openshell \ - --set supervisor.sandboxRuntime.networkPolicyEnforced=true \ --set server.credentialDrivers.kubernetesSecrets.enabled=true \ --set server.credentialDrivers.kubernetesSecrets.namespace=openshell-credentials \ --set server.credentialDrivers.kubernetesSecrets.createNamespace=true @@ -237,7 +242,6 @@ The most commonly changed values are: | `server.auth.allowUnauthenticatedUsers` | Accept user-facing calls without OIDC or mTLS credentials. Use only for trusted local development or a fully trusted access proxy. | | `server.enableLoopbackServiceHttp` | Enable local plaintext HTTP for loopback sandbox service URLs. Defaults to `true`. | | `pkiInitJob.serverDnsNames` / `certManager.serverDnsNames` | Additional gateway server DNS SANs. Wildcard SANs also enable sandbox service URLs under that domain. | -| `supervisor.sandboxRuntime.networkPolicyEnforced` | Required acknowledgement that the cluster CNI enforces ingress and egress `NetworkPolicy` in sandbox namespaces. | | `supervisor.sandboxRuntime.boundaryPort` | Non-privileged TLS port used between paired supervisor and sandbox Pods. | | `upstreamProxy` | Operator-owned corporate HTTP forward proxy for policy-approved TLS egress. Refer to [Configure a Corporate Upstream Proxy](#configure-a-corporate-upstream-proxy). | @@ -254,7 +258,6 @@ helm upgrade --install openshell \ oci://ghcr.io/nvidia/openshell/helm-chart \ --version \ --namespace openshell \ - --set supervisor.sandboxRuntime.networkPolicyEnforced=true \ --values my-values.yaml ``` @@ -350,7 +353,6 @@ helm upgrade --install openshell \ oci://ghcr.io/nvidia/openshell/helm-chart \ --version \ --namespace openshell \ - --set supervisor.sandboxRuntime.networkPolicyEnforced=true \ --set serviceAccount.create=false \ --set serviceAccount.name=my-existing-sa ``` diff --git a/docs/reference/gateway-config.mdx b/docs/reference/gateway-config.mdx index 06404a69de..e032ff8a89 100644 --- a/docs/reference/gateway-config.mdx +++ b/docs/reference/gateway-config.mdx @@ -791,13 +791,15 @@ gateway_namespace = "openshell" gateway_pod_selector = { "app.kubernetes.io/name" = "openshell", "app.kubernetes.io/instance" = "openshell" } [openshell.drivers.kubernetes.sandbox_runtime] -# Required acknowledgement that the cluster CNI enforces NetworkPolicy and the -# sandbox namespaces prevent untrusted policy, pod, label, and Secret changes. -network_policy_enforced = true # TLS-protected boundary listener reached only by the paired control pod. boundary_port = 5500 ``` +The Kubernetes driver always creates the workload `NetworkPolicy` before +releasing a sandbox Pod. Your cluster CNI must enforce ingress and egress +`NetworkPolicy` in every sandbox namespace; the Kubernetes API does not verify +enforcement. + In managed workspace mode, the Kubernetes driver copies each explicitly named `image_pull_secrets` Secret from `namespace` into the managed workspace namespace on sandbox creation. Shared and operator modes require the Secret to diff --git a/docs/reference/sandbox-compute-drivers.mdx b/docs/reference/sandbox-compute-drivers.mdx index 383f24a329..395a04edf8 100644 --- a/docs/reference/sandbox-compute-drivers.mdx +++ b/docs/reference/sandbox-compute-drivers.mdx @@ -442,6 +442,11 @@ than being staged into the guest. See the [Gateway Configuration File](./gateway Kubernetes-backed sandboxes run as pods in the configured sandbox namespace. Use Kubernetes for shared clusters, remote compute, GPU scheduling, and operator-managed environments. +The cluster CNI MUST enforce ingress and egress `NetworkPolicy` in every sandbox +namespace. OpenShell creates the workload policy, but the Kubernetes API cannot +confirm enforcement. Without it, workload pods may connect directly and bypass +supervisor network policy. + Kubernetes workspace namespaces are an administrative trust boundary. In shared and managed modes, only the OpenShell gateway and its trusted Agent Sandbox controller may administer Sandbox CRs, sandbox pods, or the configured @@ -471,7 +476,6 @@ For maintainer-level implementation details, refer to the [Kubernetes driver REA | `sandbox_runtime_image_pull_policy` | `sandboxRuntime.image.pullPolicy` | Set the Kubernetes image pull policy for the sandbox runtime image. | | `supervisor_image` | `supervisor.image.registry` / `supervisor.image.repository` / `supervisor.image.tag` / `supervisor.image.digest` | Override the image that provides `openshell-supervisor`. Individual image values take precedence over `global.image`; a digest takes precedence over the tag. | | `supervisor_image_pull_policy` | `supervisor.image.pullPolicy` | Set the canonical supervisor pull policy: `always`, `if_not_present`, or `never`. `newer` is Podman-only. | -| `sandbox_runtime.network_policy_enforced` | `supervisor.sandboxRuntime.networkPolicyEnforced` | Acknowledge that the cluster CNI enforces ingress and egress `NetworkPolicy` in sandbox namespaces. This must be `true`. | | `sandbox_runtime.boundary_port` | `supervisor.sandboxRuntime.boundaryPort` | Set the non-privileged TLS port used between the paired supervisor and sandbox Pods. | | `https_proxy` | `upstreamProxy.url` | Set the operator-owned `http://host:port` or `https://host:port` corporate forward proxy used for policy-approved TLS CONNECT egress. | | `no_proxy` | `upstreamProxy.noProxy` | Set destinations that bypass only the corporate proxy. OpenShell policy evaluation still applies. | diff --git a/e2e/docker/Dockerfile.external-kubernetes-gateway b/e2e/docker/Dockerfile.external-kubernetes-gateway index 3b860dacc7..ffa1072bb7 100644 --- a/e2e/docker/Dockerfile.external-kubernetes-gateway +++ b/e2e/docker/Dockerfile.external-kubernetes-gateway @@ -24,8 +24,7 @@ ENV OPENSHELL_COMPUTE_DRIVER=kubernetes \ OPENSHELL_SUPERVISOR_IMAGE_PULL_POLICY=if_not_present \ OPENSHELL_SANDBOX_RUNTIME_IMAGE=${SANDBOX_RUNTIME_IMAGE} \ OPENSHELL_SANDBOX_RUNTIME_IMAGE_PULL_POLICY=if_not_present \ - OPENSHELL_SUPERVISOR_SIDELOAD_METHOD=init-container \ - OPENSHELL_K8S_SANDBOX_RUNTIME_NETWORK_POLICY_ENFORCED=true + OPENSHELL_SUPERVISOR_SIDELOAD_METHOD=init-container USER 1000:1000 EXPOSE 8080 diff --git a/examples/gateway-deploy-connect.md b/examples/gateway-deploy-connect.md index 2861444d1c..2ad4f29303 100644 --- a/examples/gateway-deploy-connect.md +++ b/examples/gateway-deploy-connect.md @@ -6,6 +6,7 @@ Deploy or register an OpenShell gateway, verify it is reachable, and run your fi - OpenShell CLI installed (`openshell`) - A reachable gateway endpoint, or access to a Kubernetes cluster where you can install the Helm chart +- For Kubernetes installs, a CNI that enforces ingress and egress `NetworkPolicy` in sandbox namespaces ## Helm Deployment @@ -15,7 +16,6 @@ Install the gateway into a Kubernetes cluster you manage: kubectl create namespace openshell helm upgrade --install openshell deploy/helm/openshell \ --namespace openshell \ - --set supervisor.sandboxRuntime.networkPolicyEnforced=true \ --set server.disableTls=true \ --set service.type=ClusterIP ``` diff --git a/skills/debug-openshell-cluster/SKILL.md b/skills/debug-openshell-cluster/SKILL.md index 406e7209d2..22e35fbb27 100644 --- a/skills/debug-openshell-cluster/SKILL.md +++ b/skills/debug-openshell-cluster/SKILL.md @@ -708,9 +708,8 @@ kubectl -n get sandbox -o jsonpath='{.spec.te ``` The Kubernetes driver creates a sandbox workload Pod and a separate, directly -managed supervisor Pod. Helm must render -`network_policy_enforced = true`. This is an explicit operator assertion that -the cluster CNI enforces Kubernetes NetworkPolicy; the Kubernetes API cannot +managed supervisor Pod. The cluster CNI must enforce ingress and egress +Kubernetes NetworkPolicy in every sandbox namespace; the Kubernetes API cannot attest enforcement. Run sandboxes only in a trusted namespace where tenants cannot create Pods, copy OpenShell role labels, or read the bootstrap Secret. diff --git a/tasks/helm.toml b/tasks/helm.toml index 33b3550dff..43f01bf3f1 100644 --- a/tasks/helm.toml +++ b/tasks/helm.toml @@ -39,12 +39,12 @@ run = """ helm dependency build deploy/helm/openshell echo "--- helm lint: defaults ---" echo "values files: deploy/helm/openshell/values.yaml" - helm lint deploy/helm/openshell --set agentSandbox.preflight.enabled=false --set supervisor.sandboxRuntime.networkPolicyEnforced=true + helm lint deploy/helm/openshell --set agentSandbox.preflight.enabled=false for f in deploy/helm/openshell/ci/values-*.yaml; do variant=$(basename "$f" .yaml | sed 's/values-//') echo "--- helm lint: $variant ---" echo "values files: deploy/helm/openshell/values.yaml, $f" - helm lint deploy/helm/openshell -f "$f" --set agentSandbox.preflight.enabled=false --set supervisor.sandboxRuntime.networkPolicyEnforced=true + helm lint deploy/helm/openshell -f "$f" --set agentSandbox.preflight.enabled=false done echo "--- helm lint: workspace defaults ---" helm lint deploy/helm/openshell-workspace