Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion .agents/skills/test-release-canary/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -109,7 +109,6 @@ helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart \
--namespace openshell --create-namespace \
--set server.disableTls=true \
--set server.telemetryEnabled=false \
--set supervisor.sandboxRuntime.networkPolicyEnforced=true \
--wait --timeout 5m

kubectl wait --namespace openshell \
Expand Down
1 change: 0 additions & 1 deletion .github/workflows/release-canary.yml
Original file line number Diff line number Diff line change
Expand Up @@ -359,7 +359,6 @@ jobs:
--set server.disableTls=true \
--set server.auth.allowUnauthenticatedUsers=true \
--set "server.telemetryEnabled=${OPENSHELL_TELEMETRY_ENABLED}" \
--set supervisor.sandboxRuntime.networkPolicyEnforced=true \
--wait --timeout 5m

- name: Verify gateway pod is Ready
Expand Down
6 changes: 4 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,12 +41,14 @@ The installer installs the latest stable release by default. See [Prerelease and
**Kubernetes installation:**

> **Experimental** — the Kubernetes deployment path is under active development. Expect rough edges and breaking changes.
> **Required:** Your cluster CNI MUST enforce Kubernetes `NetworkPolicy` for
> ingress and egress in every sandbox namespace. OpenShell creates the policies,
> but Kubernetes does not verify that the CNI applies them.

Deploy the OpenShell gateway into a Kubernetes cluster from the OCI chart published to GHCR:

```bash
helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart \
--set supervisor.sandboxRuntime.networkPolicyEnforced=true
helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart
```

See [`deploy/helm/openshell/README.md`](deploy/helm/openshell/README.md) for available versions, dev tag conventions, and configuration.
Expand Down
2 changes: 1 addition & 1 deletion architecture/compute-runtimes.md
Original file line number Diff line number Diff line change
Expand Up @@ -301,7 +301,7 @@ delete, reconciliation removes the row; otherwise it can remain `Deleting`.
|---|---|---|---|
| Docker | Local development with Docker available. | Capability-free workload container. | Uses `network_mode=none`; a separate capability-free supervisor container mediates egress and access over a private daemon-local Unix socket volume. |
| Podman | Existing rootless driver. | Container. | Not converted by this isolation stack. |
| Kubernetes | Cluster deployment through Helm. | Capability-free sandbox Pod. | Uses one namespace-wide empty-egress workload NetworkPolicy and a separate capability-free supervisor Pod over mutually authenticated TLS. It requires an enforcing CNI and trusted sandbox namespace. |
| Kubernetes | Cluster deployment through Helm. | Capability-free sandbox Pod. | Always creates a namespace-wide empty-egress workload NetworkPolicy and a separate capability-free supervisor Pod over mutually authenticated TLS. It requires an enforcing CNI and trusted sandbox namespace; the Kubernetes API does not attest policy enforcement. |
| VM | Experimental microVM isolation. | Per-sandbox libkrun or QEMU VM. | The NIC-less guest runs `openshell-sandbox` as PID 1; host `openshell-supervisor` owns gateway networking and reaches the guest over vsock. |
| Extension | Out-of-tree drivers operated alongside the gateway. | Whatever boundary the driver implements. | Selected by a custom `compute_drivers = ["<name>"]` entry with `[openshell.drivers.<name>].socket_path`, or at launch time by pairing `--drivers <name>` with `--compute-driver-socket=<path>`. A launch-time endpoint may use a canonical built-in name to preserve its driver-config key while replacing in-process construction. The gateway connects to an operator-provisioned UDS, snapshots `GetCapabilities`, and dispatches all sandbox lifecycle calls through `compute_driver.proto`. The driver process and socket lifecycle are operator-owned; the gateway does not spawn, supervise, or remove unmanaged extension drivers. The trust boundary is the socket's filesystem permissions: the operator must ensure only the gateway uid can read/write it. |

Expand Down
6 changes: 3 additions & 3 deletions crates/openshell-driver-kubernetes/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -81,9 +81,9 @@ and permits OpenShell supervisor Pods to reach the sandbox TLS port. The
authenticated Sandbox Protocol binds each connection to the exact sandbox and
supervisor Pod identities. Supervisors have normal egress for gateway, DNS,
and policy-approved upstream connections unless an operator policy restricts
them. Set
`sandbox_runtime.network_policy_enforced = true` only after verifying that the cluster
CNI enforces ingress and egress `NetworkPolicy` for sandbox namespaces.
them. The cluster CNI must enforce ingress and egress `NetworkPolicy` for every
sandbox namespace. Kubernetes accepts policy objects without confirming
enforcement, so operators must verify CNI support before running sandboxes.

Each sandbox generation uses two immutable bootstrap Secrets. A trusted init
container stages the sandbox bootstrap into memory, and the sandbox removes it
Expand Down
23 changes: 0 additions & 23 deletions crates/openshell-driver-kubernetes/src/config.rs
Original file line number Diff line number Diff line change
Expand Up @@ -78,30 +78,20 @@ pub const DEFAULT_WORKSPACE_STORAGE_SIZE: &str = "2Gi";
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(default, deny_unknown_fields)]
pub struct KubernetesSandboxRuntimeConfig {
/// Explicit operator assertion that the cluster CNI enforces
/// `networking.k8s.io/v1` `NetworkPolicy` for the sandbox namespaces.
pub network_policy_enforced: bool,
/// TCP port exposed by the workload boundary to its paired control pod.
pub boundary_port: u16,
}

impl Default for KubernetesSandboxRuntimeConfig {
fn default() -> Self {
Self {
network_policy_enforced: false,
boundary_port: 5500,
}
}
}

impl KubernetesSandboxRuntimeConfig {
pub fn validate(&self) -> Result<(), String> {
if !self.network_policy_enforced {
return Err(
"sandbox_runtime.network_policy_enforced must be true after the operator has verified CNI NetworkPolicy enforcement"
.to_string(),
);
}
if self.boundary_port < 1024 {
return Err("sandbox_runtime.boundary_port must be at least 1024".to_string());
}
Expand Down Expand Up @@ -887,19 +877,6 @@ mod tests {
assert!(cfg.workspace_storage_class.is_empty());
}

#[test]
fn sandbox_runtime_requires_network_policy_enforcement_acknowledgement() {
let mut cfg = KubernetesComputeConfig::default();
assert!(
cfg.validate_proxy_uid()
.unwrap_err()
.contains("network_policy_enforced")
);

cfg.sandbox_runtime.network_policy_enforced = true;
cfg.validate_proxy_uid().unwrap();
}

#[test]
fn serde_rejects_sidecar_binary_identity_field() {
let json = serde_json::json!({
Expand Down
8 changes: 0 additions & 8 deletions crates/openshell-driver-kubernetes/src/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -128,13 +128,6 @@ struct Args {
#[arg(long, env = "OPENSHELL_SUPERVISOR_IMAGE_PULL_POLICY")]
supervisor_image_pull_policy: Option<KubernetesImagePullPolicy>,

#[arg(
long,
env = "OPENSHELL_K8S_SANDBOX_RUNTIME_NETWORK_POLICY_ENFORCED",
default_value_t = false
)]
sandbox_runtime_network_policy_enforced: bool,

#[arg(
long,
env = "OPENSHELL_K8S_SANDBOX_RUNTIME_BOUNDARY_PORT",
Expand Down Expand Up @@ -270,7 +263,6 @@ async fn main() -> Result<()> {
.unwrap_or_else(openshell_core::config::default_supervisor_image),
supervisor_image_pull_policy: args.supervisor_image_pull_policy,
sandbox_runtime: KubernetesSandboxRuntimeConfig {
network_policy_enforced: args.sandbox_runtime_network_policy_enforced,
boundary_port: args.sandbox_runtime_boundary_port,
},
https_proxy: args.https_proxy,
Expand Down
12 changes: 7 additions & 5 deletions deploy/helm/openshell/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,12 @@ multiple pre-provisioned workspace namespaces.

## Prerequisites

> **Required:** Your cluster CNI MUST enforce Kubernetes `NetworkPolicy` for
> ingress and egress in every sandbox namespace. OpenShell creates the policies,
> but Kubernetes accepts them even if no CNI enforces them. Without enforcement,
> sandbox workloads may connect directly and bypass supervisor network policy.
> Verify CNI support before installing OpenShell.

The Kubernetes Agent Sandbox CRDs and controller must be installed on the cluster before deploying OpenShell. Install them with:

```shell
Expand All @@ -35,8 +41,7 @@ where Helm cannot discover cluster APIs.
## Install on Kubernetes

```shell
helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart --version <version> \
--set supervisor.sandboxRuntime.networkPolicyEnforced=true
helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart --version <version>
```

## Install on OpenShift
Expand All @@ -49,7 +54,6 @@ oc create ns openshell

# Deploy openshell with overrides to allow SCC assignment of fsGroup and runAsUser for the gateway
helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart --version <version> -n openshell \
--set supervisor.sandboxRuntime.networkPolicyEnforced=true \
--set server.disableTls=true \
--set podSecurityContext.fsGroup=null \
--set securityContext.runAsUser=null
Expand Down Expand Up @@ -110,7 +114,6 @@ Then install the chart pointing at that Secret:
```bash
helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart --version <version> \
-n openshell \
--set supervisor.sandboxRuntime.networkPolicyEnforced=true \
--set workload.kind=deployment \
--set server.externalDbSecret=my-pg-credentials
```
Expand Down Expand Up @@ -352,7 +355,6 @@ discovery endpoint or its TLS CA.
| supervisor.image.repository | string | `"openshell/supervisor"` | Supervisor image repository. |
| supervisor.image.tag | string | `""` | Supervisor image tag. Defaults to the chart appVersion when empty. |
| supervisor.sandboxRuntime.boundaryPort | int | `5500` | Workload boundary TLS listener port. |
| supervisor.sandboxRuntime.networkPolicyEnforced | bool | `false` | Required operator acknowledgement that the cluster CNI enforces NetworkPolicy. |
| tolerations | list | `[]` | Tolerations for the gateway pod. |
| upstreamProxy | object | `{"authAllowInsecure":false,"authSecret":{"key":"","name":""},"caBundle":{"configMapName":"","key":"ca.crt"},"connectByHostname":false,"noProxy":"","url":""}` | Operator-owned corporate forward proxy for policy-approved TLS egress from Kubernetes sandboxes. The workload cannot select or override it. |
| upstreamProxy.authAllowInsecure | bool | `false` | Required when authSecret is configured because Basic auth to an HTTP proxy is cleartext. |
Expand Down
11 changes: 7 additions & 4 deletions deploy/helm/openshell/README.md.gotmpl
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,12 @@ multiple pre-provisioned workspace namespaces.

## Prerequisites

> **Required:** Your cluster CNI MUST enforce Kubernetes `NetworkPolicy` for
> ingress and egress in every sandbox namespace. OpenShell creates the policies,
> but Kubernetes accepts them even if no CNI enforces them. Without enforcement,
> sandbox workloads may connect directly and bypass supervisor network policy.
> Verify CNI support before installing OpenShell.

The Kubernetes Agent Sandbox CRDs and controller must be installed on the cluster before deploying OpenShell. Install them with:

```shell
Expand All @@ -35,8 +41,7 @@ where Helm cannot discover cluster APIs.
## Install on Kubernetes

```shell
helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart --version <version> \
--set supervisor.sandboxRuntime.networkPolicyEnforced=true
helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart --version <version>
```

## Install on OpenShift
Expand All @@ -49,7 +54,6 @@ oc create ns openshell

# Deploy openshell with overrides to allow SCC assignment of fsGroup and runAsUser for the gateway
helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart --version <version> -n openshell \
--set supervisor.sandboxRuntime.networkPolicyEnforced=true \
--set server.disableTls=true \
--set podSecurityContext.fsGroup=null \
--set securityContext.runAsUser=null
Expand Down Expand Up @@ -110,7 +114,6 @@ Then install the chart pointing at that Secret:
```bash
helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart --version <version> \
-n openshell \
--set supervisor.sandboxRuntime.networkPolicyEnforced=true \
--set workload.kind=deployment \
--set server.externalDbSecret=my-pg-credentials
```
Expand Down
3 changes: 1 addition & 2 deletions deploy/helm/openshell/ci/values-keycloak.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,7 @@
#
# Then layer this file on top of values.yaml when deploying:
# helm upgrade --install openshell . \
# -f values.yaml -f ci/values-skaffold.yaml -f ci/values-keycloak.yaml \
# --set supervisor.sandboxRuntime.networkPolicyEnforced=true
# -f values.yaml -f ci/values-skaffold.yaml -f ci/values-keycloak.yaml
#
# Or add this file to skaffold.yaml valuesFiles for iterative dev.
#
Expand Down
3 changes: 1 addition & 2 deletions deploy/helm/openshell/ci/values-openshift-scc.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,7 @@
# OpenShift SCC compatibility overlay. Removes the hardcoded runAsUser and
# fsGroup so that OpenShift's restricted-v2 SCC can inject the namespace-
# assigned UID/GID range. Layer after values.yaml:
# helm install openshell deploy/helm/openshell -f ci/values-openshift-scc.yaml \
# --set supervisor.sandboxRuntime.networkPolicyEnforced=true
# helm install openshell deploy/helm/openshell -f ci/values-openshift-scc.yaml
#
# The e2e Kubernetes harness applies this automatically when it detects an
# OpenShift cluster (route.openshift.io API present).
Expand Down
1 change: 0 additions & 1 deletion deploy/helm/openshell/ci/values-skaffold.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -20,4 +20,3 @@ supervisor:
# The local k3s cluster created by `mise run helm:k3s:create` enables its
# built-in NetworkPolicy controller for sandbox namespaces.
sandboxRuntime:
networkPolicyEnforced: true
1 change: 0 additions & 1 deletion deploy/helm/openshell/templates/gateway-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -232,7 +232,6 @@ data:
gateway_pod_selector = { "app.kubernetes.io/name" = {{ include "openshell.name" . | quote }}, "app.kubernetes.io/instance" = {{ .Release.Name | quote }} }

[openshell.drivers.kubernetes.sandbox_runtime]
network_policy_enforced = {{ .Values.supervisor.sandboxRuntime.networkPolicyEnforced }}
boundary_port = {{ .Values.supervisor.sandboxRuntime.boundaryPort | default 5500 }}

{{- if not $credentialDrivers }}
Expand Down
5 changes: 0 additions & 5 deletions deploy/helm/openshell/templates/network-policy-ack.yaml

This file was deleted.

20 changes: 0 additions & 20 deletions deploy/helm/openshell/tests/network_policy_ack_test.yaml

This file was deleted.

2 changes: 0 additions & 2 deletions deploy/helm/openshell/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -70,8 +70,6 @@ supervisor:
# -- Supervisor image digest. When set, this takes precedence over tag.
digest: ""
sandboxRuntime:
# -- Required operator acknowledgement that the cluster CNI enforces NetworkPolicy.
networkPolicyEnforced: false
# -- Workload boundary TLS listener port.
boundaryPort: 5500

Expand Down
2 changes: 0 additions & 2 deletions deploy/helm/test-split-ownership.sh
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,6 @@ trap 'rm -rf "${work_dir}"' EXIT
helm template openshell "${repo_root}/deploy/helm/openshell" \
--namespace openshell \
--set agentSandbox.preflight.enabled=false \
--set supervisor.sandboxRuntime.networkPolicyEnforced=true \
--set workspaceResources.enabled=false \
>"${work_dir}/gateway.yaml"

Expand Down Expand Up @@ -41,7 +40,6 @@ fi
helm template openshell "${repo_root}/deploy/helm/openshell" \
--namespace openshell \
--set agentSandbox.preflight.enabled=false \
--set supervisor.sandboxRuntime.networkPolicyEnforced=true \
--set-json workspaceResources=null \
>"${work_dir}/legacy-reuse-values.yaml"

Expand Down
2 changes: 0 additions & 2 deletions docs/kubernetes/high-availability.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,6 @@ helm upgrade --install openshell \
oci://ghcr.io/nvidia/openshell/helm-chart \
--version <version> \
--namespace openshell \
--set supervisor.sandboxRuntime.networkPolicyEnforced=true \
--values values-ha.yaml \
--wait
```
Expand Down Expand Up @@ -196,7 +195,6 @@ helm upgrade openshell \
oci://ghcr.io/nvidia/openshell/helm-chart \
--version <version> \
--namespace openshell \
--set supervisor.sandboxRuntime.networkPolicyEnforced=true \
--values values-ha.yaml \
--wait
```
Expand Down
2 changes: 0 additions & 2 deletions docs/kubernetes/ingress.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,6 @@ helm upgrade --install openshell \
oci://ghcr.io/nvidia/openshell/helm-chart \
--version <version> \
--namespace openshell \
--set supervisor.sandboxRuntime.networkPolicyEnforced=true \
--set grpcRoute.enabled=true \
--set grpcRoute.gateway.create=true \
--set grpcRoute.gateway.className=eg
Expand Down Expand Up @@ -114,7 +113,6 @@ helm upgrade --install openshell \
oci://ghcr.io/nvidia/openshell/helm-chart \
--version <version> \
--namespace openshell \
--set supervisor.sandboxRuntime.networkPolicyEnforced=true \
--set grpcRoute.enabled=true \
--set grpcRoute.gateway.create=true \
--set grpcRoute.gateway.className=eg \
Expand Down
2 changes: 0 additions & 2 deletions docs/kubernetes/managing-certificates.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,6 @@ helm upgrade --install openshell \
oci://ghcr.io/nvidia/openshell/helm-chart \
--version <version> \
--namespace openshell \
--set supervisor.sandboxRuntime.networkPolicyEnforced=true \
--set certManager.enabled=true
```

Expand All @@ -74,7 +73,6 @@ helm upgrade --install openshell \
oci://ghcr.io/nvidia/openshell/helm-chart \
--version <version> \
--namespace openshell \
--set supervisor.sandboxRuntime.networkPolicyEnforced=true \
--set certManager.enabled=true \
--set certManager.serverIssuerRef.name=letsencrypt-prod \
--set certManager.serverIssuerRef.kind=ClusterIssuer \
Expand Down
12 changes: 9 additions & 3 deletions docs/kubernetes/openshift.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -41,17 +41,23 @@ in the sandbox qualification output.
- [Agent Sandbox](/kubernetes/setup#install-agent-sandbox) controller and CRDs.
- A CNI that enforces ingress and egress `NetworkPolicy` in sandbox namespaces.

<Warning>
Your cluster MUST enforce ingress and egress `NetworkPolicy` in every sandbox
namespace. OpenShell creates the policies, but Kubernetes does not verify that
the CNI applies them. Without enforcement, sandbox workloads may bypass
supervisor network policy through direct connections.
</Warning>

## Install OpenShell

Pre-create the namespace, then install the chart. Keep the default restricted
security posture and acknowledge NetworkPolicy only after validating the CNI.
security posture.

```shell
oc create ns openshell
helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart \
--version <version> \
--namespace openshell \
--set supervisor.sandboxRuntime.networkPolicyEnforced=true
--namespace openshell
```

The driver reads the namespace's `openshift.io/sa.scc.uid-range` annotation and
Expand Down
7 changes: 4 additions & 3 deletions docs/kubernetes/sandbox-runtime.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -81,9 +81,10 @@ restricts them.

Kubernetes policies are additive. Keep sandbox namespaces under administrative
control so another principal cannot add permissive policies, create Pods with
OpenShell labels, or read bootstrap Secrets. Set
`supervisor.sandboxRuntime.networkPolicyEnforced: true` only after you verify that the
cluster CNI enforces both ingress and egress policies for these namespaces.
OpenShell labels, or read bootstrap Secrets. The cluster CNI MUST enforce both
ingress and egress `NetworkPolicy` in every sandbox namespace. Kubernetes accepts
policy objects even when no CNI enforces them; OpenShell does not test traffic
to verify enforcement.

## Bootstrap a Sandbox

Expand Down
Loading
Loading