Skip to content

Kubernetes driver never requests CAP_SETPCAP, so combined-topology sandboxes crash-loop with EPERM on privilege drop #2751

Description

@thoraxe

Summary

On Kubernetes, topology = "combined" sandboxes always crash on startup with EPERM because the supervisor's privilege-drop routine requires CAP_SETPCAP, which the Kubernetes driver never requests in the pod's securityContext.capabilities.add list — under any config.

Root cause

  • drop_privileges_with_identity (crates/openshell-supervisor-process/src/process.rs) calls drop_capability_bounding_set() before setuid()/setgid(), whenever enforcement_mode.uses_privileged_process_setup() is true (i.e. topology = "combined" → ProcessEnforcementMode::Full, see crates/openshell-sandbox/src/lib.rs:857-858).
  • drop_capability_bounding_set() (process.rs:277-286) calls capctl::caps::bounding::clear(), which requires CAP_SETPCAP.
  • The Kubernetes driver's capability list (crates/openshell-driver-kubernetes/src/driver.rs:2692-2699) only ever adds SYS_ADMIN, NET_ADMIN, SYS_PTRACE, SYSLOG (plus SETUID, SETGID, DAC_READ_SEARCH when enable_user_namespaces is on) — SETPCAP is never added, in any code path.
  • Since the driver also sets capabilities: { drop: ["ALL"] } unconditionally, the supervisor (running as root, runAsUser: 0) never actually holds CAP_SETPCAP, so the bounding-set clear fails with EPERM and the pod crash-loops before ever exec'ing the workload.

By contrast, the Podman driver gets this right: crates/openshell-driver-podman/container.rs:1048-1051 explicitly adds SETPCAP to cap_add with a comment referencing exactly this requirement.

Reproduction

  1. Deploy the Kubernetes driver with topology = "combined" (the documented/default topology) on any cluster where sandbox pods run under a restricted PodSecurityStandard / custom SCC that only grants the capabilities the driver actually requests (i.e. not a blanket privileged SCC).
  2. Create a sandbox via CreateSandbox.
  3. Observe the agent container crash-loop with:
    Error:   × EPERM: Operation not permitted
    
  4. oc logs <pod> -c agent shows nothing more specific — the error surfaces from validate_capability_bounding_set_clear (process.rs:289-311) with a non-empty remaining bounding set, which does not hit the "already empty" fallback path.

We also confirmed the network-policy portion of sandbox setup (which needs SYS_ADMIN/NET_ADMIN) succeeds and reports ReportPolicyStatus: status=loaded before this crash — this is specifically the privilege-drop step, not the network-policy setup step.

Notes

  • Docs (docs/kubernetes/openshift.mdx:12) note the OpenShift install path is "experimental" and recommend the privileged SCC — but since the driver unconditionally sets capabilities: { drop: ["ALL"], add: [...] } regardless of SCC, granting a more permissive SCC doesn't help; the pod's own requested capability list is still missing SETPCAP.
  • enable_user_namespaces = true does NOT fix this — it only adds SETUID/SETGID/DAC_READ_SEARCH, not SETPCAP.
  • Workaround we're using in the meantime: switching to topology = "sidecar" (ProcessEnforcementMode::NetworkOnly), which skips drop_privileges entirely since the container already runs as the target UID.

Suggested fix

Add "SETPCAP" to the capability list built in crates/openshell-driver-kubernetes/src/driver.rs (around line 2692), mirroring the Podman driver's cap_add list, for any topology whose ProcessEnforcementMode::uses_privileged_process_setup() is true.

Environment

  • OpenShell built from dev branch, commit c4b500a (2026-08-14)
  • OpenShift, custom SCC granting SYS_ADMIN, NET_ADMIN, NET_RAW, SYS_PTRACE, SYSLOG, CHOWN, FOWNER, DAC_READ_SEARCH, SETUID, SETGID (no SETPCAP)

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions