Skip to content

test(podman): run existing Podman behavioral e2e coverage in CI #3712

Description

@johntmyers

User Story

As an OpenShell maintainer, I want the existing Podman behavioral end-to-end tests to run in required CI and release qualification, so that Podman-specific networking, lifecycle, policy, identity, resource, and provider regressions cannot merge while applicable tests remain unscheduled.

Problem Statement

OpenShell currently has a required bundled-Podman branch job, but it runs a hard-coded PODMAN_CI_TESTS allowlist rather than the full set of targets eligible under the e2e-podman feature.

There are currently 39 eligible Rust test targets. The curated CI set selects 20 and omits 19. Two omissions are intentionally ignored performance benchmarks, leaving 17 executable targets outside Podman behavioral CI.

Twelve of those 17 are shared, driver-independent test targets that run from the same source files in Docker and Kubernetes CI but are explicitly omitted from Podman CI:

  • port_forward
  • provider_auto_create
  • proxy_egress_pipeline
  • sandbox_labels
  • sandbox_lifecycle
  • sandbox_templates
  • settings_management
  • sync
  • transparent_tcp
  • upload_create
  • websocket_conformance
  • workspace_lifecycle

Five are Podman-specific targets whose behavior cannot be covered by another driver:

  • podman_oci_identity
  • podman_preflight
  • podman_resource_limits
  • podman_userns
  • provider_refresh_handles

podman_preflight may need a separate lightweight invocation because it expects the standalone openshell-driver-podman binary. The older podman_userns Rust target should either run or be removed as superseded by the newer rootful/rootless tmachine user-namespace suite.

The bundled Podman behavioral job is also present only in the branch workflow. Release Dev and Release Tag do not run the bundled Podman behavioral suite.

This is a follow-up to #3663. PR #3690 added the resource-limit and daemon-failure targets and broadened rootful user-namespace testing, but the new Rust targets were not added to PODMAN_CI_TESTS. PR #3606 restored a bundled Podman branch lane after the host.openshell.internal regression, but intentionally limited it to targets known to pass at that time.

Impact / Why This Matters

This scheduling gap has already allowed a Podman host-networking regression to merge even though relevant e2e coverage existed. It creates the same exposure for transparent networking, proxy enforcement, port forwarding, lifecycle cleanup, resource enforcement, OCI identity, workspace operations, and credential refresh.

The current workaround is for contributors to know that mise run e2e:podman is broader than required CI and run it manually on a compatible Linux Podman host. That is insufficient because:

  • contributors reasonably treat required CI as the supported regression suite;
  • the omission is not visible from individual test files or Cargo feature gates;
  • many developers do not have a compatible rootless Podman/AppArmor/pasta environment;
  • Release Dev and Release Tag can qualify artifacts without bundled Podman behavioral coverage; and
  • newly added Podman-eligible targets can silently remain outside CI.

Acceptance Criteria

  • All 12 shared executable targets listed above run against the bundled Podman driver in required branch CI and pass reliably.
  • podman_oci_identity, podman_resource_limits, and provider_refresh_handles run against a real Podman-backed OpenShell gateway in CI.
  • podman_preflight runs in CI with the standalone Podman driver artifact and verifies the bounded, actionable daemon-unavailable failure path.
  • podman_userns is either enabled or removed with its assertions demonstrably covered by the rootful/rootless driver-podman tmachine suite.
  • The two intentionally ignored performance targets remain documented as manual benchmarks rather than appearing as unexplained CI omissions.
  • Release Dev and Release Tag include bundled Podman behavioral qualification, or a documented release-gating replacement provides equivalent coverage.
  • CI contains an inventory/reachability check that fails when a new non-ignored e2e-podman target is not selected by a workflow, unless it appears in a checked-in exclusion list with a rationale and tracking issue.
  • Job names and summaries report the selected test set or target count so a narrow Podman job cannot be mistaken for the complete suite.
  • Rootless and rootful scope is explicit. Any behavioral targets not run in both modes have a documented technical reason and tracking issue.

Reproduction Steps

  1. Compare the [[test]] declarations and auto-discovered files under e2e/rust/tests/ that are eligible with the e2e-podman feature against PODMAN_CI_TESTS in e2e/rust/e2e-podman.sh.
  2. Observe that 39 targets are eligible while only 20 are selected by OPENSHELL_E2E_PODMAN_TEST_SET=ci.
  3. Observe that 17 non-benchmark targets are omitted.
  4. Inspect .github/workflows/branch-e2e.yml and observe that required branch CI invokes mise run e2e:podman:ci, not the full mise run e2e:podman suite.
  5. Inspect .github/workflows/release-dev.yml and .github/workflows/release-tag.yml and observe that neither has a bundled Podman behavioral job.

Environment

  • OpenShell: main at 924486805
  • Required Podman lane: Ubuntu 26.04, Podman 5.7.0, rootless/pasta
  • Relevant files: e2e/rust/e2e-podman.sh, e2e/rust/Cargo.toml, .github/workflows/branch-e2e.yml, .github/workflows/e2e-podman-test.yml

Related Work

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions