feat(server)!: harden gateway peer transport and expand HA conformance

.agents/skills/helm-dev-environment/SKILL.md

@@ -156,6 +156,17 @@ plaintext by default. Override `server.disableTls=false` to exercise TLS/mTLS.
 |------|---------------------|----------------|
 | Skaffold dev (default) | `true` | `http://` |
 | TLS enabled | `false` (or omitted) | `https://` |
+| HA TLS overlay (`ci/values-high-availability-tls.yaml`) | `false` | `https://`, pod TLS and mTLS, no Envoy, HTTPS peers |
+
+Gateways on PostgreSQL refuse plaintext peer traffic, and the chart refuses to
+render `server.disableTls: true` with PostgreSQL, unless
+`server.peer.allowInsecureTransport` is `true`. `ci/values-skaffold.yaml` sets
+it so the `high-availability` profile and the external-PostgreSQL e2e lanes
+keep running plaintext; gateways on PostgreSQL then log a warning at every
+startup. To exercise HTTPS peer transport, layer
+`ci/values-high-availability-tls.yaml` after `ci/values-high-availability.yaml`
+without `ci/values-gateway.yaml` (the `high-availability` profile adds that
+plaintext Envoy listener), or run `mise run e2e:kubernetes:ha-tls`.
 
 ### Connecting through the forwarding task
 
@@ -296,6 +307,14 @@ KUBECONFIG=kubeconfig helm upgrade openshell deploy/helm/openshell \
 
 Use the IP that pods in that cluster use to reach listeners on the test host.
 
+Use `mise run e2e:kubernetes:ha-tls` for the TLS HA lane. It layers
+`ci/values-high-availability-tls.yaml` so gateway pods serve TLS and mTLS and
+route peers over HTTPS with the chart CA, skips Envoy, registers an mTLS CLI
+gateway over the Service port-forward (`OPENSHELL_E2E_KUBE_POD_TLS=1`), and
+runs only `kubernetes_ha_operations`: one sandbox kept across scale-up,
+scale-down, graceful and forced owner loss, and a rolling restart, with each
+HA-tested operation class driven through non-owner replicas.
+
 ### BackendTLSPolicy (end-to-end TLS)
 
 To enable end-to-end TLS between the Gateway proxy and the gateway pod, add
@@ -451,6 +470,7 @@ for dependencies still declared in `Chart.yaml`.
 | `deploy/helm/openshell/ci/values-spire-stack.yaml` | SPIRE hardened chart values for local dev |
 | `deploy/helm/openshell/ci/values-tls-disabled.yaml` | Lint-only: TLS + auth disabled (reverse-proxy edge termination) |
 | `deploy/helm/openshell/ci/values-credential-driver-vault.yaml` | Vault credential-driver validation overlay with HTTPS and private-CA trust |
+| `deploy/helm/openshell/ci/values-high-availability-tls.yaml` | HA TLS overlay: gateway pods serve TLS and mTLS so peers use HTTPS; layered after the HA overlay; used by `e2e:kubernetes:ha-tls` without Envoy |
 | `deploy/kube/manifests/envoy-gateway-openshell.yaml` | GatewayClass and BackendTrafficPolicy for Envoy Gateway (`mise run helm:gateway:apply`) |
 | `tasks/scripts/helm-k3s-local.sh` | k3d cluster create/delete/start/stop/status |
 | `tasks/scripts/keycloak-k8s-setup.sh` | Keycloak deploy, realm import, and development TLS trust anchor |

.config/nextest.toml

@@ -30,3 +30,14 @@ test-group = "kubernetes-ha"
 # Relative to the profile store dir (`e2e/rust/target/nextest/e2e-kubernetes/`).
 [profile.e2e-kubernetes.junit]
 path = "../../../../../results/e2e-kubernetes.xml"
+
+# The HA operation inventory keeps one sandbox across gateway scale-up,
+# scale-down, owner loss, and a rolling restart, which takes longer than the
+# profile's five minutes. It shares the kubernetes-ha group because it scales
+# and rolls the same gateway Deployment as the rebalancing tests. Its per-phase
+# summary lines (owner, attempts, durations) are printed on green runs too.
+[[profile.e2e-kubernetes.overrides]]
+filter = "test(/^ha_operation_inventory_/)"
+test-group = "kubernetes-ha"
+slow-timeout = { period = "60s", terminate-after = 20 }
+success-output = "final"

.github/workflows/branch-e2e.yml

@@ -461,6 +461,20 @@ jobs:
       e2e-task: e2e:kubernetes:credential-drivers
       conformance-artifact-prefix: openshell-conformance
 
+  kubernetes-ha-tls-e2e:
+    needs: [pr_metadata, build-binaries, build-images]
+    if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_kubernetes_ha_e2e == 'true'
+    permissions:
+      actions: read
+      contents: read
+      packages: read
+    uses: ./.github/workflows/e2e-kubernetes-test.yml
+    with:
+      image-tag: ${{ github.sha }}
+      job-name: Kubernetes HA E2E (TLS peers, operation inventory)
+      e2e-task: e2e:kubernetes:ha-tls
+      conformance-artifact-prefix: openshell-conformance
+
   core-e2e-result:
     name: Core E2E result
     needs:
@@ -503,7 +517,7 @@ jobs:
 
   kubernetes-ha-e2e-result:
     name: Kubernetes HA E2E result
-    needs: [pr_metadata, kubernetes-ha-e2e]
+    needs: [pr_metadata, kubernetes-ha-e2e, kubernetes-ha-tls-e2e]
     if: always() && needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_kubernetes_ha_e2e == 'true'
     runs-on: ubuntu-latest
     steps:

CI.md

@@ -29,8 +29,10 @@ Three opt-in labels enable the long-running E2E suites:
   with both managed and standalone compute drivers in `Branch E2E Checks`
 - `test:e2e-gpu` runs GPU E2E in `Branch E2E Checks`
 - `test:e2e-kubernetes` runs Kubernetes E2E with the HA Helm overlay
-  (`replicaCount: 2` and bundled PostgreSQL) and the credential-driver suite
-  (Kubernetes Secrets plus Vault) in `Branch E2E Checks`
+  (`replicaCount: 2` and an external PostgreSQL fixture) in two lanes, plaintext
+  pods behind Envoy (HA rebalancing tests) and TLS pods with HTTPS peer routing
+  (HA operation inventory), plus the credential-driver suite (Kubernetes Secrets
+  plus Vault) in `Branch E2E Checks`
 
 When multiple labels are present, `Branch E2E Checks` builds each generic multi-architecture artifact set once and fans out enabled suites in parallel. Runtime-specific reusable workflows define the Docker, Podman, VM, and Kubernetes lanes. Composite actions own the replaceable Podman, KVM, kind, and mise setup. Each lane depends only on the artifact categories it consumes: VM does not wait for container-driver artifacts or supervisor images, and GPU does not wait for the gateway image. Docker, Podman, GPU, Rust, Python, MCP, and VM E2E reuse matching prebuilt gateway and CLI binaries instead of compiling debug binaries in test jobs. Standalone-driver lanes additionally reuse driver-free gateway and compute-driver artifacts. Kubernetes managed-driver lanes consume published gateway and supervisor images, while the standalone-driver lane composes its gateway image from prebuilt binaries.
 The `OpenShell / E2E` and `OpenShell / GPU E2E` required statuses are evaluated from separate suite result jobs inside that workflow. `test:e2e-kubernetes` is optional while Kubernetes HA and credential-driver behavior are under active iteration: failures are visible in the workflow run but do not publish a required CI gate status.

TESTING.md

@@ -434,6 +434,7 @@ Available task variants:
 | `e2e:kubernetes:workspace-operator` | Operator workspace mode (pre-provisioned namespaces) |
 | `e2e:kubernetes:v1alpha1` | Agent Sandbox v1alpha1 compatibility |
 | `e2e:kubernetes:external-driver` | External Kubernetes driver sidecar |
+| `e2e:kubernetes:ha-tls` | Two TLS and mTLS gateway replicas with HTTPS peer routing and external PostgreSQL: one sandbox kept across scale-up, scale-down, graceful and forced owner loss, and a rolling restart, with non-interactive and streamed-stdin exec, file transfer, TCP forwarding, policy updates, and provider attachment status through non-owner replicas |
 
 Kubernetes e2e environment variables:
 
@@ -448,6 +449,8 @@ Kubernetes e2e environment variables:
 | `GATEWAY_IMAGE` | Kubernetes gateway image repository or complete tagged/digest-pinned image reference; digests require `OPENSHELL_E2E_KUBE_BUILD_IMAGES=0` |
 | `SUPERVISOR_IMAGE` | Gateway/supervisor image repository or complete tagged/digest-pinned image reference; Kubernetes digests require `OPENSHELL_E2E_KUBE_BUILD_IMAGES=0` |
 | `SANDBOX_IMAGE` | Trusted sandbox runtime image repository or complete tagged/digest-pinned image reference |
+| `OPENSHELL_E2E_KUBE_POD_TLS` | Set to `1` when the extra values make the gateway pods serve TLS; the harness registers an mTLS CLI gateway over the Service port-forward. Not supported with Envoy or the database scenarios |
+| `OPENSHELL_E2E_KUBE_HA_OPS_BUDGET_SECS` | Overall budget in seconds for `kubernetes_ha_operations`; default `1020` |
 
 Run a single test directly with cargo: