test: enforce restricted PSS for CI user namespace by danish9039 · Pull Request #3487 · kubeflow/community-distribution

danish9039 · 2026-05-26T21:57:32Z

Summary of Changes

This PR makes CI validate Kubeflow user workloads under Kubernetes Pod Security Standards restricted.

Relabel the CI Profile namespace kubeflow-user-example-com to pod-security.kubernetes.io/enforce=restricted after the Profile Controller creates it.
Keep the production/default Profile Controller behavior unchanged.
Update workflow path filters so shared PSS test-script and local overlay changes rerun affected component workflows.
Make CI workload examples compatible with restricted PSS across Katib, KServe, Notebooks, Workspaces, Trainer, Training Operator, Ray, and Istio validation.
Move Katib, Trainer, and Training Operator restricted-PSS test mutations into local overlays instead of live kubectl patch or generated temp-overlay logic.
Move PR-added script debug dumps into GitHub Actions failure-log artifacts while keeping test waits fail-fast.

Notes

This is intentionally CI/test focused. It does not change the default customer Profile namespace policy.

The new application changes are local Kubeflow manifests overlays under applications/*/overlays. They do not edit synchronized applications/*/upstream folders or common/* manifests.

Follow-up notes are tracked separately for keeping the copied Katib CI config in sync during future Katib syncs and for removing the temporary Training Operator overlay after the final 26.03.1 path is available.

Validation

Latest head 7adb2d55 is green on all technical checks.

DCO passes.
Formatting checks pass for YAML, Bash, and Python.
Full Kubeflow integration passes.
Component workflows pass, including Katib, Trainer, Training Operator, notebooks, pipelines, workspaces, KServe, Ray, Istio validation, dashboard, and Dex/OAuth2.
Local validation included bash -n, shellcheck, YAML parsing, kustomize build for the new overlays, rendered field assertions, git diff --check, and a guard confirming no applications/*/upstream/** changes.

tide is still waiting for review labels.

Contributor Checklist

I have tested these changes with kustomize. See Installation Prerequisites.
All commits are signed-off to satisfy the DCO check.
I have considered adding my company to the adopters page to support Kubeflow and help the community, since I expect help from the community for my issue.

Overwrites the default baseline namespace label created by the profile controller exclusively during CI tests. This guarantees that test workloads simulate strict pod security standard (PSS) restricted enforcements without modifying production default baselines. Signed-off-by: abdullahpathan22 <abdullahpathan22@users.noreply.github.com>

- Broaden katib triggers: tests/katib_install.sh -> tests/katib* - Broaden pipeline triggers: individual files -> tests/pipeline* - Add tests/pipeline* trigger to pipeline_run_from_notebook workflow - Replace dead experimental/security/PSS/* path (directory no longer exists) with actual test files: tests/kubeflow_profile_install.sh and tests/PSS_enable.sh across all affected workflows Note: Dashboard/profiles directory paths are intentionally kept as-is since applications/profiles/ does not exist yet on master. Those paths will be updated once the directory restructure lands. Signed-off-by: abdullahpathan22 <abdullahpathan22@users.noreply.github.com>

Signed-off-by: danish9039 <danishsiddiqui040@gmail.com>

google-oss-prow · 2026-05-26T21:57:38Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign kimwnasptd for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

github-actions · 2026-05-26T21:57:41Z

Welcome to the Kubeflow Manifests Repository

Thanks for opening your first PR. Your contribution means a lot to the Kubeflow community.

Before making more PRs:
Please ensure your PR follows our Contributing Guide.
Please also be aware that many components are synchronizes from upstream via the scripts in /scripts.
So in some cases you have to fix the problem in the upstream repositories first, but you can use a PR against kubeflow/manifests to test the platform integration.

Community Resources:

CNCF Slack channels: https://www.kubeflow.org/docs/about/community/#kubeflow-slack-channels
Community meetings: https://www.kubeflow.org/docs/about/community/

Thanks again for helping to improve Kubeflow.

Copilot

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

This PR updates CI to validate/enforce Kubernetes Pod Security Standards (PSS) settings for Kubeflow profile namespaces, and wires the related test scripts into multiple GitHub Actions path filters so relevant workflows run when these scripts change.

Changes:

Update kubeflow_profile_install.sh to label the Kubeflow user namespace with pod-security.kubernetes.io/enforce=restricted and enforce-version=latest, and assert the labels are applied.
Add tests/kubeflow_profile_install.sh and tests/PSS_enable.sh to workflow on.pull_request.paths filters across several test workflows.
Broaden some workflow path filters (e.g., tests/pipeline*, tests/katib*) to cover more related changes.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tests/kubeflow_profile_install.sh	Labels the profile namespace for PSS restricted + adds assertions for applied labels.
.github/workflows/workspaces_pipeline_run_test.yaml	Triggers workflow on profile/PSS test script changes.
.github/workflows/volumes_web_application_test.yaml	Triggers workflow on profile install script changes.
.github/workflows/training_operator_test.yaml	Triggers workflow on profile/PSS test script changes.
.github/workflows/trainer_test.yaml	Triggers workflow on profile/PSS test script changes.
.github/workflows/pipeline_test.yaml	Broadens pipeline path trigger + triggers on profile/PSS test script changes.
.github/workflows/pipeline_run_from_notebook.yaml	Adds pipeline wildcard + profile/PSS script triggers.
.github/workflows/kserve_test.yaml	Triggers workflow on profile/PSS test script changes.
.github/workflows/kserve_models_web_application_test.yaml	Triggers workflow on profile install script changes.
.github/workflows/katib_test.yaml	Broadens katib path trigger + triggers on profile/PSS test script changes.
.github/workflows/istio_validation.yaml	Triggers workflow on profile/PSS test script changes.
.github/workflows/dex_oauth2-proxy_test.yaml	Triggers workflow on profile/PSS test script changes.

Signed-off-by: danish9039 <danishsiddiqui040@gmail.com>

danish9039 · 2026-06-01T05:34:37Z

@juliusvonkohout

Copilot

Pull request overview

Copilot reviewed 27 out of 27 changed files in this pull request and generated 1 comment.

juliusvonkohout · 2026-06-01T14:27:06Z

 kubectl get pods -n kubeflow-system -l app.kubernetes.io/name=trainer
 kubectl get clustertrainingruntimes torch-distributed

+kubectl patch clustertrainingruntime torch-distributed --type=json -p='[


this must also be a proper overlay and maybe upstreamed.

There might already bea recent Pr for this.

Moved the torch-distributed runtime changes into applications/trainer/overlays/runtimes-restricted; upstream follow-up is noted separately.

Checked kubeflow/trainer#3066 ; it helps with override support, but this PR still needs the local runtime default overlay.

juliusvonkohout · 2026-06-01T14:29:00Z

please a proper overlay. this has to survive the next release and then we can already remove it. Please raise a follow up PR to remove it such that i can merge it afte rthe 26.03.1 release.

Moved this into applications/training-operator/overlays/kubeflow-restricted-pss and noted the follow-up removal after 26.03.1.

Signed-off-by: danish9039 <danishsiddiqui040@gmail.com>

Copilot

Pull request overview

Copilot reviewed 37 out of 37 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (2)

tests/workspaces_pipeline_run_test.sh:1

This JSONPatch adds nested fields under /spec/podTemplate/securityContext/... and /spec/podTemplate/containerSecurityContext/.... JSONPatch add will fail if the parent object (e.g., /spec/podTemplate/securityContext) does not already exist. To make this robust across different base WorkspaceKind manifests, patch the parent objects in one operation (e.g., add/merge the full securityContext / containerSecurityContext objects) or use a strategic/merge patch (--type=merge) targeting the parent keys.
tests/kserve_test.sh:1
The polling logic treats any response other than 403 as success (twice). This can still false-positive on 404/503 while routes/backends are not ready—exactly the failure mode mentioned in the comment above the loop. Prefer checking for the expected success code (typically 200) or at least explicitly excluding 404/503 (and possibly 000) from counting toward STABLE_POLL_COUNT, to prevent passing the policy wait while the service is still unreachable.

#!/bin/bash

juliusvonkohout · 2026-06-05T11:32:09Z

Can you upstream this or make sure to only patch the securitycontext, but not the image for example ?

juliusvonkohout · 2026-06-05T11:33:37Z

+        serviceAccountName: default-editor
+        containers:
+        - name: ray-worker
+          image: rayproject/ray:2.44.1-py311-cpu


may you update ray and kuberay to the latest available versions please

so not just this file, also the kuberay operator

juliusvonkohout · 2026-06-05T11:36:41Z


-cd applications/katib/upstream && kustomize build installs/katib-with-kubeflow | kubectl apply -f - && cd ../../../
-
+kustomize build applications/katib/overlays/security | kubectl apply -f -


is it possible to have 2 steps for this? since these scripts are also used for the single step install and would reach customers.

Maybe you have to use kustomize components instead of overlays to have a cleaner separation. If componnts do not work you can also stay with the overlay

you caould have an if around it as in our kind installation that you install the extra security stuff on top only if you are in GHA. That goes for all installation scripts.

juliusvonkohout · 2026-06-05T11:40:21Z

 export KSERVE_M2M_TOKEN="$(kubectl -n ${NAMESPACE} create token default-editor)"
 export KSERVE_TEST_NAMESPACE=${NAMESPACE}

+kubectl patch clusterstoragecontainer default --type=merge --patch '{


can you add this directly to the kserve installation as patch ? Maybe even upstream it ?

juliusvonkohout · 2026-06-05T11:42:17Z

+kubectl label namespace "$KF_PROFILE" \
+  pod-security.kubernetes.io/enforce=restricted \
+  pod-security.kubernetes.io/enforce-version=latest \
+  --overwrite


only when you are in the GHA environments as in our kind script.

juliusvonkohout · 2026-06-05T11:42:58Z

-kubectl -n kubeflow rollout status deployment/tensorboard-controller-deployment --timeout=60s
-kubectl -n kubeflow rollout status deployment/tensorboards-web-app-deployment --timeout=60s
-kubectl -n kubeflow rollout status deployment/volumes-web-app-deployment --timeout=60s
+kubectl -n kubeflow rollout status deployment/jupyter-web-app-deployment --timeout=180s


why do you increase the timeouts so much ?

juliusvonkohout · 2026-06-05T11:43:35Z

 kubectl wait --for=condition=Available deployment/jobset-controller-manager -n kubeflow-system --timeout=120s

-kustomize build upstream/overlays/runtimes | kubectl apply --server-side --force-conflicts -f -
+kustomize build overlays/runtimes-restricted | kubectl apply --server-side --force-conflicts -f -


the same comments as above. keep it like before for regular users, but if you are in gha add the extra security

juliusvonkohout · 2026-06-05T11:44:25Z


-cd applications/training-operator/upstream
-kustomize build overlays/kubeflow | kubectl apply --server-side --force-conflicts -f -
+kustomize build applications/training-operator/overlays/kubeflow-restricted-pss | kubectl apply --server-side --force-conflicts -f -


same here, only when you are in the GHA environment

juliusvonkohout · 2026-06-05T11:44:48Z

 kubectl apply -f /tmp/pytorch-job.yaml

-kubectl wait --for=jsonpath='{.status.conditions[0].type}=Created' pytorchjob.kubeflow.org/pytorch-simple -n $KF_PROFILE --timeout=60s
+kubectl wait --for=condition=Created pytorchjob.kubeflow.org/pytorch-simple -n "$KF_PROFILE" --timeout=180s


why do you increase the timeouts ?

juliusvonkohout · 2026-06-05T11:45:34Z

if really needed this has to be upstreamed. CC @christian-heusel

juliusvonkohout · 2026-06-05T11:46:24Z


+kubectl get clustertrainingruntime torch-distributed -o yaml


what is the purpose of this line ?

juliusvonkohout · 2026-06-05T11:47:09Z

why do you change the timeouts ?

abdullahpathan22 and others added 3 commits May 27, 2026 03:26

ci: complete restricted PSS trigger coverage

f3b8537

Signed-off-by: danish9039 <danishsiddiqui040@gmail.com>

Copilot AI review requested due to automatic review settings May 26, 2026 21:57

google-oss-prow Bot added the do-not-merge/work-in-progress label May 26, 2026

google-oss-prow Bot requested a review from juliusvonkohout May 26, 2026 21:57

google-oss-prow Bot requested a review from tarekabouzeid May 26, 2026 21:57

google-oss-prow Bot added the size/M label May 26, 2026

Copilot AI reviewed May 26, 2026

View reviewed changes

Comment thread tests/kubeflow_profile_install.sh

Comment thread tests/kubeflow_profile_install.sh

test: make CI workloads restricted-compatible

3eec781

Signed-off-by: danish9039 <danishsiddiqui040@gmail.com>

google-oss-prow Bot added size/L and removed size/M labels May 27, 2026

danish9039 added 9 commits May 27, 2026 11:53

test: patch generated workloads for restricted PSS

4ab9185

Signed-off-by: danish9039 <danishsiddiqui040@gmail.com>

test: fix restricted PSS CI follow-ups

56e3320

Signed-off-by: danish9039 <danishsiddiqui040@gmail.com>

test: fix restricted generated workload checks

5ea5779

Signed-off-by: danish9039 <danishsiddiqui040@gmail.com>

test: configure generated workload security contexts

6d5b8cf

Signed-off-by: danish9039 <danishsiddiqui040@gmail.com>

test: wait for patched katib controller rollout

c78ed42

Signed-off-by: danish9039 <danishsiddiqui040@gmail.com>

test: render katib controller security flag

f447b3d

Signed-off-by: danish9039 <danishsiddiqui040@gmail.com>

test: enable katib security context config

aed22f0

Signed-off-by: danish9039 <danishsiddiqui040@gmail.com>

test: patch katib suggestion security context

6a66e91

Signed-off-by: danish9039 <danishsiddiqui040@gmail.com>

test: avoid kserve auth policy race

2b55af8

Signed-off-by: danish9039 <danishsiddiqui040@gmail.com>

google-oss-prow Bot added size/XL and removed size/L labels May 27, 2026

danish9039 added 2 commits May 27, 2026 13:57

test: wait longer for notebooks rollout

40b41f0

Signed-off-by: danish9039 <danishsiddiqui040@gmail.com>

test: patch workspacekind for restricted PSS

50bc03b

Signed-off-by: danish9039 <danishsiddiqui040@gmail.com>

juliusvonkohout reviewed May 27, 2026

View reviewed changes

Comment thread tests/dashboard_install.sh Outdated

danish9039 added 2 commits May 28, 2026 00:10

test: add raw kserve security context

6a64a7f

Signed-off-by: danish9039 <danishsiddiqui040@gmail.com>

test: add ray autoscaler security context

3a7d2ad

Signed-off-by: danish9039 <danishsiddiqui040@gmail.com>

danish9039 marked this pull request as ready for review May 30, 2026 14:15

google-oss-prow Bot removed the do-not-merge/work-in-progress label May 30, 2026

google-oss-prow Bot requested review from juliusvonkohout and kimwnasptd May 30, 2026 14:15

juliusvonkohout requested review from Copilot and removed request for Copilot June 1, 2026 14:11

Copilot started reviewing on behalf of juliusvonkohout June 1, 2026 14:11 View session

Copilot AI reviewed Jun 1, 2026

View reviewed changes

Comment thread tests/katib_install.sh Outdated

juliusvonkohout reviewed Jun 1, 2026

View reviewed changes

Comment thread tests/katib_install.sh

juliusvonkohout reviewed Jun 1, 2026

View reviewed changes

Comment thread tests/katib_test.sh Outdated

juliusvonkohout reviewed Jun 1, 2026

View reviewed changes

Comment thread tests/notebooks_install.sh Outdated

juliusvonkohout reviewed Jun 1, 2026

View reviewed changes

juliusvonkohout mentioned this pull request Jun 4, 2026

test: enforce PSS restricted for CI user namespace #3444

Open

test: move restricted pss fixes into overlays

7adb2d5

Signed-off-by: danish9039 <danishsiddiqui040@gmail.com>

danish9039 requested a review from Copilot June 5, 2026 08:07

Copilot AI reviewed Jun 5, 2026

View reviewed changes

Comment thread .github/workflows/workspaces_pipeline_run_test.yaml

juliusvonkohout reviewed Jun 5, 2026

View reviewed changes

Comment thread tests/katib_test.sh

juliusvonkohout Jun 5, 2026

Copy link
Copy Markdown

Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why do you change the timeouts ?


		cd applications/katib/upstream && kustomize build installs/katib-with-kubeflow \| kubectl apply -f - && cd ../../../

		kustomize build applications/katib/overlays/security \| kubectl apply -f -


		kubectl get clustertrainingruntime torch-distributed -o yaml

Uh oh!

Conversation

danish9039 commented May 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary of Changes

Notes

Related

Validation

Contributor Checklist

Uh oh!

google-oss-prow Bot commented May 26, 2026

Uh oh!

github-actions Bot commented May 26, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

danish9039 commented Jun 1, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

danish9039 Jun 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

juliusvonkohout Jun 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

juliusvonkohout Jun 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

danish9039 commented May 26, 2026 •

edited

Loading

danish9039 Jun 5, 2026 •

edited

Loading

juliusvonkohout Jun 5, 2026 •

edited

Loading

juliusvonkohout Jun 5, 2026 •

edited

Loading