Skip to content

feat(initializer): validate initializer Pod uses correct volume name#3714

Open
HarshPopat23 wants to merge 4 commits into
kubeflow:masterfrom
HarshPopat23:feat-initializer-volume-validation
Open

feat(initializer): validate initializer Pod uses correct volume name#3714
HarshPopat23 wants to merge 4 commits into
kubeflow:masterfrom
HarshPopat23:feat-initializer-volume-validation

Conversation

@HarshPopat23

Copy link
Copy Markdown

What this PR does / why we need it ?:

Adds validation in the JobSet plugin's Validate() function to ensure the
dataset/model initializer ReplicatedJob has a Volume named
jobsetplgconsts.VolumeNameInitializer.

Previously VolumeNameInitializer was only a string constant with no
enforcement — a misconfigured runtime template could silently reference
the wrong volume name and only fail later at Pod scheduling/mount time,
instead of at admission time with a clear error.

This follows the same pattern already used in this function for validating
container names (rJobContainerNames), extended to also track volume
names (rJobVolumeNames) per ReplicatedJob.

Which issue(s) this PR fixes (optional, in Fixes #<issue number>, #<issue number>, ... format, will close the issue(s) when PR gets merged):
Fixes #3710

Checklist:

  • Docs included if any changes are user facing (not user facing — internal validation logic only, no API/docs change needed)

…validation in the JobSet plugin's Validate() function to ensure the dataset/model initializer ReplicatedJob has a Volume named jobsetplgconsts.VolumeNameInitializer. Previously this was only a string constant with no enforcement, meaning a misconfigured runtime template could silently reference the wrong volume name. Fixes kubeflow#3710

Signed-off-by: HarshPopat23 <musichk61@gmail.com>
Copilot AI review requested due to automatic review settings July 10, 2026 11:13
@google-oss-prow google-oss-prow Bot requested review from jinchihe and kuizhiqing July 10, 2026 11:13
@HarshPopat23 HarshPopat23 changed the title feat(runtime): validate initializer Pod uses correct volume name feat(initializer): validate initializer Pod uses correct volume name Jul 10, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR strengthens admission-time validation in the JobSet runtime plugin so misconfigured dataset/model initializer templates fail fast (with a clear validation error) rather than later during Pod scheduling/mount.

Changes:

  • Adds tracking/validation of volume names per ReplicatedJob and requires the initializer volume name to be jobsetplgconsts.VolumeNameInitializer.
  • Updates JobSet plugin validation tests to assert the new volume-name error cases.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 5 comments.

File Description
pkg/runtime/framework/plugins/jobset/jobset.go Adds initializer-volume name validation to Validate() for dataset/model initializer jobs.
pkg/runtime/framework/plugins/jobset/jobset_test.go Extends validation test expectations and adds cases covering missing initializer volume.

Comment thread pkg/runtime/framework/plugins/jobset/jobset.go Outdated
Comment thread pkg/runtime/framework/plugins/jobset/jobset.go Outdated
Comment thread pkg/runtime/framework/plugins/jobset/jobset.go
Comment thread pkg/runtime/framework/plugins/jobset/jobset.go
Comment thread pkg/runtime/framework/plugins/jobset/jobset.go Outdated
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Signed-off-by: HarshPopat23 <musichk61@gmail.com>

@andreyvelich andreyvelich left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/ok-to-test
/retest

Signed-off-by: HarshPopat23 <musichk61@gmail.com>
@google-oss-prow google-oss-prow Bot added size/L and removed size/M labels Jul 10, 2026

@andreyvelich andreyvelich left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the contribution, @HarshPopat23! 🙏 A few comments below. Note: this review was performed with the help of AI tools.

Comment thread pkg/runtime/framework/plugins/jobset/jobset.go Outdated
Comment thread pkg/runtime/framework/plugins/jobset/jobset.go
Comment thread pkg/runtime/framework/plugins/jobset/jobset_test.go
Comment thread pkg/runtime/framework/plugins/jobset/jobset.go Outdated
Signed-off-by: HarshPopat23 <musichk61@gmail.com>
@google-oss-prow

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from andreyvelich. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@HarshPopat23

Copy link
Copy Markdown
Author

Updated the PR to address the Copilot and maintainer review feedback:

  • Removed duplicate TODO noise.
  • Removed inline Pod Volume validation to avoid rejecting torchtune runtimes that use JobSet volumeClaimPolicies.
  • Added initializer container VolumeMount validation using jobsetplgconsts.VolumeNameInitializer.
  • Ensured missing initializer ReplicatedJob/container cases do not produce misleading volume/volumeMount errors.
  • Added positive tests for volumeClaimPolicies + VolumeMount runtime shape.
  • Added a dataset + model initializers together positive test.
  • Added negative tests for missing dataset/model initializer VolumeMounts.

Tests:

  • gofmt -w pkg/runtime/framework/plugins/jobset/jobset.go pkg/runtime/framework/plugins/jobset/jobset_test.go
  • go test ./pkg/runtime/framework/plugins/jobset

I also attempted:

  • go test ./pkg/runtime/...

but it failed locally because my Windows disk/cache is full:
"There is not enough space on the disk." The focused JobSet plugin tests pass.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(runtime): Add validation that initializer Pod uses the correct volume name

3 participants