feat(job): Enable MTC for Pathways Workloads

[Neelabh94]

This pull request introduces support for Multi-Tier Checkpointing (MTC) in Pathways workloads within the HPC Toolkit. The changes include new CLI flags for enabling MTC, updates to the GKE orchestrator to handle MTC-specific volume mounts and sidecar configurations, and refinements to the JobSet templates to improve reliability and integration with GKE infrastructure.

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces support for Multi-Tier Checkpointing (MTC) in Pathways workloads within the HPC Toolkit. The changes include new CLI flags for enabling MTC, updates to the GKE orchestrator to handle MTC-specific volume mounts and sidecar configurations, and refinements to the JobSet templates to improve reliability and integration with GKE infrastructure.

Highlights

• Multi-Tier Checkpointing (MTC) Support: Added CLI flags and configuration support to enable MTC for Pathways workloads, including a configurable ramdisk directory.
• Sidecar Container Refactoring: Moved the Python sidecar container from a standard container to an init container in the Pathways JobSet template to better support MTC requirements.
• Infrastructure Enhancements: Updated JobSet templates with improved annotations for Kueue and GKE, and configured blocking restart strategies for better workload management.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces support for Multi-Tier Checkpointing (MTC) in Pathways workloads by adding new CLI flags, updating documentation, and modifying the GKE job orchestrator to configure the required CSI driver, volumes, and sidecar shared memory. It also refactors the colocated Python sidecar into an init container and updates the JobSet restart strategy to BlockingRecreate. The feedback suggests avoiding hardcoding imagePullPolicy: Always on the sidecar container to prevent inefficient network pulls and reduce startup latency.

[agrawalkhushi18]

LGTM