Remote Pilot Demo Project

This is a small example project that vendors rclone_remote_pilot inside the workload repo.

The demo shows how to remote-pilot an HPC, lab server, cloud VM, or other remote Linux desktop from a controller machine such as a laptop or workstation.

The intended workflow is:

1. The user configures the pilot locally with HPC paths in mind.
2. The user commits the generated project config file.
3. The HPC side pulls the repo.
4. The HPC operator selects the project instance and starts either:
   • the plain relay
   • the Slurm job supervisor with email notifications

Keep the two sides separate:

• Controller machine Run configure.sh if you want to generate the committed project config locally, edit the shared Drive commands.sh, and commit/push config changes.
• HPC/remote machine Pull the repo, keep rclone_remote_pilot/projects/demo_project.env in the checkout, create any machine-only rclone_remote_pilot/projects/demo_project.local.env, provide secrets, and run the relay, supervisor, sync, repair, and status commands.

The committed config can be built on the controller machine, but it contains HPC/remote paths and must be present in the HPC/remote checkout before restart/status commands can load it. The shared commands.sh file can be edited from the controller side, but its contents run on the HPC/remote machine from the configured PROJECT_DIR.

Demo layout

• src/train.py: dummy training script
• configs/demo_config.json: example workload config
• scripts/run_training.sh: runs the dummy workload
• scripts/use_remote_pilot_project.sh: helper to export the active pilot project
• scripts/start_remote_pilot.sh: wrapper around relayctl.sh start
• scripts/start_job_monitoring.sh: wrapper around job_supervisor.sh
• slurm/submit_demo.slurm: example Slurm submission script
• rclone_remote_pilot/: embedded remote pilot tool

Local user setup

Run this from inside the embedded pilot directory:

cd rclone_remote_pilot
./configure.sh --project demo_project

Use HPC paths when prompted. The generated config file is:

rclone_remote_pilot/projects/demo_project.env

Commit that file after reviewing it.

If a runtime setting should apply only on the HPC/remote machine, put it in rclone_remote_pilot/projects/demo_project.local.env on that machine or export it in the HPC shell before restarting the relay. Examples include SLEEP_SECS, INTERVAL_SEC, RUN_IN_BACKGROUND, and COMMAND_FILE_NAME.

HPC usage

Runtime scripts should be launched from inside rclone_remote_pilot. The parent project directory is fine as the checkout location, but do not run bash rclone_remote_pilot/... from there because the pilot expects its supporting files relative to its own directory.

All commands in this section are meant to run on the HPC/remote machine.

Preferred direct usage on HPC:

source scripts/use_remote_pilot_project.sh
cd rclone_remote_pilot
./relayctl.sh start

For Slurm monitoring with start/finish email notifications:

source scripts/use_remote_pilot_project.sh
sbatch slurm/submit_demo.slurm

The submitted job changes into rclone_remote_pilot before launching job_supervisor.sh.

If you want to start from the project root without calling the runtime scripts directly, use the wrappers because they cd into rclone_remote_pilot before executing:

source scripts/use_remote_pilot_project.sh
./scripts/start_remote_pilot.sh

and:

source scripts/use_remote_pilot_project.sh
./scripts/start_job_monitoring.sh