globus-cluster-control

gcx runs job-control commands on HPC clusters through Globus Compute instead of SSH. It checks jobs and queues, submits and cancels batch jobs, and reads logs and files.

Why

Interactive SSH to HPC clusters is fragile. Multiplexed connections die when the network changes, and rebuilding them costs an MFA prompt per cluster, which scripts and AI agents cannot answer. With gcx, a small Globus Compute endpoint runs on each cluster's login node and holds an outbound connection to the Globus service. Your laptop talks only to that service, over stateless HTTPS:

• A dropped network costs a retry, not a login. Tokens are long-lived; there is no daily MFA for gcx itself.
• Tasks wait in the cloud until you collect their results, so a submission survives the laptop sleeping.
• Submission is at-most-once. If the connection drops after a submit may have been delivered, gcx stops and says so rather than risk running it twice.

How it is locked down

Each cluster runs only what you allowed for it when you set it up:

• Commands are small registered functions, not shell: job status (always on), submit/cancel, read files, and optionally arbitrary shell.
• Your choices are written into the registered code: which directories may be read, which scripts may be submitted, and which accounts, queues and QoS. A request outside them is refused on the cluster.
• The endpoint accepts nothing else. Its function allowlist means Globus rejects any other code with HTTP 403, and it refuses pickled payloads.
• Only you can use it. It is a single-user endpoint, owned by your Globus identity.

Details: docs/design.md.

Install

You need a laptop running macOS or Linux (WSL works), uv and git. The Globus CLI is optional: gcx status uses it where the cluster's Globus collection exposes your home directory. gcx login globus logs it in and grants it access to those collections, and gcx login globus --check shows what is missing.

git clone https://github.com/envfluids/globus-cluster-control.git
cd globus-cluster-control
uv tool install -e .          # puts `gcx` on PATH; `git pull` updates it

On a fresh computer, install git and uv first; the setup guide shows how.

Set up a cluster

gcx ssh-config <cluster> --user <username>           # preview SSH aliases
gcx ssh-config <cluster> --user <username> --apply   # write them
gcx login <cluster>                                  # you answer MFA
gcx setup <cluster>                                  # install, choose, test

gcx setup probes the login node and installs the endpoint under ~/.gcx in your cluster home (about 60 MB, nothing system-wide). It then has you:

• log in to Globus for the endpoint;
• choose which commands, directories, accounts and queues gcx may use;
• pick how the endpoint is kept alive. The options depend on the cluster: failover between two login nodes, cron on one node, or a restart the next time you use gcx.

It finishes with a health check and a short test job. Re-running it is safe, since finished steps are skipped. --dry-run shows what it would do, and --menu changes your choices later.

The full walk-through, from a fresh computer to a working Midway3, is in docs/colleague-setup.md. It also covers a second computer, troubleshooting and removing gcx. For a cluster without a profile, start with docs/new-cluster.md.

Use

gcx <cluster> jobs                         # your queued and running jobs
gcx <cluster> history --since 2026-10-01   # finished jobs
gcx <cluster> queues
gcx <cluster> submit ~/runs/job.sbatch -A <account> -p <queue> -t 01:00:00
gcx <cluster> run ~/repo/jobs/smoke.sh arg1 arg2   # script gets the args; see below
gcx launcher add <cluster> ~/repo          # let `run` submit scripts under ~/repo
gcx <cluster> cancel <job-id>
gcx <cluster> tail ~/runs/job.out -n 100   # also: head, ls, du
gcx <cluster> sh '<command>'               # only where shell is enabled
gcx doctor                                 # health check of every cluster (-v: all checks)
gcx doctor <cluster>                       # every check for one cluster
gcx status <cluster>                       # which login node holds the endpoint
gcx watch status                           # the laptop watcher: last check per cluster

• Paths are relative to your cluster home unless absolute. Quote ~ and $VARS so they expand on the cluster.
• --json gives machine-readable output. --no-wait returns a task ID to collect later with gcx result <id>.
• Results are capped at 1 MB; use Globus Transfer for large files.

Exit code	Meaning
2, "gcx refused"	Outside what you allowed for this cluster; change it with gcx setup <cluster> --menu.
75	The network dropped after a submit, run, cancel or sh may have been sent. Check jobs before retrying.
69	The endpoint is offline and could not be restarted. Run gcx login <cluster> and retry.

Staying online

Each endpoint is kept alive on the cluster where the site allows it:

• Cron allowed (midway3, stampede3, dsi): cron restarts the endpoint, and fails over to a second login node where there is one.
• No cron (delta, deltaai, derecho, polaris): nothing on the cluster can restart it after a login-node reboot. The next gcx call restarts it over SSH, if your SSH connection is up.

gcx watch install adds a small watcher on your laptop: launchd on macOS, a systemd user timer (or crontab) on Linux. Every 30 minutes it asks Globus whether each endpoint is online; that is one status call per cluster, with no tasks and nothing billed.

• An endpoint on a no-cron cluster is offline: it restarts it if your SSH connection is up; otherwise it sends a desktop notification naming the gcx login <cluster> to run.
• A cron-kept endpoint stays down for 10 minutes: it alerts you.

It notifies only when something changes, plus one daily reminder while something stays down. gcx setup offers to install it, and gcx watch status shows its last check and log.

Supported clusters

Profiles of site facts live in src/gcx/data/profiles/: delta, deltaai, derecho, dsi, midway3, polaris, stampede3. Each one records the login host, scheduler (Slurm or PBS), named login nodes, test-job resources and site quirks. Your own choices stay in ~/.config/gcx/ on your laptop. docs/clusters.md records what setting up each cluster involved.

Adding a cluster means adding a profile. Setup learns everything else by probing the cluster; docs/new-cluster.md lists the facts to gather and every profile field.

Running scripts with arguments

gcx <cluster> run SCRIPT [ARGS...] submits SCRIPT with ARGS as its "$@", which submit cannot do. Each allowed repo has a launcher: a command that prints the batch script to submit. The default, cat, submits the script as written; a repo that builds its #SBATCH header per cluster can use its own renderer instead. gcx checks the header's account and partition (both required) and refuses a few risky options (--wrap, --export, --uid/--gid, --get-user-env). gcx options (-A -p -q -t -J --after --dry-run --json) go before SCRIPT; everything after it is the script's. A ~ at the start of an argument (or after key=) means your cluster home, whether or not your laptop shell expanded it. --dry-run prints the rendered script and the sbatch command without submitting anything; try it before a script's first real run.

It is off by default and Slurm-only. Turn it on with gcx setup <cluster> --menu, which also asks for its accounts and queues and a first launcher. After that, add or remove launchers without the menu:

gcx launcher add <cluster> ~/repo                            # cat: scripts as written
gcx launcher add <cluster> ~/repo2 --timeout 60 -- sh tools/render.sh
gcx launcher add <cluster> ~/repo2 --dry-run -- sh tools/render.sh   # check only
gcx launcher list <cluster>
gcx launcher remove <cluster> ~/repo2

Launchers are part of the policy registered on the endpoint, so add and remove re-register the cluster's functions and restart its endpoint (about a minute), and need gcx login <cluster> first. The launcher runs in the endpoint's environment, not the repo's, and must finish within its timeout (120 s by default), so it should not install or sync anything. Details: docs/design.md.

AI agents

gcx skill, also run by every gcx setup and gcx launcher add/remove, writes two Claude Code skills from your configuration:

• ~/.claude/skills/gcx/SKILL.md, for using gcx. Agents in any repo then know which clusters you have, what each one allows, and the rules: respect refusals, never retry-loop submit, and leave gcx login, gcx setup and gcx launcher add to the human.
• ~/.claude/skills/gcx-run-author/SKILL.md, for making a script or repo runnable with gcx run. An agent writes or adapts the script (or a renderer), checks it against gcx's rules, checks the launcher with gcx launcher add ... --dry-run, and hands you the command to run.

Layout

Path	What
src/gcx/cli.py	the gcx command
src/gcx/transport.py	retries and at-most-once submission over an unreliable network
src/gcx/capabilities/	the code that runs on the cluster (runtime.py) and the step that writes a policy into it (build.py)
src/gcx/setup/	gcx setup: probe, command menu, idempotent install steps
src/gcx/data/	cluster profiles, endpoint templates, the keepalive script, the bundled gcx login script
docs/	setup guide, adding a cluster, design, per-cluster record
tests/	unit tests (uv run pytest), plus a live outage test (tests/outage_test.sh)

Development

uv sync
uv run pytest

The tests cover the following, and CI runs them on Linux and macOS:

• the network-safety logic, against real socket failures;
• the cluster-side code, against a fake scheduler, and under Python 3.9 and 3.13;
• failover, with two simulated login nodes;
• the setup decisions;
• the generated SSH config, checked with OpenSSH.

globus-compute-sdk in pyproject.toml must match the globus-compute-endpoint version installed on the clusters. After changing it, gcx setup <cluster> reinstalls the endpoint to match.