Skip to content

HPC_STARTER_GUIDE.md

Latest commit

 

History

History
263 lines (188 loc) · 7.01 KB

HPC_STARTER_GUIDE.md

File metadata and controls

263 lines (188 loc) · 7.01 KB
Raw

Northeastern Explorer HPC — Starter Guide

A practical reference for getting up and running on the Explorer cluster.

Access

ssh <your_username>@explorer.northeastern.edu

Use tmux so your session survives SSH disconnects and laptop sleep:

tmux new -s main          # start a named session
# Ctrl+B then D           # detach (leaves session running)
tmux attach -t main       # reconnect later
tmux ls                   # list existing sessions

Environment Setup (one-time)

# 1. Load anaconda — must do this before any conda commands
module load anaconda3/2024.06

# 2. Create a Python environment
conda create -n myenv python=3.11 -y

# 3. Activate (interactive shell only)
conda activate myenv

# 4. Install packages
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu121
pip install numpy scipy matplotlib tqdm

# 5. Verify CUDA
python -c "import torch; print(torch.__version__, torch.cuda.is_available())"

Important: conda activate does NOT work inside SLURM batch scripts. Use the full Python path instead: /home/<your_username>/.conda/envs/myenv/bin/python script.py

Storage

# Check disk usage
df -h ~
du -sh ~/*

# Home vs Scratch
#   ~/           — small quota (~50–100 GB), permanent, backed up
#   /scratch/<username>/  — large (~1–10 TB), fast, auto-purged after 30–90 days

Use home for code and results. Use scratch for large temporary datasets during active jobs.

Partitions

# See all partitions and their states
sinfo

# Summarized view — one line per partition
sinfo -s

# Formatted: partition | max time | GPUs | nodes | state
sinfo -o "%.15P %.10l %.10G %.6D %.10t"
Partition Max Time GPU Notes
gpu 8 hours A100 / V100 Main GPU training — 1 GPU/job max
gpu-short 2 hours A100 / V100 Quick tests, faster queue
gpu-interactive 2 hours A100 / V100 Interactive srun sessions
short 48 hours CPU only Data preprocessing, no GPU
courses-gpu 24 hours 34 GPUs Needs rc/courses group membership

gpu Partition QOS Limits

Limit Value Meaning
MaxTRES=gres/gpu=1 1 GPU/job No multi-GPU jobs
MaxTRESPU=gres/gpu=4 4 GPUs/user Max 4 concurrent jobs
MaxJobsPU=4 4 running 5th job waits
MaxSubmitPU=8 8 queued+running 9th sbatch fails

GPU Hardware

# Which nodes have GPUs and what state are they in
sinfo -o "%N %G %t" | grep -v "^NODELIST"

# Idle A100 nodes right now
sinfo -o "%N %G %t" | grep a100 | grep idle

# Idle V100 nodes right now
sinfo -o "%N %G %t" | grep v100 | grep idle

# Node RAM + GPU type for all gpu-partition nodes
sinfo -p gpu -o "%N %G %m"   # %m = RAM in MB

# Full detail on a specific node (RAM, GPU count, features)
scontrol show node <nodename>

# From inside a running job — GPU VRAM
nvidia-smi
nvidia-smi --query-gpu=name,memory.total,memory.used,memory.free --format=csv,noheader

# From inside a running job — system RAM
free -h
GPU VRAM Nodes
A100-SXM4 40 / 80 GB d3146, d3203
V100-SXM2 32 GB d3091–d3098
V100-PCIE 16 GB c2204–c2207

Writing a SLURM Script

#!/bin/bash
#SBATCH --job-name=my-job
#SBATCH --partition=gpu
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --gres=gpu:a100:1          # always specify GPU type
#SBATCH --cpus-per-task=8
#SBATCH --mem=64G
#SBATCH --time=07:30:00            # HH:MM:SS — leave buffer before partition limit
#SBATCH --output=logs/job_%j.out   # %j = job ID
#SBATCH --error=logs/job_%j.err

module load cuda/12.1

/home/<your_username>/.conda/envs/myenv/bin/python my_script.py

Save as job.sh, then submit:

sbatch job.sh

Job Submission & Monitoring

# Submit a job
sbatch job.sh

# Check your jobs (state + reason)
squeue -u <your_username>
squeue -u <your_username> --start            # show estimated start time
squeue -u <your_username> -o "%.18i %.9P %.20j %.2t %.10M %.6D %R"

# Cancel a job
scancel <JOB_ID>

# Cancel all your jobs
scancel -u <your_username>

# Watch a running job's output live (Ctrl+C does NOT cancel the job)
tail -f logs/job_<JOB_ID>.out

# Detailed job info (resources, node, pending reason)
scontrol show job <JOB_ID>

# Post-run stats (exit code, CPU/RAM used)
sacct -j <JOB_ID> --format=JobID,JobName,State,ExitCode,Elapsed,MaxRSS,AllocCPUS

# How many jobs are in a partition right now
squeue -p gpu | wc -l

Job States

State Meaning
PD Pending — waiting in queue
R Running
CG Completing
(gone) Finished — check logs

Pending Reasons

Reason Meaning
Priority Other jobs ahead — normal, will start soon
Resources Waiting for CPUs/RAM/GPUs to free up
QOSMaxJobsPerUser Hit 4-concurrent-job limit
QOSMaxGRESPerJob Requesting too many GPUs — use --gres=gpu:a100:1
QOSMaxSubmitJobPerUserLimit Hit 8-job submit cap
PartitionTimeLimit Requested time exceeds partition max

Interactive GPU Session

# A100 (good for development and debugging)
srun --partition=gpu-interactive --gres=gpu:a100:1 --mem=32G --cpus-per-task=4 --pty /bin/bash

# V100 (faster to get)
srun --partition=gpu-interactive --gres=gpu:v100:1 --mem=16G --cpus-per-task=4 --pty /bin/bash

Once inside: load modules, activate conda, run Python directly.

Useful One-liners

# What modules are available?
module avail

# What modules are currently loaded?
module list

# Load CUDA
module load cuda/12.1

# Check CUDA version
nvcc --version

# Python path in conda env (use this in SLURM scripts)
which python   # only valid after conda activate in interactive shell

# Run a quick GPU test (from inside a job or interactive session)
python -c "import torch; print(torch.cuda.get_device_name(0), torch.cuda.get_device_properties(0).total_memory // 1e9, 'GB')"

# Add PYTHONUNBUFFERED so print() shows up in SLURM logs without delay
export PYTHONUNBUFFERED=1

Common Pitfalls

Symptom Cause Fix
conda activate fails in batch job SLURM runs non-interactive shell Use full path: /home/<user>/.conda/envs/myenv/bin/python
Python prints nothing in .out log stdout buffering Add export PYTHONUNBUFFERED=1 to script
git pull fails with auth error Repo is private, HTTPS not supported Switch to SSH remote: git remote set-url origin git@github.com:user/repo.git
Job rejected: PartitionTimeLimit Requested time > partition max Lower --time to fit partition limit
Job rejected: QOSMaxSubmitJobPerUserLimit 8-job cap reached Wait for jobs to finish or scancel some
Job pending forever: QOSMaxGRESPerJob Requested >1 GPU, partition allows 1 Change to --gres=gpu:a100:1
conda create fails / env broken Module not loaded first module load anaconda3/2024.06 before any conda command