fixing building apptainer

Author: andrew-e

.env_example

@@ -5,6 +5,6 @@ DOCKER_VERSION=1.1.0
 # Optional variables
 QTL_DATA_DIR=
 SNAKEMAKE_PROFILE=snakemake/profiles/local/
-APPTAINER_MODULE=apptainer/1.3.1-ksax
+APPTAINER_MODULE=
 SLURM_PARTITION=
 SLURM_ACCOUNT=

README.md

@@ -42,7 +42,7 @@ There are **six** Snakemake pipelines (grouped as two tables of three). Each pip
 ### 1. Clone the repository into your personal space on BlueCrystal 4
 `git clone git@github.com:MRCIEU/GeneHackman.git && cd GeneHackman`
 
-### 2. Ensure you [have conda installed and initialised before activating](https://www.acrc.bris.ac.uk/protected/hpc-docs/software/python_conda.html)
+### 2. Ensure you [have conda installed and initialised before activating](https://www.anaconda.com/docs/getting-started/miniconda/install/overview)
 
 ```conda env create --file environment.yml```
 
@@ -84,11 +84,41 @@ Alternatives if you don’t set **`QTL_DATA_DIR`** directly: keep the same direc
 
 ### 4. Populate `.env` and `input.yaml` files
 
-`cp .env_example .env`
-* Set **`PROJECT_DIR`** to the root folder for this analysis; the pipeline uses **`PROJECT_DIR/data/`** for inputs (GWAS, clumps, …) and **`PROJECT_DIR/results/`** for outputs (finemap, coloc, plots, …).
-* Set **`PIPELINE_DATA_DIR`** to the path where you unpacked **`genehackman`** (see §3).
-* Set **`QTL_DATA_DIR`** to the path where you unpacked **`genehackman-qtl`** if you run **`qtl_mr`** (can be left empty otherwise; see [.env_example](.env_example)).
-* **Container cache:** If there is no pre-built `genehackman_<version>.sif` under `PIPELINE_DATA_DIR`, Snakemake pulls the `docker://` image and caches the SIF under `.snakemake/singularity` by default. Set **`GENEHACKMAN_SINGULARITY_PREFIX`** in `.env` to use another directory (e.g. scratch). Running `snakemake` without `./run_pipeline.sh`? Pass `--singularity-prefix /path` or add `singularity-prefix:` under **`snakemake/profiles/`** in **`config.yaml`** for the profile you use (paths are relative to the repo root).
+Copy the template and edit paths for your machine:
+
+```bash
+cp .env_example .env
+```
+
+Variables match [.env_example](.env_example):
+
+**Mandatory**
+
+| Variable | Purpose |
+|----------|---------|
+| **`PROJECT_DIR`** | Root folder for this analysis. The pipeline uses **`PROJECT_DIR/data/`** for inputs (GWAS, clumps, …) and **`PROJECT_DIR/results/`** for outputs (finemap, coloc, plots, …). |
+| **`PIPELINE_DATA_DIR`** | Path where you unpacked the shared **`genehackman`** reference data (see §3). Also used for the Apptainer image: **`PIPELINE_DATA_DIR/genomic_data/pipeline/genehackman_<DOCKER_VERSION>.sif`**. If that SIF is missing and the directory is writable, `run_pipeline.sh` builds it from `docker://mrcieu/genehackman:<DOCKER_VERSION>`. |
+| **`DOCKER_VERSION`** | Docker/Apptainer image tag (e.g. `1.1.0` or `develop`). Must match the SIF filename and the image you intend to run. |
+
+**Optional**
+
+| Variable | Purpose |
+|----------|---------|
+| **`QTL_DATA_DIR`** | Path to **`genehackman-qtl`** data if you run **`qtl_mr`**. Leave empty for other pipelines. You can instead place QTL data under **`PIPELINE_DATA_DIR/qtl_datasets/`** (see [PLATFORM_SETUP.md](PLATFORM_SETUP.md)). |
+| **`SNAKEMAKE_PROFILE`** | Snakemake profile directory. Default in `.env_example`: **`snakemake/profiles/local/`** (local Apptainer). On HPC use e.g. **`snakemake/profiles/slurm/`**. |
+| **`APPTAINER_MODULE`** | Environment module name to load Apptainer/Singularity before running on HPC (only used when the profile is not `local`). |
+| **`SLURM_PARTITION`** | Slurm partition for cluster jobs. If unset, `run_pipeline.sh` tries to detect one from `sinfo`, else uses **`compute`**. |
+| **`SLURM_ACCOUNT`** | Slurm account for cluster jobs. If unset, `run_pipeline.sh` tries to infer it from `sacctmgr`. |
+
+Example `.env`:
+
+```bash
+PROJECT_DIR=/path/to/my_project
+PIPELINE_DATA_DIR=/path/to/my_pipeline_data/
+DOCKER_VERSION=1.1.0
+QTL_DATA_DIR=/path/to/my_qtl_data/
+SNAKEMAKE_PROFILE=snakemake/profiles/local/
+```
 
 **`input.yaml`**
 

run_pipeline.sh

@@ -24,7 +24,7 @@ fi
 
 if [ -f .env ]
 then
-  export $(cat .env | xargs)
+  export $(grep -vE '^[[:space:]]*#' .env | xargs)
 else
   echo "Error: .env file missing"
   exit 1
@@ -50,10 +50,10 @@ if [[ -z "${SLURM_PARTITION:-}" ]]; then
 fi
 export SLURM_PARTITION
 
-if [[ -z "${SLURM_ACCOUNT:-}" ]]; then
+if [[ -z "${SLURM_ACCOUNT:-}" ]] && command -v sacctmgr >/dev/null 2>&1; then
   export USER=$(whoami)
-  export ACCOUNT_ID=$(sacctmgr show user withassoc format=account where user=$USER | grep '[0-9]' | head -n1)
-  export SLURM_ACCOUNT="$ACCOUNT_ID"
+  export ACCOUNT_ID=$(sacctmgr show user withassoc format=account where user="$USER" | grep '[0-9]' | head -n1)
+  export SLURM_ACCOUNT="${ACCOUNT_ID:-}"
 fi
 
 export GENEHACKMAN_EXTRA_SINGULARITY_BINDS=""
@@ -102,11 +102,13 @@ PIPELINE_GENOMIC_DIR="${PIPELINE_GENOMIC_DIR%/}"
 SIF_PATH="${PIPELINE_GENOMIC_DIR}/${SIF_NAME}"
 
 echo "SIF_PATH: ${SIF_PATH}"
-if [[ -f "${SIF_PATH}" ]] || [[ -w "${PIPELINE_GENOMIC_DIR}" ]] ; then
-  mkdir -p "${PIPELINE_GENOMIC_DIR}"
-else
-  SIF_PATH=".snakemake/singularity/${SIF_NAME}"
-fi  
+if [[ ! -f "${SIF_PATH}" ]]; then
+  if ! mkdir -p "${PIPELINE_GENOMIC_DIR}" 2>/dev/null || [[ ! -w "${PIPELINE_GENOMIC_DIR}" ]]; then
+    SIF_PATH=".snakemake/singularity/${SIF_NAME}"
+    echo "Note: caching SIF under ${SIF_PATH} (${PIPELINE_GENOMIC_DIR} missing or not writable)"
+  fi
+fi
+mkdir -p "$(dirname "${SIF_PATH}")"
 
 SINGULARITY_DOCKER_REFERENCE="docker://mrcieu/genehackman:${GENEHACKMAN_VERSION}"
 
@@ -123,7 +125,6 @@ fi
 
 if [[ ! -f "${SIF_PATH}" ]]; then
   echo "SIF file not found: ${SIF_PATH}"
-  mkdir -p "${PIPELINE_GENOMIC_DIR}"
   echo "Building container with singularity from: ${SINGULARITY_DOCKER_REFERENCE}"
   if [[ "${#SINGULARITY_BUILD_ARCH_ARGS[@]}" -gt 0 ]]; then
     echo "(host is ARM: using singularity build ${SINGULARITY_BUILD_ARCH_ARGS[*]} for amd64 image)"