Skip to content

Instrastructure

Jump to bottom
Sunyoung Yoo edited this page Sep 8, 2025 · 37 revisions

This page is about how the infrastructure is set for the summer school.

Empty ticker means not implemented yet. Should be connected to the issue or jira ticket.

Check List

All the listed task should be done before every school.
It should be done a couple of weeks in advance to the school.

  • Create a Call (ask SIMS team) for workshops this year.
  • Create a Proposal for the school under the Call using instrument WORKSHOP.
    • Ask SIMS team to approve the proposal.
      (It will allow students to use VISA)
    • Invite all students as CoProposers.
      (It will send invitation to the User Office and ask them to create an ESS account.)
  • Test the script to create the directory for each student.
  • Test all the notebooks in a VISA instance under the proposal ID.
  • Students should have write access to the upload folder so that scitacean can copy the uploaded files there (with UUID as a prefixed subdirectory).

Per Student

Before Arrival

  • Create ESS account and Join the proposal.
    It is very important that students do this before they arrive.
    Synchronization between ESS account and DMSC LDAP happens ONLY when the scientific user JOINs the proposal, not when the account is created. And the sync happens only once an hour.

At the beginning (or before arrival)

  • Instruct the student to clone the material/make the directory for the dataset.

    • They should temporarily have write access to the derived folder for this.

      In 2025, we wrote this script for students so that they can simply execute the bash script that clone the workbook repo with the right name (including their user ID) and make a soft link in their home folder.

      source /PATH/TO/THE/PROPOSAL/DIRECTORY/derived/tools/initialize_visa.sh
      Full Script (Some values are not filled) 
        export PROPOSAL_DIR=/PATH/TO/THE/PROPOSAL/DIRECTORY/derived
  export WORKBOOK_DIR_NAME=${USER}-workbook
  export WORKBOOK_DIR=${PROPOSAL_DIR}/${WORKBOOK_DIR_NAME}
  export http_proxy=PROXY_ADDRESS
  export https_proxy=PROXY_ADDRESS
  export WORKBOOK_REPO="https://github.com/ess-dmsc-dram/dmsc-school-notebooks.git"

  if [ ! -d "${WORKBOOK_DIR}" ]; then
      echo "Workbook ${WORKBOOK_REPO} is not cloned yet.";
      echo "Cloning workbook...";
      cd ${PROPOSAL_DIR};
      git clone ${WORKBOOK_REPO} ${WORKBOOK_DIR_NAME};
      git config --global --add safe.directory ${WORKBOOK_DIR};  # Issue with ownership retrieval
      cd ${HOME};
  fi

  echo "Workbook is cloned in ${WORKBOOK_DIR}"

  if [ ! -d "${HOME}/workbook" ]; then
      echo "workbook symboliclink is not set yet.";
      echo "Creating a symbolic link...";
      cd ${HOME};
      ln -s ${WORKBOOK_DIR} workbook;
  fi

Before Leave

  • Students should zip the workbook (excluding too large files) and download them using the file browser interface. 
    cd  # At home, where the soft link to the local repository exists
zip -r dmsc-school-workbook.zip workbook/

Hardware Requirement

This is the hardware spec we used for summer school 2025.

Platform Memory Cores Number of Instances
VISA 32 GB 8 20 students + 1~3 teachers

Student Accounts

Students can be invited to ESS User Office and create their ess account from ORCID ID.

  • They should be added to a proposal under workshop instrument. (DST ticket #151)
  • Once they are added to a proposal the students can log into the VISA.

ESS account is not strongly bound with ORCID ID once it's created.

Environment Setting

File Storage and Scicat

Since the derived folder is accessible in VISA instances we will simply copy over the file when uploading the dataset to scicat.
SFTP file transfer is avilable (stfp.esss.dk) but we are not using it in the school since it requires setting up ssh and is complicated to use.
And we decided to skip it in the scicat lesson as well.

  • We need to specify the location of the file from perspective of scicat backend when uploading the dataset and this is hardcoded in the widget helper. #128

Directory Structure

We did not automate the creation of directory or cloning the material.
Therefore students will be instructed to do this by themselves with the temporary write access to the derived folder.
And they can freely create files and edit the files under their own directory.

  • This will be a part of the introductory course.
Why Not Automate This? We decided not to automate this since it takes too much time and effort to maintain such pipeline.
But it may be automated later if this kind of workshop happens more often and this step becomes a problem.

workspace: cloned local repository of each student

Each course(simulation, reduction and analysis) export its result as a relevant file format under each corresponding directory like below.
And students can move or copy these files under data directory in their own workspace for uploading scicat.
Then scicat may copy this file to permanent read-only space
In this way, we prevent accidentally overwriting the file.

PROPOSAL_DIR
├── upload/
|   ├── uuid-1/uploaded_reduced_data.hdf
|   ├── uuid-2/uploaded_reduced_data.txt
│   ├── ...
|   └── uuid-3/uploaded_reduced_data.csv
└── derived/
    ├── STUDENT1-DMSC-SCHOOL
    │   ├── 1-python/
    │   ├── ...
    │   ├── 3-mcstas/
    │   │   └── simulation_data.nxs
    │   ├── ...
    │   └── data/
    │       ├── simulation_data.nxs
    │       └── reduced_data
    └── STUDENT2-DMSC-SCHOOL
        ├── 1-python/
        ├── ...
        ├── 3-mcstas/
        │   └── simulation_data.nxs
        ├── ...
        └── data/
            ├── simulation_data.nxs
            └── reduced_data

Clone this wiki locally