The cascade config structure

The config cascade structure

Use a cascade structure to separate concerns so that a task/job only need to define required environmental variables in runtime

config.${MACHINE} -> config.meshdep -> config.base -> [config.realtime if it is a realtime run] -> [config.chemistry if do_chemistry] -> [config.override if exists]
NOTE:

1. [ ] means optional
2. Many resource-related configurations (queue/account,etc) are only visible in the workflow setup process and are not needed in runtime
3. task-specific resource settings are through variable cascade in config.base. For example, we define WALLTIME for all tasks first. If we want to set a different WALLTIME for spinup fcst, we can add a variable WALLTIME_FCST_SPINUP in config/resources/config.base. setup_xml.py has a function called get_cascade_env to retrieve the correct environmental setting in a reverse cascade way. Examples:

export NODES=${NODES:-"<nodes>1:ppn=1</nodes>"} #the NODES variable has to include the tag <nodes> or <cores>
........
........
# fcst
export NODES_FCST=${NODES_FCST:-"<nodes>3:ppn=40</nodes>"}
export WALLTIME_FCST=${WALLTIME_FCST:-"00:50:00"}

exp.setup

Set up top-level options for an experiment, such as directories, VERSION, TAG, days for a realtime run or retro period for a retro run.

config.${MACHINE}

Define high-level common machine-dependent settings

config.meshdep

Set mesh dependent settings in this file

config.base

Most default system settings go into this file

config.realtime

This file defines DEADLINE and STARTTIME for rocoto-based realtime runs, not needed by and does not affect NCO

config.chemistry

This config file provides a centralized location to configure options for chemistry applications. Some settings can be reconfigured in this file to meet the chemistry need. This file takes effect only when DO_CHMISTRY is set to true in the exp file.

config.override

This file is optional and will NOT be included in rrfs-workflow. But users can link their own config.override to workflow/config_resources and setup_rocoto.py will use all variables in this file to override their previous settings. This file is useful for running other meshes not maintained by the authoritative rrfs-workflow (such as ar3.5km, ea5km, eu12km, etc).

Users who use config.override are expected to understand the cascade config structure and use it carefully to avoid unexpected results. Especially, don't override top-level variables, such as EXP_NAME, EXPDIR, COMROOT, NET, VERSION, RUN, TAG, WGF, DO_ENSEMBLE, etc, which should be configured in the exp files. Also, if a variable is used to define other variables in the exp file, all related variables should be overwritten.

Here is an example config.override used to run RRFSv1 initialized experiments:

#!/usr/bin/env bash
export ZETA_LEVELS="L60.txt"

export IC_OFFSET=0
export LBC_OFFSET=0
export LBC_LENGTH=24

export LBC_UNGRIB_GROUP_TOTAL_NUM=6
export LBC_GROUP_TOTAL_NUM=2
export WALLTIME_UNGRIB_LBC="1:30:00"

export RETRO_PERIOD="2025111418-2025111418"
export LBC_CYCS="00"
export COLDSTART_CYCS="00"

export IC_EXTRN_MDL_NAME="RRFS"
export IC_EXTRN_MDL_NAME_PATTERN="rrfs.@Y@m@d/@H/rrfs.t@Hz.natlev.3km.ffHHH.na.grib2"
export IC_EXTRN_MDL_NAME_PATTERN_B=""
export LBC_EXTRN_MDL_NAME=${IC_EXTRN_MDL_NAME}
export LBC_EXTRN_MDL_NAME_PATTERN=${IC_EXTRN_MDL_NAME_PATTERN}
export LBC_EXTRN_MDL_NAME_PATTERN_B=${IC_EXTRN_MDL_NAME_PATTERN_B}

export IC_EXTRN_MDL_BASEDIR="/scratch4/BMC/rtrr/RRFS2_RETRO_DATA/Nov2025/RRFS"
export LBC_EXTRN_MDL_BASEDIR=${IC_EXTRN_MDL_BASEDIR}