cycle4

[hakaseh]

@aekiss After setting and testing master+bgc, I'm thinking to create a branch cycle4 for our upcoming cycle 4 run. let me know if you have a better branch name in mind.

For this, I'd like to create diag_table. can you locate me to diag_table_source.yaml that has all the physical diagnostics for cycle 4? I can add BGC diagnostics to it.

Also do you have any recommendation for the duration of 0.1-deg run for one job? For physics only, it seems walltime is set to 5:00:00 for 3 months? If it's good to keep 3 months at a time, and BGC requires approximately nearly doubling, I can try 10:00:00 for 3 months?

[hakaseh]

I just noticed that there is already a branch for Cycle 4, https://github.com/COSIMA/01deg_jra55_iaf/tree/01deg_jra55v140_iaf_cycle4.

Should I pull this and start adding BGC diagnostics and all other bgc settings?

[aekiss]

Yes, https://github.com/COSIMA/01deg_jra55_iaf/tree/01deg_jra55v140_iaf_cycle4 is the branch intended for this purpose.

diag_table_source.yaml in that branch has the physical diagnostics from cycle3 so that is a good starting point for you to add BGC diags to.

There's a 5-hour walltime limit with this number of cores https://opus.nci.org.au/display/Help/Queue+Limits
so I think we'll need to use 1mo runs, at least to start with.

[aekiss]

FYI the actual walltime for cycles 2 and 3 is in these tables
https://github.com/COSIMA/01deg_jra55_iaf/blob/01deg_jra55v140_iaf_cycle2/run_summary_home_156_aek156_payu_01deg_jra55v140_iaf_cycle2.csv
https://github.com/COSIMA/01deg_jra55_iaf/blob/01deg_jra55v140_iaf_cycle3/run_summary_home_156_aek156_payu_01deg_jra55v140_iaf_cycle3.csv
it's under 3hr on average but gets close to 4hr when we need to reduce the timestep.

[aekiss]

If we have 1mo runs we'll need to set new_file_freq: 1 in the global section of diag_table_source.yaml

[hakaseh]

my test runs with master+bgc took just under 3 hours for 1 month simulation. but cycle4 will take longer because of substantially larger diagnostics.

[aekiss]

Wow, so 3x slower... What timestep did you use?

[hakaseh]

300s, i did not change from the default. should i try increase?

[aekiss]

I guess you were starting from rest? If so, 300s is a good initial choice. 540s will probably only work after ~1.5yr, as it needs time to settle down from the IC. See timesteps in first cycle: https://github.com/COSIMA/01deg_jra55_iaf/blob/01deg_jra55v140_iaf/run_summary_home_156_aek156_payu_01deg_jra55v140_iaf.csv

So that makes it more like 1.7x slower if we take timestep into account.
How many cores are you using?

[hakaseh]

yes, from rest, using the same number of cores as master of https://github.com/COSIMA/01deg_jra55_ryf

[hakaseh]

I haven't sorted out the walltime for Cycle 4 yet, but I have created an BGC-enabled branch for Cycle 4 in my fork repo:
https://github.com/hakaseh/01deg_jra55_iaf/tree/01deg_jra55v140_iaf_cycle4_bgc
This should have all the changes needed to enable BGC runs. The next step is to set up restart files for this run, which requires manually adding sea-ice BGC tracers to sea-ice restart file, which I have tested with 1-degree in the past.

Will this be the restart directory used for the start of cycle 4?: /g/data/ik11/outputs/access-om2-01/01deg_jra55v140_iaf_cycle3/restart731

[aekiss]

I've just merged master+bgc into 01deg_jra55v140_iaf_cycle4, so your https://github.com/hakaseh/01deg_jra55_iaf/tree/01deg_jra55v140_iaf_cycle4_bgc could be merged into 01deg_jra55v140_iaf_cycle4 via a pull request when we're ready.

Yes, /g/data/ik11/outputs/access-om2-01/01deg_jra55v140_iaf_cycle3/restart731 is the restart we'll start from.
See https://github.com/COSIMA/01deg_jra55_iaf/blob/01deg_jra55v140_iaf_cycle3/run_summary_home_156_aek156_payu_01deg_jra55v140_iaf_cycle3.csv

metadata.yaml will need to be updated to document the BGC initial conditions used.

[hakaseh]

I actually created https://github.com/hakaseh/01deg_jra55_iaf/tree/01deg_jra55v140_iaf_cycle4_bgc so that it can be merged into 01deg_jra55v140_iaf_cycle4 without conflicts. There will be conflicts in BGC setup between master+bgc and cycle4 for example diag_table.

I can do pull request but is it possible to try merge with the version of 01deg_jra55v140_iaf_cycle4 before merging with master+bgc? I think this should be easier than resolving conflicts between master+bgc and https://github.com/hakaseh/01deg_jra55_iaf/tree/01deg_jra55v140_iaf_cycle4_bgc.

I have updated metadata.yaml to include BGC initial conditions in https://github.com/hakaseh/01deg_jra55_iaf/tree/01deg_jra55v140_iaf_cycle4_bgc.

[hakaseh]

Regarding the restart files for sea-ice BGC tracers, I added them and created the following new files based on /g/data/ik11/outputs/access-om2-01/01deg_jra55v140_iaf_cycle3/restart731/ice/:

iced.2019-01-01-00000.nc, i2o.nc, o2i.nc.

The shell script to create these files are below (I couldn't attach *.sh here). Should I add this script to input_om2-bgc or create a separate github repo?

#Add sea-ice BGC tracers to the restart file

#Define the path to the restart file you want to add BGC tracers
path2restart=/g/data/ik11/outputs/access-om2-01/01deg_jra55v140_iaf_cycle3/restart731/ice

#Define the name of the restart file
filename=iced.2019-01-01-00000.nc

#Copy the restart file to pwd
cp ${path2restart}/${filename} .

#Add 2D fields.
for j in algalN nit
do
	ncap2 -O -s ${j}=iceumask*0 ${filename} ${filename}
done

#Add 3D fields.
for j in bgc_N_sk bgc_Nit_sk
do
	ncap2 -O -s ${j}=aicen*0 ${filename} ${filename}
done

#Next i2o.nc and o2i.nc

filename=i2o.nc

#Copy the restart file to pwd
cp ${path2restart}/${filename} .

for j in wnd10_io nit_io alg_io
do
    ncap2 -O -s ${j}=licefh_io*0 ${filename} ${filename}
done

filename=o2i.nc

#Copy the restart file to pwd
cp ${path2restart}/${filename} .

for j in ssn_i ssalg_i
do
    ncap2 -O -s ${j}=sst_i*0 ${filename} ${filename}
done