Implement Cylc Driver

[bschroeter]

This is the intended target, however, we should still investigate https://github.com/ACCESS-NRI/access-am3-configs/issues/232 as an interim measure.

[bschroeter]

Having met with @jo-basevi and @Qian-HuiChen today, I have the following notes to share:

• Currently, long-running PBS jobs are sub-optimal for reproducibility tests, rather a submit-poll strategy is preferred.
• There is the possibility of a self-hosted runner where GH minutes are no longer an issue, @CodeGat, would be a resource here.
• If we were to do this, we may need to split the workflow in two, submitting the job, then processing outputs for reproducibility. It is unclear how this would work in the current setup.
• At the lowest levels, Payu is invoked via subprocess calls. As such, it is possible to replace these calls with Cylc equivalents. However, it is the remaining tooling in model-config-tests that is strongly-coupled to Payu and would require the most work.

There isn't much developer documentation (at least from what I've seen) for model-config-tests, however, in general a Cylc version would require the following:

1. https://github.com/ACCESS-NRI/model-config-tests/blob/main/src/model_config_tests/exp_test_helper.py would need a Cylc equivalent/child class, or both classes (Payu and Cylc) would need to be generalised from a base class and abstracted accordingly.
2. There would need to be an AM3 script added to this directory: https://github.com/ACCESS-NRI/model-config-tests/tree/main/src/model_config_tests/models
3. Likewise, QA (pre-run sanity checks run on GH runners) will need to be developed for AM3 here: https://github.com/ACCESS-NRI/model-config-tests/tree/main/tests/config_tests/qa
4. Model checksums are stored on the model repository itself (i.e. https://github.com/ACCESS-NRI/access-om3-configs/tree/release-MC_25km_jra_iaf/testing/checksum), these checksums can be whatever we decide (true checksums, Helmholtz numbers, field sums etc.).
5. If we were to go with the interim solution of translating from Cylc to Payu, then we would need to inject an additional step here for the translation: https://github.com/ACCESS-NRI/model-config-tests/blob/main/.github/workflows/test-repro.yml

In general, there are different classes of repro tests:

• Default run comparison (i.e. Known Good Outputs - KGOS)
• Determinism (run twice, same results?)
• Restart reproducibility (run twice, different time periods, same restart?)
• Restart determinism

However, the tests are flexible and we could define our own appropriate tests for AM3.

In short, it is possible to create a Cylc driver for Payu (and may still be preferable to alternatives), but knowing that it would be a substantial package of work to generalise a tool built specifically for Payu to work with Cylc.

[bschroeter]

@jo-basevi, please feel free to correct me on anything above.

[jo-basevi]

Thanks @bschroeter, the above seems correct to me

[bschroeter]

Renaming to describe current efforts, closing redundant issues elsewhere.

[bschroeter]

@jo-basevi, I do not have adequate permissions on the model-configs-repo to assign this. Can you please assign yourself, @Qian-HuiChen and myself to the issue. I will likely lead efforts on this, but I will be relying on your expertise. Thanks.

[jo-basevi]

Done @bschroeter. I've asked one of the repo admins to give you write access to this repository so hopefully you'll have the permissions in future.