Add a test that compares the restart files against a known-good output

[manodeep]

This issue uncovered a case where even though the bitwise repro checks passed, the restart files were actually corrupted. Normally such a situation would be caught by a test that compares, say 1 run with 2 days against the output from 2 consecutive runs with 1 day each; however, ESM1.6 is known to be not restart-reproducible - so we don't have that test.

In absence of restart reproducibility, such a case would have been caught if we compared the restart at the end of 1-day (say checksum) against a known-good checksum that's committed into the relevant config-branch.

[anton-seaice]

Some options

1. Presumably its simplest, to just change the historical repro test to be two consecutive runs with a restart in the middle (and save the checksums from the end of the run)

2. It would be better to be saving two sets of checksums, one unchanged from the current historical test, and one for the restart tests.

3. (A third option would be to calculate checksums for all restart fields in the historical tests (rather than only the ocean fields), which also would catch this problem)

[aidanheerdegen]

Can we just use the MD5 hash in the manifest?

Related: #86

[anton-seaice]

In my daydreams we add metadata (like the UUID, or the author/rundate/git repo etc) to restart files which breaks that (we may already in some cases)

[aidanheerdegen]

In my daydreams we add metadata (like the UUID, or the author/rundate/git repo etc) to restart files which breaks that (we may already in some cases)

Hrm. Yeah real plusses and minuses territory there.

So we'd probably have to go down the route of adding variable level MD5 checksums to the metadata. netCDF4 supports HDF5 Fletcher 32 data integrity checksums, but they are calculated at the chunk level, so might be a bit of overkill for this use case.

[anton-seaice]

I got the impression that they were quite slow / overkill ? Although don't really remember now

The mom approach of calculating a global sum, or a global checksum for each restart field seems like the best balance to me

[manodeep]

@blimlim and I have uncovered yet another interesting bug with the AMIP config. One run produces bitwise identical results and restart files for the first year, but then diverges immediately upon startup for the second year. The difference seems to come from the land-use change code path, which only gets executed after the first year, i.e., we would need to run a 1 year experiment, followed by a restart to catch this bug.

May be a starting point would be to figure out whether the configs execute different code-paths depending on the simulated period.

[manodeep]

More details are here

[aidanheerdegen]

Options for longer runs

1. Self-hosted runners using our kubernetes cluster. The jobs would require very little resources as they're just submitting a PBS job and waiting for them to finish. I know @CodeGat had some concerns around security using self-hosted runners, so not sure if there is any progress there, e.g. using ssh certificate authentication.
2. Submit PBS job but don't wait, and then poll for completion (there are some existing actions to poll for HTTP endpoints, but none look well maintained)
3. Submit PBS job but don't wait, but run some sort of API/Webhook call after job has completed to call next part of workflow. Would require internet access ...

Option 2 would use/block a GitHub runner instance for a long period. Options 1 and 3 don't block/use a GitHub runner instance but are probably more work to implement.

[CodeGat]

Option 1 sounds pretty good. I think I'd stand up another cluster just so model-config-tests runs don't step on the toes of build-cis stuff, and if the jobs are very lightweight I see no reason that the cluster couldn't be, as well! As for the security side of things, I think that if we lock down the workflow running self-hosted runners via runner groups to branches that are well protected via and can't be pushed to (outside of members), I think they're secure enough. We do this with build-ci and I'm pretty happy with it!

[CodeGat]

Hey gang, I've set up a low resource kubernetes cluster for self-hosted runners, for the express purpose of repro CI. To use it in a workflow, instead of:

runs-on: ubuntu-latest

You would do:

runs-on:
  group: repro-ci

This cluster of runners is explicitly for low-resource jobs (i.e., sshing into Gadi and doing work on there). I've restricted the usage of those runners to only run workflows on the main branch of model-config-tests, so they're secure.

There will have to be updates to that workflow, so it chooses to use the self-hosted runners (or maybe an input to pick between GitHub and self hosted, like in https://github.com/ACCESS-NRI/build-ci/blob/v3/.github/workflows/ci.yml#L120-L126 and https://github.com/ACCESS-NRI/build-ci/blob/v3/.github/workflows/ci.yml#L194