compare-exp-tests looks in non-existent directory for restart files

[manodeep]

When trying to compare two test runs that I just finished, I get a FileNotFoundError from compare-exp-tests -

>       with open(mom_restart_pointer) as f:
E       FileNotFoundError: [Errno 2] No such file or directory: '/scratch/tm70/ms2335/access-om3/archive/25km_jra_iaf_May5_35days/restart000/rpointer.ocn'

From what I can see, the underlying issue is that the root filepath is /scratch rather than the sym-linked archive path within the experiment directory, and hence is missing the branch-name and UUID in the archive path (-dev-MC_25km_jra_iaf-a73dd521) - like so:

lrwxrwxrwx 1 ms2335 tm70   93 May  6 19:52 archive -> /scratch/tm70/ms2335/access-om3/archive/25km_jra_iaf_May5_35days-dev-MC_25km_jra_iaf-a73dd521

Here is the full traceback:

(test-venv-model-config-tests) [~/perf-om3/25km/test-nonblocking-updates @gadi-login-08]$ compare-exp-tests --dirs "25km_jra_iaf_May5_35days 25km_jra_iaf_May5_35days_non_blocking_updt"
Test session starts (platform: linux, Python 3.10.0, pytest 9.0.3, pytest-sugar 1.1.1)
rootdir: /home/593/ms2335/test-venv-model-config-tests/lib/python3.10/site-packages/model_config_tests/compare_exp_tests
plugins: sugar-1.1.1
collected 1 item                                                                                                                                                                           


――――――――――――――――――――――――――――――――――――――――――――――― test_pairwise_repro[25km_jra_iaf_May5_35days vs 25km_jra_iaf_May5_35days_non_blocking_updt] ――――――――――――――――――――――――――――――――――――――――――――――――

experiment_1 = PosixPath('/g/data/tm70/ms2335/performance_optimisation_runs/oneapi/2025.2.0/om3/25km/test-nonblocking-updates/25km_jra_iaf_May5_35days')
experiment_2 = PosixPath('/g/data/tm70/ms2335/performance_optimisation_runs/oneapi/2025.2.0/om3/25km/test-nonblocking-updates/25km_jra_iaf_May5_35days_non_blocking_updt')

    def test_pairwise_repro(experiment_1: Path, experiment_2: Path):
        """
        Compare combinations of experiments to check for reproducibility.
        This is parametrised in conftest with pytest_generate_tests to
        dynamically generate pairs of experiments to compare.
        """
    
        lab_path1 = get_lab_path(experiment_1)
        exp1 = ExpTestHelper(
            control_path=experiment_1, lab_path=lab_path1, disable_payu_run=True
        )
    
        lab_path2 = get_lab_path(experiment_2)
        exp2 = ExpTestHelper(
            control_path=experiment_2, lab_path=lab_path2, disable_payu_run=True
        )
    
        # Compare the two experiments - compares checksums from output000
>       exp1_checksums = exp1.model.extract_full_checksums()

/home/593/ms2335/test-venv-model-config-tests/lib/python3.10/site-packages/model_config_tests/compare_exp_tests/test_repro.py:39: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/home/593/ms2335/test-venv-model-config-tests/lib/python3.10/site-packages/model_config_tests/models/accessom3.py:129: in extract_full_checksums
    return self.extract_checksums(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <model_config_tests.models.accessom3.AccessOm3 object at 0x7fb85093c7c0>, output_directory = None, schema_version = '1-0-0'

    def extract_checksums(
        self,
        output_directory: Path = None,
        schema_version: str = None,
    ) -> dict[str, Any]:
        """Parse output file and create checksum using defined schema"""
        if output_directory:
            mom_restart_pointer = output_directory / self.mom_restart_pointer_filename
        else:
            mom_restart_pointer = self.mom_restart_pointer
    
        # MOM6 saves checksums for each variable in its restart files.
        # In unified (collated) restart files, the checksum is stored once per variable.
        # In split (per-processor) restarts, the global checksum is duplicated
        # into every tile file, hence reading one tile (e.g. .0000) is sufficient.
        output_checksums: dict[str, list[any]] = defaultdict(list)
    
>       with open(mom_restart_pointer) as f:
E       FileNotFoundError: [Errno 2] No such file or directory: '/scratch/tm70/ms2335/access-om3/archive/25km_jra_iaf_May5_35days/restart000/rpointer.ocn'

/home/593/ms2335/test-venv-model-config-tests/lib/python3.10/site-packages/model_config_tests/models/accessom3.py:101: FileNotFoundError

 test_repro.py ⨯                                                                                                                                                             100% ██████████
================================================================================= short test summary info ==================================================================================
FAILED ../../../../../../../../../../home/593/ms2335/test-venv-model-config-tests/lib/python3.10/site-packages/model_config_tests/compare_exp_tests/test_repro.py::test_pairwise_repro[25km_jra_iaf_May5_35days vs 25km_jra_iaf_May5_35days_non_blocking_updt] - FileNotFoundError: [Errno 2] No such file or directory: '/scratch/tm70/ms2335/access-om3/archive/25km_jra_iaf_May5_35days/restart000/rpointer.ocn'

Results (0.64s):
       1 failed
         - test_repro.py:21 test_pairwise_repro[25km_jra_iaf_May5_35days vs 25km_jra_iaf_May5_35days_non_blocking_updt]