1.7.0 seems slower

[DJDavies2]

What happened?

I built ectrans and ran the ctests for version 1.7.0 and on one of the machines I ran on the ctests seemed to run much slower. There are multiple tasks taking several hundred seconds; for 1.6.2 nothing takes more than 50 apart from the install task.

What are the steps to reproduce the bug?

I built with gnu 12.2.0, latest tags of eckit, fckit and fiat.

Version

1.7.0

Platform (OS and architecture)

Linux cazldf0000CG 5.14.0-570.49.1.el9_6.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Sep 23 01:11:18 EDT 2025 x86_64 x86_64 x86_64 GNU/Linux

Relevant log output

This is just a selection:

184/189 Test #147: ectrans-benchmark-cpu-sp_T47_O48_mpi1_omp8_callmode1_nfld10_nlev20_flt ...........   Passed  244.22 sec
185/189 Test #153: ectrans-benchmark-cpu-sp_T47_O48_mpi1_omp8_callmode2_nfld10_nlev20_flt ...........   Passed  242.56 sec
186/189 Test #101: ectrans-benchmark-cpu-dp_T47_O48_mpi2_omp8_callmode2_nfld10_nlev20_flt ...........   Passed  396.21 sec
187/189 Test  #95: ectrans-benchmark-cpu-dp_T47_O48_mpi2_omp8_callmode1_nfld10_nlev20_flt ...........   Passed  398.89 sec
188/189 Test #173: ectrans-benchmark-cpu-sp_T47_O48_mpi2_omp8_callmode1_nfld10_nlev20_flt ...........   Passed  346.12 sec
189/189 Test #179: ectrans-benchmark-cpu-sp_T47_O48_mpi2_omp8_callmode2_nfld10_nlev20_flt ...........   Passed  292.95 sec

Accompanying data

No response

Organisation

Met Office

[samhatfield]

Hi @DJDavies2, this is not supposed to happen. Here's a comparison between standard (no features passed to CMake) builds of 1.6.2 and 1.7.0:

1.6.2

100% tests passed, 0 tests failed out of 163

Label Time Summary:
ectrans       =  72.05 sec*proc (162 tests)
executable    =   7.14 sec*proc (15 tests)
mpi           =  47.12 sec*proc (102 tests)
openmp        =  64.22 sec*proc (144 tests)
script        =  64.91 sec*proc (147 tests)

Total Test time (real) =  79.25 sec

1.7.0

100% tests passed, 0 tests failed out of 187

Label Time Summary:
Fortran       =   3.82 sec*proc (6 tests)
cpu           =   3.82 sec*proc (6 tests)
ectrans       = 145.78 sec*proc (180 tests)
executable    =   8.89 sec*proc (20 tests)
mpi           =  97.81 sec*proc (114 tests)
openmp        = 134.47 sec*proc (156 tests)
script        = 136.89 sec*proc (160 tests)

Total Test time (real) = 156.14 sec

Running on an Apple MacBook M1 Pro, GNU 15.1.0.

1.7.0 is twice as slow. This is expected as we've added tests for the first "call mode" of ecTrans and added checksum comparisons between different MPI cases. However no CPU tests should be taking 100s of seconds.

Maybe you're seeing file system latency from the checksum file dumping. I think it's unlikely, as the wall times are huge, but worth a try. If this is an issue then we might have to add an option to disable checksum tests.

Could you try deleting all examples of --dump-checksums {filename} from tests/CMakeLists.txt and also delete these lines, then repeat the tests? You better delete this line too.

Also could you share the log with the full list of tests so I can try and spot any patterns?

[samhatfield]

For reference I did the same (removed checksum dumping as per the instructions above) and this is what I get:

100% tests passed, 0 tests failed out of 186

Label Time Summary:
Fortran       =   4.31 sec*proc (6 tests)
cpu           =   4.31 sec*proc (6 tests)
ectrans       =  67.55 sec*proc (179 tests)
executable    =   9.55 sec*proc (20 tests)
mpi           =  45.64 sec*proc (114 tests)
openmp        =  57.33 sec*proc (156 tests)
script        =  57.99 sec*proc (159 tests)

Total Test time (real) =  78.46 sec

[DJDavies2]

ectrans.ctest.log

[DJDavies2]

I've attached the full ctest output. It is also output for #322 for reference.

[DJDavies2]

I also ran with the changes suggested (removing --dump-checksums and whatnot). The tests run much more quickly, the longest is about 110 seconds.

[DJDavies2]

Just to comment, it appears that forcing jobs to run sequentially seems to make the jobs run much quicker. We normally run ctest jobs in parallel.

[samhatfield]

I think you are just seeing file system latency then. We always run ctest serially so I guess that's why we never spotted it. Is that a workable solution for you, or is the test suite still too long? If so you could either exclude tests to reduce the wall time, or we could perhaps create a new feature which disables checksum dumping to accelerate testing.

[DJDavies2]

Are the jobs writing to different files or the same files? If the latter maybe they could be changes to write to different files?

Running sequentially is okay for us. However making jobs write to different files is perhaps fixable if that is the issue.

[samhatfield]

Each job writes to a separate file (actually two, one for the inverse transform and one for the direct transform). I'll try running ctest in parallel to see if our system shows a similar problem.