Skip to content

Threaded exact restart tests that change task/thread count fail with DGLC or CISM when run to a year or more #4160

New issue
New issue
Closed
Bug
1 of 1 issue completed
#4224
Closed
Threaded exact restart tests that change task/thread count fail with DGLC or CISM when run to a year or more#4160
Bug
1 of 1 issue completed
#4224
Labels
bugsomething is working incorrectlycode healthimproving internal code structure to make easier to maintain (sustainability)
Milestone
ctsm6.0.0 (code freeze)

Description

@ekluzek
ekluzek
opened on Aug 6, 2026

Brief summary of bug

In bringing in #4110 I found that some of the longer threaded tests fail when using DGLC.

ERP_Ly3_P64x2.f10_f10_mg37.IHistClm50BgcCrop.derecho_intel.clm-cropMonthOutput  ( COMPARE_base_rest )
ERP_Ly3_P64x2.f10_f10_mg37.IHistClm60BgcCrop.derecho_intel.clm-cropMonthOutput--clm-matrixcnOn_ignore_warnings  ( COMPARE_base_rest )
ERP_P64x2_Ld1096.f10_f10_mg37.I2000Clm50BgcCrop.derecho_intel.clm-clm50cropIrrigMonth_interp    ( COMPARE_base_rest )
ERP_P64x2_Ld1096.f10_f10_mg37.I2000Clm50BgcCrop.derecho_intel.clm-irrig_o3falk_reduceOutput     ( COMPARE_base_rest )
ERP_P64x2_Ld366.f10_f10_mg37.I2000Clm50BgcCrop.derecho_intel.clm-irrig_alternate_monthly        ( COMPARE_base_rest )
ERP_P64x2_Ld396.f10_f10_mg37.IHistClm60Bgc.derecho_gnu.clm-monthly      ( COMPARE_base_rest )
ERP_P64x2_Ld765.f10_f10_mg37.I2000Clm60BgcCrop.derecho_intel.clm-monthly        ( COMPARE_base_rest )

It seems to be only tests that go beyond a year, at least that's my speculation.

General bug information

CTSM version you are using: ctsm5.4.049-98-g48ed6abb9
Does this bug cause significantly incorrect results in the model's science? No
Configurations affected: ERP restart tests and PEM tests

Details of bug

Looking at the threaded tests that fail comparison of rest to base. It's the cpl history files that fail for 4 fields:

tail -n20 /glade/derecho/scratch/erik/tests_ctsm545acl/ERP_P64x2_Ld366.f10_f10_mg37.I2000Clm50BgcCrop.derecho_intel.clm-irrig_alternate_monthly.GC.ctsm545acl_int/run/ERP_P64x2_Ld366.f10_f10_mg37.I2000Clm50BgcCrop.derecho_intel.clm-irrig_alternate_monthly.GC.ctsm545acl_int.cpl.hi.2001-01-02-00000.nc.base.cprnc.out
grep RMS run/ERP_P64x2_Ld366.f10_f10_mg37.I2000Clm50BgcCrop.derecho_intel.clm-irrig_alternate_monthly.GC.ctsm545acl_int.cpl.hi.2001-01-02-00000.nc.base.cprnc.out
 RMS rofImp_Forr_rofi_glc             7.0515E-21            NORMALIZED  8.4114E-14
 RMS rofExp_Fgrg_rofi                 7.3536E-22            NORMALIZED  9.1454E-15
 RMS glc1Imp_Fgrg_rofi                5.0408E-21            NORMALIZED  1.3781E-15
 RMS glc1Exp_Flgl_qice                4.7167E-21            NORMALIZED  4.4018E-16

so it's a roundoff difference somehow due to DGLC when threading is on.

Important details of your setup / configuration so we can reproduce the bug

This was found in #4110 with this comment:

#4110 (comment)

Important output or errors that show the problem

These are examples of tests that fail:

ERP_P64x2_Ld366.f10_f10_mg37.I2000Clm60Sp.derecho_intel.clm-nofireemis  ( COMPARE_base_rest )
PEA_P64x2_Ld366.f10_f10_mg37.I2000Clm50BgcCrop.derecho_intel.clm-crop   ( RUN)
PEM_P64x2_Ld366.f10_f10_mg37.I2000Clm50BgcCrop.derecho_intel.clm-crop   ( COMPARE_base_modpes )

and these are tests that PASS

ERI_P64x2_Ld366.f10_f10_mg37.I2000Clm50BgcCrop.derecho_intel.clm-crop
ERS_P64x2_Ld366.f10_f10_mg37.I2000Clm50BgcCrop.derecho_intel.clm-crop
PET_P64x2_Ld366.f10_f10_mg37.I2000Clm50BgcCrop.derecho_intel.clm-crop
REP_P64x2_Ld366.f10_f10_mg37.I2000Clm50BgcCrop.derecho_intel.clm-crop

So it's not just exact restart that fails, or a fail when processors change, it's exact restart with a processor change. But, the PEM test fails while PET passes and those are with a change in processors. It's odd that PET passes (only changes OMP threads) while the PEM test fails (only changes MPI tasks), since it's only seen in threaded tests. I would think that would be switched.

Activity

  1. ekluzek
    added this to the ctsm6.0.0 (code freeze) milestone on Aug 6, 2026
  2. ekluzek
    added the
    Bug
    issue type on Aug 6, 2026
  3. ekluzek
    added
    code healthimproving internal code structure to make easier to maintain (sustainability)
    bugsomething is working incorrectly
    on Aug 6, 2026

  4. ekluzek commented on Aug 9, 2026

    @ekluzek
    ekluzek
    on Aug 9, 2026
    ContributorAuthor

    Until this is resolved we should add tests for the longest shorter period that works, to make sure other problems don't come up. From testing I see that the shortest period is 364 days. Once it hits the 365 mark there's a change in answers.

  5. ekluzek
    added a sub-issue on Aug 9, 2026
  6. ekluzek
    added a commit that references this issue on Aug 10, 2026

    Add shorter tests that pass until ESCOMP#4160 is handled

    690d727
  7. ekluzek
    changed the title [-]Threaded exact restart tests that change task/thread count fail with DGLC%NOEVOLVE fail when run over a year[/-] [+]Threaded exact restart tests that change task/thread count fail with DGLC%NOEVOLVE fail when run to a year or more[/+] on Aug 10, 2026
  8. ekluzek
    changed the title [-]Threaded exact restart tests that change task/thread count fail with DGLC%NOEVOLVE fail when run to a year or more[/-] [+]Threaded exact restart tests that change task/thread count fail with DGC or CISM fail when run to a year or more[/+] on Aug 10, 2026

  9. ekluzek commented on Aug 10, 2026

    @ekluzek
    ekluzek
    on Aug 10, 2026
    ContributorAuthor

    This problem occurs for both DGLC and CISM (From running ERP_P64x2_Ld365.f10_f10_mg37.I1850Clm50BgcCropG.derecho_intel.clm-monthly that fails similarly). For CISM two additional fields fail, and the difference is large for glc1Imp_Fgrd_rofi on the CPL history files. And there are differences in the CISM history files.

    cpl.hist:

     RMS glc1Imp_Fgrg_rofi                2.8454E-09            NORMALIZED  1.8555E-03
 RMS glc1Imp_Fgrg_rofl                9.4595E-25            NORMALIZED  9.1221E-14
 RMS glc1Imp_Sg_topo                  3.0714E-14            NORMALIZED  3.8788E-17
 RMS glc1Exp_Flgl_qice                1.4286E-20            NORMALIZED  1.2046E-15

    cism.gris.h:

    grep RMS /glade/derecho/scratch/erik/cases/ERP_P64x2_Ld365.f10_f10_mg37.I1850Clm50BgcCropG.derecho_intel.clm-monthly.GC.ctsm5451delist/run/ERP_P64x2_Ld365.f10_f10_mg37.I1850Clm50BgcCropG.derecho_intel.clm-monthly.GC.ctsm5451delist.cism.gris.h.0002-01-01-00000.nc.base.cprnc.out
 RMS acab_applied                     1.4666E-23            NORMALIZED  1.5132E-15
 RMS beta_internal                    3.4559E-02            NORMALIZED  6.1832E-16
 RMS bmlt                             3.0218E-27            NORMALIZED  2.4181E-13
 RMS bmlt_applied                     2.9614E-27            NORMALIZED  2.3699E-13
 RMS bpmp                             2.3830E-17            NORMALIZED  4.4896E-17
 RMS bwat                             2.6119E-20            NORMALIZED  8.4187E-14
 RMS calving_rate                     8.2596E-15            NORMALIZED  2.5834E-13
 RMS smb                              4.1938E-13            NORMALIZED  1.4883E-15
 RMS tempstag                         2.6461E-15            NORMALIZED  9.0780E-16
 RMS thk                              3.0386E-14            NORMALIZED  5.0167E-17
 RMS total_bmb_flux                   3.5527E-14            NORMALIZED  7.6889E-16
 RMS total_calving_flux               2.7940E-09            NORMALIZED  6.1879E-16
 RMS total_smb_flux                   3.3528E-08            NORMALIZED  1.1699E-15
 RMS usurf                            3.0714E-14            NORMALIZED  3.8788E-17
 RMS uvel                             4.9838E-22            NORMALIZED  4.1782E-15
 RMS vvel                             4.2730E-22            NORMALIZED  6.2141E-15

  10. ekluzek commented on Aug 10, 2026

    @ekluzek
    ekluzek
    on Aug 10, 2026
    ContributorAuthor

    Since the problem shows up for CPL fields on the CISM grid (which CLM doesn't know abiout I think this means the problem is in CMEPS. But, it's odd that it shows up for threading tests only as CMEPS doesn't do threading

    Pinging: @billsacks @Katetc

  11. ekluzek commented on Aug 10, 2026

    @ekluzek
    ekluzek
    on Aug 10, 2026
    ContributorAuthor

    I thought the problem might be because of GLC_TWO_WAY_COUPLING==TRUE, but I reran a test case doing that and the problem didn't go away (but I'm not sure how safe it is to rerun a ERP threaded case).

  12. billsacks commented on Aug 10, 2026

    @billsacks
    billsacks
    on Aug 10, 2026
    Member

    It's puzzling that you only see this in threaded tests, but the fact that you see this in PEM and not PET tests makes me think that the fundamental issue is related to changing processor count. And that makes me wonder if you're seeing the same issue that @Katetc found, reported here: ESCOMP/CMEPS#650

    You could check whether you're hitting the same issue by turning off the smb renormalization: set glc_renormalize_smb = 'off' in user_nl_cpl.

  13. ekluzek commented on Aug 10, 2026

    @ekluzek
    ekluzek
    on Aug 10, 2026
    ContributorAuthor

    @billsacks you are right -- it is that issue!

    It's puzzling that you only see this in threaded tests, but the fact that you see this in PEM and not PET tests makes me think that the fundamental issue is related to changing processor count. And that makes me wonder if you're seeing the same issue that @Katetc found, reported here: ESCOMP/CMEPS#650

    You could check whether you're hitting the same issue by turning off the smb renormalization: set glc_renormalize_smb = 'off' in user_nl_cpl.

    Yes, the test passes correctly when I do that.

    So what does this mean? @KateC and @billsacks? Should we run with this setting off? Does this make ESCOMP/CMEPS#650 a higher priority to solve? Or is this restricted enough that it's still a low priority to fix?

  14. ekluzek commented on Aug 10, 2026

    @ekluzek
    ekluzek
    on Aug 10, 2026
    ContributorAuthor

    Also @billsacks is 650 the right number for this issue? That CMEPS issue seems to be slightly different maybe? It doesn't reference the glc_renormalize_smb namelist and is about reproducable sums. Although I suppose maybe that's the cause for the difference there are sums being done for glc_renormalize_smb that aren't reproducable on a change in number of tasks?

  15. billsacks commented on Aug 10, 2026

    @billsacks
    billsacks
    on Aug 10, 2026
    Member

    Yes, ESCOMP/CMEPS#650 is the right issue number. It didn't directly reference glc_renormalize_smb but I just edited it to reference that variable directly for future searchability. The problem is that glc_renormalize_smb does a global sum, and it's done in the normal, simple way that leads to the sum differing for different processor counts. As that issue notes, I'm actually surprised that this doesn't come up in the runoff coupling, since there is also a global sum in handling negative runoff, but maybe something in the details of that implementation mean that roundoff-level changes typically don't matter there.

    I do feel like this makes ESCOMP/CMEPS#650 higher priority. I'll see if I can resolve at least the glc_renormalize_smb portion of this quickly....

  16. billsacks commented on Aug 10, 2026

    @billsacks
    billsacks
    on Aug 10, 2026
    Member

    I'll see if I can resolve at least the glc_renormalize_smb portion of this quickly....

    This is turning out to be more involved than I had imagined it would be. (Why do things like that continue to surprise me?) But with the help of Claude, I think I'm making progress....

  17. billsacks commented on Aug 11, 2026

    @billsacks
    billsacks
    on Aug 11, 2026
    Member

    I realized that, in addition to the CMEPS changes, I think we also need similar changes in DGLC for the allreduce there... I'll work on that, too.

  18. billsacks commented on Sep 17, 2026

    @billsacks
    billsacks
    on Sep 17, 2026
    Member

    PRs to get bit-for-bit answers with changing processor count when using DGLC:

  19. billsacks commented on Sep 18, 2026

    @billsacks
    billsacks
    on Sep 18, 2026
    Member

    PRs to get bit-for-bit answers with changing processor count when using DGLC:

    Those PRs have been merged now.

  20. ekluzek commented on Sep 18, 2026

    @ekluzek
    ekluzek
    on Sep 18, 2026
    ContributorAuthor

    Thanks @billsacks I've put in a placeholder to bring this into CTSM. I'm looking forward to this fixing some of our expected fails.

  21. billsacks
    changed the title [-]Threaded exact restart tests that change task/thread count fail with DGC or CISM fail when run to a year or more[/-] [+]Threaded exact restart tests that change task/thread count fail with DGLC or CISM fail when run to a year or more[/+] on Sep 18, 2026
  22. ekluzek
    added a commit that references this issue on Sep 23, 2026

    Remove the 364 day tests that were there until ESCOMP#4160 was resolved

    06b45eb
  23. ekluzek
    changed the title [-]Threaded exact restart tests that change task/thread count fail with DGLC or CISM fail when run to a year or more[/-] [+]Threaded exact restart tests that change task/thread count fail with DGLC or CISM when run to a year or more[/+] on Sep 24, 2026
  24. ekluzek
    closed this as completedin #4224on Sep 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugsomething is working incorrectlycode healthimproving internal code structure to make easier to maintain (sustainability)

    Type

    Bug

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions