DEBUG: instrument SPBase.allreduce_or to localize LOR_bug

[DLWoodruff]

Do not merge. Diagnostic patch for a spurious-shutdown bug under investigation. Revert before merging anything else from this branch.

How to use

Run your case with this branch and capture stdout to a file, then summarize with the bundled helper:

python lor_bug_report.py <stdout_file>

The script parses every [LOR_bug ...] block emitted by the instrumented SPBase.allreduce_or and prints:

• a per-comm summary (one row per (cls, mpicomm name)),
• a section per hypothesis (H1–H4) with counts and a few example offending calls,
• a one-line verdict naming which hypotheses (if any) were triggered.

A clean run prints Clean log: no anomalies on any of the four hypotheses. (or notes that nonzero results were observed but consistent with legitimate shutdown signals). Any other verdict is the lead to chase.

What this does

Replaces SPBase.allreduce_or with an instrumented variant that, on every call, does four collectives instead of one and dumps a one-block diagnostic to stdout from cyl_rk == 0 of the cylinder's mpicomm.

Originally the function was:

def allreduce_or(self, val):
    local_val = np.array([val], dtype='int8')
    global_val = np.zeros(1, dtype='int8')
    self.mpicomm.Allreduce(local_val, global_val, op=MPI.LOR)
    return bool(global_val[0])

Now each call also does:

1. Allgather of (world_rk, cyl_rk, local_int) — shows exactly which world ranks participate and what each one packed.
2. SUM, MAX, LOR reductions in parallel — MAX distinguishes "many small" from "few large"; LOR confirms call-site behavior.
3. Rank-sum sanity — each rank contributes its mpicomm rank; expected sum is n*(n-1)/2. Mismatch flags a corrupt SUM reducer.
4. Cross-check: gather_sum (sum of locally-reported values from the Allgather) vs reduce_sum (the Allreduce SUM). Divergence isolates the bug to the reducer path.

Hypotheses being tested (in priority order)

1. self.mpicomm has wider membership than the cylinder it should — i.e., includes ranks from another cylinder. Detected if mpicomm size exceeds the cylinder's rank count, or if world_ranks includes ranks outside the cylinder's slice.
2. Buffer memory underneath local_val is not 0 when MPI reads it. Detected if gather_sum > 0 (some rank reports nonzero) AND max > 0 (the rank's value was nonzero from MPI's perspective).
3. The Allreduce reducer path is malfunctioning. Detected if gather_sum == 0 but reduce_sum != 0, OR if rank_sum != expected_rank_sum.
4. Duplicate rank participation in self.mpicomm. Detected if world_ranks: unique < count.

Reading the output

Operator-friendly greps:

# 1. Did anything print at all?
grep '^\[LOR_bug' out.log | head

# 2. What does each cylinder think its comm size is?
#    (one printer per cylinder; size should be that cylinder's rank count)
#    If you see size=150 here, hypothesis (i) is the bug.
grep 'mpicomm size=' out.log | sort -u

# 3. World-rank membership of each cylinder.
#    count==unique is required; unique<count means duplicate participation (iv).
#    The range [min..max] tells you which slice of WORLD is in this comm.
grep 'world_ranks:' out.log

# 4. Sanity check that SUM works at all on this comm.
#    rank_sum should equal expected_rank_sum (n*(n-1)/2). If not, the
#    SUM reducer itself is broken on this comm - hypothesis (iii).
grep 'rank_sum=' out.log | head

# 5. The main reduce result vs. the gather-truth.
#    sum == gather_sum is required. Divergence pins the bug to the
#    reducer path (gather honest, reduce wrong).
#    max>1 means at least one rank contributed >1 (not boolean) -
#    hypothesis (ii) on that rank specifically.
grep -E 'reductions:|gather:' out.log

# 6. Which ranks actually contributed nonzero (only printed on anomaly).
#    Tells you world_rk + cyl_rk of every guilty contributor.
grep 'nonzero: world_rk' out.log

# 7. Full participant list (only printed on anomaly).
#    Use this if comm membership is suspicious.
grep 'ALL world_ranks:' out.log

Cost / overhead

One Allreduce call now does 4 reductions + 1 Allgather. In the target environment the run-launch overhead dominates the per-call cost, so this is acceptable. Output is only printed on cyl_rk == 0 (one line block per cylinder per call), so log volume is bounded.

Control run (MPISPPY_LOR_CONTROL)

SPBase.allreduce_or has an env toggle so this single branch produces both
the instrumented and the control data point (no re-push needed):

export MPISPPY_LOR_CONTROL=1   # in the sbatch, alongside PYTHONMALLOC=debug

• unset (default): the instrumented path above (4 reductions + 1 Allgather per call).
• set: the original minimal path only — a single Allreduce(LOR), no extra
  collectives, no per-call numpy allocations. Canary guards and the buffer/window
  teardown fixes are unaffected either way.

Run the same case twice (toggle on vs. off) and compare:

• control survives Iter 0 → the corruption tracks the instrumentation's extra
  collective volume; real but volume-dependent. Next: shrink and run ASan/valgrind.
• control also aborts at Iter 0 (in the real RMA path, not the diagnostic
  Allgather) → the bug is independent of the diagnostic; go straight at the RMA
  window / teardown code.

Reproduction notes

• PYTHONMALLOC=debug (no valgrind) reproduces the failure deterministically at
  Iter 0 at 150 ranks. The instrumented run aborts inside got_kill_signal -> allreduce_or -> Allgather with MVAPICH2 MPIC_Sendrecv: Negative count from
  MPIR_Allgather_RD_MV2 — MPI's internal count arithmetic went negative, i.e.
  its internal heap was already corrupted. The per-iteration field-buffer canary
  did not fire, so the OOB write lands in MPI-internal allocations, outside
  the guarded field buffers.
• valgrind at 150 ranks does not work: every rank aborts during comm setup
  (_make_comms) with libpsm2 psm_ep_connect ... Operation timed out, because
  valgrind's slowdown blows past PSM connection timeouts. Use PYTHONMALLOC=debug,
  or shrink to the smallest repro before adding valgrind/ASan (and set
  PYTHONMALLOC=malloc under valgrind).

Followups

• Once the hypothesis is pinned, fix in a separate PR.
• Revert this file change before any non-diagnostic merge to main.

🤖 Generated with Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 71.25000% with 46 lines in your changes missing coverage. Please review.
✅ Project coverage is 72.78%. Comparing base (ade8219) to head (a8de453).

Files with missing lines	Patch %	Lines
mpisppy/debug_utils/heap_probe.py	36.36%	21 Missing ⚠️
mpisppy/spbase.py	78.57%	12 Missing ⚠️
mpisppy/cylinders/spcommunicator.py	76.59%	11 Missing ⚠️
mpisppy/__init__.py	80.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #717      +/-   ##
==========================================
- Coverage   72.82%   72.78%   -0.04%     
==========================================
  Files         164      165       +1     
  Lines       20959    21108     +149     
==========================================
+ Hits        15263    15364     +101     
- Misses       5696     5744      +48     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.