Make AppLauncher teardown truthful and re-entrancy-safe; document mGPU NCCL workaround

[hujc7]

Summary

• Consolidates AppLauncher process-lifecycle handling into one nested _SimulationAppLifecycle class: startup announcements (CI marker, Kit version diagnostics) plus the entire exit-path policy, with the policy table as the class docstring.
• Fixes every exit path that misreported failure as success or destroyed its own diagnostics (full failure-case table below). Absorbs the atexit exit-code fix from Fix AppLauncher exception exit status #6634.
• Documents the NCCL cuMem workaround for multi-GPU RTX training on NUMA-spanning GPU allocations (NCCL_CUMEM_HOST_ENABLE=0 first, NCCL_CUMEM_ENABLE=0 as fallback).
• The two isaacsim-dependent pieces are marked WORKAROUND(isaac-sim) and fail loudly if upstream drift disables them; both become deletable once Preserve nonzero exit status through fast-shutdown close IsaacSim#717 lands.

Failure cases and how this PR addresses them

Root mechanism: Kit fast shutdown terminates the process with exit code 0 from inside SimulationApp.close(), so any death funneled through an unqualified close() is reported as success; additionally, the abort-signal handler was unguarded against re-entrancy.

How the process ends	Before this PR	After this PR
Unhandled Python exception	atexit close overwrote the pending failure with exit 0 (CI false-green)	exits 1 (sys.last_exc detected; absorbed from #6634; SystemExit documented as not yet covered)
Single SIGTERM (torchrun teardown, SLURM preemption, kill)	graceful close → exit 0; launcher marks the killed rank SUCCEEDED; surviving ranks hang until the NCCL watchdog	full teardown, then dies by the signal (128+signum)
Second signal while a close is running (repeated SIGTERM; fault inside the replicator stop/wait)	handler re-entered close() → infinite recursion → SIGKILL-only shutdown, spurious SIGSEGV, logs flooded (~975 recursion frames/job observed on OSMO pods)	guard: re-entrant signal falls back to SIG_DFL
Signal racing the normal atexit close	nested full second teardown of a half-closed app	atexit arms the same guard
kill -ABRT	graceful close → exit 0	same truthful-close path as SIGTERM
Real SIGSEGV, main thread	Python handler can never run → process spins forever at 100% CPU, crash reporter clobbered (no minidump)	SIGSEGV no longer intercepted → default action, minidumps restored
Real SIGSEGV, worker thread	handler ran on the main thread → exit 0 for a crashed process	same: default action, truthful signal death
Ctrl-C	SimulationApp's handler exits 0 before user finally/KeyboardInterrupt code runs	Python default handler restored: KeyboardInterrupt unwinds user code, nonzero exit

Not addressed here (tracked elsewhere): sys.exit(N)/SystemExit still exits 0 (gap inside the #6634 mechanism, documented at the detection site); the fast-shutdown disable on the signal path is an interim cost removed by isaac-sim/IsaacSim#717.

Implementation notes

1. All exit-path logic lives in AppLauncher._SimulationAppLifecycle; the class docstring is the policy table, and each decision carries its rationale in place.
2. The signal path disables Kit fast shutdown so close() returns, then re-raises with the default action (WORKAROUND(isaac-sim), removable once IsaacSim#717 lands). If the override stops taking effect, a [ISAACLAB] WARNING is printed instead of silently regressing.
3. Docs: distributed camera training fails deterministically when the allocated GPUs span NUMA nodes and passes on a single-switch set; disabling NCCL cuMem host allocations rescues the failing shape in paired same-node experiments. Added to the multi-GPU NCCL troubleshooting section.

Testing. Five kitless unit tests (test_simulation_app_lifecycle.py: re-entrancy both directions, killed-by-signal contract, exit-code selection, drift fallback) plus two real-Kit integration tests (test_app_launcher_exit_status.py: SIGTERM → dies by SIGTERM with no recursion; unhandled exception → exits 1). All fail against the previous behavior — the SIGTERM test observes exit code 0 on develop — and pass with the fix. The integration tests also pass unchanged against an IsaacSim build patched with IsaacSim#717, confirming no lab change is needed when it lands.

Type of change

• Bug fix (non-breaking change which fixes an issue)
• Documentation update

Checklist

• I have read and understood the contribution guidelines
• I have run the pre-commit checks with ./isaaclab.sh --format
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• I have added a changelog fragment under source/<pkg>/changelog.d/ for every touched package (do not edit CHANGELOG.rst or bump extension.toml — CI handles that)
• I have added my name to the CONTRIBUTORS.md or my name already exists there

[greptile-apps]

Greptile Summary

This PR fixes a re-entrancy bug in AppLauncher._abort_signal_handle_callback where a repeated signal (e.g. a second SIGTERM from torchrun during distributed teardown) caused unbounded recursion into SimulationApp.close(), leading to stack overflows and masked failure signals. It also adds NCCL_CUMEM_ENABLE=0 to the multi-GPU troubleshooting docs for the NUMA-topology / RTX-rendering watchdog-timeout scenario.

• Signal handler fix: An _abort_in_progress flag is set before close() is called; any re-entrant signal restores SIG_DFL and re-raises, terminating the process with the original signal rather than overflowing the stack.
• Regression test: A kitless unit test using types.SimpleNamespace and monkeypatch verifies that close() is called exactly once and that the fallback path fires correctly for re-entrant signals.
• Docs update: New paragraph in multi_gpu.rst describes the NCCL watchdog-timeout symptom signature and the NCCL_CUMEM_ENABLE=0 workaround, placed appropriately within the existing NCCL troubleshooting section.

Confidence Score: 5/5

Safe to merge. The re-entrancy guard is minimal, correct, and does not change the happy-path shutdown sequence.

The flag is set before close() is called so any re-entrant signal immediately takes the fallback path; the rename of the signal parameter to signum is both a style improvement and a correctness prerequisite for the new signal.signal() and signal.raise_signal() calls inside the handler. The regression test exercises the re-entrant path end-to-end without needing a live Kit instance. The docs change is accurately scoped to the observed symptom and workaround. No existing behaviour is altered on the first signal delivery.

No files require special attention.

Important Files Changed

Filename	Overview
source/isaaclab/isaaclab/app/app_launcher.py	Adds _abort_in_progress flag to prevent re-entrant signal handler recursion; renames the shadowed signal parameter to signum (required for the new signal.signal() / signal.raise_signal() calls to resolve to the module, not the parameter).
source/isaaclab/test/app/test_signal_handlers.py	New kitless regression test: uses types.SimpleNamespace + monkeypatch to verify close() is called exactly once and that the re-entrant fallback (SIG_DFL + raise_signal) fires correctly.
docs/source/features/multi_gpu.rst	Adds a new paragraph and NCCL_CUMEM_ENABLE=0 workaround for NCCL watchdog-timeout failures on NUMA-spanning multi-GPU hosts with RTX rendering enabled; well-placed within the existing troubleshooting section.
source/isaaclab/changelog.d/fix-abort-handler-reentrancy.rst	Changelog fragment accurately describes the re-entrancy bug and the fix; follows the existing format.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant OS as OS / torchrun
    participant SH as _abort_signal_handle_callback
    participant Flag as _abort_in_progress
    participant App as SimulationApp.close()

    OS->>SH: SIGTERM (first delivery)
    SH->>Flag: check False
    SH->>Flag: set True
    SH->>App: close() [starts long teardown]
    Note over App: e.g. replicator orchestrator.stop() blocks...

    OS-->>SH: SIGTERM (re-entrant, during close)
    SH->>Flag: check True
    SH->>OS: signal.signal(SIGTERM, SIG_DFL)
    SH->>OS: signal.raise_signal(SIGTERM)
    Note over OS: process terminates with original signal

    App-->>SH: close() returns (if re-entrant signal did not kill yet)
    SH-->>OS: handler returns

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant OS as OS / torchrun
    participant SH as _abort_signal_handle_callback
    participant Flag as _abort_in_progress
    participant App as SimulationApp.close()

    OS->>SH: SIGTERM (first delivery)
    SH->>Flag: check False
    SH->>Flag: set True
    SH->>App: close() [starts long teardown]
    Note over App: e.g. replicator orchestrator.stop() blocks...

    OS-->>SH: SIGTERM (re-entrant, during close)
    SH->>Flag: check True
    SH->>OS: signal.signal(SIGTERM, SIG_DFL)
    SH->>OS: signal.raise_signal(SIGTERM)
    Note over OS: process terminates with original signal

    App-->>SH: close() returns (if re-entrant signal did not kill yet)
    SH-->>OS: handler returns

Loading

_{Reviews (1): Last reviewed commit: "Document NCCL_CUMEM_ENABLE=0 workaround ..." | Re-trigger Greptile}