DPointNet: add opt-in multi-GPU sample-parallel training

[shixnya]

Summary

Add explicitly opt-in, single-host sample-parallel DPointNet training:

• MPI (recommended starting point): one TensorFlow process per GPU, using BMTK's shared communicator and mpirun -n N python -m bmtk.utils.dpointnet_mpi .... The launcher binds inherited GPU allocation tokens before TensorFlow imports and aborts the job on uncaught worker failure.
• MirroredStrategy (alternative): one process with explicitly selected devices.
• Each replica holds the complete network and receives disjoint samples. run.batch_size remains per replica, optional global size must match, and one global batch produces one optimizer update. This is not neuron partitioning or pooled GPU memory.
• No runtime-default change: omitting run.distribution preserves previous execution and batch behavior. Neither new sample-parallel mode is automatically selected.

Implementation

• Split globally generated host inputs and targets together, retaining existing generator seeds and Poisson counts. Preserve rank-specific replay-safe internal Poisson streams.
• Reduce canonical dense FP32 gradients from every trainable surface with one packed host-MPI mean per update, outside TensorFlow graphs; scalar metrics are reduced separately. Reuse staging buffers with completed-upload boundaries so later collectives cannot mutate earlier GPU snapshots.
• Synchronize chief model variables, all recurrent/named-input masters (including frozen inputs), optimizer slots and noise state; refresh compute shadows. Explicit complete checkpoints and collective synchronization remain required after chief restore.
• Make normal Config.build_env()/directory/log/resolved-config handling rank-aware and chief-owned. Reject incorrect launcher/config pairs before output writes; use noncollective fatal logging so worker failures reach the launcher.
• Support replica-local native CSR metadata and immutable rollout captures. Remove repeated peer/host transfers from temporal/checkpoint execution; widen only compact-integer checkpoint storage losslessly, retaining public state dtypes and exact BPTT semantics. Native C++/CUDA sources are unchanged.
• Add the optional dpointnet-mpi dependency extra, a complete SONATA JSON/runner example, real-process CPU CI for Keras 2/3, and a dedicated multi-GPU manual linked from the guide, training and performance pages. Preserve existing documentation anchors.

Supported scope and explicit limits

Initial admitted scope: single host, full FP32, BPTT, one condition, equal full local batches, zero-state initialization, no optimizer clipping, and independently qualified sample-separable mean losses. TargetFiringRate retains its default Hz objective; spikes_per_step is an explicit alternative.

Unqualified mixed precision, condition accumulation/parallelism, local learning rules, cached/warmup initial states and nonlinear batch-statistic losses fail explicitly. MPI combined training/inference is rejected to prevent duplicate output ownership. This backend uses host MPI, not CUDA-aware MPI or NCCL. Multi-node execution, all-GPU qualification and convergence are not claimed.

Validation

The publication tree is based directly on current develop (8381b54c439df0745417f62e96a92f4e69ceafac). All 663 tracked Python files match the immutable CPU-qualified source; the later manual revisions change documentation only.

Validation	Result
Complete CUDA-disabled Keras 3 suite, TF 2.21	1702 passed / 982 skipped
Complete actual Python 3.8 / TF 2.13 / Keras 2 suite	1693 passed / 991 skipped
Fresh focused CPU/MPI tests, including real configuration/error subprocesses and serial/MPI JSON examples	108 passed / 0 skipped
Fresh independent two-CPU-rank oracle	All 8 terminal states, 2 canonical gradient surfaces, 3 Adam updates, constraints, destructive complete restore, input/callback ownership and rank-specific Poisson replay passed; worst relative L2 1.163e-7
Python 3.8 publication compilation	Passed
Full native Sphinx site, notebook execution disabled	Passed; 12 unchanged baseline diagnostics, no new diagnostics
Exact-base patch/source hashes and whitespace	Passed; 29 changed files

The extra nine local Keras 2 skips are explicit real-process cases because the existing immutable environment lacks mpi4py; it was not modified. The new OpenMPI CPU workflow installs the optional dependency and must supply real Keras 2 MPI coverage on this PR. GitHub Actions results are pending, not claimed green.

Historical physical-GPU evidence

Frozen pre-I/O-integration qualification on two V10032 GPUs used full V1 (66,658 neurons / 24,450,554 recurrent edges), 500 timesteps, FP32/NEST, checkpoint interval 25, recurrent-only Adam, seeded host LGN and count-valued Poisson background, with full spike outputs/diagnostic writes. Twenty synchronized samples followed three excluded warmups; MPI timings use rank-MAX walltime.

Mode	Single-GPU local B16 control	Two-GPU local16/global32	Throughput scaling
MPI	3.686884 s	3.919032 s	1.881528x
Optimized MirroredStrategy	3.696960 s	4.765848 s	1.551438x

These are separate qualification runs with matched single-GPU controls, not a simultaneous head-to-head benchmark or proof that MPI transport is intrinsically superior. Physical-GPU numerical/state/update/restore gates passed on those frozen parent snapshots; no new physical-GPU benchmark is claimed for subsequent config-I/O/documentation revisions. The earlier GTX1080Ti 1.953x prototype used a different counts-only workload and is not a matched comparison.