Skip to content

Tier-1 regression: ParticleGAN mode-fill coverage floor (paper watch) - #570

Open
mikkel wants to merge 4 commits into
developfrom
paper-watch/t1-coverage-floor
Open

mikkel wants to merge 4 commits into
developfrom
paper-watch/t1-coverage-floor

Conversation

@mikkel

@mikkel mikkel commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

What this is

A standing tier-1 regression test promoted from the ParticleGAN paper watch (cycles 1–2), per Mikkel's call (2026-11-18): the coverage gap "seems like it should be a t1 test."

The finding. On low-dimensional mixture toys, ParticleGAN's learned-prior system reaches every mode — mode counts tie a plain GAN control at the ceiling — while covering far fewer of the real distribution's neighborhoods:

Mode count alone certifies these runs as healthy. This test guards the fill, not the reach: it asserts mode reach AND a calibrated coverage floor, records the plain-control gap, and pins the mechanism that explains the gap (below).

The test — tests/test_tier1_coverage_floor.py

One new file, CPU-pinned, fixed seeds, public API only (GANTrainer / get_recipe / init, as in PRs #559–#564). Five tests:

  1. test_naeem_coverage_metric_sanity — the metric itself (identity ≈ 1, one-mode collapse < 0.3, disjoint = 0).
  2. test_particle_reaches_every_mode — 8/8 modes, every seed.
  3. test_particle_coverage_stays_above_floor — per-seed Naeem coverage ≥ 0.14.
  4. test_particle_coverage_scales_to_plain_parity_with_count — the scaling assertion (below).
  5. test_plain_control_gap_is_measured_and_recorded — plain control trained identically; the default-count gap is printed (T1-COVERAGE-GAP ...), never asserted. Parity at the default count is the aspiration, not today's truth.

Toy choice (measured, not convenience)

The 25-Gaussian grid from PR #563 was probed on pop-os CPU: one particle run at 1500/6000 steps (batch 256) took 110 s (~0.073 s/step; regime already present: 25/25 modes, coverage 0.208), so the shipped budget costs ~7.5 min/run and ~45 min for a six-run fixture — outside any tier-1 budget. The 8-Gaussian ring (PR #560 configuration: radius 2.0, σ 0.15, batch 128, 2400 steps) reproduces the same regime at ~1/6 the cost.

Calibration (pop-os CPU, this exact protocol, master @ 52c26c1)

Default count (256 particles):

arm seed modes min-mode count coverage
particle 0 8/8 451 0.1807
particle 1 8/8 459 0.2129
particle 2 8/8 433 0.2095
plain 0 8/8 410 0.6667
plain 1 8/8 424 0.6646
plain 2 8/8 409 0.5815

Particle: mean 0.201, min 0.181, sd 0.018 (matches PR #560's independent 0.201 ± 0.014). Floor = 0.14 = observed minimum − 0.04 safety margin (~2.3 seed-sigmas; also absorbs CPU thread-count numeric drift). The floor sits far above the failure values it exists to catch (0.04–0.08, PR #564; 0.048, collapsed BOLT arm).

The scaling assertion — count is the lever

A particle-count sweep (grid, BOLT harness, 6000 steps, seeds 0/1/2, eval 4096 vs 4096, k=5) found: coverage 256 → 0.128 ± 0.020, 1,024 → 0.208, 4,096 → 0.347, 16,384 → 0.510 ± 0.049 — full parity with the plain control (0.507 ± 0.058) at 64×, wall time flat (~3 min/run). Within-mode dispersion never moves (0.61–0.70 at all counts vs plain 1.32): per-particle concentration is fixed; the gap closes by cluster count, not by within-mode spread.

It composes cleanly with this test, so the assertion is included. Ring confirmation at this test's protocol (pop-os CPU):

arm seed modes coverage train secs
16,384 particles 0 8/8 0.7891 49
16,384 particles 1 8/8 0.7224 34
16,384 particles 2 8/8 0.6941 47

Mean 0.735, worst seed 0.694 — above the plain control's mean (0.638) in every seed, at the same ~45 s/run as 256 particles. The assertion uses bands well inside these margins (per-seed ≥ plain mean − 0.10; mean ≥ plain mean − 0.05; lever ≥ +0.35 over the default-count mean, measured +0.53). The docstring states the consequence: the floor is a statement about the default configuration, not a ceiling on the formulation.

Verification

Full file verified end-to-end on pop-os from a clean copy of this branch (byte-identical md5): 5 passed in 336 s (5m36s; wall 5m42s) for the 9-run fixture — inside the tier-1 budget. The fixture reproduced the calibration means exactly: T1-COVERAGE-SCALING default=0.201 16k=0.735 plain=0.638 and T1-COVERAGE-GAP particle=0.201 plain=0.638 gap=+0.437.

Notes

Update (2026-11-20): rebased onto develop; GIF + tier1 marker

  • Base is now develop. The PR's single commit was rebased onto origin/develop (git rebase --onto origin/develop <merge-base with master>).
  • Actual-training GIF. Seed 0 of each arm (256 particles, plain control, 16,384 particles) is snapshotted every 200 updates through a separate evaluation stream (GANTrainer.sample(generator=...) never consumes a training RNG, so trajectories and the calibration are unchanged). test_actual_training_gif renders target ring vs observed API samples, labelled with update, modes, live coverage and floor/gate status; the final frame is the gated endpoint sample (asserted). Written to $T1_COVERAGE_GIF if set, else pytest's tmp dir.
  • Marker. Module carries the repository's existing tier1 marker (pytestmark = pytest.mark.tier1) for selection (-m 'not tier1' skips the ~3–5 min CI fixture). It is not deselected by default: the CI shard receipt check requires every collected test to run.
  • Cost. The real-real kNN radius is computed once per run instead of per snapshot (identical arithmetic for the gated metric).
  • Unchanged: COVERAGE_FLOOR = 0.14, all parity/lever bands, seeds, protocol.

🤖 Generated with Claude Code

@mikkel
mikkel changed the base branch from master to develop November 20, 2026 03:34
mikkel and others added 2 commits November 20, 2026 03:36
Promotes the cycles 1-2 paper-watch finding to a standing regression
test: ParticleGAN reaches every mode on the mixture toys while its
Naeem coverage sits far below a plain control. The test asserts mode
reach AND a calibrated coverage floor (0.14) at the default particle
count, records the plain-control gap without asserting parity, and
pins the count-scaling law (16,384 particles reach plain parity;
per-particle concentration is fixed, count is the lever).
Snapshot seed 0 of each arm every 200 updates through the separate
evaluation stream (GANTrainer.sample never consumes a training RNG), and
render target-vs-observed frames labelled with modes, coverage and gate
status; the last frame is the gated endpoint. Mark the module tier1 for
selection. Calibrated floor (0.14) and all bands unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mikkel
mikkel force-pushed the paper-watch/t1-coverage-floor branch from 2e63e04 to eabee6f Compare November 20, 2026 04:10
mikkel and others added 2 commits November 20, 2026 05:52
…oor test

Identical arithmetic for the gated metric; removes the repeated
4096x4096 real-real distance pass from each GIF snapshot.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tted)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant