Repository navigation
Conversation
Promotes the cycles 1-2 paper-watch finding to a standing regression test: ParticleGAN reaches every mode on the mixture toys while its Naeem coverage sits far below a plain control. The test asserts mode reach AND a calibrated coverage floor (0.14) at the default particle count, records the plain-control gap without asserting parity, and pins the count-scaling law (16,384 particles reach plain parity; per-particle concentration is fixed, count is the lever).
Snapshot seed 0 of each arm every 200 updates through the separate evaluation stream (GANTrainer.sample never consumes a training RNG), and render target-vs-observed frames labelled with modes, coverage and gate status; the last frame is the gated endpoint. Mark the module tier1 for selection. Calibrated floor (0.14) and all bands unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mikkel
force-pushed
the
paper-watch/t1-coverage-floor
branch
from
November 20, 2026 04:10
2e63e04 to
eabee6f
Compare
…oor test Identical arithmetic for the gated metric; removes the repeated 4096x4096 real-real distance pass from each GIF snapshot. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tted) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
A standing tier-1 regression test promoted from the ParticleGAN paper watch (cycles 1–2), per Mikkel's call (2026-11-18): the coverage gap "seems like it should be a t1 test."
The finding. On low-dimensional mixture toys, ParticleGAN's learned-prior system reaches every mode — mode counts tie a plain GAN control at the ceiling — while covering far fewer of the real distribution's neighborhoods:
Mode count alone certifies these runs as healthy. This test guards the fill, not the reach: it asserts mode reach AND a calibrated coverage floor, records the plain-control gap, and pins the mechanism that explains the gap (below).
The test —
tests/test_tier1_coverage_floor.pyOne new file, CPU-pinned, fixed seeds, public API only (
GANTrainer/get_recipe/init, as in PRs #559–#564). Five tests:test_naeem_coverage_metric_sanity— the metric itself (identity ≈ 1, one-mode collapse < 0.3, disjoint = 0).test_particle_reaches_every_mode— 8/8 modes, every seed.test_particle_coverage_stays_above_floor— per-seed Naeem coverage ≥ 0.14.test_particle_coverage_scales_to_plain_parity_with_count— the scaling assertion (below).test_plain_control_gap_is_measured_and_recorded— plain control trained identically; the default-count gap is printed (T1-COVERAGE-GAP ...), never asserted. Parity at the default count is the aspiration, not today's truth.Toy choice (measured, not convenience)
The 25-Gaussian grid from PR #563 was probed on pop-os CPU: one particle run at 1500/6000 steps (batch 256) took 110 s (~0.073 s/step; regime already present: 25/25 modes, coverage 0.208), so the shipped budget costs ~7.5 min/run and ~45 min for a six-run fixture — outside any tier-1 budget. The 8-Gaussian ring (PR #560 configuration: radius 2.0, σ 0.15, batch 128, 2400 steps) reproduces the same regime at ~1/6 the cost.
Calibration (pop-os CPU, this exact protocol, master @ 52c26c1)
Default count (256 particles):
Particle: mean 0.201, min 0.181, sd 0.018 (matches PR #560's independent 0.201 ± 0.014). Floor = 0.14 = observed minimum − 0.04 safety margin (~2.3 seed-sigmas; also absorbs CPU thread-count numeric drift). The floor sits far above the failure values it exists to catch (0.04–0.08, PR #564; 0.048, collapsed BOLT arm).
The scaling assertion — count is the lever
A particle-count sweep (grid, BOLT harness, 6000 steps, seeds 0/1/2, eval 4096 vs 4096, k=5) found: coverage 256 → 0.128 ± 0.020, 1,024 → 0.208, 4,096 → 0.347, 16,384 → 0.510 ± 0.049 — full parity with the plain control (0.507 ± 0.058) at 64×, wall time flat (~3 min/run). Within-mode dispersion never moves (0.61–0.70 at all counts vs plain 1.32): per-particle concentration is fixed; the gap closes by cluster count, not by within-mode spread.
It composes cleanly with this test, so the assertion is included. Ring confirmation at this test's protocol (pop-os CPU):
Mean 0.735, worst seed 0.694 — above the plain control's mean (0.638) in every seed, at the same ~45 s/run as 256 particles. The assertion uses bands well inside these margins (per-seed ≥ plain mean − 0.10; mean ≥ plain mean − 0.05; lever ≥ +0.35 over the default-count mean, measured +0.53). The docstring states the consequence: the floor is a statement about the default configuration, not a ceiling on the formulation.
Verification
Full file verified end-to-end on pop-os from a clean copy of this branch (byte-identical md5): 5 passed in 336 s (5m36s; wall 5m42s) for the 9-run fixture — inside the tier-1 budget. The fixture reproduced the calibration means exactly:
T1-COVERAGE-SCALING default=0.201 16k=0.735 plain=0.638andT1-COVERAGE-GAP particle=0.201 plain=0.638 gap=+0.437.Notes
naeem_coverageis implemented inside the test file: master'slib/toy_metrics.pydoes not carry it yet (it exists on PR Paper watch: EmbeddGAN salvage — Gini distance correlation as an evaluation metric #559's unmerged branch). When Paper watch: EmbeddGAN salvage — Gini distance correlation as an evaluation metric #559 lands, this file should switch to the lib version and delete the local copy — the docstring says so.test_actual_training_gif, see update below).Update (2026-11-20): rebased onto develop; GIF + tier1 marker
develop. The PR's single commit was rebased ontoorigin/develop(git rebase --onto origin/develop <merge-base with master>).GANTrainer.sample(generator=...)never consumes a training RNG, so trajectories and the calibration are unchanged).test_actual_training_gifrenders target ring vs observed API samples, labelled with update, modes, live coverage and floor/gate status; the final frame is the gated endpoint sample (asserted). Written to$T1_COVERAGE_GIFif set, else pytest's tmp dir.tier1marker (pytestmark = pytest.mark.tier1) for selection (-m 'not tier1'skips the ~3–5 min CI fixture). It is not deselected by default: the CI shard receipt check requires every collected test to run.COVERAGE_FLOOR = 0.14, all parity/lever bands, seeds, protocol.🤖 Generated with Claude Code