Skip to content

Paper watch: TARP-CvM coverage loss (arXiv:2611.15829) — diagnostic cohort, partial transfer on CPU - #569

Open
mikkel wants to merge 5 commits into
developfrom
paper-watch/tarp-cvm
Open

mikkel wants to merge 5 commits into
developfrom
paper-watch/tarp-cvm

Conversation

@mikkel

@mikkel mikkel commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Status update (rebased onto develop; base is develop)

Scope: diagnostic / assisted cohort — not qualification evidence, no longer "proven transfer".

  • The training loop is caller-owned (hinge loss + hand-built Adam on G/D/particle table) and the TARP-CvM term and its frozen b(y) live in the host. The ka2 recipe only supplies make_prior; its optimizer, penalty, noise and prior policy never run. TARP is not a declared recipe capability: GANTrainer is unconditional, and a faithful port needs a recipe-driven paired conditional host (make_optimizers/make_loss/make_critic_penalty, as the DDGAN conditional-energy host does) plus a declared recipe field and evidence identity. That is more than a modest change, so I left it out of this PR.
  • Device: the round 1–5 numbers below were measured on CUDA and are kept as a historical CUDA cohort. The test now pins DEVICE=cpu and runs the gauntlet with one intra-op thread, so CI and local runs follow the same deterministic law. The earlier CI failures came from CUDA-calibrated thresholds meeting CPU control arms that behave differently.
  • The CPU cohort is hardware-dependent. With the same code, seeds, single thread and torch build, GitHub runners reproduce two distinct CPU states. The control arms are chaotic, so last-bit matmul differences change which basin a run settles in. In state B (this workstation and some runners), coverage was particle 0.321 / plain 0.446 / tarp 0.818 and mode-mass error 0.250 / 0.245 / 0.071. In state A (other runners), the tarp-plain coverage margin was 0.157 and tarp lost mode-mass to plain in all 3 seeds. Verdict: PARTIAL. No threshold was loosened:
    • Holds in both states (asserted): TARP wins coverage in every seed against both controls (≥0.65 each, ≥0.25 margin over particle), keeps mode-mass ≤0.12 and dispersion in [0.90, 1.25] in every seed, and beats particle's held-out TARP statistic by ≥4×.
    • Fails in both states (xfail(strict=True) with measured numbers): tarp wins mode-mass in every seed vs particle; the particle control concentrates in every seed.
    • Flips with the runner (xfail(strict=False), both outcomes recorded): every claim against the plain control, namely the coverage margin, mode-mass wins and margins, dispersion and the held-out statistic. These claims are not established on CPU.
    • The fixture prints a one-line [tarp-cvm-cohort] JSON to stderr, so each runner's numbers can be read from CI logs.
  • GIF: seed-0 frames are recorded on a dedicated RNG stream. test_training_gif renders the target and the three arms at y = −0.8 / 0 / +0.8. The CPU-cohort GIF and compact metrics are in docs/paper-watch/tarp-cvm/ (kept out of reports/, which is a Forge history source root).

Paper

"Training Calibrated Conditional Generative Models with a Sample-Only Coverage Loss" — Remy, Rossellini, Chang, Willett (2026). arXiv:2611.15829 — https://arxiv.org/abs/2611.15829

Mechanism (TARP-CvM)

For a paired example (θ_i, y_i) and a reference rule ψ, the credibility value s_i is the fraction of M model draws θ̂ ~ q(·|y_i) lying closer to ψ(y_i) than the true θ_i does. If q is the true posterior, s ~ U(0,1). The loss is the Cramér–von Mises discrepancy (paper Eq. 5, plug-in form) between the empirical CDF of {s_i} and the uniform, computed separately per rule and averaged over rules — never pooled, so per-condition errors cannot cancel. Rules are built around a frozen point estimate b(y) (a posterior-mean MLP regressor trained first and frozen), with scale-matched random displacements ς = per-coordinate RMS residual (Eq. 9): switch rules (one displacement per side of a random hyperplane in y, Eq. 8), prior rules (one marginal point shared by the batch), and posterior rules (a detached own draw, excluded from its own credibility). Indicators are relaxed with a sigmoid at τ = 0.05 × median sample-to-reference distance. Total loss = base loss + λ · mean-rule CvM. Theorem 1: the averaged loss is a strict divergence — zero iff q equals the true posterior — and it needs only samples, which is exactly why it can attach to a GAN/particle generator at all. This test uses the paper's SBI configuration: λ=30, M=16, R=12 rules/step (6 switch / 3 prior / 3 posterior, scaled down from the paper's 48 for toy step cost).

Why here: conditional averaging is the conditional generator's signature failure — mass parked at the conditional mean, between the modes. An averaging generator's credibility values are non-uniform, so TARP-CvM punishes precisely that.

Harness note

The repo's GANTrainer is unconditional-only (it rejects any recipe whose conditioning is not "scalar"), so this test uses the caller-owned conditional host the trainer's own error message points to: the condition is concatenated into the generator and critic inputs, and the particle arms keep the repo's learned ParticlePrior table, sampled and trained exactly as in the unconditional harness. The plain arm swaps the table for the repo's frozen fresh-Gaussian code law (same control construction as the pairing gauntlet, PR #560). No formulation math changes.

Toy

y ~ U(−1, 1); θ | y is a two-component Gaussian mixture in R² with closed-form ground truth per condition:

  • c₁(y) = (−1.1, 0.9y), c₂(y) = (1.1, −0.9y), σ = 0.18
  • w₁(y) = 0.5 + 0.3y (ranges over [0.2, 0.8])

A conditionally-averaging generator visibly parks at w₁c₁ + w₂c₂, on the bridge between the modes. Training data = paired (θ, y) samples from the true joint.

Arms (same budget: 1,800 steps, batch 128, seeds 0/1/2)

  • (a) particle — conditional ParticleGAN control (learned 256-particle prior)
  • (b) plain — conditional plain GAN control (frozen fresh-Gaussian codes)
  • (c) tarp — (a) + TARP-CvM regularizer; b(y) = small frozen MLP regressor trained on the training pairs

Metrics (cycle-3 rule: coverage / dispersion / mode-mass, not mode counts)

Per-condition (5 y-bins) Naeem coverage; within-mode dispersion ratio (mean pairwise distance inside a mode / σ√π; 1.0 ideal); conditional mode-mass error (|generated mass within 3σ of each true mode − true weight|). The held-out hard-indicator TARP-CvM statistic is reported as a diagnostic only.

Iteration log

Rounds 1–3 ran on a mislabelled toy and are superseded. The component pick (rand < w).long() casts True → 1, so the sampler drew component 0 with probability 1 − w while the metrics compared against w: a weight-flipped world. (Coverage / dispersion / the TARP statistic are world-consistent, so those rounds' qualitative reading survived; the mode-mass margins did not.) The b(y) unit check caught it — the frozen regressor had learned the exact negative of the analytic conditional mean. Both pick sites are fixed, and a sampler unit test now guards the flip. Contributions of the flipped rounds: M=16→32 was a wash; λ=30→60 helped there. Both were re-measured on the corrected toy:

Corrected-toy controls (identical at both configs):

metric particle plain
coverage [0.151, 0.073, 0.308] mean 0.177 [0.633, 0.165, 0.378] mean 0.392
dispersion [0.433, 0.470, 0.671] mean 0.525 [0.952, 1.059, 0.913] mean 0.975
mode-mass err [0.175, 0.499, 0.208] mean 0.294 [0.025, 0.427, 0.498] mean 0.317
held-out TARP stat mean 0.055 mean 0.059

Both controls are erratic: the particle control concentrates within modes in every seed (dispersion ≤ 0.67) and its coverage collapses; the plain control is genuinely good in seed 0 and collapses on mode-mass in seeds 1/2 (0.43 / 0.50).

Round 4 — corrected toy, λ=60, M=32 (the flipped rounds' pick): coverage 0.808 | dispersion 1.172 | mode-mass 0.087 | TARP stat 0.003.

Round 5 — corrected toy, the paper's config, λ=30, M=16: coverage [0.764, 0.780, 0.825] mean 0.790 | dispersion [0.994, 1.065, 1.139] mean 1.066 | mode-mass [0.049, 0.077, 0.089] mean 0.072 | TARP stat mean 0.003. The paper's config is as good or better on every primary metric, so it is the shipped configuration: the paper's SBI settings transfer as-is.

Final margins (round 5, shipped config)

metric particle (a) plain (b) tarp (c) margin vs (a) margin vs (b)
coverage ↑ 0.177 0.392 0.790 +0.613 +0.398
dispersion (ideal 1.0) 0.525 0.975 1.066 relieved from ≤0.67/seed —
mode-mass err ↓ 0.294 0.317 0.072 −0.222 −0.245
held-out TARP stat ↓ (diagnostic) 0.055 0.059 0.003 20× lower 20× lower

Verdict: PROVEN

TARP beats both controls on mean coverage and mean mode-mass error, wins coverage in every seed against both controls, and is consistent in a way neither control is: coverage ≥ 0.76, mode-mass ≤ 0.09, dispersion within [0.99, 1.14] in every seed. The single exception, encoded in the assertions rather than hidden: plain seed 0 posts a lower mode-mass error (0.025 vs 0.049) — that seed's plain run is good, and TARP's claim is the mean margin plus per-seed consistency. The test assertions encode exactly these margins with headroom.

Seam found (fixed in this PR)

The component-pick bool→index flip described above — in this test's own sampler, caught by its own b(y) unit check before any conclusion was drawn from the corrupted metric. A regression unit test (test_sampler_matches_analytic_conditional_mean) now guards it. No repo code outside this new test file is touched.

How to run

python -m pytest tests/test_paperwatch_tarp_cvm.py

~7 min for the nine trainings on one A6000 (the TARP arm draws M samples per pair per step); much slower on CPU. Iteration knobs via env: TARP_STEPS, TARP_LAMBDA, TARP_M, TARP_TAU_MULT. Verified: 9 passed on pop-os (RTX A6000, torch 2.15).

Nothing merges — paper-watch protocol.

@mikkel
mikkel changed the base branch from master to develop November 20, 2026 03:34
@mikkel
mikkel force-pushed the paper-watch/tarp-cvm branch from 265349f to 961e33f Compare November 20, 2026 04:45
@mikkel mikkel changed the title Paper watch: TARP-CvM coverage loss (arXiv:2611.15829) — proven transfer Paper watch: TARP-CvM coverage loss (arXiv:2611.15829) — diagnostic cohort, partial transfer on CPU Nov 20, 2026
mikkel and others added 3 commits November 20, 2026 05:17
…ven transfer

Conditional three-arm toy (ParticleGAN / plain GAN / ParticleGAN +
TARP-CvM) on a y-dependent two-component Gaussian mixture with
closed-form conditional truth, 1,800 steps, seeds 0-2. The repo's
GANTrainer is unconditional-only, so the test uses a caller-owned
conditional host with the repo's learned ParticlePrior; the paper's
mechanism (per-rule CvM of TARP credibilities, switch/prior/posterior
rules around a frozen posterior-mean regressor, lambda=30, M=16) is
implemented exactly. Assertions encode the gauntlet's measured
margins: TARP beats both controls on coverage and mode-mass error
and is consistent per seed where both controls collapse somewhere.
Nothing merged.
- Pin DEVICE to CPU and run the gauntlet with one intra-op thread so CI
  and local runs share one deterministic law (the CUDA-calibrated
  thresholds failed 4 tests on CPU CI; control arms are chaotic across
  devices and thread counts).
- Keep the original CUDA-cohort thresholds; split the claims and mark the
  three the CPU cohort does not support as strict xfails carrying the
  measured numbers (seed-0 mode-mass win over particle, particle
  concentration in every seed, tarp dispersion > plain).
- Declare the evidence scope: caller-owned hinge/Adam conditional host
  with a host-side TARP term; the ka2 recipe only supplies the prior, so
  this is a diagnostic/assisted cohort, not qualification evidence.
- Record seed-0 frames on a dedicated RNG stream and render an
  actual-training GIF (target vs arms at fixed conditions); commit the
  CPU-cohort GIF and compact metrics under reports/paper-watch/tarp-cvm.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
reports/ (outside reports/forge/) is a Forge history source root, so
adding files there changes the published source digest and stales the
Forge readouts. docs/ is not a source root.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mikkel
mikkel force-pushed the paper-watch/tarp-cvm branch from 961e33f to 7c52cf3 Compare November 20, 2026 05:48
mikkel and others added 2 commits November 20, 2026 06:59
…e non-strict xfails

Identical pinned CPU runs (one thread, same seeds) reproduce two distinct
states across GitHub runner hardware. Claims holding in both stay
assertions; claims failing in both stay strict xfails; claims against
the plain control that flip with the runner become non-strict xfails
naming both measured outcomes (not established on CPU). Thresholds are
unchanged. The gauntlet fixture prints a one-line [tarp-cvm-cohort] JSON
summary to stderr so each runner's numbers are readable in CI logs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rter

sys.__stderr__ is fd-captured under CI, so the [tarp-cvm-cohort] line
never reached the job log.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant