Repository navigation
Paper watch: TARP-CvM coverage loss (arXiv:2611.15829) — diagnostic cohort, partial transfer on CPU - #569
Open
mikkel wants to merge 5 commits into
Open
Paper watch: TARP-CvM coverage loss (arXiv:2611.15829) — diagnostic cohort, partial transfer on CPU#569mikkel wants to merge 5 commits into
mikkel wants to merge 5 commits into
Conversation
mikkel
force-pushed
the
paper-watch/tarp-cvm
branch
from
November 20, 2026 04:45
265349f to
961e33f
Compare
…ven transfer Conditional three-arm toy (ParticleGAN / plain GAN / ParticleGAN + TARP-CvM) on a y-dependent two-component Gaussian mixture with closed-form conditional truth, 1,800 steps, seeds 0-2. The repo's GANTrainer is unconditional-only, so the test uses a caller-owned conditional host with the repo's learned ParticlePrior; the paper's mechanism (per-rule CvM of TARP credibilities, switch/prior/posterior rules around a frozen posterior-mean regressor, lambda=30, M=16) is implemented exactly. Assertions encode the gauntlet's measured margins: TARP beats both controls on coverage and mode-mass error and is consistent per seed where both controls collapse somewhere. Nothing merged.
- Pin DEVICE to CPU and run the gauntlet with one intra-op thread so CI and local runs share one deterministic law (the CUDA-calibrated thresholds failed 4 tests on CPU CI; control arms are chaotic across devices and thread counts). - Keep the original CUDA-cohort thresholds; split the claims and mark the three the CPU cohort does not support as strict xfails carrying the measured numbers (seed-0 mode-mass win over particle, particle concentration in every seed, tarp dispersion > plain). - Declare the evidence scope: caller-owned hinge/Adam conditional host with a host-side TARP term; the ka2 recipe only supplies the prior, so this is a diagnostic/assisted cohort, not qualification evidence. - Record seed-0 frames on a dedicated RNG stream and render an actual-training GIF (target vs arms at fixed conditions); commit the CPU-cohort GIF and compact metrics under reports/paper-watch/tarp-cvm. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
reports/ (outside reports/forge/) is a Forge history source root, so adding files there changes the published source digest and stales the Forge readouts. docs/ is not a source root. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mikkel
force-pushed
the
paper-watch/tarp-cvm
branch
from
November 20, 2026 05:48
961e33f to
7c52cf3
Compare
…e non-strict xfails Identical pinned CPU runs (one thread, same seeds) reproduce two distinct states across GitHub runner hardware. Claims holding in both stay assertions; claims failing in both stay strict xfails; claims against the plain control that flip with the runner become non-strict xfails naming both measured outcomes (not established on CPU). Thresholds are unchanged. The gauntlet fixture prints a one-line [tarp-cvm-cohort] JSON summary to stderr so each runner's numbers are readable in CI logs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rter sys.__stderr__ is fd-captured under CI, so the [tarp-cvm-cohort] line never reached the job log. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Status update (rebased onto
develop; base isdevelop)Scope: diagnostic / assisted cohort — not qualification evidence, no longer "proven transfer".
ka2recipe only suppliesmake_prior; its optimizer, penalty, noise and prior policy never run. TARP is not a declared recipe capability:GANTraineris unconditional, and a faithful port needs a recipe-driven paired conditional host (make_optimizers/make_loss/make_critic_penalty, as the DDGAN conditional-energy host does) plus a declared recipe field and evidence identity. That is more than a modest change, so I left it out of this PR.DEVICE=cpuand runs the gauntlet with one intra-op thread, so CI and local runs follow the same deterministic law. The earlier CI failures came from CUDA-calibrated thresholds meeting CPU control arms that behave differently.xfail(strict=True)with measured numbers): tarp wins mode-mass in every seed vs particle; the particle control concentrates in every seed.xfail(strict=False), both outcomes recorded): every claim against the plain control, namely the coverage margin, mode-mass wins and margins, dispersion and the held-out statistic. These claims are not established on CPU.[tarp-cvm-cohort]JSON to stderr, so each runner's numbers can be read from CI logs.test_training_gifrenders the target and the three arms at y = −0.8 / 0 / +0.8. The CPU-cohort GIF and compact metrics are indocs/paper-watch/tarp-cvm/(kept out ofreports/, which is a Forge history source root).Paper
"Training Calibrated Conditional Generative Models with a Sample-Only Coverage Loss" — Remy, Rossellini, Chang, Willett (2026). arXiv:2611.15829 — https://arxiv.org/abs/2611.15829
Mechanism (TARP-CvM)
For a paired example (θ_i, y_i) and a reference rule ψ, the credibility value s_i is the fraction of M model draws θ̂ ~ q(·|y_i) lying closer to ψ(y_i) than the true θ_i does. If q is the true posterior, s ~ U(0,1). The loss is the Cramér–von Mises discrepancy (paper Eq. 5, plug-in form) between the empirical CDF of {s_i} and the uniform, computed separately per rule and averaged over rules — never pooled, so per-condition errors cannot cancel. Rules are built around a frozen point estimate b(y) (a posterior-mean MLP regressor trained first and frozen), with scale-matched random displacements ς = per-coordinate RMS residual (Eq. 9): switch rules (one displacement per side of a random hyperplane in y, Eq. 8), prior rules (one marginal point shared by the batch), and posterior rules (a detached own draw, excluded from its own credibility). Indicators are relaxed with a sigmoid at τ = 0.05 × median sample-to-reference distance. Total loss = base loss + λ · mean-rule CvM. Theorem 1: the averaged loss is a strict divergence — zero iff q equals the true posterior — and it needs only samples, which is exactly why it can attach to a GAN/particle generator at all. This test uses the paper's SBI configuration: λ=30, M=16, R=12 rules/step (6 switch / 3 prior / 3 posterior, scaled down from the paper's 48 for toy step cost).
Why here: conditional averaging is the conditional generator's signature failure — mass parked at the conditional mean, between the modes. An averaging generator's credibility values are non-uniform, so TARP-CvM punishes precisely that.
Harness note
The repo's
GANTraineris unconditional-only (it rejects any recipe whose conditioning is not"scalar"), so this test uses the caller-owned conditional host the trainer's own error message points to: the condition is concatenated into the generator and critic inputs, and the particle arms keep the repo's learnedParticlePriortable, sampled and trained exactly as in the unconditional harness. The plain arm swaps the table for the repo's frozen fresh-Gaussian code law (same control construction as the pairing gauntlet, PR #560). No formulation math changes.Toy
y ~ U(−1, 1); θ | y is a two-component Gaussian mixture in R² with closed-form ground truth per condition:
A conditionally-averaging generator visibly parks at w₁c₁ + w₂c₂, on the bridge between the modes. Training data = paired (θ, y) samples from the true joint.
Arms (same budget: 1,800 steps, batch 128, seeds 0/1/2)
Metrics (cycle-3 rule: coverage / dispersion / mode-mass, not mode counts)
Per-condition (5 y-bins) Naeem coverage; within-mode dispersion ratio (mean pairwise distance inside a mode / σ√π; 1.0 ideal); conditional mode-mass error (|generated mass within 3σ of each true mode − true weight|). The held-out hard-indicator TARP-CvM statistic is reported as a diagnostic only.
Iteration log
Rounds 1–3 ran on a mislabelled toy and are superseded. The component pick
(rand < w).long()casts True → 1, so the sampler drew component 0 with probability 1 − w while the metrics compared against w: a weight-flipped world. (Coverage / dispersion / the TARP statistic are world-consistent, so those rounds' qualitative reading survived; the mode-mass margins did not.) The b(y) unit check caught it — the frozen regressor had learned the exact negative of the analytic conditional mean. Both pick sites are fixed, and a sampler unit test now guards the flip. Contributions of the flipped rounds: M=16→32 was a wash; λ=30→60 helped there. Both were re-measured on the corrected toy:Corrected-toy controls (identical at both configs):
Both controls are erratic: the particle control concentrates within modes in every seed (dispersion ≤ 0.67) and its coverage collapses; the plain control is genuinely good in seed 0 and collapses on mode-mass in seeds 1/2 (0.43 / 0.50).
Round 4 — corrected toy, λ=60, M=32 (the flipped rounds' pick): coverage 0.808 | dispersion 1.172 | mode-mass 0.087 | TARP stat 0.003.
Round 5 — corrected toy, the paper's config, λ=30, M=16: coverage [0.764, 0.780, 0.825] mean 0.790 | dispersion [0.994, 1.065, 1.139] mean 1.066 | mode-mass [0.049, 0.077, 0.089] mean 0.072 | TARP stat mean 0.003. The paper's config is as good or better on every primary metric, so it is the shipped configuration: the paper's SBI settings transfer as-is.
Final margins (round 5, shipped config)
Verdict: PROVEN
TARP beats both controls on mean coverage and mean mode-mass error, wins coverage in every seed against both controls, and is consistent in a way neither control is: coverage ≥ 0.76, mode-mass ≤ 0.09, dispersion within [0.99, 1.14] in every seed. The single exception, encoded in the assertions rather than hidden: plain seed 0 posts a lower mode-mass error (0.025 vs 0.049) — that seed's plain run is good, and TARP's claim is the mean margin plus per-seed consistency. The test assertions encode exactly these margins with headroom.
Seam found (fixed in this PR)
The component-pick bool→index flip described above — in this test's own sampler, caught by its own b(y) unit check before any conclusion was drawn from the corrupted metric. A regression unit test (
test_sampler_matches_analytic_conditional_mean) now guards it. No repo code outside this new test file is touched.How to run
~7 min for the nine trainings on one A6000 (the TARP arm draws M samples per pair per step); much slower on CPU. Iteration knobs via env:
TARP_STEPS,TARP_LAMBDA,TARP_M,TARP_TAU_MULT. Verified: 9 passed on pop-os (RTX A6000, torch 2.15).Nothing merges — paper-watch protocol.