Skip to content

feat(benchmarks): add reproducible CIFAR-style Transformer-G / ResNet18-feature-D benchmark - #586

Merged
martyn merged 2 commits into
developfrom
feat/cifar-style-toy-benchmark
Nov 20, 2026
Merged

martyn merged 2 commits into
developfrom
feat/cifar-style-toy-benchmark

Conversation

@martyn

@martyn martyn commented Nov 20, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds a fast (~1.5–4.5 min/run on 1× RTX A6000), reproducible CIFAR-style 32×32 image toy benchmark (benchmarks/cifar_toy/) and contract tests (tests/test_cifar_toy_benchmark.py) matching the structural regime of HyperGAN's cifar10-tiny-transformer, using only ParticleGAN's public API (GANTrainer, get_recipe, MoGParticlePrior, particlegan.init.deterministic_orthogonal_).

Benchmark Structure

  • Generator (TinyTransformerGenerator32): 2-block 64-token Transformer -> 32×32 RGB (16 matrix/conv params, 2,489,731 trainable params).
  • Critic (ResNet18FeatureCritic32): Frozen SHA-256-verified ImageNet ResNet-18 (layer1/2/3 at 64px, strictly locked in eval() mode) + 3 trainable feature heads (9 matrix/conv params, 171,779 trainable params). Zero torchvision dependency.
  • Evaluator (ResNet34FeatureEvaluator): Independent frozen SHA-256-verified ImageNet ResNet-34 512-dim avgpool Feature-FID, unbiased polynomial KID (×1e3), and 10-class centroid mode coverage/TV (real-vs-real CIFAR-10 oracle floor: FID=82.66, KID×1e3=61.2±206.3, 10/10 cov, TV=0.088).
  • Protocol (AGENTS.md compliant): Protocol seed 0, deterministic public init, isolated critic/generator data + trainer RNG streams, bit-identical initial_g_sha256, initial_d_sha256, initial_prior_sha256, and batch_sequence_digest across trainers, zero task-host repairs, zero bulk logs in Git. Registered "cifar_toy" in experiments/forge/history.py with refreshed Forge catalog.

Runnable CLI

python -m benchmarks.cifar_toy \
  --recipes bispf_optimized empf unified_v11 \
  --num-particles 200 16384 \
  --batch-size 128 --steps 800 --eval-every 200 --eval-samples 2048 \
  --device cuda:0 --output-dir runs/cifar_toy_bench/baseline_v1

Matched Baseline Leaderboard (Real CIFAR-10, 1× RTX A6000, Seed 0, steps=800, B=128)

Step-0 FID: 2977.78 (N=200) / 2965.82 (N=16384). Real-vs-Real oracle FID: 82.66.

Run Recipe MoG N Status Final FID ↓ Best FID ↓ (step) KID×1e3 ↓ Cov /10 TV ↓ D acc Margin G grad D grad it/s ↑ Peak MiB
bispf_optimized_p200_b128_s800 bispf_optimized 200 COMPLETE 439.36 439.36 (800) 3521.34 10/10 0.121 0.502 -0.000 0.033 0.092 2.75 1485
empf_p200_b128_s800 empf 200 COMPLETE 454.21 440.22 (600) 3636.87 10/10 0.196 0.502 -0.001 0.028 0.093 4.91 1496
empf_p16384_b128_s800 empf 16384 COMPLETE 856.34 836.06 (400) 22058.32 10/10 0.276 0.586 +0.521 1.516 3.278 3.92 1508
bispf_optimized_p16384_b128_s800 bispf_optimized 16384 COMPLETE 916.03 916.03 (800) 33862.31 9/10 0.312 0.623 +0.620 2.310 4.198 2.27 1493
unified_v11_p16384_b128_s800 unified_v11 16384 COMPLETE 2618.93 2618.93 (800) 71734.14 2/10 0.876 0.477 -0.037 0.080 14.884 8.87 3489
unified_v11_p200_b128_s800 unified_v11 200 COMPLETE 2730.72 2730.72 (800) 66737.60 1/10 0.897 0.000 -0.121 0.069 30.375 8.82 3898

Verification

  • python scripts/check_lint.py — PASS (294 baseline, 0 additions)
  • python -m pytest -q tests/test_cifar_toy_benchmark.py tests/test_forge_history.py — 19 passed
  • python -m experiments.forge validate && python -m experiments.forge compile --check — PASS (CURRENT, 11861/11861)

@martyn
martyn merged commit 89c730a into develop Nov 20, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant