PR #305 (recompute-don't-trust) changed published SPS values (e.g. the 3-model ensemble on globalopinionqa moved from ~0.747 rank-6 to ~0.81 rank-1; opinionsqa/subpop ensembles ~0.877/0.879). Hand-maintained numbers in FINDINGS.md, the arXiv draft under docs/papers/, and the Q2 report are now stale/contradicted by the leaderboard.
- Regenerate the findings tables +
publish.py::_build_findings narrative from leaderboard-results/ at publish time; CI-diff FINDINGS.md against artifacts so they can't drift.
- Baseline-anchor headline claims (random baseline scores ~0.71–0.76 on the same scale — surface
normalized_sps).
- Version the metric (SPS-3 vs SPS-5) and recompute all tables on one scale.
See gap-analysis P2-1/P2-2/P2-3.
PR #305 (recompute-don't-trust) changed published SPS values (e.g. the 3-model ensemble on globalopinionqa moved from ~0.747 rank-6 to ~0.81 rank-1; opinionsqa/subpop ensembles ~0.877/0.879). Hand-maintained numbers in
FINDINGS.md, the arXiv draft underdocs/papers/, and the Q2 report are now stale/contradicted by the leaderboard.publish.py::_build_findingsnarrative fromleaderboard-results/at publish time; CI-diff FINDINGS.md against artifacts so they can't drift.normalized_sps).See gap-analysis P2-1/P2-2/P2-3.