Commit bb05045
data(subpop): clean Haiku constituent re-run + ensemble re-blend (#321)
* data(subpop): clean Haiku constituent re-run + ensemble re-blend
Replace the validity-excluded subpop ensemble constituent
(subpop_synthpanel_claude-haiku-4-5-20251001_20260411_073013, 80%
uniform-garbage, SPS 0.773 fabricated) with a clean re-run on today's
harness: synthpanel via openrouter/anthropic/claude-haiku-4-5, same
config (n=200, s=30, t=default, question_set_hash 6d991a90..), 5.0%
uniform, NOT flagged by run_validity, tier-3 validation 0 errors.
Regenerate the 3-model blend from the clean constituent (SPS 0.8307,
still rank 1 on subpop) and drop the two identical stale blend files —
publish dedup keeps the first tie by n_evaluated, so the old blend
would otherwise shadow the new one. The excluded constituent file stays
committed for excluded_runs transparency.
Also fix the ensemble command to round distributions before computing
per-question metrics: metrics from unrounded values can flip
Kendall-tau tie handling when two blended options round to the same
4-dp published value, producing PER_Q_TAU validation errors on an
otherwise-correct blend (hit on BIDENATTN_b_W85).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chore(publish): regenerate leaderboard.json + FINDINGS.md after subpop re-blend
subpop ensemble SPS 0.879423 -> 0.830664 (recomputed); the
excluded-constituent caveat is gone now that all three blend
constituents pass the validity filter.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chore(vrt): update baselines via vrt-baseline-update label
* ci: re-trigger checks on committed VRT baseline
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: dataviking-infra-terraform[bot] <274205567+dataviking-infra-terraform[bot]@users.noreply.github.com>1 parent 0fb54d9 commit bb05045
8 files changed
Lines changed: 5322 additions & 3892 deletions
File tree
- leaderboard-results
- site
- e2e/visual/__screenshots__/smoke.visual.spec.ts/desktop-chromium
- src/data
- src/synthbench
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
30 | 30 | | |
31 | 31 | | |
32 | 32 | | |
33 | | - | |
| 33 | + | |
34 | 34 | | |
35 | 35 | | |
36 | 36 | | |
| |||
206 | 206 | | |
207 | 207 | | |
208 | 208 | | |
209 | | - | |
| 209 | + | |
210 | 210 | | |
211 | 211 | | |
212 | | - | |
213 | | - | |
214 | 212 | | |
215 | 213 | | |
216 | 214 | | |
| |||
291 | 289 | | |
292 | 290 | | |
293 | 291 | | |
294 | | - | |
| 292 | + | |
295 | 293 | | |
296 | 294 | | |
297 | 295 | | |
| |||
0 commit comments