Skip to content

Commit bb05045

Browse files
openclaw-dvclaudedataviking-infra-terraform[bot]
authored
data(subpop): clean Haiku constituent re-run + ensemble re-blend (#321)
* data(subpop): clean Haiku constituent re-run + ensemble re-blend Replace the validity-excluded subpop ensemble constituent (subpop_synthpanel_claude-haiku-4-5-20251001_20260411_073013, 80% uniform-garbage, SPS 0.773 fabricated) with a clean re-run on today's harness: synthpanel via openrouter/anthropic/claude-haiku-4-5, same config (n=200, s=30, t=default, question_set_hash 6d991a90..), 5.0% uniform, NOT flagged by run_validity, tier-3 validation 0 errors. Regenerate the 3-model blend from the clean constituent (SPS 0.8307, still rank 1 on subpop) and drop the two identical stale blend files — publish dedup keeps the first tie by n_evaluated, so the old blend would otherwise shadow the new one. The excluded constituent file stays committed for excluded_runs transparency. Also fix the ensemble command to round distributions before computing per-question metrics: metrics from unrounded values can flip Kendall-tau tie handling when two blended options round to the same 4-dp published value, producing PER_Q_TAU validation errors on an otherwise-correct blend (hit on BIDENATTN_b_W85). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore(publish): regenerate leaderboard.json + FINDINGS.md after subpop re-blend subpop ensemble SPS 0.879423 -> 0.830664 (recomputed); the excluded-constituent caveat is gone now that all three blend constituents pass the validity filter. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore(vrt): update baselines via vrt-baseline-update label * ci: re-trigger checks on committed VRT baseline Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: dataviking-infra-terraform[bot] <274205567+dataviking-infra-terraform[bot]@users.noreply.github.com>
1 parent 0fb54d9 commit bb05045

8 files changed

Lines changed: 5322 additions & 3892 deletions

File tree

FINDINGS.md

Lines changed: 3 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -30,7 +30,7 @@ asymmetries in how LLMs represent different demographic groups.
3030
|---------|---------------------|-------------------|-----------------|
3131
| globalopinionqa (n=100) | **0.813** | 0.786 (SynthPanel (GPT-4o-mini), product) | 0.710 (n=10) |
3232
| opinionsqa (n=684) | **0.877** | 0.829 (Gemini 2.5 Flash, raw) | 0.763 (n=684) |
33-
| subpop (n=200) | **0.879** | 0.821 (SynthPanel (Gemini Flash Lite), product) | 0.757 (n=200) |
33+
| subpop (n=200) | **0.831** | 0.821 (SynthPanel (Gemini Flash Lite), product) | 0.757 (n=200) |
3434
<!-- END GENERATED: headline -->
3535

3636
Note on interpretation: SPS is a composite parity score, not a
@@ -206,11 +206,9 @@ three runs costs roughly 3× a single-model run.
206206
|---------|-------------------|-------------|-------------|-----------------|
207207
| globalopinionqa (100q) | 0.786 (SynthPanel (GPT-4o-mini), product) | **0.813** | **+2.7 pts** | 0.710 (n=10) |
208208
| opinionsqa (684q) | 0.829 (Gemini 2.5 Flash, raw) | **0.877** | **+4.8 pts** | 0.763 (n=684) |
209-
| subpop (200q) | 0.821 (SynthPanel (Gemini Flash Lite), product) | **0.879** | **+5.8 pts** | 0.757 (n=200) |
209+
| subpop (200q) | 0.821 (SynthPanel (Gemini Flash Lite), product) | **0.831** | **+1.0 pts** | 0.757 (n=200) |
210210

211211
Comparison set: best_single = highest recomputed-SPS non-ensemble, non-baseline leaderboard row (raw or product framework) evaluated on at least as many questions as the ensemble, after per-(model, framework, dataset) dedup; random_baseline = recomputed SPS of the random-baseline run for the same dataset (n in random_baseline_n).
212-
213-
> **Caveat**: The subpop ensemble blends constituent run 'subpop_synthpanel_claude-haiku-4-5-20251001_20260411_073013', which the run-validity filter excludes from the leaderboard as uniform-distribution garbage. The blended per-question rows still include its (uniform) distributions, so the published ensemble score for subpop understates a clean re-blend.
214212
<!-- END GENERATED: ensemble -->
215213

216214
### Key Findings
@@ -291,7 +289,7 @@ more temperature — it just adds noise.
291289
<!-- BEGIN GENERATED: levers -->
292290
| Lever | Effect size (SPS pts) | Cost | Status |
293291
|-------|----------------------|------|--------|
294-
| **Ensemble blending** | +2.7–5.8 | zero | done |
292+
| **Ensemble blending** | +1.0–4.8 | zero | done |
295293
| **Per-model optimal temperature** | +0.8–4.5 | low | actionable |
296294
| **Demographic conditioning** | +1.9–7.3 | moderate | scientific |
297295
| **Persona template** | 'current' template already optimal (+11.0 pts over the best alternative); no further gain available from this lever. | zero | done |

0 commit comments

Comments
 (0)