Skip to content

data(subpop): clean Haiku constituent re-run + ensemble re-blend #321

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
openclaw-dv merged 4 commits into from
Jul 14, 2026
Merged

data(subpop): clean Haiku constituent re-run + ensemble re-blend #321

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
337dada
data(subpop): clean Haiku constituent re-run + ensemble re-blend
openclaw-dv Jul 14, 2026
aac64c6
chore(publish): regenerate leaderboard.json + FINDINGS.md after subpo…
openclaw-dv Jul 14, 2026
2006102
chore(vrt): update baselines via vrt-baseline-update label
dataviking-infra-terraform[bot] Jul 14, 2026
193500e
ci: re-trigger checks on committed VRT baseline
openclaw-dv Jul 14, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 3 additions & 5 deletions FINDINGS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ asymmetries in how LLMs represent different demographic groups.
|---------|---------------------|-------------------|-----------------|
| globalopinionqa (n=100) | **0.813** | 0.786 (SynthPanel (GPT-4o-mini), product) | 0.710 (n=10) |
| opinionsqa (n=684) | **0.877** | 0.829 (Gemini 2.5 Flash, raw) | 0.763 (n=684) |
| subpop (n=200) | **0.879** | 0.821 (SynthPanel (Gemini Flash Lite), product) | 0.757 (n=200) |
| subpop (n=200) | **0.831** | 0.821 (SynthPanel (Gemini Flash Lite), product) | 0.757 (n=200) |
<!-- END GENERATED: headline -->

Note on interpretation: SPS is a composite parity score, not a
Expand Down Expand Up @@ -206,11 +206,9 @@ three runs costs roughly 3× a single-model run.
|---------|-------------------|-------------|-------------|-----------------|
| globalopinionqa (100q) | 0.786 (SynthPanel (GPT-4o-mini), product) | **0.813** | **+2.7 pts** | 0.710 (n=10) |
| opinionsqa (684q) | 0.829 (Gemini 2.5 Flash, raw) | **0.877** | **+4.8 pts** | 0.763 (n=684) |
| subpop (200q) | 0.821 (SynthPanel (Gemini Flash Lite), product) | **0.879** | **+5.8 pts** | 0.757 (n=200) |
| subpop (200q) | 0.821 (SynthPanel (Gemini Flash Lite), product) | **0.831** | **+1.0 pts** | 0.757 (n=200) |

Comparison set: best_single = highest recomputed-SPS non-ensemble, non-baseline leaderboard row (raw or product framework) evaluated on at least as many questions as the ensemble, after per-(model, framework, dataset) dedup; random_baseline = recomputed SPS of the random-baseline run for the same dataset (n in random_baseline_n).

> **Caveat**: The subpop ensemble blends constituent run 'subpop_synthpanel_claude-haiku-4-5-20251001_20260411_073013', which the run-validity filter excludes from the leaderboard as uniform-distribution garbage. The blended per-question rows still include its (uniform) distributions, so the published ensemble score for subpop understates a clean re-blend.
<!-- END GENERATED: ensemble -->

### Key Findings
Expand Down Expand Up @@ -291,7 +289,7 @@ more temperature — it just adds noise.
<!-- BEGIN GENERATED: levers -->
| Lever | Effect size (SPS pts) | Cost | Status |
|-------|----------------------|------|--------|
| **Ensemble blending** | +2.7–5.8 | zero | done |
| **Ensemble blending** | +1.0–4.8 | zero | done |
| **Per-model optimal temperature** | +0.8–4.5 | low | actionable |
| **Demographic conditioning** | +1.9–7.3 | moderate | scientific |
| **Persona template** | 'current' template already optimal (+11.0 pts over the best alternative); no further gain available from this lever. | zero | done |
Expand Down
Loading
Loading