data(subpop): clean Haiku constituent re-run + ensemble re-blend#321
Merged
Conversation
Replace the validity-excluded subpop ensemble constituent (subpop_synthpanel_claude-haiku-4-5-20251001_20260411_073013, 80% uniform-garbage, SPS 0.773 fabricated) with a clean re-run on today's harness: synthpanel via openrouter/anthropic/claude-haiku-4-5, same config (n=200, s=30, t=default, question_set_hash 6d991a90..), 5.0% uniform, NOT flagged by run_validity, tier-3 validation 0 errors. Regenerate the 3-model blend from the clean constituent (SPS 0.8307, still rank 1 on subpop) and drop the two identical stale blend files — publish dedup keeps the first tie by n_evaluated, so the old blend would otherwise shadow the new one. The excluded constituent file stays committed for excluded_runs transparency. Also fix the ensemble command to round distributions before computing per-question metrics: metrics from unrounded values can flip Kendall-tau tie handling when two blended options round to the same 4-dp published value, producing PER_Q_TAU validation errors on an otherwise-correct blend (hit on BIDENATTN_b_W85). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…p re-blend subpop ensemble SPS 0.879423 -> 0.830664 (recomputed); the excluded-constituent caveat is gone now that all three blend constituents pass the validity filter. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
openclaw-dv
added
the
vrt-baseline-update
github-actions
Bot
removed
the
vrt-baseline-update
Contributor
SynthBench submission validationNo adapter-based submissions detected in this PR — legacy provider-keyed submissions are validated by the |
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This was referenced Jul 14, 2026
This was referenced Jul 14, 2026
openclaw-dv
added a commit
that referenced
this pull request
Jul 15, 2026
…nche, ≤$20) (#323) * data(scorecards): demographic-conditioned subpop runs for 5 cheap models (#255) Five conditioned runs (n=100, s=15, --demographics CREGION,EDUCATION,INCOME,POLPARTY,SEX) reproducing the April SubPOP conditioning mechanism for: raw GPT-4o-mini, raw Gemini 2.5 Flash, raw Llama 3.3 70B, SynthPanel (GPT-4o-mini), SynthPanel (Gemini Flash Lite). Each carries a demographic_breakdown with 5 dimensions x 2 subgroups (10 cells). Gated human_distribution stripped per #308. Total LLM spend for the tranche: $1.87 (hard budget $20). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(validation): RAW_RESPONSES_MODE — flag impossible samples, not legitimate minority draws The runner records the FIRST non-refusal sample as the raw_responses audit trail, so any option that actually received samples is a legitimate recorded value. The old gap-vs-top heuristic (>1/sqrt(s) below the mode) statistically guaranteed false positives: at s=15, ~1 question in 12 records a first draw outside that envelope, so an honest 100-question schema-v2 run failed validation almost surely once RAW_RESPONSES_MODE graduated to ERROR (sb-88fw) — exactly what happened to this tranche's runs (8/100 and 6/100 flagged on the two raw runs). Replace it with the sound arithmetic invariant: a recorded sample contributes at least one count, so its option must carry >= ~1/n_samples of the distribution's mass (with 1e-3 slack for published rounding, e.g. 1/11 stored as 0.0909). Absent or zero-mass options — the actual fabrication signal — still flag and still graduate to ERROR in v2. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore(publish): regenerate leaderboard.json + FINDINGS generated sections for #255 tranche Regenerated on top of the merged subpop re-blend (#321). Five more vendors now carry a populated demographic_scorecard (5 dimensions x 10 cells each): GPT-4o-mini, Gemini 2.5 Flash, Llama 3.3 70B, SynthPanel (GPT-4o-mini), SynthPanel (Gemini Flash Lite). (site/public/data run-explorer artifacts are CI-generated, not committed.) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore(vrt): update baselines via vrt-baseline-update label * ci: re-trigger checks on committed VRT baseline Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: dataviking-infra-terraform[bot] <274205567+dataviking-infra-terraform[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces the validity-excluded constituent of the subpop 3-model ensemble with a clean re-run, and regenerates the blend. Numbers below are verified against the committed artifacts (they correct the earlier draft narrative, which misquoted the 2-metric parity mean as SPS and overstated refusal rates).
Constituent: old → new
…20260411_073013, excluded)…20260714_171636)openrouter/anthropic/claude-haiku-4-5— same leaderboard identity)Ensemble
Subpop 3-model blend: 0.879 → 0.831, still rank 1 (best single: Gemini Flash Lite 0.821),
verifiedbadge. The old 0.879 was inflated by the garbage constituent's uniform rows acting as a hedge in the blended distributions. The findings excluded-constituent caveat is gone (findings.caveats: []); drift guard green.Validation & hygiene
--tier3 --peers→ 0 errors; strip check clean;pytest -q1099 passed / 28 skipped; ruff + biome clean.synthbench ensemblenow computes per-question metrics from the same 4-dp-rounded distributions it publishes (a rounding-tie previously produced a PER_Q_TAU validation error).subpop_ensemble_3blend_20260412_*files removed (publish dedup would have shadowed the new blend).Refs #309 (the tracked re-blend follow-up).
🤖 Generated with Claude Code