Skip to content

Regenerate FINDINGS.md / paper numbers from recomputed artifacts (post-#305) #309

Description

@openclaw-dv

PR #305 (recompute-don't-trust) changed published SPS values (e.g. the 3-model ensemble on globalopinionqa moved from ~0.747 rank-6 to ~0.81 rank-1; opinionsqa/subpop ensembles ~0.877/0.879). Hand-maintained numbers in FINDINGS.md, the arXiv draft under docs/papers/, and the Q2 report are now stale/contradicted by the leaderboard.

  • Regenerate the findings tables + publish.py::_build_findings narrative from leaderboard-results/ at publish time; CI-diff FINDINGS.md against artifacts so they can't drift.
  • Baseline-anchor headline claims (random baseline scores ~0.71–0.76 on the same scale — surface normalized_sps).
  • Version the metric (SPS-3 vs SPS-5) and recompute all tables on one scale.
    See gap-analysis P2-1/P2-2/P2-3.

Metadata

Metadata

Assignees

No one assigned

    Labels

    documentationImprovements or additions to documentationpmf-researchSurfaced from synthpanel-driven PMF research 2026-05-14

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions