Skip to content

ci(bench): grid benchmark reports false regressions, compare build time over repeated runs #1132

Description

@ZhuchkaTriplesix

Problem

.github/workflows/benchmark.yml comments "regression" on PRs that did not touch the measured code. Evidence:

Run base p50 PR p50 Result
#1131, run 1 11.39 ms 9.35 ms -17.9%
#1131, run 2 (same code, 4 min later) 15.75 ms 16.78 ms +6.5%, p99 +18.1%, comment posted
#1106 11.63 ms 7.26 ms p50 -37.6%, p90 +5.9%, comment posted

#1131 did not change ResultGridView / ResultsTab; the base itself went from 11.4 to 15.8 ms between two runs.

Causes

  1. The compared metric is FrameTiming.totalSpan, which includes vsync wait and the raster queue. The Dart side is build (p50 0.42-0.47 ms in every run, no base/PR difference). raster is software rendering (Mesa llvmpipe under xvfb) on a shared VM.
  2. One run per side, always PR first, then base: any runner spike lands entirely on one side.
  3. Threshold too tight: >5% and >0.3 ms. p99 of ~300 frames is 3 frames.
  4. stutters(>12.5ms) assumes 120 Hz, but xvfb presents at 60 Hz, so ~every frame (290/291) counts.

Proposal (benchmark.yml, scripts/ci/bench_compare.py; no app code)

  • Flag regressions on build p50 / p90 only; keep raster, total and p99 in the table as information.
  • 3 runs per side, interleaved (base, PR, base, PR, base, PR); compare medians.
  • Regression = median worse by >10% and >0.2 ms, and PR slower in all three pairs.
  • Count stutters as gaps >25 ms (a missed vsync at 60 Hz), or drop the row.
  • Still informational, never blocks a merge.

Cost: roughly 4-5 extra minutes per run on the benchmark job.

Related: #1065 (introduced the workflow), #987.

Activity

  1. added
    performanceTheme parser epic label: performance
    testsTheme parser epic
    P3Low priority / Polish & Enhancements
    on Oct 8, 2026
  2. added a commit that references this issue on Oct 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P3Low priority / Polish & EnhancementsperformanceTheme parser epic label: performancetestsTheme parser epic

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions