Skip to content

ci: reset transitional perf-gate ceilings after engine reroute landed#22

Merged
yipjunkai merged 1 commit into
mainfrom
ci/reset-perf-gate-ceilings
Jun 11, 2026
Merged

ci: reset transitional perf-gate ceilings after engine reroute landed#22
yipjunkai merged 1 commit into
mainfrom
ci/reset-perf-gate-ceilings

Conversation

@yipjunkai

@yipjunkai yipjunkai commented Jun 11, 2026

Copy link
Copy Markdown
Owner

What

Resets the six (T) 1.80 perf-gate ceilings that #19 bumped for the normalised-Black engine baseline shift. Touches only .github/scripts/check_perf.py (PER_BENCH_THRESHOLDS).

key before after rationale
bsm_price_scalar 1.80 (T) default 0.10 engine + erfcx now in base → ~0% head-vs-base
black76_price_scalar 1.80 (T) default 0.10 same
bsm_price_vec 1.80 (T) default 0.10 same
parallel_bsm_price 1.80 (T) removed → default 0.10 experiment sub-bench; see note
cdf_branch_experiment 1.80 (T) 0.50 experiment-harness ceiling restored
bsm_price_flag_dispatch 1.80 (T) 0.50 experiment-harness ceiling restored
parallel 0.50 removed (dead key) matched nothing — see note
iv_solve_scalar_atm 0.85 0.85 kept (out of scope)
iv_solve_vec 0.80 0.80 kept (out of scope)

The ENGINE-REROUTE TRANSITION comment block is deleted and the header comment reframed around the head-vs-base principle, so the table documents why it stays small rather than accreting transient exceptions.

Why

The gate compares PR-head against PR-base. Once #19 merged, both the engine reroute and the earlier erfcx deep-OTM cdf tail live in the base, so a normal PR's price benches return to ~0%. The 1.80 (T) ceilings were scaffolding to let that one baseline-shifting PR through — #19's own comment marked them "MUST be reset … in the next PR". Leaving them is precisely the stale-threshold trap the 2026-06 audit flagged: a 1.80 ceiling silently masks a real +50% price regression.

Note: parallel vs parallel_bsm_price (corrected from the gate's own log)

criterion flattens benchmark_group("parallel/bsm_price") to the on-disk path parallel_bsm_price (the / in a group name collapses to _). So:

  • The explicit parallel_bsm_price key did gate those serial/rayon experiment benches (at 1.80). Removing it tightens them to the default 0.10 — what the audit asked for.
  • The parallel key (0.50) matched nothing (no path starts with parallel/) — it was the genuinely dead one. Dropped.

The whole parallel_* serial-vs-rayon harness now runs on the default 10%. These are experiment benches, not production gates (production price/greeks/iv are gated by bsm_price_vec / bsm_greeks_all_vec / iv_solve_vec). The in-file comment documents how to widen a specific noisy sub-bench (its own parallel_<sub> key) if needed — open question for review: do we want the experiment harness back at 0.50 to avoid flaky-gate risk on the noisy rayon-small-N arms? (the original design intent, defeated all along by the dead key).

Kept (deliberately out of scope)

iv_solve_scalar_atm (0.85) and iv_solve_vec (0.80) are head-vs-base stale by the same logic — the LBR cost is in the base — and are flagged in-comment as future reset candidates. Left as-is here.

Verification

  • ruff check ., ruff format --check, pyright — all green locally (the three CI lint gates).
  • Behavioral test of threshold_for against the exact paths the perf-gate log prints: price + parallel_* → 0.10, experiments → 0.50, IV overrides intact, both parallel/parallel_bsm_price keys gone.
  • The perf workflow triggers on check_perf.py, so the gate runs head-vs-base over identical Rust (~0% deltas) under the restored ceilings — confirmed green.

🤖 Generated with Claude Code

#19's normalised-Black price engine is now on `main`, i.e. the perf gate's
base ref. The gate compares PR-head vs PR-base, so with the engine (and the
earlier `erfcx` deep-OTM cdf tail) baked into the base, the price benches
return to ~0% on a normal PR. The 1.80 `(T)` ceilings #19 added were
scaffolding to let that one baseline-shifting PR through, and were explicitly
marked for reset in the next PR -- leaving them is the stale-threshold trap
the 2026-06 audit flagged: a 1.80 ceiling silently masks a real +50% price
regression.

Changes to PER_BENCH_THRESHOLDS in .github/scripts/check_perf.py:
- bsm_price_scalar, black76_price_scalar, bsm_price_vec: drop the 1.80 (T)
  overrides -> default 0.10, same as every other production bench.
- parallel_bsm_price: drop the 1.80 (T) override. criterion flattens
  `benchmark_group("parallel/bsm_price")` to the on-disk path
  `parallel_bsm_price`, so this explicit key DID gate those experiment benches;
  removing it tightens them to the default 0.10 (what the audit asked for). The
  sibling `parallel` key (0.50), by contrast, matched nothing -- no path starts
  with `parallel/` -- so it was the dead one; dropped too. The parallel
  serial-vs-rayon harness now runs on the default 10% (it is an experiment, not
  a production gate; widen a sub-bench with its own `parallel_<sub>` key if a
  noisy rayon arm flakes).
- cdf_branch_experiment, bsm_price_flag_dispatch: restore 1.80 (T) -> 0.50, the
  experiment-harness ceiling.
- Delete the ENGINE-REROUTE TRANSITION comment block; reframe the header comment
  around the head-vs-base principle so the table documents why it stays small
  instead of accreting transient exceptions.

Kept: iv_solve_scalar_atm (0.85) and iv_solve_vec (0.80); flagged in-comment as
head-vs-base-stale reset candidates, but out of scope here.

Verification: ruff check / ruff format --check / pyright green; a behavioral
test of threshold_for against the exact paths the perf-gate log prints; and the
perf workflow itself (it triggers on check_perf.py) runs head-vs-base over
identical Rust (~0% deltas) under the restored ceilings.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
@yipjunkai
yipjunkai force-pushed the ci/reset-perf-gate-ceilings branch from 5ae22d0 to bc8dbf6 Compare June 11, 2026 05:06
@yipjunkai
yipjunkai merged commit ff6f279 into main Jun 11, 2026
25 checks passed
@yipjunkai
yipjunkai deleted the ci/reset-perf-gate-ceilings branch June 11, 2026 05:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant