ci: reset transitional perf-gate ceilings after engine reroute landed#22
Merged
Conversation
#19's normalised-Black price engine is now on `main`, i.e. the perf gate's base ref. The gate compares PR-head vs PR-base, so with the engine (and the earlier `erfcx` deep-OTM cdf tail) baked into the base, the price benches return to ~0% on a normal PR. The 1.80 `(T)` ceilings #19 added were scaffolding to let that one baseline-shifting PR through, and were explicitly marked for reset in the next PR -- leaving them is the stale-threshold trap the 2026-06 audit flagged: a 1.80 ceiling silently masks a real +50% price regression. Changes to PER_BENCH_THRESHOLDS in .github/scripts/check_perf.py: - bsm_price_scalar, black76_price_scalar, bsm_price_vec: drop the 1.80 (T) overrides -> default 0.10, same as every other production bench. - parallel_bsm_price: drop the 1.80 (T) override. criterion flattens `benchmark_group("parallel/bsm_price")` to the on-disk path `parallel_bsm_price`, so this explicit key DID gate those experiment benches; removing it tightens them to the default 0.10 (what the audit asked for). The sibling `parallel` key (0.50), by contrast, matched nothing -- no path starts with `parallel/` -- so it was the dead one; dropped too. The parallel serial-vs-rayon harness now runs on the default 10% (it is an experiment, not a production gate; widen a sub-bench with its own `parallel_<sub>` key if a noisy rayon arm flakes). - cdf_branch_experiment, bsm_price_flag_dispatch: restore 1.80 (T) -> 0.50, the experiment-harness ceiling. - Delete the ENGINE-REROUTE TRANSITION comment block; reframe the header comment around the head-vs-base principle so the table documents why it stays small instead of accreting transient exceptions. Kept: iv_solve_scalar_atm (0.85) and iv_solve_vec (0.80); flagged in-comment as head-vs-base-stale reset candidates, but out of scope here. Verification: ruff check / ruff format --check / pyright green; a behavioral test of threshold_for against the exact paths the perf-gate log prints; and the perf workflow itself (it triggers on check_perf.py) runs head-vs-base over identical Rust (~0% deltas) under the restored ceilings. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
yipjunkai
force-pushed
the
ci/reset-perf-gate-ceilings
branch
from
June 11, 2026 05:06
5ae22d0 to
bc8dbf6
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Resets the six
(T)1.80 perf-gate ceilings that #19 bumped for the normalised-Black engine baseline shift. Touches only.github/scripts/check_perf.py(PER_BENCH_THRESHOLDS).bsm_price_scalar(T)black76_price_scalar(T)bsm_price_vec(T)parallel_bsm_price(T)cdf_branch_experiment(T)bsm_price_flag_dispatch(T)paralleliv_solve_scalar_atmiv_solve_vecThe
ENGINE-REROUTE TRANSITIONcomment block is deleted and the header comment reframed around the head-vs-base principle, so the table documents why it stays small rather than accreting transient exceptions.Why
The gate compares PR-head against PR-base. Once #19 merged, both the engine reroute and the earlier
erfcxdeep-OTM cdf tail live in the base, so a normal PR's price benches return to ~0%. The 1.80(T)ceilings were scaffolding to let that one baseline-shifting PR through — #19's own comment marked them "MUST be reset … in the next PR". Leaving them is precisely the stale-threshold trap the 2026-06 audit flagged: a 1.80 ceiling silently masks a real +50% price regression.Note:
parallelvsparallel_bsm_price(corrected from the gate's own log)criterion flattens
benchmark_group("parallel/bsm_price")to the on-disk pathparallel_bsm_price(the/in a group name collapses to_). So:parallel_bsm_pricekey did gate those serial/rayon experiment benches (at 1.80). Removing it tightens them to the default 0.10 — what the audit asked for.parallelkey (0.50) matched nothing (no path starts withparallel/) — it was the genuinely dead one. Dropped.The whole
parallel_*serial-vs-rayon harness now runs on the default 10%. These are experiment benches, not production gates (production price/greeks/iv are gated bybsm_price_vec/bsm_greeks_all_vec/iv_solve_vec). The in-file comment documents how to widen a specific noisy sub-bench (its ownparallel_<sub>key) if needed — open question for review: do we want the experiment harness back at 0.50 to avoid flaky-gate risk on the noisy rayon-small-N arms? (the original design intent, defeated all along by the dead key).Kept (deliberately out of scope)
iv_solve_scalar_atm(0.85) andiv_solve_vec(0.80) are head-vs-base stale by the same logic — the LBR cost is in the base — and are flagged in-comment as future reset candidates. Left as-is here.Verification
ruff check .,ruff format --check,pyright— all green locally (the three CI lint gates).threshold_foragainst the exact paths the perf-gate log prints: price +parallel_*→ 0.10, experiments → 0.50, IV overrides intact, bothparallel/parallel_bsm_pricekeys gone.perfworkflow triggers oncheck_perf.py, so the gate runs head-vs-base over identical Rust (~0% deltas) under the restored ceilings — confirmed green.🤖 Generated with Claude Code