Repository navigation
test(mcp): the repeat-reads A/B and timing tests stop flaking - #56
Merged
Merged
Conversation
…timing test alternates its arms and compares medians The A/B test read 16 files and needed three in each arm, which fails 0.4% of runs because the arm comes from the path and the paths are new every run. It now takes files until each arm has four, and checks that the arm a file lands in is the arm the read was answered as. The timing test ran all reads of one arm, then all of the other, so a load spike hit one arm only, and a single stalled read decided the result. It now alternates the arms read by read and compares medians. A repeat path that costs 400 ms more a read still fails it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Two tests in
crates/lumen-mcp/tests/repeat_reads.rsfail without anything being wrong in the product. Test-only change; no product code.A/B split (
ab_withholds_the_answer_from_control_files_and_records_what_it_would_have_done): it read 16 files and needed at least 3 in each arm. The arm is a function of the path, and the paths are new on every run (temp dirs), so with 16 fair coin flips one side gets fewer than 3 in 274 of 65,536 outcomes: it fails 0.42% of runs for any code.Timing (
a_repeat_costs_no_more_time_than_a_first_read): it ran 40 reads with repeat reads off, then 40 with them on, and compared the two totals against1.5 x off + 1 s. A load spike lands on one arm only, and one stalled read can decide the result. It failed in a local gate at machine load 77.Change
lumen_core::repeat::treated(Mode::Ab, path), the function the product uses: a treated file gets the "Unchanged since" answer, a control file the outline. The test now fails when the split is wrong, not when it is uneven.on_median < 2 x off_median + 25 ms. Load lands on both alike, and one stalled read moves no median.Evidence
Gate
Local gate green: fmt, clippy
-D warningson the workspace with all targets, workspace tests withRUSTFLAGS=-D warnings, frontend build and tests with thresholds, e2e (read path, ingest path, decision metering), sidecars, Tauri crate tests and clippy, hook e2e.Not run locally, because only a test file changed: the debug app build, install verify, the full MSRV build.
cargo check -p lumen-mcp --tests --lockedon 1.94.0 passes; CI's MSRV job runs the full build.No issue was opened for this flake; say if you want one.