Repository navigation
feat(flaky): auto-prune, Java wiring, live rerun smoke - #83
Merged
Merged
Conversation
Build the three pending flake-triager follow-ons from the spec's Not-yet list:
- Auto-prune: new 'prune' mode re-arms quarantined tests gone stable. gate now
appends a per-run pass/fail line per quarantined test that ran to
.foundry/flaky-streaks.jsonl (best-effort, never alters the exit code); prune
<baseline> [--min-green M] removes ids with >= M trailing consecutive greens (a
fail resets). Removal-only, so the snooze guard passes a prune label-free. Fed by
pre-push /ship (persistent .foundry/); CI needs .foundry/ cached to contribute.
- Java wiring: parse_report now accepts a DIRECTORY of JUnit XML (Gradle's
build/test-results/test/TEST-*.xml), merged; mise/java.toml gains the Java
test:flaky task + the gradlew --tests classify recipe (globs the report dir, fails
if empty; notes parameterized bracket ids don't round-trip).
- Live rerun smoke: tests/test_foundry_flaky.py (16 cases) drives the real
subprocess rerun loop with a synthetic flipping command and a spaced id (proving
the {test} quoting contract), plus parse/gate/streak/prune coverage. Runner-filter
semantics (vitest -t, gradlew --tests) remain unverified - documented.
Design doc updated (Implemented/Not-yet + the honest caveats). Full tests/: 36 pass.
…le XML - prune reports quarantined ids never seen in the streak log as 'stale' (renamed/ deleted/never-run dead debt) for human review, without auto-removing them. - prune writes flaky-baseline.json with newline="\n" so a Windows-run prune can't emit a phantom CRLF diff on a committed ratchet file; consumers carry flaky-baseline.json text eol=lf (documented). - java test:flaky rm -rf build/test-results/test before gradlew: '|| true' swallows a compile failure, and Gradle leaves per-class XML behind, so stale results could otherwise pass the glob and gate green. Also hoist 'import glob'; fix designs/ -> docs/designs/ drift in my comment; design-doc caveats (streak accrues only once test:flaky is folded into test). tests/: 39 pass (3 new).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Builds the three pending follow-ons from
docs/designs/flake-triager.md's own "Not yet" list.What's here
foundry-flaky prune <baseline> [--min-green M], default M=10): re-arms quarantined tests gone stable.gatenow appends a per-run pass/fail line for each quarantined test that ran to.foundry/flaky-streaks.jsonl(best-effort; never changes the exit code).pruneremoves any id with ≥ M trailing consecutive greens — a single fail resets the streak. Removal-only, so ruleset-guard'ssnoozekind passes a prune label-free.parse_reportaccepts a directory of JUnit XML (Gradle'sbuild/test-results/test/TEST-*.xml, merged);mise/java.tomlgains the Javatest:flakytask + the./gradlew test --tests "{test}"classify recipe.tests/test_foundry_flaky.pydrives the real subprocess rerun loop with a synthetic flipping command and a spaced id (proving the{test}quoting contract), plus parse / gate / streak / prune coverage. Fulltests/: 39 pass.Reviewer caveats
gateruns against the baseline — i.e. once a consumer foldstest:flakyintotest./shiprunsgate; CI contributes to a streak only if.foundry/is cached across runs. Not automatic — documented.{test}subprocess contract is verified;vitest -t/gradlew --testsbehavior against a real suite is still unverified (no JVM/node project here).prunereports (never auto-removes)stalequarantines (renamed/deleted tests that can never reach the threshold); writes the baseline LF to avoid a Windows phantom-CRLF diff (consumers carryflaky-baseline.json text eol=lf); the Java task clears stale XML before the run so|| truecan't gate green on a prior run's results.method[1]) only glob-match--testsand don't round-trip — noted in the task, not engineered for.Independent of #81 (arch adoption); both branch off the same
origin/main.