Repository navigation
evidence_report: Db2 det verdicts from the coverage ledger, kept (stale) across push refreshes (#4758) - #4761
Merged
Conversation
…tale like its coverage (#4758) The refresh job rewrote the det-sweep coverage ledger from both sweeps but built the report with no sweep, so every Db2 case stayed "not run" (L1) even after the db2 job proved it. The ledger holds an entry only for a case its last sweep proved; the report now takes "equal" for a Db2 case with no sweep row and no det_sweep_baseline entry from that entry while it is readable (current, or stale on harness / oracle only), so a push refresh that skips Db2 keeps the last Db2 level, marked stale per #4730, instead of reverting it to L1. A local sweep still takes precedence. - det_coverage_ledger update drops the entry of a case a sweep ran and did not prove (cases it did not run are kept), making "entry = last sweep proved it" an invariant. - A LINKed run never takes its case's (main program's) coverage (#4270): it is L2 at most, stale with the ledger verdict it rests on. Previously masked because no LINKed Db2 run was ever equal. - evidence-refresh.yml: comment only. Fixes #4758 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MQNoe4wPJgr7dbUvs3DG6D
Contributor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #4758
Root cause
The
refreshjob passes both sweeps (non-Db2 + Db2) todet_coverage_ledger.py update, butevidence_report.py --refresh --livetakes no sweep (the report's sweeps are the frozenmeasuredblock). For a Db2 case with no sweep row and nodet_sweep_baseline.jsonentry,proof_blockers.judge_equivalenceadds "not proven in CI", which the report rendered as "det port not run" -> L1, although the ledger already held that case's coverage.Design (persistent, ledger-based)
det_coverage()/last_coverage()from Evidence report: keep the last measured level, marked stale, on harness / oracle staleness #4734: current, or stale on harness/oracle only). A blocking change (case, corpus) makes it "not run" again.L3*) once its inputs move (evidence report: keep a program's last measured level marked stale instead of dropping L3/L4 to zero on any harness change #4730). A local sweep row still takes precedence (unchanged path).det_coverage_ledger.py updatenow drops the entry of a case a sweep ran and did NOT prove (cases it did not run are kept), so "entry = last sweep proved it" is an invariant, not a convention.Local verification (
--refresh --livelevels, not committed)With the #4756 Db2 ledger, fingerprints refreshed to this tree (= what a full refresh at HEAD writes):
The #4756 ledger as-is against this tree (harness moved since: a push run): the same levels, every Db2-derived one marked stale (
*), e.g. GenApp lgacus01 L1 -> L3* (3/3 paragraphs, 4/8 branches), UPDACC L1 -> L4*. Main's current ledger gives the same histogram, all Db2 levels stale.Note: once this lands,
--check --live(release gate) will list stale Db2 levels until the next nightly full refresh makes them current -- intended #4730 behaviour (they were L1 before).Tests
tests/tools/test_evidence_report.py: Db2 with no sweep/no entry stays "not run"; Db2 verdict from a sweep input (L3, local sweep); persistence from the ledger on a run without Db2 (L3 current); stale ledger entry ->L3*with stale inputs; baseline/sweep outrank the ledger; LINKed program never inherits coverage and is stale with its ledger verdict.tests/tools/test_det_coverage_ledger.py: update drops a not-proven swept case, keeps an unswept one. 106 passed (1 skip: the advisory committed-reports check).Gates:
pr_gates --fast5/5 pass;--ratchets6 pass,portsFAIL from the shared cics-crucible checkout being off its v0.6.0 pin (environment; no ports/generator code touched).🤖 Generated with Claude Code
https://claude.ai/code/session_01MQNoe4wPJgr7dbUvs3DG6D