Conversation
…, never a safe fold (#679) An off-enum corrected finding ('Probably Fine', a model-drift spelling, or a corrupted 'error' string) fell into the plain-disagreed arm — which the scanner folds into safe (safe += disagreed) — so every operator surface read the unit as safe while the recount ladder counted it as an error. The false-clean class: the envelope, the recount, and the report disagreed about the same row. The fix, consumer-side per the maintainer's preference (no new counter): - verifier: an unparseable corrected finding reaches error_count BEFORE the agreement arms — agreed or not, an unclassifiable verdict is a visible error. The existing error-marker path (a top-level r['error']) is unchanged. - report (cli.py): the unknown-verdict rows join a visible ERROR group (a local display group — the canonical enum is untouched) and the false-clean remediation message is suppressed when unparseable rows exist ('N unit(s) have an unrecognized verdict'). - the scanner needs no change: with the off-enum no longer 'disagreed', the safe fold never sees it. RED->GREEN: test_off_enum_corrected_counts_as_error_never_disagreed and test_off_enum_disagreement_is_not_a_false_positive_eliminated FAIL on pristine 6240682, PASS at HEAD. The sibling suites (#622/#623/#286) hold. Full suite: 4273 passed / 2 failed (the pre-existing SDK-env pins, reproduced identically at the merge-base). Fixes #679 Agent: ISSUES-TO-PR ses_f3e7372a5ffe3b2I6lP4Wm3w2d
… safe fold (#679) An off-enum corrected finding ('Probably Fine', a model-drift spelling, or a corrupted 'error' string) fell into the plain-disagreed arm — which the scanner folds into safe (safe += disagreed) — so the envelope read safe while the recount read error (the false-clean class: every surface disagreed about the same row). The fix, consumer-side per the maintainer's preference (no new counter): - verifier: an unparseable corrected finding reaches error_count BEFORE the agreement arms — agreed or not, an unclassifiable verdict is a visible error. The error-marker path (a top-level r['error']) is unchanged. - report (cli.py): the visible not-classified group carries BOTH the canonical error rows and the off-enum spellings (the T1 F1 split: the canonical 'error' is RECOGNIZED — the wording distinguishes 'N errored' from 'M with an unrecognized verdict', the html_report convention); the false-clean message is suppressed when either exists. Both via the extracted helpers _unrecognized_verdict_rows + _remediation_for_unrecognized. - the scanner: unchanged — the off-enum no longer lands in disagreed so the safe fold never sees it. The stale #622-era docstrings updated (verifier: the residual arm is now safe-only; scanner: same). RED->GREEN: the off-enum counter tests FAIL on pristine 6240682, PASS at HEAD; the real-helper tests drive the extraction (the T1 F5 fix: the for-pass stub removed). 19 tests in the harness file; the sibling suites (#622/#623/#286) hold; the full suite 4273/2 (the SDK-env pins, identical at base). Declared residues (the T1 round): the Go chart/stats still iterate the canonical verdict_order only (the error group is visible in the findings list but not the pie — the html_report standalone already charts errors; the gap is the Go renderer's, named for the follow-up); the html_report.py standalone is direct-invocation-only (no production callers) and partially handles the off-enum message already. Fixes #679 Agent: ISSUES-TO-PR ses_f3e7372a5ffe3b2I6lP4Wm3w2d
gadievron
requested review from
dgeyshis,
shahar-davidson and
sounil
as code owners
September 26, 2026 20:37
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
…e CSV exports the Stage-1 verdict — the counter gains the consistency_{protected,safe,inconclusive} buckets (gated on the record the apply loop writes), the envelope folds them (extracted _post_verify_metrics), the CSV reads the consistency_update.from AFTER the note parse (the F1 precedence: on disagreed-then-rewritten rows the record's from is Stage-2's correction); the VerifyResult serialization threads the counts; the #622 suite honestly narrowed
gadievron
force-pushed
the
fix/681-682-consistency-provenance
branch
from
September 26, 2026 20:43
ece77c3 to
35b6ca3
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A Stage-2 consistency reply that volunteers
findings_to_updatecan rewrite an agreed row's finding (e.g. vulnerable -> protected) — and the rewritten row previously reached no scanner-envelope bucket (#681): the envelope'sprotected = analyze.protected + disagreed_protectedcounts neither term (the analysis metrics counted the row vulnerable at Stage 1; the rewrite isagree=True, not a disagreement), so the envelope said protected=0 while the recount (which reads the finding field) said protected=1 — two numbers across three surfaces, the envelope the outlier. And the CSV'sstage1_verdictcolumn exported the rewritten finding as the Stage-1 verdict (#682).The fix (the #654-named shape): source the envelope from the buckets it already owns.
core/verifier.py—_count_verification_outcomesgains the consistency-rewrite buckets by destination (consistency_protected/consistency_safe/consistency_inconclusive), gated on theconsistency_updaterecord the apply loop already writes (never a guess from the verdict delta; a disagreed row never lands in them).core/scanner.py— the post-verify envelope (extracted into_post_verify_metrics, one testable authority) folds the buckets into their columns — the envelope and the recount agree; the sum-to-total property restored.core/schemas.py—VerifyResultcarries the counts;to_dict()andverify_step_summary()serialize them (the standaloneverifysurface agrees too).report/csv_export.py—get_stage1_verdictreads theconsistency_update.fromprovenance the verifier already records — after theChanged fromnote parse (the ordering matters: on a disagreed-then-rewritten row the record'sfromholds Stage-2's correction, and the note — which always carries the true Stage-1 original — must win; the review's F1 caught the inverse ordering before it shipped).The tests
test_issue681_consistency_envelope.py— the bucket lattice: the rewritten row reaches its bucket (agreed + the destination bucket, never double-counted), the plain-agreed/disagreed controls touch no new bucket, the envelope folds (_post_verify_metricsdriven directly).test_issue682_csv_stage1_verdict.py— the five shapes: agreed+rewritten (the from), disagreed-then-rewritten (the note wins), the no-rewrite and note-path controls, the The JSON rescue schema offers a verdict enum the analysis prompt never does, and INSUFFICIENT_CONTEXT hits four consumers that disagree (errors / completed / dropped / absent) #623 verbatim-spelling guard.Declared notes (honest)
repro: fixer-inferred— the failure shape was inferred from the issue's mechanism, not an issue-supplied repro.558381b, PR fix(verify): an off-enum Stage-2 verdict is a visible error, never a safe fold (#679) #749) — merge fix(verify): an off-enum Stage-2 verdict is a visible error, never a safe fold (#679) #749 first, then this.Fixes #681, fixes #682. Cross-linked: #654 (the named fix), #622 (the sibling bucket), #679 (the stack base).