Repository navigation
Conversation
Define exact references and resolve inputs with current Scope authorization. Generate pinned bilingual calibration and held-out cases with documented label mappings and independent groups. Keep expanded inputs outside Git. Refs oceanbase#1930
Reuse Runtime gate semantics to compare mappings and confidence thresholds. Report incomplete input coverage separately from negative judgments and measure error denominators, fallback, latency, known cost and sample friction. Cover authorization, immutable revisions and fallback behavior with tests. Refs oceanbase#1930
Evaluate pinned public inputs through the existing Jev adapter and expose the calibration and held-out workflow. Read frozen calibration records with matching data, model and assessment-code identities. Keep private inputs outside hosted calls and run outputs outside Git. Refs oceanbase#1930
Document configuration comparisons, held-out measurements and evidence gaps. Recommend retaining shadow/advisory for the evaluated configuration and provide operational rollback. Refs oceanbase#1930
hidb4ai
requested changes
Oct 14, 2026
hidb4ai
left a comment
Contributor
There was a problem hiding this comment.
The private-manifest split validation accepts duplicate evaluation inputs across calibration and held-out, which undermines the independence of held-out measurements. Please reject overlapping inputs as described inline.
AlexStocks
approved these changes
Oct 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue or RFC does this PR close?
Closes #1930. Follows RFC #1770 and PR #1921.
Rationale for this change
The Memory write gate needs reproducible evaluation and measured evidence before promotion to
enforcing. This adds an offline evaluation workflow and reports Jev measurements supporting continuedshadow/advisoryoperation for the evaluated configuration.What changes are included in this PR?
used_fallback=true => coverage=unadjudicated.Are there any user-facing changes?
Contributors can run input-coverage checks and Jev evaluation from a checkout. Existing Memory writes, policy modes and public APIs are unchanged.
How was this change tested?
hold_on=yes;threshold=none. Held-out results included 107/595 false holds (17.98%), including 82/299 Chinese false holds (27.42%), and 176/1,200 false passes including fallback (14.67%).make checkpassed, including lock verification, pre-commit checks, type checks and 49 integration-manifest tests. Changed-file checks andgit diff --checkpassed.AI usage statement
OpenAI Codex ChatGPT 6.1 Sol assisted implementation and documentation