Skip to content

feat(evaluation): add offline evaluation workflow for Memory write gate - #2001

Merged
hidb4ai merged 7 commits into
oceanbase:masterfrom
lizining1231:docs/memory-write-gate-promotion-decision
Oct 15, 2026
Merged

hidb4ai merged 7 commits into
oceanbase:masterfrom
lizining1231:docs/memory-write-gate-promotion-decision

Conversation

@lizining1231

Copy link
Copy Markdown
Contributor

Which issue or RFC does this PR close?

Closes #1930. Follows RFC #1770 and PR #1921.

Rationale for this change

The Memory write gate needs reproducible evaluation and measured evidence before promotion to enforcing. This adds an offline evaluation workflow and reports Jev measurements supporting continued shadow/advisory operation for the evaluated configuration.

What changes are included in this PR?

  • Add a read-only evaluator using current Scope authorization, exact Artifact revisions and citation checks. Report unavailable inputs separately from negative judgments and preserve used_fallback=true => coverage=unadjudicated.
  • Prepare 600 calibration and 1,800 held-out English/Chinese cases from pinned XNLI data, with documented labels and sampling rules. Keep corresponding translations in the same split.
  • Evaluate public inputs through the existing Jev adapter and reuse Runtime scoring rules. Compare both hold directions and four confidence thresholds, then freeze the selected configuration before held-out evaluation.
  • Report coverage, false holds, false passes, fallback reasons, latency, usage/cost and HOLD/FLAG frequency. Preserve bounded checkpoints and validate code/data/model identities when reusing calibration results.
  • Add necessary behavior tests, usage instructions and one report covering measurements, the promotion recommendation and rollback steps. Keep raw data, expanded manifests and run records outside Git.

Are there any user-facing changes?

Contributors can run input-coverage checks and Jev evaluation from a checkout. Existing Memory writes, policy modes and public APIs are unchanged.

How was this change tested?

  • 177 focused tests passed, covering authorization, exact revisions, missing inputs, fallback behavior, dataset separation, scoring and calibration reuse:
    uv run --no-sync pytest tests/evaluation tests/examples/systemone tests/builtin/runtime/test_memory_write_gate_contract.py tests/e2e/test_memory_write_gate_evaluation.py -q
    
  • Live Jev 1.13 evaluation used 600 calibration and 1,800 held-out cases. The frozen configuration was hold_on=yes;threshold=none. Held-out results included 107/595 false holds (17.98%), including 82/299 Chinese false holds (27.42%), and 176/1,200 false passes including fallback (14.67%).
  • Last-attempt P50/P95 latency was 0.706/1.380 seconds; known evaluation cost was USD 0.049940184. Missing usage prevents a complete cost/token total;
  • make check passed, including lock verification, pre-commit checks, type checks and 49 integration-manifest tests. Changed-file checks and git diff --check passed.

AI usage statement

OpenAI Codex ChatGPT 6.1 Sol assisted implementation and documentation

Define exact references and resolve inputs with current Scope authorization.
Generate pinned bilingual calibration and held-out cases with documented
label mappings and independent groups. Keep expanded inputs outside Git.

Refs oceanbase#1930
Reuse Runtime gate semantics to compare mappings and confidence thresholds.
Report incomplete input coverage separately from negative judgments and
measure error denominators, fallback, latency, known cost and sample friction.

Cover authorization, immutable revisions and fallback behavior with tests.

Refs oceanbase#1930
Evaluate pinned public inputs through the existing Jev adapter and expose
the calibration and held-out workflow. Read frozen calibration records with
matching data, model and assessment-code identities.

Keep private inputs outside hosted calls and run outputs outside Git.

Refs oceanbase#1930
Document configuration comparisons, held-out measurements and evidence gaps.
Recommend retaining shadow/advisory for the evaluated configuration and
provide operational rollback.

Refs oceanbase#1930
@CLAassistant

CLAassistant commented Oct 14, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@hidb4ai hidb4ai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The private-manifest split validation accepts duplicate evaluation inputs across calibration and held-out, which undermines the independence of held-out measurements. Please reject overlapping inputs as described inline.

Comment thread evaluation/memory/write_gate/dataset.py

@hidb4ai hidb4ai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@hidb4ai
hidb4ai merged commit 55a8160 into oceanbase:master Oct 15, 2026
36 checks passed
@hidb4ai hidb4ai added the 1.4.0 v1.4.0 release tracking label Oct 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

1.4.0 v1.4.0 release tracking

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(runtime): evaluate Memory write-gate promotion evidence

4 participants