Improve agent-memory evidence and benchmark integrity - #225
Conversation
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d244b8b811
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 62a7a474fe
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 692efe5474
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 192cc7e3f4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b65f8dd9f2
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
You are seeing this message because GitHub Code Scanning has recently been set up for this repository, or this pull request contains the workflow file for the Code Scanning tool. What Enabling Code Scanning Means:
For more information about GitHub Code Scanning, check out the documentation. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0989033b27
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c56da42ad9
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9e8ffde57f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 549012095b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 98e8f5c241
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a19a8078b4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f17de56906
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 833e3e922e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c3a86295f5
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c35d840a1e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: dd74eb40d9
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3d7f5fbfd8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: dabc9a1cfe
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 950c9c9a34
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 95bff91362
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2be5953da5
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
2 participants
Engraphis can currently expose an exact value after packing has omitted a condition from its source. This PR requires the complete meaningful source before advertising an exact binding, preserving literal bytes and original offsets in any language. Coverage withholds bound occurrences that cannot fit; legacy retains its selected text but clears incomplete binding metadata. Content corrections require explicit replacement bindings, and compact responses keep bindings only with admitted source context.
The PR also makes benchmark results reproducible and attributable: datasets, fixtures, oracles, reports and charts use verified byte snapshots; producer names are unique; scope, trust, case identity, retry receipts and observed failure usage remain consistent across fresh and resumed runs. Checkpoint locking prevents competing execution and permits explicit recovery after process death. Campaign pricing uses disjoint ordinary/cached/written input categories and a single final rounding step. Recall receipts retain the selected recipe, effective depth and packing mode, with strict validation of persisted mode labels.
Local queue and capacity locks now release automatically on process death while retaining interrupted receipts. Completed jobs remain cached, and interrupted jobs still require explicit reconciliation. Upstream diagnostics are verified while holding their released producer lock; persistent markers no longer cause an endless wait. Prerequisite paths stay within the repository, and pre-open marker disappearance or replacement fails closed. Real-process tests cover live exclusion, forced termination, preserved receipts and cached resume.
Benchmark analysis also rejects malformed scoring flags and binds repair digests to the identified manifest, including agreement with retained dataset-integrity metadata. Chart transport claims require actual JSON booleans. All 36 retained benchmark envelopes and eight external diagnostics remain compatible; the focused analysis and renderer suite passes 192 tests.
External analysis requires finite rates within [0,1], including when invalid rows agree with their aggregate. Capacity completion requires 24 unique cell identities and an exact matching hash index. Paid campaign and continuation runners use a shared persistent OS-held lock, preserving unfinished reservations, budget approval, single-child allocation and cached no-dispatch behavior after process death. The shared lock implementation is included in new frozen campaign source snapshots.
Campaign preparation derives public execution metadata locally and rejects all caller-supplied configuration overrides before binding. A bounded CLI-version grammar prevents warning banners or arbitrary process output from entering public manifests. Preparation and manifest loading share strict field, type, basename, digest and timeout validation; the three retained OAuth campaign manifests remain compatible. The CodeQL finding also exposed a name-based metadata classification; helpers now describe execution metadata accurately. No security query, gate or suppression was changed.
Provider usage is counted only after validating the complete TokenUsage schema, finite nonnegative counters, token relationships and consistent invocation metadata. Empty or malformed checksummed checkpoints cannot publish complete zero-cost accounting. Fresh failures preserve valid measurements and explicit missingness through terminal resume; legacy unknown rows stay distinct from proven zero invocations. Explicit zero-call rows cannot be scored complete. Reserved estimates remain distinct from observed usage. The combined campaign, continuation, API, ledger, audit, metadata, lock and CodeQL regression group passes 236 tests without provider calls.
Campaign history mutations now enforce the exact effective workspace, repository, session and scope. Logical child labels resolve under the declared parent; foreign or unknown physical child IDs fail closed. Core correction lineage uses mapped memory IDs, retaining fixture labels separately. Duplicate write IDs, missing targets, invalid intervals and backdated or already-closed corrections fail validation before the first memory write. Repeated invalidation preserves the earliest close. Memory creation and its validity closures use the existing engine transaction hooks, with target revalidation under the transaction and rollback on closure failure. The canonical ledger follows the same scope and temporal rules. All 400 retained ledger snapshots are unchanged; real-engine replay validates 1,040 stored records and all 80 correction chains. These are offline integrity checks, not new answer-quality scores or a promise of whole-batch rollback on arbitrary backend failures. Focused campaign and replay validation: 423 passed, 1 skipped in 22.95s.
Candidate code could forge a passing oracle result through stdout or shutdown hooks. A versioned expression interpreter now treats candidate source as data, exposes only primitive operations, and reserves stdout for one trusted result frame. Expected values remain in the host. Unsupported syntax/resource limits are unscored and cannot trigger corrections. Both Docker campaigns and the local corpus helper use the same contract; reader instructions disclose it and producer snapshots bind it. All 400 v1 and 400 generated v2 deterministic repairs pass, while stale fixtures remain incorrect. Historical campaign scores are unchanged; new execution requires a newly frozen manifest. Focused campaign validation passes 456 tests with one Docker-dependent skip.
Measured development fixtures:
These comparisons use identical inputs and budgets against
c3a86295. The three tight query-retention losses remain explicit. The earlier five unpunctuated retention fixtures also remain 0/5 versus 5/5 on98e8f5c2; omission is not counted as retention. Prior safety populations remain 5/5, 8/8 and 16/16; all 14 action and four source-integrity cases pass. A separate 35-token packing fixture retains three sources in 33 tokens with coverage versus one in 35 with legacy. The CI gate independently checks rendered token counts, full binding/span retention, suppression and roomy controls.Evidence is retained in
evidence-contracts-20260921-v11.json,offline-fixtures-v70.jsonanduser-journeys-20260921-v4.json. The offline registry binds 579 producer files; all 122 historical artifact checksums verify. All seven local user journeys pass with 116 unique producer identities. Historical results and run provenance remain unchanged.Validation at
655fb8ee: campaign accounting, continuation, audit, API, checkpoint recovery, compact REST/MCP binding, source identity, and evidence/documentation regressions pass, along with Ruff, Pyright 1.1.414, all eight offline evaluation gates, commercial boundary checks, and dashboard asset checks. Full local suite: 6,470 passed, 63 skipped. GitHub: 29/29 checks pass on this head.These are deterministic development and functional results, not new external QA, provider billing, deployment or untouched-holdout qualification. Production retrieval and packing defaults remain unchanged. The synchronized Railway fixes from #224 are included; dependency updates remain in #226, #227 and #228.