Skip to content

oracle: remove the historical Mall item files from the public tree (#100) - #101

Merged
DJAscendance merged 2 commits into
masterfrom
fix/mall-asset-remediation
Oct 29, 2026
Merged

DJAscendance merged 2 commits into
masterfrom
fix/mall-asset-remediation

Conversation

@DJAscendance

Copy link
Copy Markdown
Member

Summary

This PR removes the historical Mall item files from the public tree. They have no redistribution permission. They are the same files as in Ascendance3D/X_ITE_R#168.

removed path files bytes sha256
oracle/corpus-items/ 859 69,552,646 0a8127f2…b66f534 (tree listing)
oracle/cm5-cmd-ops-baseline.json (embeds about 4.2 MB of corpus text) 1 8,666,084 59a1d53e…27ed5

Primary issue: Closes #100.

  • oracle/private-inputs.tsv (new) lists both paths, with kind, file count, bytes and sha256. Both paths are gitignored.
  • cm-evidence::private_input (new) finds and verifies each input. It looks in $CM_PRIVATE_INPUTS/<name> when the variable is set. Otherwise it looks in oracle/<name>, then in the main worktree. A missing input fails in strict mode. In CI mode it skips only when CM_EVIDENCE_ALLOW names the input itself, and each skip is logged. A mismatched input always fails.
  • The cm-modes baseline check reads corpus files through the resolver. When the corpus is skipped, the cases and the non-corpus files still run. cmd_fuzz_oracle reads its baseline and its start documents through the resolver.
  • tools/evidence_summary.py accepts the keyed reason private-input-absent only for an allowed key. It always refuses private-input-mismatch.
  • No assertion changes, and no baseline is re-recorded.

Kept after a check: the oracle/corpus-items/... file rows in cm5-vrml-baseline.json and cm5-javascript-baseline.json hold a path, counts and sha256 digests only. docs/provenance/corpus-items-ffe080b.sha256 holds hashes and paths only. cm5-doc-ops-baseline.json and oracle/cm5-modes-inputs/ are synthetic.

Owner decision: new CI allow-list entries

CI has no copy of the private inputs. This PR adds two names to CM_EVIDENCE_ALLOW in .github/workflows/rust-tests.yml: corpus-items and cm5-cmd-ops-baseline.json. The CI run then skips by name, and logs:

  • vrml_tokens_match_js_vrml_mode: the 606 corpus files. Its 531 cases still run.
  • javascript_tokens_match_cm5_runmode: the 253 corpus files. Its 821 cases and 294 upstream files still run.
  • cmd_fuzz_oracle: the whole test, because its baseline is private.

The strict local suite still runs all three in full. Please approve or reject these entries. This PR is not merged until you decide.

Verification

  • Strict local suite, data present (tools/fetch-upstream.sh, then cargo test --workspace --no-fail-fast, CM_EVIDENCE unset): 803 passed, 0 failed, 0 ignored. This includes 3 new cm-evidence tests.
  • Data present: vrml 531 cases + 606 files, 0 mismatches. javascript 821 cases + 547 files, 0 mismatches. cmd fuzz 600 sequences, 22,189 steps, 0 mismatching (2 known). The same vrml result comes from CM_PRIVATE_INPUTS pointed at the private backup.
  • Data absent (standalone clone), strict: the 3 tests fail with evidence private-input-absent (corpus-items) or (cm5-cmd-ops-baseline.json). With the old CI allow list (no-gpu-adapter,tool-absent), they also fail.
  • Data absent, CI mode with the new entries: all pass. The log has 3 named lines. vrml runs 531 cases + 0 files, and javascript runs 821 cases + 294 files. tools/evidence_summary.py exits 0.
  • cargo fmt --all --check: clean.

Private evidence

A verified private backup is outside the repository: 860 files, 78,218,730 bytes, and every file is in its SHA256SUMS. It equals the Git blobs at 32f3cca, and it is byte-identical to the X_ITE_R #168 backup of the same files. It is not committed.

Limits

Removal from master does not erase the files from Git history. Older commits, forks, clones and GitHub caches still hold them. Erasure needs a history rewrite and separate owner authorization.

CLAUDE.md says "Keep the repository PRIVATE", but the repository is public. This PR does not change visibility or that rule.

Mandatory Reverse Engineer Anything (REA) Evidence — Not Applicable

This PR removes assets and changes how tests find their inputs. No CodeMirror behavior changes. The CodeMirror 5.65.21 pin, every baseline and every assertion are unchanged.

)

Removes oracle/corpus-items (859 files, 69,552,646 bytes) and
oracle/cm5-cmd-ops-baseline.json (8,666,084 bytes; it embeds about 4.2 MB
of corpus-items text). They have no redistribution permission
(Ascendance3D/X_ITE_R#168).

Both are rows of the new oracle/private-inputs.tsv and are gitignored.
cm-evidence gains private_input: $CM_PRIVATE_INPUTS/<name>, else
oracle/<name>, else the main worktree's; each is verified by kind, file
count, bytes and sha256 (the X_ITE_R answer-key measure). Absent: fails in
strict mode; skips in ci mode only when CM_EVIDENCE_ALLOW names the input,
and logs the skip. Mismatched: always fails.

The cm-modes baseline check reads corpus files through it (the cases and
the other files still run when the corpus is skipped); cmd_fuzz_oracle
reads its baseline and start documents through it. CI allows the two
inputs by name, and tools/evidence_summary.py checks the keyed reason.
No assertion changes; no baseline is re-recorded. A verified private
backup is outside the repository. Git history is not rewritten.
An input path whose symlink target does not exist was treated as absent, so ci mode skipped it. It now fails as a mismatch, as any unreadable entry does. Unit tests cover the broken symlink, an extra file and a file where a directory is wanted.
@DJAscendance

Copy link
Copy Markdown
Member Author

Independent review and owner decision (2026-10-29)

MiniMax: failed on its context limit. This is not a passed review.

Independent reviewer: a Claude subagent with a fresh context. It reviewed the 13 non-corpus files at e84695e4 and ran adversarial cases. Verdict: no blocker and no major finding.

# finding severity disposition
1 resolve_private_in treated a dangling symlink as absent, so ci mode skipped it minor fixed in 10f28bd2: a broken symlink or unreadable entry is now a mismatch, which always fails
2 cmd_fuzz_oracle returns when the corpus is absent, including the inline-text sequences minor accepted limit: logged, not silent; CI has neither input
3 resolver unit tests lack extra-file, wrong-kind and broken-symlink cases minor fixed in 10f28bd2 for those three cases
4 input names share the allow list with reason names nit latent; no skip("corpus-items") caller exists
5 walk ignores empty directories and opens non-regular files nit deferred; the corpus has neither
6 the main-worktree fallback trusts git worktree list nit harmless: content is digest-verified
7 oracle/cm5-cmd-ops.mjs and cm5-modes-tokens.mjs read oracle/corpus-items directly nit out of scope (re-record only)

Confirmed correct by running: strict mode fails on each absent input; a mismatch (extra file, wrong kind, truncated file, one changed byte) fails in every mode; the allow list matches names exactly; every permitted skip writes one log line; the original assertions are unchanged; the corpus digest reproduces with sha256sum.

Public CI coverage lost (logged skips, not passes): javascript_tokens_match_cm5_runmode and vrml_tokens_match_js_vrml_mode (corpus files only; 253 JavaScript corpus files and all 606 VRML files), and cmd_fuzz_oracle (all of it). CI on e84695e4 shows 803 cargo passes, of which 26 are logged skips: 3 private-input, 19 no-GPU, 4 tool-absent.

Owner decision: the owner approved corpus-items and cm5-cmd-ops-baseline.json in CM_EVIDENCE_ALLOW as temporary exceptions.

Mandatory Reverse Engineer Anything (REA) Evidence — Not Applicable: asset removal and test input resolution only. No JavaScript-to-Rust behavior changed. The CodeMirror 5.65.21 pin in tools/reference.lock is unchanged.

@DJAscendance
DJAscendance merged commit 91899af into master Oct 29, 2026
1 of 3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Remove the historical Mall item files from the public tree

1 participant