DST harness — generative simulation suite + shared omnigraph-dst crate + cross-backend walk (tests-only; consolidates #303 + #305)

[ragnorc]

Consolidated, TESTS-ONLY DST harness. Formerly the stacked PRs #303 → #305 → #309; retargeted to main as a single reviewable unit since all three carry zero production-code changes. The one engine-src change in the effort — the deterministic-replay seam — is intentionally kept separate in #304 so the only production change gets isolated review.

Supersedes (closed): #303 (comprehensive harness), #305 (fuzz + S3 context). Stays separate: #304 (determinism seam, engine src).

What's in here (all under crates/omnigraph/tests/dst* + the new crates/omnigraph-dst)

• Generative DST harness — seeded op-alphabet walk + white-box invariant battery after every op (HEAD==manifest, Dataset::validate, unique live _rowid [RC-X class], scalar-index probe, count/content/edges==model, @key uniqueness) + a structured known/novel classifier. Pins the known bugs as characterization regressions (RC-1, RC-X, dup-@key/MR-714, self-loop-not-traversable).
• Fault injection (FaultAdapter), proptest-state-machine shrinking, a D3×D2 read-shape battery, a concurrent multi-actor walk, failpoint recovery cells (dst_recovery, incl. the fix(recovery): converge roll-forward on concurrent manifest advance #296 concurrent-opens-converge cell), parser/loader fuzz, and an env-gated S3-context battery.
• crates/omnigraph-dst (dev-only, not a default-member) — the reusable harness behind a black-box Backend trait (Embedded + Cli), enabling…
• cli_cross_backend_walk — the same seeded walk driven against the embedded SDK AND the omnigraph CLI subprocess, asserting per-step black-box agreement.

The dev-dependency cycle (engine dev-dep → omnigraph-dst → engine) is supported by cargo (lib builds without dev-deps; validated). White-box battery is embedded-only by construction; the CLI arm runs the black-box oracles.

Review findings addressed

• count==model masking lost-writes — the Person message carried the dup-@key marker that classify() allow-lists, so a genuine lost write was recorded as known MR-714 instead of failing. Marker dropped; real dup detection stays in check_content + no_duplicate_keys.
• swallowed errors/panics — the concurrent walk now classifies actor JoinError panics, and s3_battery_holds classifies op errors, instead of discarding (both could false-pass).
• S3 battery never ran in CI — added a step to the rustfs_integration job + the dst paths to the RustFS change filter (the env-gated skip itself is by-design).
• stale MATRIX recovery row + FaultAdapter scope wording tightened (the seam is the StorageAdapter conditional text-object write, not the Lance manifest-publish CAS — that's dst_recovery's failpoints).

Verification

• cargo test -p omnigraph-engine --test dst → 16 passed.
• cargo test -p omnigraph-engine --features failpoints --test dst_recovery → 2 passed.
• cargo test -p omnigraph-cli --test cli_cross_backend_walk → 1 passed.
• Crate builds clean; no cargo cycle; CI YAML validated.

Note for merge: #304 (determinism) is based on #303's commits; once this lands on main, #304 needs a rebase onto main (its #303-based commits become redundant).

---

Note

Low Risk
Changes are confined to test crates, dev-dependencies, CI, and docs; production binaries are unaffected unless a separate determinism PR lands.

Overview
Adds a tests-only deterministic-simulation (DST) stack: a shared dev crate omnigraph-dst (op alphabet, reference model, white-box invariant battery, FaultAdapter, and a Backend trait with Embedded + Cli backends) plus engine integration suites under tests/dst* and CLI parity tests.

The engine dst target runs seeded generative walks with known-vs-novel classification (allow-listing RC-1, RC-X, dup-@key, etc.), characterization regressions, concurrent structural walks, read-shape × morphology battery, parser/loader fuzz, optional CAS-fault walks, env-gated s3_battery_holds, and a separate dst_recovery binary (failpoints) for manifest sidecar / concurrent-open convergence cells. cli_dst_parity and cli_cross_backend_walk lockstep the same op stream against embedded SDK vs CLI subprocess (black-box oracles only on the CLI arm).

CI extends RustFS change detection to DST paths and runs s3_battery_holds with a grep guard against vacuous passes. docs/dev/testing.md documents run modes and the coverage ledger in MATRIX.md.

^{Reviewed by Cursor Bugbot for commit 9fd1ac7. Bugbot is set up for automated code reviews on this repo. Configure here.}

Greptile Summary

This PR adds a shared DST test harness and wires it into engine, CLI, CI, and docs. The main changes are:

• New dev-only omnigraph-dst crate with shared backend, model, op, fault, and invariant helpers.
• New engine DST suites for generative walks, recovery cells, fuzzing, read-shape coverage, and S3 coverage.
• New CLI parity tests that compare embedded and subprocess-backed execution.
• RustFS CI wiring for the S3 DST battery.
• Updated testing docs for the new harness.

Confidence Score: 5/5

This looks safe to merge.

• No blocking issues found in the changed code.

Important Files Changed

Filename	Overview
crates/omnigraph-dst/src/backend.rs	Adds the shared backend trait and embedded/CLI implementations used by the DST walks.
crates/omnigraph-dst/src/invariants.rs	Adds the known/novel classifier and the embedded white-box invariant battery.
crates/omnigraph-dst/src/model.rs	Adds the reference model and black-box count, content, and edge oracles.
crates/omnigraph-dst/src/op.rs	Defines the seeded operation alphabet shared by embedded and CLI test paths.
crates/omnigraph/tests/dst.rs	Adds the main engine DST suite, including regressions, walks, concurrent coverage, and the S3 battery.
crates/omnigraph-cli/tests/cli_cross_backend_walk.rs	Adds the generative embedded-vs-CLI parity walk.
.github/workflows/ci.yml	Adds DST paths to RustFS change detection and runs the DST S3 battery in that job.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[omnigraph-dst crate] --> B[Embedded backend]
    A --> C[CLI backend]
    A --> D[Reference model]
    A --> E[Invariant battery]
    B --> F[Engine dst.rs tests]
    C --> G[CLI cross-backend tests]
    E --> F
    D --> F
    D --> G
    F --> H[RustFS S3 CI battery]

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
    A[omnigraph-dst crate] --> B[Embedded backend]
    A --> C[CLI backend]
    A --> D[Reference model]
    A --> E[Invariant battery]
    B --> F[Engine dst.rs tests]
    C --> G[CLI cross-backend tests]
    E --> F
    D --> F
    D --> G
    F --> H[RustFS S3 CI battery]

Loading

_{Reviews (5): Last reviewed commit: "fix(dst): honor Cli branch; tighten repa..." | Re-trigger Greptile}

Context used:

• Context used - AGENTS.md (source)
• Context used - CLAUDE.md (source)

[chatgpt-codex-connector]

💡 Codex Review

omnigraph/crates/omnigraph-dst/src/model.rs

Lines 84 to 85 in 99abd2a

	pub fn persons_vec(&self) -> Vec<usize> {
	self.persons.iter().copied().collect()

Make model iteration deterministic for lockstep walks

When cli_cross_backend_walk drives two independent Model::new() instances with the same RNG seed, edge ops sample from these vectors by index. Because HashSet/HashMap iteration is randomly seeded per map, the embedded and CLI models can produce different persons_vec/free_froms orders even with identical logical contents; with the current seeds the first InsertKnows can pick different endpoints, so the supposedly identical op stream diverges and the per-step edge oracle fails/flakes. Sort these vectors or store the model sets in an ordered collection before sampling.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[greptile-apps]

Lost Writes Become Known Duplicates

classify now allow-lists any logical finding containing dup-@key, but the count oracle emits that marker in the ambiguous lost-write or dup-@key message. When a real lost write produces count < model, the embedded walk can classify it as the known duplicate-key issue and continue instead of failing.

Context Used: AGENTS.md (source)

[cursor]

Cursor Bugbot has reviewed your changes and found 3 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 58a7a3c. Configure here.}

[cursor]

Delete updates model on no-op

Low Severity

DeletePerson updates the reference Model on every successful mutate, while UpdateDoc only updates the model when the target doc exists in the model. A successful delete that matches zero rows still calls model.del_person, which can desync the model from the engine when the slug was never live in the model but the op returns Ok.

 

^{Reviewed by Cursor Bugbot for commit 58a7a3c. Configure here.}