feat(holograph): production polish — sled + fetch fallback + shutdown

[data-bot-coasys]

Summary

Fourth slice of the Holograph stack, stacked on PR-C (#841). Production-polish round addressing the v1 substrate's known soft spots from SPIKE §2.5 exit-checks and the Wake-18 dispatch.

• D1 — KvOpStore::open retries on sled lock contention (50/100/200/400/800ms backoff, 5 attempts, ~1.55s budget) with a best-effort stale-lock cleanup after the first failed attempt. Two concurrent HolographSpace::new calls against the same data directory no longer race to a hard error.
• D2 — `SpaceConfig.fetch_fallback_policy: FetchFallbackPolicy` lifts the previously-implicit `fallback_timeout` + `max_retry_peers` knobs into one structured policy (`initial_timeout` / `max_attempts` / `retry_budget`). `fallback_pass` now round-robins through all arc-overlapping peers within a single tick instead of one-peer-per-tick.
• D3 — `HolographSpace::shutdown()`: flag-flip → stop watcher → drain queue (10s timeout) → sled flush → transport teardown. `Drop` is the safety net: best-effort sync flush, logs on error, never panics.
• D4 — New integration test `tests/restart_survives.rs` — 100 ops across 3 logical agents, full shutdown / reopen / retrieve round-trip. Formalizes SPIKE §2.5 exit-check Integration test suit for Link Languages #6 at substrate scale.
• D5 — When fetch-fallback exhausts `max_attempts` OR `retry_budget` for a pending entry, the entry is dropped and `NotifyUp::notify_parent_fetch_permanent_failure(op_id, missing_parents, last_error)` fires. The trait method has a default no-op so older `NotifyUp` impls compile.
• D6 — `SpaceConfig.iroh_relay_url: Option` plus a `resolve_iroh_relay()` helper that reads `HOLOGRAPH_IROH_RELAY` (preferred) or `HOLOGRAPH_IROH_RELAY_URL` (back-compat alias). `HolographSpace::new` resolves from env if the field is `None` and folds the value into `space.config()` so consumers don't have to re-read the env. New `rust-executor/crates/holograph/README.md` with a "Configuration" section + env-vars table.

Test plan

• `cargo test -p holograph --release -- --test-threads=1` — 53 tests green (47 lib + 4 pdiff_parity + 1 restart_survives + 1 two_node)
• `cargo check -p ad4m-executor` — clean (with baseline `CUSTOM_DENO_SNAPSHOT.bin` + `dapp/dist` artifacts in place; same baseline as PR-C / `dev`)
• `cargo fmt --all --check` — clean (after `a4a755840` fmt fixup)
• JS multi-conductor regression — unchanged surface from PR-C; CI will produce the result

Commits

```
9e2cade feat(holograph): retry sled lock contention in KvOpStore::open (Wake-18 D1)
feb5a66 feat(holograph): fetch-fallback policy + round-robin + permanent-failure event (Wake-18 D2/D5)
32c0d4f feat(holograph): graceful shutdown + sync-flush Drop (Wake-18 D3)
dfa9358 test(holograph): restart_survives_state — 100 ops, 3 agents, full round-trip (Wake-18 D4)
8db80bf feat(holograph): HOLOGRAPH_IROH_RELAY env config surface + README (Wake-18 D6)
a4a7558 chore: cargo fmt --all
```

Notable behavioural changes for reviewers

• Bool wins on env: `HOLOGRAPH_IROH_RELAY` takes precedence over the older `HOLOGRAPH_IROH_RELAY_URL`. Existing deployments setting only the long name keep working unchanged.
• NotifyUp surface widened with one default-impl method (`notify_parent_fetch_permanent_failure`). Existing impls in the wires layer compile unchanged.
• `IntegrationQueueConfig` / `HolographSpaceConfig` swapped `fallback_timeout` + `max_retry_peers` for `fallback_policy`. Callers inside this crate were updated; `holograph_wires.rs` is not affected (it doesn't construct queue configs directly).
• `on_local_commit` after `shutdown` returns `K2Error::other("... shutdown in progress")`. Callers should treat this as a terminal signal, not a retry.

Stacked split

• PR-A (PR-A: Algorithm crate extraction (substrate-neutral perspective-diff-algorithm) #839) — algorithm crate extraction
• PR-B (PR-B: Holograph substrate crate (K2 OpStore + KitsuneRetreiver + IntegrationQueue + Space) #840) — Holograph substrate
• PR-C (PR-C: holograph-link Language + JS integration + default-switch (Step 5-12) #841) — Language + JS
• PR-D (this) — production polish

🤖 Generated with Claude Code

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 028c53be-0b16-467d-989f-6a2850a970c0

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch holograph-pr-d-production-polish

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}