feat(holograph): K2-backed perspective sync substrate (spike, feasibility-proven)#836
Draft
data-bot-coasys wants to merge 41 commits into
Draft
feat(holograph): K2-backed perspective sync substrate (spike, feasibility-proven)#836data-bot-coasys wants to merge 41 commits into
data-bot-coasys wants to merge 41 commits into
Conversation
…p 0.5) New `rust-executor/crates/holograph` crate added as workspace member. Modules: - `config`: `SpaceConfig` with `ArcPolicy`, `LocFnPolicy`, `ValidationRegime`, and `gossip_initiate_interval_ms`. `full_replication_single_doc()` is the v1 default and is also the `Default` impl. Honors sharding-ready commitments 1, 2, 6 from SPIKE.md §1.5 (DhtArc-aware, loc-fn policy wired, space takes config struct). - `envelope`: `OpEnvelope` CBOR-encoded via ciborium with optional `doc_id` field. Honors commitment 3 — `#[serde(default, skip_serializing_if)]` keeps v1 envelopes (no `doc_id`) interop-compatible with v1.5 readers. Op-ids are raw bytes on the wire rather than going through `kitsune2_api::OpId`'s base64-string serde (which requires borrowed `&str` deserialization that CBOR cannot provide). Tests cover SpaceConfig defaults, ArcPolicy round-trip, envelope round-trip with/without doc_id, legacy-envelope forward-compat, and malformed-bytes error path.
…eiver (Step 1)
The trait definition no longer carries the HDK-shaped trait bounds it used
to require on every implementer:
- `T: TryFrom<SerializedBytes, Error = SerializedBytesError>` on `get` and
`get_with_timestamp`. The only `T` the algorithm ever fetched was
`PerspectiveDiffEntryReference`, so those methods are now concretely typed
to return it — no generic, no bound.
- `ScopedEntryDefIndex/EntryVisibility/Entry: TryFrom<I>/WasmError: From<E,E2>`
on `create_entry`. The fn now takes the `EntryTypes` integrity-zome union
directly; all call sites already passed `EntryTypes::Foo(…)`.
The `HolochainRetreiver` impl still uses HDK internally (decodes via
`to_app_option::<PerspectiveDiffEntryReference>()`, calls the HDK host
`create_entry`). The trait surface itself no longer imposes HDK conversions,
so the upcoming `KitsuneRetreiver` (Step 2) and the in-process
`MockPerspectiveGraph` can implement it without inheriting the HDK type
machinery through the trait.
Call-site updates are mechanical: `Retriever::get::<PerspectiveDiffEntryReference>(h)`
becomes `Retriever::get(h)` across `lib.rs`, `link_adapter/{commit,pull,chunked_diffs}.rs`,
and the mock test fixtures.
All 36 `perspective_diff_sync` unit tests stay green.
…m crate (Step 1.5, narrowed)
Adds new `crates/perspective-diff-algorithm` workspace crate, the foundation
for hosting the perspective-diff DAG algorithm in a form usable from both
the Holochain-backed `p-diff-sync` language and the upcoming Kitsune-backed
`holograph` substrate (SPIKE.md Step 2).
This commit lands the *foundation* of the extraction — the trait
abstractions plus a first migrated module — and explicitly narrows the
broader move per SPIKE.md §2.6 ("narrow the move") and risk-register
guidance. The remaining link_adapter modules (`workspace`, `chunked_diffs`,
`revisions`, `snapshots`, `render`, `pull`, `commit`) stay in p-diff-sync
for now; see `.spike-status/step-1.5-status.md` for the deferred-work list
and rationale.
What lands:
- `crates/perspective-diff-algorithm/`:
* `OpId` marker trait — Clone+Eq+Ord+Hash+Debug+Display+Serialize+
DeserializeOwned+Send+Sync+'static. Blanket-impl'd so any conforming
type (e.g. `HoloHash<Action>`, `kitsune2_api::OpId`) is automatically
an `OpId`.
* `HasDiffParents<O>` trait — the only structural property the DAG-walk
algorithms need from a node type. Lets the algorithm crate stay
ignorant of whether the node is `PerspectiveDiffEntryReference` or
something else.
* `topo_sort_diff_references<O, V>` — Kahn's algorithm, generic over
`(O: OpId, V: HasDiffParents<O>)`. Moved verbatim from
`p-diff-sync::link_adapter::topo_sort`. Has its own
substrate-independent tests (linear chain, diamond, no-root,
missing-parent), all green.
- `perspective_diff_sync_integrity`:
* New dep on `perspective-diff-algorithm`.
* `HasDiffParents<HoloHash<Action>>` impl on `PerspectiveDiffEntryReference`.
Lives here (not p-diff-sync) because both the trait and the type are
foreign to p-diff-sync; the orphan rule forces this placement.
- `perspective_diff_sync`:
* New dep on `perspective-diff-algorithm`.
* `link_adapter::topo_sort` is now a thin Holochain-side adapter that
delegates into the algorithm crate and maps `TopoSortError` ->
`SocialContextError` for backwards compatibility. All call sites and
the existing `test_topo_sort` test are unchanged on the outside.
Tests:
- `cargo test --release -p perspective-diff-algorithm -- --test-threads=1` —
4 tests pass (linear, diamond, no-root, missing-parent).
- `cargo test -p perspective_diff_sync --lib -- --test-threads=1` —
36 tests pass (chunked_diffs, pull, topo_sort, workspace, mock-graph).
- `cargo build --release -p holograph` clean.
…tore (Step 2a)
New `op_store` module wires Kitsune2's 11-method `OpStore` trait onto a
single `sled::Db` per space (two trees: `ops` for op records, `slice_hashes`
for K2's gossip Merkle bookkeeping).
Persistence-not-optional per SPIKE §0: the load-bearing
`state_persists_across_reopen` test closes the db handle, reopens at the
same path, and verifies stored ops + op-ids round-trip. The
`bob_asks_alice_alice_serves` smoketest exercises the
`process_incoming_ops -> retrieve_ops -> process_incoming_ops` round-trip
end-to-end between two store instances (SPIKE §2.5 exit check, lite
version — no DynSpace involved here, just the OpStore surface).
Storage shape:
- `ops`: `op_id_bytes -> ciborium-encoded OpRecord {created_at_micros,
stored_at_micros, op_data}`. No secondary time indexes; v1 scale doesn't
need them, query methods scan + filter.
- `slice_hashes`: composite key `arc_prefix(9) || slice_id_be(8)`. The
arc prefix lets us prefix-scan all slices for a given arc in one cursor
pass.
Envelope decoding is injected as a closure (`EnvelopeDecoder`) at
construction time — keeps the OpStore generic-free and lets
HolographSpace (Step 4) own envelope semantics while letting tests use
deterministic test-only decoders.
Sharding-ready commitment 1 (SPIKE §1.5) honored in
`process_incoming_ops`: the arc-policy is consulted before storing rather
than hardcoding "yes." v1 `Full` arc lets everything through; v1.5
sharded mode filters here without further code changes.
16 tests pass (7 prior + 9 new KvOpStore tests covering round-trip,
dedup, filter, time-slice query, earliest-timestamp, slice-hash
round-trip, empty-hash rejection, restart persistence, Bob-asks-Alice).
…er (Step 2b) New `retriever_kitsune` module + `KitsuneRetreiver` marker type that lets the p-diff-sync algorithm run on top of `KvOpStore` (the sled-backed K2 op-store from Step 2a) without further changes to algorithm call sites. Bridges three things: - **Sync trait, async store**: the `PerspectiveDiffRetreiver` methods are static-sync; K2's `OpStore` returns `BoxFuture`. The retriever state owns a *dedicated* `tokio::runtime::Runtime` (built with `new_multi_thread().worker_threads(2)`) and every async K2 call goes through `runtime.block_on(...)`. This is the SPIKE §2.6 tokio-nesting mitigation — `block_on` runs on the *caller's* thread (the sync algorithm thread), so the inner runtime's worker threads are always distinct from it; no deadlock. - **Static method, per-substrate state**: matches the existing `HolochainRetreiver`/`MockPerspectiveGraph` pattern by stashing state in a process-global `RwLock<Option<Arc<…>>>` slot. `install()` is one-shot in production; tests use `reset_for_test` + a `Mutex` for serialization. - **Hash <-> OpId mapping + envelope decoding**: `Hash` (= 36-byte `HoloHash<Action>`) maps 1:1 to `kitsune2_api::OpId` via the raw 36 bytes. The op-id is `sha256(envelope.payload) || [0xdb;4]` so the same serialized `PerspectiveDiffEntryReference` always hashes to the same id — matching `MockPerspectiveGraph::create_entry`'s scheme. OpEnvelope grows a `created_at_micros: i64` field (default 0 for backward compatibility) so peers derive identical timestamps from identical envelope bytes. `OpEnvelope::new_at` is the explicit-timestamp constructor; `new` keeps existing call sites unchanged. Holograph crate now depends on `perspective_diff_sync`, `perspective_diff_sync_integrity`, `hdk`, `holo_hash`, and `holochain_serialized_bytes` (path/git deps). Per the orchestrator's Option A (see `.spike-status/blocker-step-1.5.md`), accepting this transitive HDK pull for the spike is preferred over blocking on the deeper data-type extraction. `perspective_diff_sync`'s `errors` and `retriever` modules promoted from `mod` to `pub mod` so external crates can `use` the trait and error types. No semantic changes to the modules themselves. 20 tests pass (16 prior + 4 new KitsuneRetreiver tests covering create+get round-trip, deterministic hashing, revisions round-trip, and get_with_timestamp accuracy).
…r (Step 2d) Adds an integration-test suite (`tests/pdiff_parity.rs`) that drives the real `link_adapter::workspace::Workspace::build_diffs::<KitsuneRetreiver>` algorithm against entries seeded through `KitsuneRetreiver::create_entry`. This proves the trait surface is substrate-agnostic — the same algorithm code that the existing `MockPerspectiveGraph` tests in p-diff-sync exercise also runs on the new Kitsune-backed substrate. Why not literally re-run the existing pdiff tests against KitsuneRetreiver: `MockPerspectiveGraph` derives hashes from DOT node-id strings via `node_id_hash`, while `KitsuneRetreiver` hashes by content (SHA-256 of the serialized `PerspectiveDiffEntryReference`). The test fixtures are not portable across hash schemes; the algorithm under test is. The parity tests reconstruct the same DAG shapes (linear chain, fork, merge node with two parents) and verify the same structural invariants (common-ancestor identification, entry-map completeness, parent walking) that the existing tests check. `commit`/`pull`/`render`/`get_snapshot` aren't covered here — they call HDK runtime fns directly (create_link/emit_signal/get_links/hash_entry) and stay HDK-bound per the Step 1.5 narrowing. The trait surface that KitsuneRetreiver implements *is* covered. Visibility tweaks: - `link_adapter::workspace` promoted from `pub(crate)` to `pub` so the parity tests can reach `Workspace::build_diffs`. No semantic change. - `KitsuneRetreiver::__clear_state_for_tests__` added as a `#[doc(hidden)]` escape hatch for integration-test crates; the `#[cfg(test)]`-only `reset_for_test` isn't reachable across the test-crate boundary. Tests: - `cargo test --release -p holograph -- --test-threads=1` — 20 unit + 4 integration = 24 tests pass. - `cargo test -p perspective_diff_sync --lib -- --test-threads=1` — 36 tests pass (HC-side unchanged). Combined: SPIKE.md §2.5 exit check #3 ("clean for pdiff-sync against both HolochainRetreiver AND KitsuneRetreiver") satisfied for the algorithm code paths the trait covers.
… multi-peer fallback (Step 3)
New `integration_queue` module — the K2-facing integration layer sitting
above `KvOpStore`. K2's fetch path will hand inbound envelopes to the
queue, not directly to the store; the queue owns sig-check, arc-filter,
parent-presence, pend-or-store, and cascade-promote semantics. The
store stays as the lowest-level decoder/persistence layer.
Pipeline per inbound envelope:
1. Decode + signature verify (`SigVerifier`; v1 `AlwaysValid`).
2. Arc filter — sharding-ready commitment 1, consult `SpaceConfig.arc`
policy; ops outside the local arc are dropped.
3. Dedup against op-store and pending tree (no double-store, no
double-fetch).
4. Parent presence check via `KvOpStore::filter_out_existing_ops`:
- all present → store + notify-up + cascade-promote pending children
- some missing → pend in sled `pending` tree keyed by op-id, call
`OpFetcher::request_ops(missing_parents, source)`
Pending tree (sled) holds `PendingEntry {envelope_bytes,
missing_parents, source, first_seen_micros, tried_peers}` ciborium-encoded
under op-id keys. Restart-survives-state — the sled persistence carries
pending entries across queue reinstantiation.
Cascade promotion is a worklist (not recursion) so depth-N chains don't
blow the stack; promoting one parent op can unblock any number of
children, each of which may further unblock grandchildren.
Multi-peer fallback (SPIKE §1.1's load-bearing fix for K2's source-bound
fetch): a `tokio::spawn`'d watcher periodically scans pending entries
whose `first_seen_micros` exceeded `fallback_timeout`; for each, picks
an alternative arc-overlap peer via `PeerPicker` (skipping any URL in
`tried_peers`), and re-issues `request_ops` against the new peer.
Bounded by `max_retry_peers`. The watcher runs on the runtime handle
passed to the queue at construction — Step 4 will hand it the same
dedicated runtime `KitsuneRetreiverState` already uses, per the
tokio-nesting risk register from SPIKE §2.6.
Trait surface designed so Step 4 can wire K2's real modules without
glue: `OpFetcher::request_ops` matches `kitsune2_api::Fetch::request_ops`
verbatim; `PeerPicker::pick_arc_overlap_peer` is a thin abstraction
over `PeerStore::get_by_overlapping_storage_arc`. `NotifyUp` is the
bridge to AD4M's perspective-diff emit; Step 4 plugs the real impl.
12 unit tests pass (against mock fetcher/peer/notify):
- happy path root op
- one missing parent → pend + fetch → cascade promote on parent arrival
- depth-3 missing chain → topo-ordered promotion when root arrives
- signature failure drops the op entirely
- fallback pass re-requests via alt peer
- fallback bounded by max_retry_peers
- pending tree persists across queue restart
- duplicate-pending no double-fetch
- duplicate-stored is no-op
- outside-arc dropped
- watcher start/stop lifecycle (idempotent)
- watcher loop end-to-end (tokio runtime + tick → re-request)
Total: 32 holograph unit tests pass (was 20), plus the prior 4
pdiff_parity integration tests and the 36 p-diff-sync HC-side tests.
…mestamp (Step 4a/b/c)
Step 4 lifecycle wiring: `HolographSpace` ties together the Step-2
`KvOpStore`, the Step-3 `HolographIntegrationQueue`, a K2 `DynSpace`
sink, and the new adapter traits that bridge the queue's mocked-in-Step-3
trait surface to real K2 modules.
New module `src/space.rs` ships:
- `K2FetcherAdapter` — thin `OpFetcher` newtype over `DynFetch`.
Signature matches `Fetch::request_ops` verbatim; no logic.
- `K2PeerPickerAdapter` — wraps `DynPeerStore`, asks for arc-overlap
agents on a 1-loc arc and skips any URL in the queue's `tried` set.
- `ChannelNotifier` + `EmittedOp` — the queue's `NotifyUp` emits an
`EmittedOp { op_id, created_at, envelope_bytes }` onto a
`tokio::sync::mpsc::UnboundedSender` the Step-5 Language module will
drain.
- `HolographSpaceHandler` + `TelepresenceNotification` — K2
`SpaceHandler::recv_notify` passthrough into an mpsc; Step 5/6 owns
the JS side. Returns `Ok(())` even when the sink receiver is gone so
K2 doesn't tear down the peer connection on a local failure.
- `LocalCommitTarget` trait (sealed via `K2DynSpaceTarget` for prod)
separating the K2-side commit sink (inform_ops_stored + publish_ops)
from the rest of the space — lets unit tests verify the commit-side
logic without standing up the full K2 stack.
- `HolographSpace::on_local_commit` — decodes envelope via the shared
`EnvelopeDecoder`, queues it (parents always present for local
commits so the queue takes the all-parents-present branch and
stores+notifies straight away), then `inform_ops_stored` (gossip
bookkeeping) + `publish_ops_to_peers` (eager hint to known peers
so the user-perceived commit-to-propagate latency beats gossip
cadence).
- `HolographSpace::process_incoming_ops` — entry for K2 fetch/gossip
inbound, delegates to the queue.
- `K2OpStoreShim` — the `OpStore` impl K2 sees in its builder slot.
`process_incoming_ops` routes through the integration queue when
installed, falls back to direct `KvOpStore` storage otherwise
(handles the brief construction window before the queue is built).
All other 10 `OpStore` methods delegate to the underlying store
unchanged (storage + Merkle bookkeeping stay where they are).
`HolographIntegrationQueue::NotifyUp` extended to carry
`created_at: Timestamp` alongside the op-id + bytes — needed because K2's
`StoredOp` requires `created_at` for gossip and the value is derived from
the envelope on the queue's decode side. The cascade promote path
re-derives the timestamp from the pending envelope so promoted children
get the correct `EmittedOp.created_at`.
`Cargo.toml` adds `kitsune2 = "0.4..."` and `kitsune2_core = "0.4..."`
on the same git rev workspace already uses for `kitsune2_api`. These
are needed by the upcoming integration test against `MemTransport` +
`MemBootstrap` (Step 4d).
11 new unit tests pass under `cargo test --release -p holograph --lib`:
- on_local_commit stores/informs/publishes (verifies the three K2-side
side-effects on a commit)
- process_incoming_ops routes through queue (incoming path, no publish)
- ChannelNotifier delivers EmittedOp tuple
- SpaceHandler recv_notify forwards TelepresenceNotification
- K2OpStoreShim routes through queue when installed
- K2OpStoreShim falls through to store pre-install
- K2OpStoreShim passthrough retrieve_ops (K2 fetch-response path)
- K2OpStoreShim full passthrough surface (slice-hash, filter, count)
- K2DynSpaceTarget impls LocalCommitTarget (compile-time bound)
- K2 adapters impl queue traits (compile-time bound)
Total: 43 lib tests + 4 pdiff_parity integration tests still green.
Step 4d (real two-node K2 integration test with MemTransport +
MemBootstrap) is the next sub-piece.
…ep 4d) End-to-end verification for SPIKE.md §2.5 exit check #4: two `HolographSpace`s in one process exchange a committed op via the real K2 publish + fetch path. Test stack (`tests/space_two_node.rs`): - `kitsune2_core::default_test_builder()` baseline (mem_transport, mem_bootstrap, mem_peer_store, mem_peer_meta_store, mem_blocks, core_publish, core_fetch, core_gossip_stub, core_local_agent_store, core_kitsune, core_space, core_report) with our `K2OpStoreShim` swapped into the op-store slot. - `mem_bootstrap` shared `test_id` so Alice and Bob's peer stores discover each other; poll_freq overridden to 100ms. - `mem_transport` is process-global via OnceLock — both nodes auto-share without further wiring. - `TestLocalAgent` from `kitsune2_test_utils::agent` configured with `DhtArc::FULL` so each node participates in the gossip model for everything (v1 sharding-ready commitment 1, default arc). Test flow: 1. Build Alice and Bob, share `mem_bootstrap` test_id. 2. Trigger immediate mem-bootstrap poll + 800ms settle. 3. Alice `on_local_commit(root_envelope)` → her queue stores + emits + `inform_ops_stored` + `publish_ops_to_peers` (which fans out `Publish::publish_ops` to Bob). 4. Bob's K2 receives publish hint → triggers `Fetch::request_ops` → Alice's K2 serves via her shim → Bob's K2 calls Bob's shim's `process_incoming_ops` → Bob's queue → `ChannelNotifier` emit. 5. Test waits up to 30s on Bob's `emit_rx`; on a quiet laptop the round-trip completes in <1s. 6. Repeat with a child envelope whose parent = root. Bob's queue recognizes the parent is already present and takes the all-parents-present branch. Asserts: - Both nodes' `op_count == 2` after the chain. - Both ops' `EmittedOp` reach Bob's `ChannelNotifier`. - The K2-Bob-already-has-parent path works without requiring the multi-peer fallback (the parent arrived first via publish_ops). Additional fmt drift on `space.rs`, `space/tests.rs`, `integration_queue.rs` from `cargo fmt --all`. Adds `tracing-subscriber` and `kitsune2_test_utils` as dev-deps. Totals after Step 4 complete: 43 unit + 4 pdiff_parity + 1 two-node integration in holograph; 4 in perspective-diff-algorithm; 36 in perspective_diff_sync. 88 tests, all green.
Replaces the mem_bootstrap-mediated peer discovery in the two-node integration test with manual `peer_store().insert(...)` cross- registration of each side's `AgentInfoSigned`. This matches the two-node pattern K2's own `core_space::test` uses and tightens the test focus on the bits we actually need to prove: - the publish_ops → fetch → process_incoming_ops round-trip through the real K2 transport - the queue's parent-presence check on the child envelope - both ends' `ChannelNotifier` mpsc receiving the EmittedOp Bootstrap discovery (mem_bootstrap test_id sharing, poll cadence, etc.) is K2's responsibility and is exercised by K2's own test suite — we were paying its variance in our smoke path for no signal. Captures the local URL via a `KitsuneHandler::new_listening_address` hook so we know which `Url` to publish into the other side's peer store. Builds an `AgentInfoSigned` per node using `AgentBuilder::default().with_url(url).build(TestLocalAgent)`. Total result: still 1 integration test passing, runs in <1s instead of 1.5–2s (no 800ms bootstrap settle).
…ub (Step 5)
The JS-facing Language module + Rust wire-surface sketch the orchestrator
asked for in Step 5: a thin `defineLanguage()`-shaped facade that
delegates everything to the host imports `holographCreateNeighborhood`,
`holographCommit`, `holographRender`, `holographNextEmitted`,
`holographJoinAgent`, `holographCurrentRevision`,
`holographLatestRevision`, `holographCloseNeighborhood`. Step 6 (next
dispatch) wires the real `HolographSpace` instance behind these.
### Rust side (`rust-executor`)
- New `src/holograph_wires.rs`:
- `HolographHandle(u64)` — opaque per-neighborhood handle the JS
side threads through every call.
- `EmittedOpWire { op_id_b64, created_at_ms, envelope_b64 }` — the
serializable JSON shape of `holograph::space::EmittedOp` returned
by `nextEmitted`.
- `HolographWireError` — `NotImplemented`, `UnknownHandle`,
`InvalidEnvelope`, `Substrate(String)`. Step 5 only emits
`NotImplemented`; Step 6 widens.
- `HolographDelegate` trait — 8 methods documenting Step 6's
contract against `HolographSpace`.
- `NotImplementedHolographDelegate` — stub impl returning
`NotImplemented` everywhere. 2 unit tests verify the stub and
the wire-shape serde round-trip.
- `src/js_core/host.js` — new section "Holograph (Spec section 7.8 --
OPTIONAL EXTENSION)". Wraps `globalThis.__holographDelegate__` with
the same lazy-accessor pattern `__holochainDelegate__` already uses;
throws a descriptive error if the runtime hasn't installed the
delegate yet.
- `src/lib.rs` — `pub mod holograph_wires;`.
### Host import surface (`ad4m-ldk`)
- `js/src/host.d.ts` — adds the `holograph*` declarations + the
`EmittedOpWire` interface.
- `js/src/imports.ts` — re-exports the holograph wires + the
`EmittedOpWire` type.
### Language module (`bootstrap-languages/holograph-link`)
- `package.json` (`@coasys/holograph-link@0.1.0`), `tsconfig.json`,
`esbuild.ts`, `README.md`, `.gitignore`.
- `index.ts` — `defineLanguage()`-shaped Language exposing the standard
AD4M LinkLanguage capabilities (`sync`, `commit`, `peers`,
`telepresence`). Zero polling, zero `setInterval`, zero
peer-revision walks: the subscriber loop is an `await
holographNextEmitted(handle)` inside Rust (awaiting an mpsc receiver,
not sleeping), and propagation is driven by the Step-3 queue +
Step-4 publish/fetch glue.
- `tests/smoke.test.ts` — Deno tests that load the bundled module
with a data-URL host stub and verify (1) the bundle is non-empty,
(2) all required flat exports are present, (3) init → commit →
render → sync → currentRevision → teardown round-trips through the
wires without errors.
### Bundle
`pnpm run build` (`deno run --allow-all esbuild.ts`) produces
`build/bundle.js` (~350 lines). The bundle imports `ad4m:host`
externally, matching the executor's StringModuleLoader contract.
### Tests
- `cargo test --features generate_snapshot holograph_wires` — 2 passed.
- `cargo check -p ad4m-executor --features generate_snapshot --lib` — clean.
- `cargo test --release -p holograph --lib -- --test-threads=1` — 43 passed
(Step 4 baseline still green).
- `deno test --allow-all tests/smoke.test.ts` — 8 passed.
### Step 5 vs Step 6 explicitly
- Step 5 lands: JS module, wire-surface sketch, host imports, bundle,
smoke tests.
- Step 5 does NOT flip the neighborhood default (Step 6).
- Step 5 does NOT install `__holographDelegate__` (Step 6 owns the
isolate-install plumbing on the deno-core side).
- Step 5 does NOT run the multi-conductor integration test (Step 7).
Language address scheme: `hash("@coasys/holograph-link@<version>")`;
the canonical AD4M content-address hash function is what every other
content-addressed Language uses, so the holograph-link Language picks
its address from that namespace.
…Step 6b)
`rust-executor/src/holograph_wires.rs` now hosts a real
`HolographRuntime` instead of the Step-5 `NotImplementedHolographDelegate`.
Architecture:
- `HolographRuntime` is process-global (lazy_static), owns a dedicated
multi-thread tokio runtime (2 worker threads) and a
`DashMap<HolographHandle, Arc<NeighborhoodState>>`.
- Each `NeighborhoodState` holds an `Arc<HolographSpace>` plus the
`mpsc::UnboundedReceiver<EmittedOp>` half of `ChannelNotifier::new()`.
- `create_neighborhood(space_id, storage_dir)` builds:
* sled-backed `KvOpStore` under `<storage_dir>/h<N>/ops/`,
* sled-backed `pending` tree under `<storage_dir>/h<N>/pending/`,
* K2 `DynSpace` via `kitsune2_core::default_test_builder()` (mem
transport + mem peer store + core fetch/publish + stub gossip)
with our `K2OpStoreShim` substituted into the op-store slot,
* `HolographSpace` wired via `HolographSpaceConfig::defaults` + Step-4
K2 adapters,
* a sentinel `TestLocalAgent` joined to FULL arc (real AD4M-DID-bound
agent identity is Step 7 territory).
- `commit(handle, WireDiff)` encodes a CBOR `OpEnvelope` whose payload
is JSON of the diff, drives `HolographSpace::on_local_commit`, returns
the URL-safe base64 op-id.
- `next_emitted(handle)` awaits the mpsc receiver inside the dedicated
runtime — JS subscriber loops never spin.
- `render`/`current_revision`/`latest_revision` return v1 placeholders;
full Perspective render needs PR-B's algorithm-crate wiring.
`HolographDelegate` trait + `NotImplementedHolographDelegate` stub
removed — the deno op surface (Step 6c) calls `HolographRuntime`
directly, no trait indirection.
Wire surface evolved:
- `holograph_envelope_decoder` re-exported from
`holograph::space` so the executor can build envelopes that decode
against the same Op-ID scheme `HolographSpace` already uses.
- `EmittedOpWire` now carries a decoded `WireDiff` instead of raw
base64 bytes — Step 6e's "envelope construction moves to Rust"
landed in this commit too, so JS sees typed diff data on both ends.
Cargo.toml additions:
- `holograph` path dep + `kitsune2_api`/`kitsune2_core`/
`kitsune2_test_utils` git deps (same rev workspace already uses)
+ `sled` + `ciborium` + `dashmap` + `bytes`.
Tests (`cargo test -p ad4m-executor --features generate_snapshot --lib
holograph_wires -- --test-threads=1`):
- `wire_diff_serde_round_trips` — JSON shape stable across ser/de.
- `encode_decode_envelope_round_trip` — CBOR envelope wraps/unwraps a
diff verbatim.
- `invalid_envelope_decode_returns_error` — bad bytes yield typed err.
- `unknown_handle_returns_error` — wires error on stale handle.
- `create_commit_and_emit_round_trip` — load-bearing E2E: spin up a
neighborhood, commit a diff, observe the emit on the receiver, verify
op-id + diff round-trip through the substrate.
- `render_returns_empty_links_v1` — placeholder shape stable.
- `close_neighborhood_releases_handle` — cleanup correctness.
- `revisions_default_to_none` — null pointers when no commit history.
8 tests pass. Step-4 baseline (43 holograph + 4 pdiff_parity + 1
space_two_node) still green.
…te__ install (Step 6c)
The Rust-to-JS bridge for the holograph wires lands:
- New `rust-executor/src/js_core/holograph_service_extension.rs`:
Eight `#[op2(async)]` ops that forward verbatim to
`HolographRuntime::{create_neighborhood, commit, render, next_emitted,
join_agent, current_revision, latest_revision, close_neighborhood}`.
Wires use `#[bigint] u64` for handles and `#[serde] WireDiff` for
the typed perspective-diff payload. The deno_core::extension! macro
is named `holograph_service`, matching the holochain naming.
Revision-pointer ops use `#[string]` (empty string ↦ JS null) because
`op2 #[serde] Option<String>` isn't supported by the current macro.
- New `rust-executor/src/js_core/holograph_service_extension.js`:
Installs `globalThis.HOLOGRAPH_SERVICE` with thin async wrappers
around the ops. The JS shim widens handles back from BigInt to
Number and converts the empty-string sentinel back to null. Pure
ASCII so the `ascii_str_include` macro check passes.
- `rust-executor/src/js_core/mod.rs`: `pub mod
holograph_service_extension;`.
- `rust-executor/src/js_core/options.rs`: `holograph_service::init()`
alongside `holochain_service::init()` in
`language_worker_options::extensions`.
- `rust-executor/src/js_core/language_bootstrap.js`: new
`createHolographDelegate(languageAddress)` factory (mirroring
`createHolochainDelegate`) that wraps the per-handle HOLOGRAPH_SERVICE
in the spec-shaped delegate object. `initLanguage` installs the
delegate on `globalThis.__holographDelegate__` alongside
`__holochainDelegate__`, and exports `createHolographDelegate` as a
global mirror of `createHolochainDelegate`.
The Step-5 host.js's `holographDelegate()` accessor (which reads
`__holographDelegate__`) now resolves cleanly: the runtime installs
the delegate at language-init time, the Language module calls
`holographCommit(handle, diff)` etc., the call routes through
HOLOGRAPH_SERVICE -> op2 op -> HolographRuntime -> HolographSpace.
No new tests in this commit -- the runtime-side tests landed in 6b;
the JS-side round-trip lands in Step 6f. `cargo check -p ad4m-executor
--features generate_snapshot --lib` clean; 8 holograph_wires tests
still passing.
…(Step 6d) `rust-executor/src/neighbourhoods.rs` gains three small helpers implementing the SPIKE.md §2.2 Step 6 default switch: - `HOLOGRAPH_LINK_PACKAGE_ID = "@coasys/holograph-link@0.1.0"` -- the spike's canonical identity string for the holograph-link Language. - `holograph_link_default_address()` computes the canonical AD4M content-address (SHA-256 -> CIDv1 -> base58btc with the `Qm` prefix) so the address matches whether produced from Rust here or from the JS `hash()` host function. - `holograph_default_enabled()` reads `HOLOGRAPH_DEFAULT_NEIGHBORHOOD=1` from the process environment. - `resolve_link_language(Option<String>) -> Result<String>` is the load-bearing entry point: explicit address wins, empty/None with the flag set substitutes the holograph-link default, empty/None without the flag errors out cleanly (matches pre-Step-6 behavior for callers that don't opt in). API surface change: `PublishNeighbourhoodRequest.link_language` is now `Option<String>` with `#[serde(default)]`, so callers may omit the field entirely or send empty-string and rely on the default switch. Existing callers that pass a populated address continue to work unchanged. `publish_neighbourhood` in `rust-executor/src/api/neighbourhoods_ws.rs` runs the input through `resolve_link_language` before forwarding to `neighbourhoods::neighbourhood_publish_from_perspective_with_context`. Resolution errors return `WsRpcError::bad_request` so a missing link_language without the env flag surfaces a clear 400 to the client rather than a confusing downstream failure. What this DOESN'T do (intentional, for Step 7): - `install_language` will fail when called against the synthetic holograph address because the holograph-link bundle isn't published to the language-language store. Step 7 wires bootstrap pre-install for the holograph-link bundle (or short-circuits install_language when the address matches the synthetic one). This commit gets the default-switch decision point landed; the install-side plumbing belongs to the multi-conductor work. Tests (`cargo test -p ad4m-executor --features generate_snapshot --lib neighbourhoods::tests -- --test-threads=1`): - `holograph_link_default_address_is_stable_qm` - `holograph_default_disabled_by_default` - `holograph_default_enabled_with_flag_one` - `holograph_default_disabled_with_flag_other_value` - `resolve_passes_through_explicit_address` - `resolve_substitutes_default_when_flag_set_and_empty_input` - `resolve_errors_when_flag_unset_and_empty_input` 7 pass. Existing `parse_publish_neighbourhood_request` test updated to the new `Option<String>` shape; still passes.
…p 6e)
The substrate-side `HolographRuntime` from Step 6b takes a typed
`WireDiff { additions, removals }` and emits `EmittedOpWire {
op_id_b64, created_at_ms, diff: WireDiff }`. CBOR envelope wrap+unwrap
(timestamp + signature + payload encoding) is owned by Rust now. This
commit propagates the new shape to JS:
- `ad4m-ldk/js/src/host.d.ts`:
* Adds `WireDiff` interface alongside `EmittedOpWire`.
* `holographCommit` signature is now `(handle, diff: WireDiff)`
instead of `(handle, envelopeB64: string)`.
* `EmittedOpWire` swaps `envelope_b64` for a typed `diff: WireDiff`.
- `ad4m-ldk/js/src/imports.ts`: re-exports the new `WireDiff` type
next to `EmittedOpWire`.
- `bootstrap-languages/holograph-link/index.ts`:
* Removes the Step-5 `encodeEnvelope`/`decodeEnvelope` JS helpers and
the `envelopeToBase64`/`base64ToBytes` byte-juggling.
* Replaces them with `toWireDiff`/`fromWireDiff` -- pure shape
coercions between the Language-facing `PerspectiveDiff` class and
the wire-facing `WireDiff` interface.
* `commit` hands the typed diff straight across the wire; the Rust
side runs `encode_envelope` and `HolographSpace::on_local_commit`.
* Subscriber loop reads `next.diff` directly -- no envelope decode
step.
* Fixes the `asssertHandle` typo (extra `s`) to `assertHandle`.
- `bootstrap-languages/holograph-link/tests/smoke.test.ts`:
* `holographCommit` host stub takes a typed `diff` argument (not a
base64 string). All 8 smoke tests still pass against the rebuilt
`build/bundle.js`.
Tests:
- `pnpm run build` (deno esbuild) -- bundle rebuilds cleanly.
- `deno test --allow-all tests/smoke.test.ts` -- 8/8 pass.
… 6f) Exercises the same public surface the deno op layer calls into: create_neighborhood + commit + next_emitted + close_neighborhood for two distinct handles on the same process-global HolographRuntime. Complements crates/holograph/tests/space_two_node.rs (which proves K2 cross-node propagation directly against HolographSpace) by confirming the wire-level handle dispatch keeps emit channels isolated between neighborhoods and that close_neighborhood is idempotent. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- print_holograph_address: CLI that emits the canonical AD4M address for @coasys/holograph-link@0.1.0 so the JS integration test can hardcode a verified value without re-deriving the SHA-256/CIDv1/base58btc hash client-side. - generate_snapshot: include holograph_service in the v8 snapshot extension list alongside the other Step 6c-installed ops. - lib.rs: expose neighbourhoods module so the bin can call holograph_link_default_address(). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- utils/utils.ts: startExecutor now accepts an opts bag with custom env
vars and an initData flag. Lets tests pass
HOLOGRAPH_DEFAULT_NEIGHBORHOOD=1 per-conductor and skip the
rmSync+init wipe for restart-survives-state tests.
- tests/holograph-link.test.ts: documents the intended single-conductor
end-to-end shape. Boots one executor with the env flag, pre-installs
the holograph-link bundle to <data>/ad4m/languages/<addr>/bundle.js
so install_language's disk-fast-path finds it, then exercises:
1. agent reaches initialized state with the flag on
2. publishFromPerspective without linkLanguage resolves via the
Step 6d default switch
3. perspective records the holograph address as its link_language
4. Alice's own addLink round-trips through the subscriber loop
(commit -> on_local_commit -> ChannelNotifier mpsc ->
holographNextEmitted -> bundle subscriber loop ->
emitPerspectiveDiff -> runtime listener)
5. the link survives a query round-trip
6. restart-survives-state: kill+restart preserves sled-backed
perspective state and new commits still flow
The test does not boot a second conductor: K2 mem transport (the
current default_test_builder choice) is in-process only, so
cross-process sync needs a real transport (iroh/tx5) wired into the
HolographSpace builder. That swap is PR-B work and is documented in
blocker-step-7.md as the dispatch's tests 1+2 gap.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…Step 7c) Three corrections discovered by running the JS test end-to-end: 1. generate_snapshot.rs: holograph_service was at end-of-list, but the runtime extension order in js_core/options.rs has it between holochain_service and signature_service. deno verifies snapshot ext order at runtime load and panics on mismatch — fixed by matching the runtime order. Re-generate the snapshot (`target/debug/generate_snapshot` from rust-executor/) before any ad4m-executor build that uses the snapshot. 2. print_holograph_address.rs: add an optional file-path arg. With no args it prints the Step 6d package-id-derived address (unchanged). With a file-path arg it prints the SHA-256/CIDv1/base58btc content address of that file's bytes — the same algorithm LanguageController::calculate_language_hash uses, so a bundle pre-installed at that address passes install_language's hash verification. 3. tests/holograph-link.test.ts: drop the package-id address. Shell out to print_holograph_address with the bundle path to derive the content hash, then pre-install + publishFromPerspective with that. Document that the env-default-switch (resolve_link_language) is unit-tested separately because routing it through the JS path needs the executor to itself derive the bundle's content hash at startup — a PR-B-shape config change. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…s (Step 8a) host.js's holographCommit(handle, diff) forwards both positional args to the delegate. The Step 6c delegate's commit(diff) only declared the diff arg, so JS positional binding mapped the handle (Number) into `diff` and the actual diff was discarded. The Rust-side serde_v8 decoder then errored with "expected: object, got: Number" at first real commit, leaving the diff queued in pending forever. - delegate.commit now takes (_handleArg, diff) and uses the captured handle as source of truth (defensive — must equal the passed one). - Same shape applied to render, nextEmitted, joinAgent, currentRevision, latestRevision, closeNeighborhood so all single- and multi-arg signatures line up positionally with host.js. - host.js parameter rename: envelopeB64 -> diff (Step 6e moved envelope construction Rust-side; the wire takes typed diff data). Surfaced by tests/js/tests/holograph-link.test.ts: with the fix the "Alice's addLink round-trips through the subscriber loop" path commits cleanly through HolographSpace::on_local_commit instead of queueing for retry. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
… 8b, Gap C)
Step 6d's resolve_link_language substituted a package-id-derived
address (`hash("@coasys/holograph-link@<v>")`) under
HOLOGRAPH_DEFAULT_NEIGHBORHOOD=1. But install_language content-hashes
the bundle and rejects any address that doesn't match the bundle's
hash -- so the substituted address could never resolve to an
installable Language.
Gap C cheap fix (option b1 per blocker-step-7.md): add an env-var hook
so the default-switch substitutes the bundle's actual content hash.
- ad4m_content_address(bytes): private helper mirroring
LanguageController::calculate_language_hash (SHA-256 -> CIDv1 ->
base58btc -> Qm prefix).
- holograph_link_resolved_address(): reads HOLOGRAPH_LINK_BUNDLE_PATH,
computes the bundle's content hash, caches via OnceLock. Falls back
to the package-id-derived address (with a warn-level log) when the
env var is unset, so existing unit tests still see the legacy
address.
- resolve_link_language now calls holograph_link_resolved_address()
for the substitution branch. The explicit-address and
error-when-unset branches are unchanged.
- holograph-link.test.ts: drop the explicit content-hash pin on the
publish path. The test now sets HOLOGRAPH_LINK_BUNDLE_PATH +
HOLOGRAPH_DEFAULT_NEIGHBORHOOD=1 in the conductor env and passes
`undefined` as linkLanguage. The neighborhood assertion reads the
correct NeighbourhoodExpression.data.linkLanguage path.
Result: holograph-link.test.ts is 7/7 green end-to-end:
publish-without-linkLanguage routes through the WS handler ->
resolve_link_language -> install_language -> bundle loads ->
PerspectiveDiff round-trips through the subscriber loop ->
state survives a kill+respawn cycle.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…rt (Step 9a, Gap B)
build_dyn_space_inner now picks the transport stack at first call based
on env. Two modes:
HOLOGRAPH_SBD_URL=<ws://sbd-url> set:
transport -> Tx5TransportFactory (WebRTC via SBD signal server)
bootstrap -> CoreBootstrapFactory (peer discovery via
kitsune2-bootstrap-srv; URL from HOLOGRAPH_BOOTSTRAP_URL
or derived from SBD URL by swapping scheme)
gossip -> kitsune2_gossip::K2GossipFactory (replaces the
CoreGossipStub which silently does nothing)
HOLOGRAPH_SBD_PLAINTEXT=1 allows ws:// instead of wss:// (test
harness runs bootstrap-srv on loopback).
HOLOGRAPH_SBD_URL unset (default for in-process tests):
unchanged from Step 6b -- default_test_builder, mem transport,
mem bootstrap, stub gossip. Step 4d's space_two_node and Step 6f's
two_node_via_wires both keep using this path and stay green.
NeighborhoodState gains `dyn_space: kitsune2_api::DynSpace` so
join_agent can return `dyn_space.current_url()` (the real reachable
URL the transport publishes) instead of the
"ws://holograph-local:0" placeholder. join_agent still falls back to
the placeholder when current_url is None (mem path, or before Tx5
finishes the SBD handshake).
ShimFactory + NoopSpaceHandler + NoopKitsuneHandler hoisted out of
build_dyn_space_inner into module scope so both transport branches
share them.
The kitsune-handle Box::leak is still spike-acceptable per
blocker-step-7.md; PR-B turns it into a real owned lifetime as part of
the transport-config polish.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
tests/js/tests/holograph-link-multi.test.ts. Boots one
kitsune2-bootstrap-srv (which doubles as the SBD signal server) and
two ad4m-executor conductors with HOLOGRAPH_SBD_URL +
HOLOGRAPH_BOOTSTRAP_URL pointing at it, plus the env hooks from
Step 8c so the default-switch substitutes the bundle's content
address.
Current state: 2/4 mocha cases green --
1. Alice publishes a holograph-backed neighbourhood (publish-side
setup + the Step 6d default switch via bundle content hash)
2. Bob joins via the neighbourhood URL (neighbourhood-language
resolves the metadata, holograph-link is installed on Bob's
side, the perspective is registered)
Failing: cross-process op propagation (Alice's addLink does not
reach Bob's subscriber within 60 s, and the same for the return
direction). The setup is healthy -- both executors print
"DynSpace built with Tx5 (sbd=...) + CoreBootstrap (server=...)" --
so the gap is somewhere between CoreBootstrap peer discovery and
K2 publish/fetch firing across the SBD/WebRTC link. See
.spike-status/blocker-step-9.md for the wake-10 debugging
shopping list.
The single-conductor test (tests/holograph-link.test.ts) and the
substrate baseline (Step 4d / Step 6f + the holograph_wires lib
tests) all stay green, so this commit does not regress any prior
exit checks.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Surface peer-store population and per-iteration send/skip counts so the wake-10 cross-process op-flow debug has a single line per commit that says "did we even find peers? how many? what URLs?". Kept at info level so it shows up in the default RUST_LOG of the JS test harness without bumping to debug. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…th (Step 10c, Gap-B closer) Two bugs and one perf knob, all blocking cross-process op-flow: 1. **Process-unique agent id.** TestLocalAgent::default() uses an in-process atomic counter, so every fresh ad4m-executor starts at "test-1". With two conductors both publishing AgentInfo for agent "test-1" to the bootstrap server, CoreBootstrap can't distinguish them and either dedupes or overwrites silently. Fix: when the Tx5 path is selected (HOLOGRAPH_SBD_URL set), spin up an Ed25519LocalAgent::default() instead -- random SigningKey on each call gives a process-unique 32-byte AgentId. 2. **Verifier paired with agent.** TestVerifier only accepts the literal TEST_SIG bytes; Ed25519LocalAgent produces real ed25519 signatures. AgentInfo verification across processes therefore silently fails with mismatched verifier/agent. Fix: pair the Ed25519LocalAgent swap with Ed25519Verifier on the Tx5 path. The in-process mem path (Step 4d / Step 6f) keeps the TestVerifier + TestLocalAgent pair so those tests don't churn. 3. **CoreBootstrap backoff.** Production default is 5000ms minimum (sensible). Cold-start convergence between two conductors on loopback at that interval pushed Test 3/4 past the 15s deadline. Lower to 500ms for the Tx5 path, overridable via HOLOGRAPH_BOOTSTRAP_BACKOFF_MIN_MS. Plus a robust agent-id log: AgentId Display invokes HoloHash-shaped decoding (32B only); switched to URL-safe base64 of the raw bytes + explicit byte length so the log doesn't panic for either the TestLocalAgent (13B) or Ed25519LocalAgent (32B) shape. Result: holograph-link-multi.test.ts is 4/4 then 5/5 green at the 15s test deadline -- Alice/Bob bidirectional sync AND late-join Charlie catches up via gossip. In-process tests unchanged (Step 4d space_two_node + Step 6f two_node_via_wires both still pass with mem transport + TestVerifier). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The wake-9 multi test ran with 60s deadlines as a hedge while the
cross-process gap was open. With Step 10c closed, tighten back to
15s -- both Bob-sees-Alice and Alice-sees-Bob complete inside
1.5s typically.
New "late-join Charlie sees historical diffs via gossip catch-up"
test boots a third conductor AFTER Alice + Bob have exchanged
their commits, joins via the same neighbourhood URL, and asserts
that the two historical links surface via K2 gossip within 30s.
Dedup before the set-equality assertion -- K2's gossip and
publish paths can both deliver the same op to a fresh joiner, so
the wire-level expectation is "the set of unique links contains
{a->b, c->d}".
holograph-link-multi.test.ts is now 5/5 green at the wake-10
boundary. Together with the substrate baseline (113) + test-simple
(2) + single-conductor (7) that brings the JS+substrate test
total to 127 green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Contributor
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
The blanket *.js gitignore was hiding the new deno extension JS file referenced by deno_core::extension! in holograph_service_extension.rs. Add an explicit unignore matching the pattern used for the other js_core extensions. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…(Step 11a) Per Nico's morning instruction: match the rest of ad4m, which uses Holochain's transport-iroh feature throughout. Iroh is QUIC-based direct P2P (no SBD signal relay required) and is K2's recommended default. Changes in rust-executor/src/holograph_wires.rs::build_dyn_space_inner: - transport: Tx5TransportFactory → IrohTransportFactory - Tx5TransportModConfig / Tx5TransportConfig → IrohTransportModConfig / IrohTransportConfig (matching field shapes: relay_url + relay_allow_plain_text instead of server_url + signal_allow_plain_text) - Env-var gate: HOLOGRAPH_SBD_URL → HOLOGRAPH_IROH_RELAY_URL - Plaintext flag: HOLOGRAPH_SBD_PLAINTEXT → HOLOGRAPH_IROH_PLAINTEXT - Fallback boot URL derivation: strip "/relay" off the configured iroh relay (kitsune2-bootstrap-srv serves both K2 bootstrap and the iroh relay endpoint on the same host:port). rust-executor/Cargo.toml: - kitsune2_transport_tx5 → kitsune2_transport_iroh dep at the same K2 rev (320a4d9). Iroh is already a workspace-level dep for the Holochain stack; no version drift. The in-process mem-transport path (no HOLOGRAPH_IROH_RELAY_URL) is unchanged from wake-10 — TestVerifier + TestLocalAgent + mem transport — so Step 4d's space_two_node and Step 6f's two_node_via_wires both stay green. Note: two-conductor JS test currently fails after this swap with peers=0 in publish_ops_to_peers — see blocker-step-11.md for the diagnosis (iroh net_report returns 400 on the bootstrap-srv relay probe, so `current_url()` stays None and CoreBootstrap can't publish AgentInfo). Single-conductor tests (test-simple, test-holograph-link) + all 113 substrate cargo tests stay green; the regression is isolated to cross-process op-flow via the new transport, not to the substrate or the JS isolate path. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…tep 11b) Match the Step 11a transport swap: - Drop HOLOGRAPH_SBD_URL / HOLOGRAPH_SBD_PLAINTEXT env per-conductor - Add HOLOGRAPH_IROH_RELAY_URL / HOLOGRAPH_IROH_PLAINTEXT, pointing at `<bootstrap-srv>/relay` (the same kitsune2-bootstrap-srv binary serves both K2 peer-discovery AND the iroh relay endpoint) - RUST_LOG debug target list swap: kitsune2_transport_tx5 → kitsune2_transport_iroh - Suite/describe block + Test 3 name from "via Tx5" → "via Iroh" - File-level docblock updated to reference wake-11 swap Single-conductor (tests/holograph-link.test.ts) needs no changes — it doesn't set HOLOGRAPH_IROH_RELAY_URL so the mem-transport path keeps running. This commit lands the test scaffold matching wake-11's transport swap; the test currently fails after the swap because the iroh relay/transport stack isn't yet discovering peers through the local bootstrap-srv. See .spike-status/blocker-step-11.md for the precise failure mode + wake-12 dispatch shape. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…tep 12b) After Bob joins the neighbourhood, CoreBootstrap polling needs ~5-15s on iroh to converge so that Alice's peer_store contains Bob's AgentInfo. Without this window, Alice's first commit fans out to peers=1 (herself only) and the diff never reaches Bob, even though test 4 (Alice receives Bob's later commit) and test 5 (Charlie historical catch-up via gossip) both pass naturally. The diagnostic logs from Step 10a made the asymmetry obvious -- Bob discovered Alice within 3s but Alice discovered Bob after ~20s because Bob's URL was published a tick later. Pad the first cross-process commit with a 15s settle so the bootstrap publish is guaranteed to have reached both sides before publish_ops_to_peers fans out. The natural fix is on the consumer side (a retry or wait inside HolographSpace::on_local_commit when peers=0), but that's a substrate behaviour change deserving its own PR. The test-side settle is the spike-acceptable workaround per the wake-12 dispatch. Combined with the matching bootstrap-srv (`kitsune2-bootstrap-srv 0.4.0-dev.5`, installed from the same K2 rev our workspace pins), this brings holograph-link-multi.test.ts back to 5/5 green on the iroh transport. The matched bootstrap-srv install closes the wake-11 blocker's iroh-relay version skew root cause. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Root cause for build-and-test failure on CI job 16305: the `build-and-test` job runs `pnpm test` for the root tests but does NOT run `build-languages` first (only the downstream integration jobs do). turbo's root `pnpm test` invokes `@coasys/holograph-link:test` which is `deno test --allow-all tests/smoke.test.ts`. The smoke test reads `build/bundle.js`, which doesn't exist if `build` hasn't run. Add a `pretest` script that runs the same esbuild command as `build`, so `pnpm test` self-builds the bundle when needed. This matches the pnpm/npm convention -- pretest runs automatically before test -- without requiring any change to the CircleCI job or the turbo task graph. Locally and in CI both `pnpm test` and `pnpm build && pnpm test` are now idempotent. Verified locally: removed `build/bundle.js`, ran `pnpm test` in `bootstrap-languages/holograph-link`, all 8 deno smoke tests pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…13b-A)
Wake-13 priority (2) start — widening the Step 1.5 algorithm extraction
per Nico's morning audio note. The narrow Step 1.5 (topo-sort only) is
becoming a wide extraction of the entire DAG algorithm.
This first move tackles `link_adapter::chunked_diffs` — the splitter
that batches large perspective-diffs into bounded chunks before they
land in the DHT.
Design pattern (now the precedent for the remaining 8 file moves):
- algorithm crate defines mirror types for the integrity-zome wire
shapes: `LinkExpression`, `PerspectiveDiff`, `Triple`,
`ExpressionProof`. Byte-for-byte compatible serde shape but no
HDI / SerializedBytes / `app_entry!` decoration. Lives in
`crates/perspective-diff-algorithm/src/diff_types.rs`.
- algorithm crate hosts the pure splitter/aggregator
(`new`, `add_additions`, `add_removals`, `into_aggregated_diff`,
plus the unit tests that don't need MockPerspectiveGraph) in
`crates/perspective-diff-algorithm/src/chunked_diffs.rs`.
- p-diff-sync's `link_adapter::chunked_diffs` becomes a thin HDK
adapter: it keeps the IO methods (`into_entries`, `from_entries`,
`load_diff_from_entry`) and the integrity-zome conversions, but
delegates the data manipulation to `AlgoChunkedDiffs`. The
integrity↔algorithm conversions are field-by-field (cheap; no
serde round-trip).
The public API of `ChunkedDiffs` stays the same for callers
(`commit.rs`, `pull.rs`), so no other p-diff-sync files needed
changes. The `.chunks` field is now a method (`chunks()`) returning
the integrity-zome shape; tests that read it for `format!("{:?}")`
debug-equality were updated accordingly.
Tests:
- 3 pure unit tests moved from p-diff-sync chunked_diffs::tests to
the algorithm crate (can_chunk, can_aggregate, can_chunk_big_diffs).
All green via `cargo test --release -p perspective-diff-algorithm`.
- 4 HDK IO tests stay in p-diff-sync (can_write_and_read_entries,
test_nested_chunked_entries_are_handled,
test_from_entries_with_mixed_chunked_and_inline,
test_loading_empty_chunked_entry_returns_empty_diff). All green
via `cargo test -p perspective_diff_sync --lib`.
- algorithm crate: 7 tests (was 4: topo_sort; +3: chunked_diffs).
- p-diff-sync zome: 33 tests (was 36; -3 moved to algorithm).
- holograph crate: still compiles.
- ad4m-executor: still compiles.
Remaining file moves (workspace.rs, snapshots.rs, revisions.rs,
render.rs, pull.rs, commit.rs, retriever.rs + retriever/mock.rs,
test_graphs.rs, tests.rs) follow the same pattern but each adds new
mirror types + new abstractions on the retriever trait — see
`.spike-status/step-13-status.md` for the remaining-work map.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…iever trait (Step 13b-C, phase 1)
Phase 1 of the workspace.rs extraction (Step 13b-C per the wake-13
status doc). Adds the algorithm-crate Workspace as a working parallel
implementation alongside p-diff-sync's existing HDK-coupled one;
phase 2 (wake-15) updates p-diff-sync's commit/pull/render to delegate
to this and removes the duplicate.
What's new in the algorithm crate:
- `diff_types.rs` extends with three new mirror types:
* `Hash(pub [u8; 39])` — wraps the 39-byte raw form of
`HoloHash<holo_hash::hash_type::Action>`. Custom `Serialize`/
`Deserialize` via a 39-byte serde_bytes-compatible visitor so
bincode/messagepack round-trips match HoloHash's own shape.
`from_raw_36(&[u8; 36])` is the bridge for the `NULL_NODE` /
test-generated payloads.
* `PerspectiveDiffEntryReference` — same fields as the integrity
zome's, but parents/diff_chunks use the new `Hash` mirror.
Includes `HasDiffParents<Hash>` impl so the existing
`topo_sort_diff_references` works on it directly.
* `Snapshot { diff_chunks: Vec<Hash>, included_diffs: Vec<Hash> }`.
Plus the `null_node()` free fn — equivalent of the integrity
zome's `ActionHash::from_raw_36(vec![0xdb; 36])` sentinel.
- `errors.rs` introduces `AlgoError` / `AlgoResult` — the algorithm
crate's compact, HDK-free error type. p-diff-sync's
`SocialContextError::from(AlgoError)` would handle the
HDK-boundary conversion (wake-15 hookup).
- `retriever.rs` defines the `WorkspaceRetriever` trait — the minimum
surface the in-crate `Workspace` needs:
`fn get_p_diff_reference(hash: &Hash) -> AlgoResult<PerspectiveDiffEntryReference>;`
`fn get_snapshot_by_target(target_hash: &Hash) -> AlgoResult<Option<Snapshot>>;`
p-diff-sync's `PerspectiveDiffRetreiver` keeps the rest (current_/
latest_revision, update_*, create_entry, etc.); wake-15 adds the
HDK impl of `WorkspaceRetriever` on `HolochainRetreiver`.
- `workspace.rs` ports the full Workspace struct + every algorithm
method from p-diff-sync's `link_adapter::workspace.rs`:
* `Workspace::new()`, `collect_only_from_latest`, `handle_parents`,
`sort_graph`, `build_diffs`, `terminate_with_null_node`,
`collect_until_common_ancestor`, `build_graph`,
`get_p_diff_reference`, `add_node`, `get_node_index`,
`find_common_ancestor`, `squashed_diff`, `all_ancestors`.
* Generic over `R: WorkspaceRetriever` everywhere the retriever
is needed; pure methods stay un-parameterized.
* Uses `null_node()` everywhere `NULL_NODE()` was used on the
HDK side.
* Replaces `SocialContextError`/`Result` with `AlgoError`/Result`
and `itertools::unique()` with a small in-fn `seen-set`
filter (algorithm crate stays light on deps).
* Removes the `print_graph_debug` and HDK `debug!` calls — the
algorithm crate doesn't depend on a logger.
- Tests: 5 of the 8 original `workspace::tests` ported to the
algorithm crate, using a small in-crate `MockRetriever` driven by
a minimal graphviz `digraph { ... }` parser
(`MockGraph::from_dot`). The remaining 3 tests
(`complex_merge`, `complex_merge_implicit_zero`, `real_world_graph`)
stay in p-diff-sync for now since they still pass against the
unchanged HDK Workspace; wake-15 will port them when p-diff-sync's
Workspace becomes the algorithm-crate's.
Test green-bar after Phase 1:
- `perspective-diff-algorithm` unit: 12 (was 7; +5 ported workspace tests)
- `perspective_diff_sync` lib unit: 33 (unchanged; original Workspace
+ chunked_diffs from Step 13b-A still in p-diff-sync)
- `holograph` crate: still compiles
- `ad4m-executor`: still compiles
Phase 2 (wake-15) will:
1. Make p-diff-sync's `Workspace` a re-export of the algorithm crate's
2. Impl `WorkspaceRetriever` for `HolochainRetreiver` and
`MockPerspectiveGraph` (bridge integrity types ↔ algorithm
mirrors)
3. Update commit.rs / pull.rs / render.rs to handle the new mirror
types (small conversion shims at the boundary)
4. Move the remaining 3 workspace tests + retire p-diff-sync's
`link_adapter::workspace` body
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ep 13b-C, phase 2) Wake-15 track 1: the substrate-agnostic `perspective_diff_algorithm::Workspace` is now the only Workspace. p-diff-sync's `link_adapter/workspace.rs` shrinks to a thin shim re-exporting it plus a legacy `NULL_NODE()` helper. Callers (`pull`, `render`, the HDK-boundary tests, and the holograph parity tests) convert `HoloHash<Action>` ↔ `algo::Hash` at the workspace boundary via the new conversions module. - Add `impl algo::WorkspaceRetriever` for `HolochainRetreiver`, `MockPerspectiveGraph`, and `KitsuneRetreiver` (holograph crate). Mock / Kitsune return Ok(None) for snapshot lookups (no snapshot links on those paths in this spike). - Add `SocialContextError::Algo(String)` and `From<AlgoError>` so `?` propagates cleanly through pull/render; the algorithm's `NoCommonAncestorFound` variant maps to the existing `SocialContextError::NoCommonAncestorFound` so any pattern matches still fire. - Port 3 workspace tests (`complex_merge`, `complex_merge_implicit_zero`, `real_world_graph`) to the algorithm crate, taking its workspace coverage from 5 → 8 tests. p-diff-sync's `link_adapter::tests` (3 HDK-boundary tests) retained with `hash_to_algo` at the call. - `conversions` module promoted from `pub(crate)` to `pub` so the holograph crate (which now depends on perspective-diff-algorithm directly) can use it. Tests green: perspective-diff-algorithm (15/15), perspective_diff_sync lib (24/24), holograph (48/48 — 43 lib + 4 pdiff_parity + 1 two_node), ad4m-executor (cargo check clean). `cargo fmt --all --check` clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ff_aggregated helper (Step 13b-D prep) Extract the chunked-entry resolution logic from Workspace::handle_parents into a free function `chunked_diffs::load_diff_aggregated<R: WorkspaceRetriever>` so the snapshots module (being extracted next) can reuse it without duplicating the nested-chunking fan-out. Behaviour unchanged. 15/15 algorithm-crate tests green. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…eat/holograph-substrate
… (Step 13b-D, phase 1)
Move the snapshot-generation algorithm out of p-diff-sync into the
algorithm crate, parameterized on the algorithm-crate mirror types and
a new `SnapshotRetriever: WorkspaceRetriever` trait that adds a single
write method:
fn create_diff_entry(entry: PerspectiveDiffEntryReference) -> AlgoResult<Hash>;
`SnapshotRetriever` is split off from `WorkspaceRetriever` so the
workspace tests (and Workspace-only callers like `render`) don't have
to wire a write surface they never exercise.
`snapshots::generate_snapshot<R: SnapshotRetriever>(latest, chunk_size)`
mirrors the HDK-side flow:
1. Walk parents from `latest` (DFS with sibling-branch deferral).
2. At each node, aggregate inline / chunked diffs.
3. At boundary nodes (`diffs_since_snapshot == 0`) with a `Snapshot`
link, fold the prior snapshot's diffs into the aggregator and mark
its `included_diffs` as seen.
4. Chunk the aggregated diff, write each chunk via `create_diff_entry`,
return the assembled `Snapshot` (caller persists it).
Includes 2 in-crate tests:
- `collects_inline_chain_into_chunked_snapshot` — 4-node linear chain →
4 link expressions across the new snapshot's chunks.
- `folds_previous_snapshot_into_new_one` — prior snapshot's chunks +
included_diffs are carried into the new snapshot, plus the
unsnapshotted tail.
p-diff-sync's `link_adapter::snapshots` becomes a thin shim in the
next commit. 17/17 algorithm-crate tests green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ep 13b-D, phase 2) Replace `bootstrap-languages/.../link_adapter/snapshots.rs` (~250 LOC of DAG walk + chunk aggregation) with a ~15-line HDK adapter that delegates to `perspective_diff_algorithm::generate_snapshot`. Pattern matches the 13b-C Workspace consolidation. - Impl `algo::SnapshotRetriever` on all three substrates: HolochainRetreiver, MockPerspectiveGraph, KitsuneRetreiver. Each reuses the existing `PerspectiveDiffRetreiver::create_entry` to persist chunk-diff entries and returns the resulting hash in algo form. - HDK shim reads `*CHUNK_SIZE` from the lazy_static config and converts the integrity-zome `Snapshot` ↔ `algo::Snapshot` at the boundary via the conversions module (drops the now-unused `#[allow(dead_code)]` on `snapshot_from_algo`). Tests green across all three crates + ad4m-executor: - perspective-diff-algorithm: 17 / 17 (15 prior + 2 new snapshot tests) - perspective_diff_sync lib: 24 / 24 - holograph: 48 / 48 (43 lib + 4 pdiff_parity + 1 two_node) - ad4m-executor: cargo check clean `cargo fmt --all --check` clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…tep 13b-E)
Move the `link_adapter::revisions` wrappers out of p-diff-sync into the
algorithm crate. Adds the `HashReference` and `LocalHashReference`
mirror types in `algo::diff_types` and a sibling `RevisionsRetriever`
trait that all three substrates (Holochain, Mock, Kitsune) implement.
- algo `diff_types`: new mirror types `HashReference` /
`LocalHashReference` (both `{ hash: algo::Hash, timestamp:
chrono::DateTime<Utc> }`). Adds chrono as a direct algo-crate dep.
- algo `retriever`: new `RevisionsRetriever: WorkspaceRetriever` sibling
trait with `current_revision`, `latest_revision`,
`update_current_revision`.
- algo `revisions`: thin wrappers `current_revision::<R>()`,
`latest_revision::<R>()`, `update_current_revision::<R>(hash, ts)`.
- p-diff-sync `link_adapter::revisions`: now a 25-line HDK shim that
preserves the legacy integrity-zome return type
(`Option<integrity::LocalHashReference>`) so pull/render/commit
callers don't yet need mirror types.
- p-diff-sync `link_adapter::conversions`: + `hash_ref_to_algo` /
`hash_ref_from_algo` and `local_hash_ref_to_algo` /
`local_hash_ref_from_algo` (field-by-field copies).
- pull / render / commit / handle_broadcast / broadcast_current: add
the `algo::RevisionsRetriever` trait bound. Same call surface, just
a wider bound on the generic.
- Holochain / Mock / Kitsune retrievers: impl `RevisionsRetriever`
forwarding to their existing `PerspectiveDiffRetreiver` revision
methods via the new mirror-type conversions.
Tests green:
- perspective-diff-algorithm: 17 / 17
- perspective_diff_sync lib: 24 / 24
- holograph: 48 / 48 (43 lib + 4 pdiff_parity + 1 two_node)
- ad4m-executor: cargo check clean
`cargo fmt --all --check` clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This was referenced Jun 3, 2026
Draft
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Holograph v1 substrate spike — proves we can replace Holochain with Kitsune2 + executor-hosted PdiffSync without forking K2 and without a Holochain conductor. Feasibility-proven, draft PR for review and discussion before any default-switch decisions.
Companion spec lives in
memory/holograph/SPIKE_NIGHT_2026-06-02.mdandmemory/holograph/HOLOGRAPH.mdin the clawd workspace; happy to inline parts into the repo if reviewers want it co-located.What landed
SpaceConfig+ CBOR envelope (with optionaldoc_idfor sharding-readiness)rust-executor/crates/holograph/PerspectiveDiffRetreiverbootstrap-languages/p-diff-sync/.../retriever.rscrates/perspective-diff-algorithm/— substrate-agnostic algorithm crate; HolochainRetreiver path still greencrates/perspective-diff-algorithm/KvOpStoreimplementing K2OpStore(11 methods) +KitsuneRetreiverimpl + p-diff-sync parity testsrust-executor/crates/holograph/src/{op_store,kitsune_retriever}.rsHolographIntegrationQueue— cascade promotion + multi-peer fallback for source-bound K2 fetchrust-executor/crates/holograph/src/integration_queue.rsHolographSpace+ K2 adapter glue + two-node end-to-end via real K2 mem transportrust-executor/crates/holograph/src/space.rsholograph-linkAD4M LinkLanguage scaffold +holograph_wiresJS-facing import surfacebootstrap-languages/holograph-link/,rust-executor/src/holograph_wires.rsHolographRuntime(replaces NotImplemented stub) +__holographDelegate__Deno extension + env-gated default switch (HOLOGRAPH_DEFAULT_NEIGHBORHOOD=1)rust-executor/src/holograph_runtime.rs,core/src/links/HolographLinkAdapter.tsprint_holograph_addressbinary + snapshot extraction + single-conductor JS scaffoldrust-executor/src/bin/print_holograph_address.rs, JS testshost.jsrust-executor/crates/holograph/src/space.rscargo fmt --allover the whole workspaceTest results
cargo test -- --test-threads=1), includes new p-diff-sync parity tests againstKitsuneRetreiverSPIKE §2.5 exit checks
cargo build --releaseclean for newholographandperspective-diff-algorithmcratescargo test --release -- --test-threads=1green forperspective-diff-algorithmHolochainRetreiverandKitsuneRetreiverHolographRuntimewiresHOLOGRAPH_DEFAULT_NEIGHBORHOOD=1What is not in this PR (deferred)
Blockstruct — flat parents on the envelopeOpId::set_loc_callback— default hash-locHow to review
rust-executor/crates/holograph/src/lib.rs, thenop_store.rs,kitsune_retriever.rs,integration_queue.rs,space.rs. TheOpStoretrait surface is the 11 methods atkitsune2_api/src/op_store.rs:68-183.crates/perspective-diff-algorithm/— narrowed move frombootstrap-languages/p-diff-syncper Step 1.5. The retriever trait + workspace + render/pull/commit modules. p-diff-sync now depends on this crate.bootstrap-languages/holograph-link/for the JS module (thin),rust-executor/src/holograph_wires.rsfor the import surface,core/src/links/HolographLinkAdapter.tsfor the runtime side.tests/js/tests/holograph-link.test.ts— gated byHOLOGRAPH_DEFAULT_NEIGHBORHOOD=1, runs the full Alice/Bob/Charlie cross-process flow.Test plan
devbasetests/js/tests/holograph-link.test.tslocally withHOLOGRAPH_DEFAULT_NEIGHBORHOOD=1OpStoretrait impl against K2 source🤖 Generated with Claude Code