Skip to content

Commit 700b59a

Browse files
skhazwolfy-j
andauthored
feat(store): KV stores on the shared raft (store.kv.raft + store.kv.crdt) (#307)
* feat(raft): multiplex FSM router + optional fs-durable storage Foundation for store.kv.raft riding the single cluster raft. - cluster/raft/multiplex: domain-tagged FSM router (untagged -> global registry, 0xB2 -> kv). Combined snapshot framing with legacy-bare fallback for in-place upgrade; downgrade is one-way. - cluster/raft/storage: openStores selects diskless in-mem (default, unchanged) or fs-durable raft-wal log + bbolt stable store + file snapshots under Config.DataDir. - raft.Node Start/Stop use the strategy and close durable handles. - boot wires the router transparently (nil kv FSM) so behavior is unchanged until the kv FSM is plugged in. Deps: hashicorp/raft-wal, hashicorp/raft-boltdb/v2. * feat(kv): re-port coordination KV engine Port the wip in-memory KV engine onto its own api surface: - api/store/kv: Engine interface (string keys / byte values, versions, leases, watch) + sentinel errors. The api/store wrappers will transcode registry.ID / payload onto this. - system/kv: in-memory Service implementing Engine (single-goroutine event loop + atomic snapshot reads), plus the state/lease/watcher primitives the raft and crdt backends build on. Fixes latent lint from the wip branch (fieldalignment, unused) and ports the engine test suite. * feat(store): add store.kv.raft backend on the shared raft store.kv.raft is a raft-replicated, namespace-isolated, fs-durable KV store riding the single cluster raft via the multiplex router. - system/kv: raft command codec, RaftFSM (deterministic apply + gob snapshot/restore over the shared state), RaftEngine (local reads, leader proposes; leader-side lease expiry sweeper). Follower writes return a retryable ErrNotLeader (leader-forwarding is a planned follow-up). - service/store/kv: api/store.Store/Scanner/Atomic wrapper with namespace prefixing (apps see bare keys; @sys/* reserved) and payload<->JSON codec, plus the registry.EntryListener manager. - api/service/store/kv: store.kv.raft / store.kv.crdt kinds + config with namespace validation. - boot: raft component builds the kv FSM into the router and exposes the shared engine; store component registers the store.kv.raft kind. Tested: command round-trip, FSM apply/snapshot/restore, engine CRUD/CAS/ SetIfAbsent/scan/lease-auto-expiry, namespace isolation, and 3-node raft replication through the router over real consensus. * feat(store): add store.kv.crdt gossip backend store.kv.crdt is a gossip-CRDT, namespace-isolated KV store with optional fs durability, over the existing system/crdt substrate. - system/kv: CRDTEngine implementing kvapi.Engine (epidemic delta broadcast + full-state push/pull anti-entropy, local-reaper leases). A per-node wall tiebreak gives crdt.State.Apply a total order so concurrent same-millisecond cross-node writes converge instead of diverging. Per-namespace fs snapshots when durable. Plus the membership user-delegate (kind 0xD2). - service/store/kv: CRDTManager reusing the shared Store wrapper; durable entries mark their namespace for persistence. - boot: kv-crdt system component builds the engine + registers the gossip delegate; store component registers the store.kv.crdt kind. CAS/SetIfAbsent are best-effort under eventual consistency (use store.kv.raft for linearizable conditional writes). Tested: ops, two-replica convergence incl. concurrent conflicts, full-state anti-entropy, per-namespace durable restart, and lease auto-expiry. * chore(raft): field-align Config after DataDir addition Reorder Config fields so govet fieldalignment is clean with the new DataDir string field. No behavior change (JSON tags preserved). * fix(store): bring store.kv.* to spec — durable-by-default raft, ":" keys, _sys - Raft is now fs-durable by default: node data_dir resolves to ~/.wippy/store (or cluster.raft.data_dir), shared raft + kv under <data_dir>/_sys/raft. Satisfies "raft = durable, period" (the global registry, sharing this raft, is now fs-persisted too — intended under the unified-KV direction). - Physical key separator "/" -> ":" to match the spec: <namespace>:<id>. - Reserve the _sys namespace for system coordination (app namespaces already excluded by the leading-[a-z] rule); _sys/kvcrdt for durable crdt snapshots. - Example store.kv.* config (docs). fs.directory-entry resolution for data_dir is a follow-up; explicit path and the ~/.wippy/store default are in. * feat(store): kv leader-forwarding for store.kv.raft writes Follower writes to store.kv.raft previously returned a retryable ErrNotLeader. The engine now forwards a write to the raft leader over the relay (host "storekv", req/resp topics), awaits the applied result, and re-resolves the leader on a mid-flight election. Error sentinels (ErrKeyNotFound, etc.) are preserved across the wire via kind codes so errors.Is still works. This makes store.kv.raft writes work from any node and is the prerequisite for migrating registry/locks/leases onto the shared kv (Part B). Tested: follower->leader forward applies on the leader and returns the version; forwarded delete-missing surfaces ErrKeyNotFound; follower with no router still returns ErrNotLeader. * test(store): real-stack multi-node cluster harness + E2E proofs cluster/clustertest boots N real raft nodes over an in-process internode mesh (async FIFO delivery + partition/kill/restart injection), each with the multiplex(primary+kv) FSM, the kv RaftEngine registered as a relay host for leader-forwarding, and a durable per-node DataDir. Faithfully models gossip NodeLeft by tearing down raft mesh sessions on partition/heal/kill/restart so they rebuild. E2E proofs (real consensus + real relay forwarding, -race): - follower write forwarded to leader, committed, replicated to all nodes - leader hard-kill failover; writes survive and continue - partition a follower, write on quorum, heal -> isolated node catches up - single-node durable kill+restart recovers from raft-wal+bbolt on disk - one shared raft multiplexes a non-kv command alongside kv commands * fix(crdt): deterministic total-order merge tiebreak + property tests Property testing found a real store.kv.crdt non-convergence bug: on an equal-wall, cross-origin conflict, State.Apply kept whatever entry the replica saw first (MergeNoop) — not a total order — so replicas could diverge on a key under multi-writer churn. Fix: when walls tie, resolve deterministically — delete beats live write (symmetric), else by global origin string then counter (comparing the origin STRING, not the replica-local interned id, so the order is identical on every replica). Idempotent for identical dots. The registry shares crdt.State; its eventual + global suites stay green. Adds property/fuzz tests: command codec round-trip, decode-never-panics on arbitrary bytes, namespace round-trip + cross-namespace isolation, and the CRDT convergence property (random ops/order converge on all replicas). * test(store): crdt real-gossip E2E + raft concurrency/linearizability stress - crdt_gossip: 3-node real memberlist cluster with the crdt delegate; a write converges to all nodes via delta + anti-entropy, including a node that joins after the write (full-state backfill). Proves the gossip delegate wiring. - stress (real raft, -race): concurrent CAS-increments of a shared key from random nodes (forwarded to the leader) are exactly-once — final value equals total increments, no lost updates (linearizability + forwarding under load); and many concurrent distinct-key writes from all nodes replicate everywhere. * feat(store): B1 — distributed locks on the shared kv (_sys:lock) system.lock now lives in the one shared KV instead of a separate global.Strong registration: - system/kv LockService: Acquire = SetIfAbsent(_sys:lock:<name>, holderPID) (linearizable via raft + leader-forwarding); Release verifies the holder; Holder reads the owner. Auto-release on holder process exit (topology Monitor -> the "syslock" relay host -> ReapPID) and on node leave (cluster.NodeLeft -> ReapNode); reaps are idempotent across nodes. PIDs compared by String() (the struct carries an unexported cache). - Lua system.lock module retargeted to the LockService (same acquire/release contract); global.Strong stays available until the B5 cutover. - boot registers the lock relay host, reaps on NodeLeft, and exposes the service in context. Proof: unit (acquire/release/holder/ReapPID/ReapNode), retargeted Lua module tests, and a 3-node E2E (mutual exclusion + replication + auto-release after the holder's node is killed — no deadlock). * fix(store): B2 — leases auto-expire cluster-wide regardless of grant node The lease sweeper only re-armed deadlines on a leadership change, so a lease granted on a follower (e.g. a TTL'd store.Set forwarded to the leader) was never scheduled by the current leader and would not expire. The leader now re-arms from replicated state every sweep tick (idempotent), so any lease — granted on any node — is swept and its revoke replicated. Leases are the substrate for _sys:lease (reserved system namespace) and TTL'd store.Set; there are no separate external lease consumers to migrate. Proof: 3-node E2E grants a lease + set-with-lease on a FOLLOWER and asserts the key auto-expires on every node. * feat(kv): transaction primitives for the registry on kv Add compare-and-delete, per-key epoch (= raft log index), atomic multi-key txn (two-pass all-or-nothing), and barriered GetLinearizable / ScanAtIndex on a LinearizableEngine. CRDT backend rejects Txn. Snapshot gains an additive gob Epoch field (no snapshotVersion bump; legacy snapshots still restore). LockService.Release now uses CompareAndDelete, closing a read-then-delete race. Tests: txn codec, CAD, all-or-nothing + atomic-promote, epoch stamping, scan-at-index, snapshot+legacy-restore; cluster E2E for txn/CAD replication from a follower and cross-node epoch agreement. * feat(registry): kv-backed CONSISTENT name registry New system/topology/namereg/kvbacked package implementing the globalapi.Registry facade on the shared kv under _sys:registry:* — register/lookup/unregister/remove/removenode via atomic kv txns with reverse PID/node indexes, first-write-wins conflict resolution with an override-resolve CAS swap, and ByPID lookup. Adds a forwarded leader-read (GetViaLeader) to the kv engine for read-your-writes on a follower after a forwarded write; in-mem Service now satisfies LinearizableEngine. Tests: consistent register/lookup/dedupe/conflict/override/ByPID/ unregister/remove/removenode incl. colon-containing names; cluster E2E proves follower-register replicates with a real raft-index epoch, conflict is rejected cluster-wide, and remove clears every node. * feat(registry): kv-backed STRONG name registry Strong-scope reservations on the shared kv: a pending header key, one ack key per required node (each ack is a raft-replicated kv write — no ack relay), atomic leader promotion via a conditional multi-key txn, deadline expiry, and cross-scope reject. Per-node exclusions block LOCAL/EVENTUAL shadowing during the promotion window; reconcile(name) is the single idempotent state-machine step driven by a kv-Watch reconciler. Conflict surfaces StrongConflictError, timeout StrongRegistrationTimeoutError. Tests: promote, ack-missing timeout, reject-on-conflict, reservation window + release, unregister; 3-node E2E proves a follower-opened reservation collects every node ack and promotes atomically, resolvable cluster-wide. * feat(registry): non-member dissem read-through + boot backend flag Phase 4: kv-backed registry resolves names on non-member nodes (no raft FSM) by reusing the 0xC1 dissem plane. A kv-Watch reconciler translates active-binding changes into BindingEvents (dot = raft index via the new WatchEvent.Index), Lookup falls through to the dissem cache on a local miss, and tombstones gossip on delete. Phase 5 (wiring): cluster.raft.registry_backend flag (fsm default | kv). When kv, boot builds the kv-backed Service with Strong (membership + leader + cross-scope local-conflict), dissem, the process-exit/node- leave reaper, and the watch reconciler, registered on the same facades; default fsm leaves the existing path unchanged. Adds a process-exit relay host + node-leave DropNode (drop-required so promotion completes on survivors). Tests: non-member resolves via gossip + tombstone; reconciler broadcasts on register. Full -race suite green (default path unchanged). docs/store-kv-registry-traceability.md maps requirements to proofs. * fix(registry): adversarial-review findings + watch publish-ordering Multi-agent review (6 confirmed bugs, all fixed): - dissem RunGC never started on the kv path -> start + ctx-stop in the reconciler (tombstones now swept). - forwarded read/write select race could drop a response that arrived with the timeout -> final non-blocking drain. - resolveConflict retry exhaustion -> more retries + retryable error. - entryToAPI/entryToCluster dropped Epoch -> watch put-dot was 0; copy Epoch and use the op raft index as the dissem dot. - reconcile blocked the watch goroutine on forwarded reads -> local reads on the watch path; register drives the just-opened pending directly (a local read cannot see the freshly-forwarded write). - strong deadline expiry used non-linearized reads -> barrier only on the deadline path before giving up. Root-cause fix exposed by the above: the kv FSM emitted watch events before publishing the new snapshot, so a watch-driven local read missed its own change. Buffer events and flush after snap.Store. Full -race suite green; lint clean. * test(registry): capstone — kv registry survives leader kill Real-raft 3-node E2E: a CONSISTENT name registered on the leader stays resolvable cluster-wide after the leader is killed, and registration continues on a survivor after failover. * test(registry): real-binary kv boot smoke + traceability tests/capstone/kv-boot-smoke.sh boots the wippy binary with registry_backend=kv and runs the app suite: process registry suite 46/46 and security:registry pass, no kv-path panic (7 env-only failures: docker/hub). Documents the deferred default-flip + FSM decommission as a deliberate one-way operational cutover. * fix(registry): round-2 review — register fail-fast + reap owner-guard - register(): decode the pending before installing the waiter; on an (impossible) decode failure, fail fast + remove the bad entry instead of blocking the waiter forever. - deleteBinding(): always clear the dead PID reverse-index keys, but delete the active binding only if it still belongs to that PID (owner + version guarded) — reaping a dead PID can no longer clobber a name already reassigned to a live PID. Regression test added. Full -race suite green; lint clean. * fix(registry): round-3 review — bounded wait, join barrier, leader sweep - register(): bound the wait with a backstop deadline (grace after the Strong deadline) so a skipped-delivery reconcile path cannot hang. - NameReady(): gate on the reconciler having seeded (learned+latched the cluster Strong reservations) instead of always true — join-barrier semantics; a node cannot shadow a Strong name before learning it. - leader sweep: periodically re-drive in-flight pendings while leader, so a follower promoted to leader re-arms deadline timers and resumes promotion/expiry; backstops missed watch events. - deleteBinding(): local read (not leader-forwarded) so the reaper never blocks the relay-receiver goroutine; CondVersion still guards. - StrongRegistrationTimeoutError now reports MissingAcks. Full -race suite green; lint clean. * fix(registry): round-4 review — deterministic terminal + non-member forward - onTerminal delivers an epoch-less Expired signal; finalize reads the authoritative epoch/reason/missing-acks from terminal tracking, so concurrent onTerminal calls (watch + recursive reconcile + timer) yield a deterministic outcome. (Rejected the suggested had-guard, which would have broken the fast reject path.) - Lookup gains a non-member cold-miss forward-resolve via GetViaLeader (gated by SetNonMember, default off) so a client that joined before gossip converged still resolves an active name; members never forward. Regression tests for both. Full -race suite green; lint clean. * fix(registry): round-6 review — recover active Strong exclusions on seed - seed() now also scans active Strong names and reconciles them, so a node restart / snapshot-restore re-latches their in-memory exclusions before NameReady flips. Without this, recovered active Strong names reported IsStrongReserved=false, letting a LOCAL/EVENTUAL register shadow a live Strong name. Recovery regression test added. - handleWatchEvent surfaces a (design-violating, never-expected) WatchExpired on a registry key as a debug log; still handled safely as a delete. Full -race suite green; lint clean. * fix(registry): round-7 review — invalid-PID guard + drop-node key cleanup - reconcile() skips an active Strong entry with an unparseable PID (debug log) instead of latching a zero-PID exclusion. - dropNodeFromPending() atomically deletes the departed node’s ack/reject keys along with removing it from RequiredNodes, so they are not orphaned (promote/expire only sweep keys for remaining required nodes). Full -race suite green; lint clean. * fix(registry): round-8 review — cross-scope CONSISTENT/STRONG + read bound - registerConsistent refuses (ErrPendingConflict) when a STRONG reservation is pending for the name; resolveConflict reports it instead of retrying forever (cross-scope invariant: a pending owns the name). - leaderPromote resolves a genuine conflict (active held by another binding) to a terminal expiry instead of retrying until the deadline, so the waiter never hangs. - handleReadResp surfaces a truncated forwarded-read value as an error rather than a silent empty value. Cross-scope regression test added. Full -race suite green; lint clean. * fix(registry,kv): round-9 review — STRONG precedence + CAS watch Previous - resolveConflict refuses to displace a STRONG-scope active owner with a CONSISTENT register even under a custom ResolveFunc (precedence Strong>Consistent enforced). - CompareAndSwap watch events now carry the correct Previous entry (was nil in the raft FSM, post-mutation value in the in-mem service) — a general kv watch-correctness fix. Regression tests: cross-scope no-displace; CAS watch Previous. Full -race suite green; lint clean. * fix(dissem): round-10 review — deterministic same-index LWW tie-break Dissem merge: at an equal RaftIndex (a degenerate case — distinct ops carry distinct raft indices), a live binding deterministically wins over a tombstone regardless of gossip arrival order. Defensive hardening of the shared dissem plane; global + kvbacked dissem suites stay green. Also: traceability doc updated with the 10-round review convergence (26 bugs fixed) and the deferred operational/cross-cutting items. * feat(registry): default to the kv-backed name registry Flip cluster.raft.registry_backend default fsm->kv: the kv-backed registry on the shared raft (_sys:registry) is now the live name registry; set registry_backend=fsm to fall back to the dedicated FSM. Verified: full go test ./... -race green (291 pkgs, 0 fail) with kv as default; real binary boots with the kv default (capstone, process registry suite 46/46, no kv-path panic); lint 0 issues. * test(registry): non-member (client) resolution E2E over the real relay Proves a registry non-member (no raft FSM, all ops forwarded to the leader via the relay) resolves a member-registered name through the cold-miss forward-resolve, and registers its own name (forwarded) that members then resolve. Verifies the non-member resolution mechanism end-to-end in-process. * feat(kv): server-side re-forwarding so a non-member reaches the leader via any member A registry non-member (role=client) has no raft state and cannot call raft.Leader(), so it forwards ops to an arbitrary member. A non-leader that receives a forwarded read/write now re-forwards it to the leader it can resolve, hop-bounded (maxForwardHops) and off the relay Send goroutine so the re-forward's own reply (delivered via the same Send) cannot self-deadlock. Proven E2E under -race: a client targeting a follower has both cold-miss forward-resolve and register-via-forward relayed to the leader, and members then resolve the client-registered name. * feat(boot): wire the kv name registry on role=client nodes A node that runs no raft Node (cluster.raft.role=client / raft.enabled=false) previously wired no global name registry at all. loadClientRegistry now builds the forward-only kv engine (systemkv.ClientSubmitter), a non-member kvbacked.Service, the dissem read-through delegate, and the two registry facades -- mirroring the server path so consumers stay backend-agnostic. The forward target is sysraft.PickForwardTarget(membership.Nodes(), self): a pure, deterministic, eligible-only, self-excluding choice. Membership reports only live peers, so a departed target drops out and the next call re-picks with no failover bookkeeping; server-side re-forwarding makes any member a valid entry point. Guards no-op (preserving prior client behavior) when the kv backend is not selected or relay/membership/topology are absent. Verified: PickForwardTarget, ClientSubmitter, and the no-op guards by unit tests; the production ClientSubmitter end-to-end (resolve+register via leader and via a re-forwarding follower) in cluster/clustertest. Full build + go test ./... -race green; golangci-lint 0 issues. Not verified here: a real client process joining a real cluster (the in-process harness has no gossip membership layer). * test(capstone): real-binary boot smoke for the role=client kv registry Boots the actual wippy binary with cluster.raft.role=client + registry_backend=kv and asserts loadClientRegistry runs in a real process (logs the wiring line) with no panic. Verified: the binary emits 'raft(client): kv name registry wired (non-member, forward-resolve)' at boot. shellcheck clean. * test(store/kv): prove reserved _sys namespace rejection Why: the <namespace>:<key> isolation rule and the reserved _sys coordination namespace are enforced by namespacePattern but were never exercised by a test. What: table tests on RaftConfig/CRDTConfig.Validate asserting _sys*, colon-bearing and malformed namespaces are rejected while deploy/sess/kb/kb_index pass; durability is orthogonal to the name rule. * fix(registry): complete strong reservation on survivors across a leader change Why: leaderDrive checks completion against the stored RequiredNodes; a required node that departed coincident with a leadership change was only pruned via the old leader's NodeLeft, so a new leader kept the stale set and the reservation expired at its deadline instead of promoting on the surviving acks. What: pruneDepartedRequired drops from a pending reservation's RequiredNodes any node absent from the live membership (reusing dropNodeFromPending), called on the leader in reconcile before leaderDrive, so it runs on seed takeover and every leaderSweep tick. A node still present but merely un-acked is never pruned, so the all-live-must-ack guarantee is preserved. Proof: unit TestStrong_PromotesOnSurvivorsWhenRequiredNodeDeparts (fails pre-fix, passes post-fix); cluster TestE2E_KVRegistry_StrongPromotesOnSurvivorsAfterLeaderKill (real leader kill, 3x clean) and ConsistentReapedOnNodeDrop; restart-durability refutations TestE2E_DurableRestartPreservesEpoch (epoch never resets) and TestCRDTEngine_DurableTombstoneSurvivesRestartNoResurrection. * fix(raft): use a bbolt log store on Windows where raft-wal can't fsync its dir Why: raft-wal fsyncs the WAL directory during init, which returns "Access is denied" on Windows (directory fsync is not permitted there), so store.kv.raft durability failed to initialize and CI Test (windows-latest) failed in TestOpenStores_DurableRoundTrip. What: openStores now dispatches the LogStore by OS — raft-wal on Linux/macOS (the production platforms, higher write throughput), a bbolt LogStore (the same cross-platform engine already used for the stable store) on Windows. Non-Windows behavior is unchanged. openBoltLogStore is exercised by a round-trip test on every OS so the Windows path is proven without a Windows runner; cluster/raft cross-compiles for windows/amd64. * test(raft): sound anti-inflation bound for the concurrent dead-streak test Why: TestPeerStateTracker_ConcurrentTripBoundedByStreak asserted dead-streak <= ceil(G/failureLimit)=2, but the dead window is published just after the trip is claimed via CAS, so a scheduling-dependent number of failures already in flight past isDead can claim a few more trips before gating engages — CI legitimately observed 3, failing the too-tight bound. What: assert the deterministic invariant the CAS guarantees — G goroutines' failures collapse into strictly fewer than G trips (anti-inflation) — instead of a fixed numeric count. Stress-verified 200x under -race. * chore(build): module-first go.mod + gremlins mutation-testing target Why: mutation testing was requested. gremlins reads the module name from the FIRST line of go.mod, but the SPDX comment preceded the module directive, so it mis-parsed the name and matched zero coverage to mutants (0% on every run). What: move the `module` directive to go.mod line 1 (the Go convention; the SPDX identifier is retained just below it) so gremlins — and any tool that naively reads go.mod line 1 — resolves the module name correctly; add a `make mutation` target (gremlins via `go run`, coverage scoped per MUTATE_DIR, workers=1 + a generous timeout coefficient for deterministic, contention-free scores). Verified: service/store/kv scores 18 killed / 7 lived / 72% efficacy, repeatable. * fix(eventual): gossip-replicate registry deltas to all members EVENTUAL deltas were emitted one-shot to GossipNodes peers and never re-broadcast by receivers, so nodes outside that set (incl. raft-less client nodes) converged only via slow anti-entropy (a 7-node post-join name took ~38s to reach the tail). - applyIncoming re-broadcasts entries that change local state (epidemic, loop-free via CRDT idempotency). - LocalState carries the authoritative sender id; MergeRemoteState targets shard-pulls at it instead of a max-counter heuristic, and skips self. - PushPullInterval 15s -> 5s backstops the rare epidemic tail-miss. Convergence drops to sub-second (common) / a few seconds (worst). Deltas become process.send(name,...)-resolvable on client nodes. * test(eventual): client-publish + bidirectional reg across non-raft nodes Why: the gossip-replicate fix proved a raft-less client can RESOLVE a server-published eventual name, but not that it can PUBLISH a name servers resolve. The publish direction is the other half of "eventual name reg works on non-raft nodes" and was untested. What: add TestEventual_ClientPublishResolvedByServers (client registers, both servers resolve via epidemic forward + anti-entropy) and TestEventual_ClientServerBidirectional (client and server each register, every node resolves both), reusing the membership eventual harness. Green with -race. * fix(store): version-guard the distributed-lock reaper (B1) Why: the lock auto-release reaper ran on every node off a possibly-stale local snapshot and deleted matched locks unconditionally. A reap that raced a release+reacquire — or two nodes reaping the same departed holder — could delete a lock a live holder had just acquired, breaking mutual exclusion (two holders). What: reap now captures the version observed during the scan and deletes via CompareAndDelete, so a stale victim is a no-op (OK=false) instead of a clobber — the same guard Release already uses. Adds TestLock_ReapDoesNotClobberReacquired Lock, which interleaves a reacquire between the reaper's scan and delete and fails on the old unconditional path. * fix(store): GC store.kv.crdt tombstones to bound memory (B2) Why: every Delete (and lease expiry) on the crdt backend left a tombstone that was never garbage-collected — State.ReapTombstones existed but had no caller for this engine — so put/delete churn grew local state, every gossip frame, and every durable snapshot without bound. What: add a periodic wall-floor tombstone reaper to the crdt engine. A tombstone is dropped only once it is older than crdtTombstoneFloor (15m, mirroring the eventual registry's DefaultWallFloor), well past the anti-entropy window, so a lagging peer cannot resurrect it. CV-gated (safe-counter) reaping is not used: the store.kv.crdt delegate exchanges no per-origin digests, so safeByNode is nil and the wall floor is the sole bound. Adds TestCRDTEngine_TombstoneGCBoundsMemory asserting recent tombstones are retained and aged ones reaped. * fix(store): make forwarded writes safe to retry (B3) Why: a follower forwarding a write to the leader treated a reply TIMEOUT as a not-leader rejection and retried it. But a timeout is ambiguous — the leader may have committed the op — so the retry re-applied a non-idempotent write (Set/CAS/ Delete double-bumped the version or returned a spurious ErrKeyNotFound). What: split the two outcomes. A not-leader reply means the op was provably not applied (a leader that loses leadership mid-Apply does not commit), so it stays retryable; a genuine timeout now returns errForwardTimeout, which the caller does NOT retry and surfaces as an honest write error. The follower's wait is also sized to raftApplyTimeout + grace so a slow-but-committed reply arrives instead of spuriously timing out. Adds TestRaftEngine_ForwardTimeoutDoesNotDoubleApply (leader commits, reply dropped → exactly one apply). * fix(store): persist absolute lease deadline across leader change (B4) Why: lease expiry used a leader-local now+TTL deadline with nothing replicated. On every leadership change the new leader re-armed from TTL, discarding elapsed time, so a lease could outlive its TTL by up to one failover — and ~N×TTL under N failovers (a flapping leader during a partition), contradicting "leases auto-expire cluster-wide regardless of grant node". What: the grant/renew command now carries an absolute deadline (grantor wall + TTL); the FSM stores it in lease state and snapshots it; leaseSnapshot returns it and a re-arming leader honors that deadline instead of resetting to now+TTL. Adds TestRaftEngine_LeaseDeadlineHonoredAcrossReArm (a past-deadline lease expires on re-arm) and TestRaftFSM_LeaseDeadlineSurvivesSnapshot; command round-trip test covers the new field. * fix(raft): reset an absent child domain on multiplex restore (H1) Why: raft calls FSM.Restore to REPLACE all state, but the multiplex router only restored the child domains physically present in the snapshot. A framed snapshot written by a kv-disabled node (or a legacy bare-primary snapshot) carries no kv section, so restoring it over a node that already had kv state left that state intact — the kv FSM then diverged from the cluster after an InstallSnapshot. What: the router now resets the kv child when no kv section was seen (framed and legacy paths). The kv FSM treats an empty Restore stream as reset-to-empty rather than erroring. The primary section is always written, so only the kv child can be absent. Adds router tests that restore over an existing kv FSM and a kv-FSM empty-restore test; the previous restore tests only used fresh child FSMs. * test(raft): kill surviving multiplex mutants (100% efficacy) Why: gremlins left 4 live mutants in the router — the empty-entry routing guard, the short-stream magic-read guard, and the kv-section persist error check were unverified. What: add tests for empty-Data Apply routing to primary, short/empty Restore streams treated as legacy, and a Persist that must not cancel the sink on a successful kv write. Multiplex mutation efficacy 87% -> 100%. * test(store): cover forwarding, durable crdt + tolerant decode (mutation) Why: gremlins left the leader-side forward/read handlers, hop re-forwarding, durable-namespace reload, and the tolerant lease-deadline decode unexercised. What: add forwarded-read, hop re-forward (client->member->leader), forwarded conditional-write (incl. the version-mismatch wire path), durable-namespace reload, OnFrame apply+emit, and a decode-without-trailing-ExpiresAtMs test. * test(store): cover lock relay receiver + context helpers (mutation) Why: gremlins left the LockService.Send exit-event reaping, its topic/payload filters, and the WithLockService/GetLockService context helpers unexercised. What: add tests for a topology exit event reaping the holder's locks (and not others'), unrelated topics/payloads being ignored, and the context round-trip with and without an app context. * feat(store): expose lean Lua KV surface --------- Co-authored-by: Wolfy-J <wolfy.jd@gmail.com>
1 parent dd1dc2c commit 700b59a

109 files changed

Lines changed: 14400 additions & 382 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

Makefile

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -46,6 +46,17 @@ test-network:
4646
lint:
4747
go run github.com/golangci/golangci-lint/v2/cmd/golangci-lint@v2.8.0 run --timeout=10m --build-tags=race ./...
4848

49+
# Mutation testing with gremlins. Coverage is scoped to the directory gremlins
50+
# runs from, so target a package subtree via MUTATE_DIR. workers=1 keeps per-
51+
# mutant timing free of contention and the generous timeout coefficient stops
52+
# fast packages (whose cached coverage baseline is tiny) from spuriously timing
53+
# out while each mutant recompiles — together these make the score deterministic.
54+
# Example: make mutation MUTATE_DIR=system/topology/namereg/kvbacked
55+
MUTATE_DIR ?= service/store/kv
56+
.PHONY: mutation
57+
mutation:
58+
cd $(MUTATE_DIR) && go run github.com/go-gremlins/gremlins/cmd/gremlins@v0.6.0 unleash --workers 1 --timeout-coefficient 30
59+
4960
mock:
5061
go tool mockgen -destination tests/mock/identityv1connect/identityv1connect.go github.com/wippyai/module-registry-proto-go/registry/identity/v1/identityv1connect OrganizationServiceClient
5162
go tool mockgen -destination tests/mock/modulev1connect/modulev1connect.go github.com/wippyai/module-registry-proto-go/registry/module/v1/modulev1connect ModuleServiceClient,CommitServiceClient,LabelServiceClient,DownloadServiceClient

api/cluster/raft/config.go

Lines changed: 10 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -6,9 +6,10 @@ import "time"
66

77
// Config holds configuration for a Raft node.
88
//
9-
// Raft runs with a diskless control plane (in-memory stores): cluster state
10-
// is ephemeral, restarts rejoin from peer state, persistence-vs-quorum
11-
// failure modes are removed by construction. There is no data_dir.
9+
// Raft defaults to a diskless control plane (in-memory stores): cluster state
10+
// is ephemeral, restarts rejoin from peer state, persistence-vs-quorum failure
11+
// modes are removed by construction. Setting DataDir opts into fs-durable
12+
// storage, which a store.kv.raft entry requires.
1213
//
1314
// The transport is the wippy internode mesh exclusively: peers are addressed
1415
// by NodeID over the existing internode connection, so there is no bind
@@ -25,38 +26,17 @@ import "time"
2526
// see existing peers as raft_status="in" and skip bootstrap; the leader's
2627
// reconciler adds them via AddVoter.
2728
type Config struct {
29+
DataDir string `json:"data_dir,omitempty"`
2830
CommitTimeout time.Duration `json:"commit_timeout"`
2931
SnapshotInterval time.Duration `json:"snapshot_interval"`
3032
ElectionTimeout time.Duration `json:"election_timeout"`
3133
HeartbeatTimeout time.Duration `json:"heartbeat_timeout"`
3234
SnapshotThreshold uint64 `json:"snapshot_threshold"`
33-
// TrailingLogs caps how many log entries are retained after a snapshot.
34-
// hashicorp/raft default is 10240 which keeps a lot of memory under
35-
// partition; lower this for memory-constrained nodes. Zero means use
36-
// the hashicorp/raft library default.
37-
TrailingLogs uint64 `json:"trailing_logs"`
38-
// BootstrapExpect is the expected size of the initial quorum. The
39-
// gossip-driven bootstrap watcher uses this to decide when to form
40-
// the cluster:
41-
// 0 -> never self-bootstrap; this node joins an existing cluster
42-
// (waits for the leader's reconciler to AddVoter it).
43-
// 1 -> single-node mode; bootstrap immediately with [self].
44-
// N -> wait for exactly N raft-eligible alive members advertising
45-
// the same BootstrapExpect and raft_status="pre", then all
46-
// N call BootstrapCluster with the deterministically-sorted
47-
// server list. After a full-cluster cold-start they all
48-
// converge on the same configuration at log index 1.
49-
BootstrapExpect int `json:"bootstrap_expect,omitempty"`
50-
SnapshotRetain int `json:"snapshot_retain"`
51-
MaxPool int `json:"max_pool"`
52-
// MaxAppendEntries caps how many log entries the leader packs into a
53-
// single AppendEntries RPC. The hashicorp/raft default is 64 which,
54-
// when a follower restarts with an empty log and needs to catch up
55-
// hundreds of entries, lets the leader queue large batches in
56-
// memory simultaneously. Setting this to 16 throttles the catch-up
57-
// throughput so under chaos a returning pod doesn't OOM the leader
58-
// while it ships history. Zero means use the library default.
59-
MaxAppendEntries int `json:"max_append_entries"`
35+
TrailingLogs uint64 `json:"trailing_logs"`
36+
BootstrapExpect int `json:"bootstrap_expect,omitempty"`
37+
SnapshotRetain int `json:"snapshot_retain"`
38+
MaxPool int `json:"max_pool"`
39+
MaxAppendEntries int `json:"max_append_entries"`
6040
}
6141

6242
// InitDefaults fills zero-valued fields with sensible defaults.

api/service/store/kv/config.go

Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,69 @@
1+
// SPDX-License-Identifier: MPL-2.0
2+
3+
// Package kv provides configuration for the store.kv.* store kinds.
4+
package kv
5+
6+
import (
7+
"regexp"
8+
9+
apierror "github.com/wippyai/runtime/api/error"
10+
"github.com/wippyai/runtime/api/registry"
11+
"github.com/wippyai/runtime/api/supervisor"
12+
)
13+
14+
// Store kind constants.
15+
const (
16+
// KindRaft is a raft-replicated, fs-durable kv store.
17+
KindRaft registry.Kind = "store.kv.raft"
18+
// KindCRDT is a gossip-CRDT kv store with optional fs durability.
19+
KindCRDT registry.Kind = "store.kv.crdt"
20+
)
21+
22+
// namespacePattern constrains an app namespace so it can never collide with the
23+
// reserved _sys system namespace (a leading underscore is excluded by the
24+
// required leading [a-z]). System coordination lives under _sys:* keys.
25+
var namespacePattern = regexp.MustCompile(`^[a-z][a-z0-9._-]*$`)
26+
27+
// ErrInvalidNamespace is returned for a missing or malformed namespace.
28+
var ErrInvalidNamespace = apierror.New(apierror.Invalid,
29+
"store.kv namespace must match ^[a-z][a-z0-9._-]*$").WithRetryable(apierror.False)
30+
31+
// RaftConfig configures a store.kv.raft entry. Namespace isolates this store's
32+
// keys within the shared node-wide kv state.
33+
type RaftConfig struct {
34+
Namespace string `json:"namespace"`
35+
Lifecycle supervisor.LifecycleConfig `json:"lifecycle"`
36+
}
37+
38+
// Validate checks the configuration.
39+
func (c *RaftConfig) Validate() error {
40+
if !namespacePattern.MatchString(c.Namespace) {
41+
return ErrInvalidNamespace
42+
}
43+
return nil
44+
}
45+
46+
// InitDefaults fills lifecycle defaults.
47+
func (c *RaftConfig) InitDefaults() {
48+
c.Lifecycle.InitDefaults()
49+
}
50+
51+
// CRDTConfig configures a store.kv.crdt entry.
52+
type CRDTConfig struct {
53+
Namespace string `json:"namespace"`
54+
Lifecycle supervisor.LifecycleConfig `json:"lifecycle"`
55+
Durable bool `json:"durable"`
56+
}
57+
58+
// Validate checks the configuration.
59+
func (c *CRDTConfig) Validate() error {
60+
if !namespacePattern.MatchString(c.Namespace) {
61+
return ErrInvalidNamespace
62+
}
63+
return nil
64+
}
65+
66+
// InitDefaults fills lifecycle defaults.
67+
func (c *CRDTConfig) InitDefaults() {
68+
c.Lifecycle.InitDefaults()
69+
}
Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
// SPDX-License-Identifier: MPL-2.0
2+
3+
package kv
4+
5+
import (
6+
"errors"
7+
"testing"
8+
)
9+
10+
// TestConfigValidate_NamespaceReservation pins the prefix-separation guarantee:
11+
// app namespaces match ^[a-z][a-z0-9._-]*$ so they can never collide with the
12+
// reserved _sys:* system namespace, and the key separator ':' is excluded so a
13+
// namespace can never span into another store's keyspace. Both store.kv kinds
14+
// share the rule; durability is orthogonal to it.
15+
func TestConfigValidate_NamespaceReservation(t *testing.T) {
16+
reserved := []string{"_sys", "_sys:raft", "_sys:registry", "_lock", "_"}
17+
malformed := []string{"", "1bad", "BadCase", "a:b", "ns/with/slash", "has space", "UPPER"}
18+
valid := []string{"deploy", "sess", "kb", "kb_index", "good.ns", "ok-ns", "a"}
19+
20+
check := func(t *testing.T, name string, validate func(ns string) error) {
21+
t.Helper()
22+
for _, ns := range append(append([]string{}, reserved...), malformed...) {
23+
if err := validate(ns); !errors.Is(err, ErrInvalidNamespace) {
24+
t.Errorf("%s namespace %q: got err=%v, want ErrInvalidNamespace", name, ns, err)
25+
}
26+
}
27+
for _, ns := range valid {
28+
if err := validate(ns); err != nil {
29+
t.Errorf("%s namespace %q: got err=%v, want nil", name, ns, err)
30+
}
31+
}
32+
}
33+
34+
t.Run("raft", func(t *testing.T) {
35+
check(t, "raft", func(ns string) error {
36+
c := RaftConfig{Namespace: ns}
37+
return c.Validate()
38+
})
39+
})
40+
41+
t.Run("crdt is identical and orthogonal to durability", func(t *testing.T) {
42+
for _, durable := range []bool{false, true} {
43+
check(t, "crdt", func(ns string) error {
44+
c := CRDTConfig{Namespace: ns, Durable: durable}
45+
return c.Validate()
46+
})
47+
}
48+
})
49+
}

api/store/command.go

Lines changed: 81 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -12,17 +12,20 @@ import (
1212

1313
func init() {
1414
dispatcher.MustRegisterCommands("store",
15-
Get, Set, Delete, Has,
15+
Get, Set, Delete, Has, EntryCommand, ListCommand, PutCommand,
1616
)
1717
}
1818

1919
// Command IDs for store operations.
2020
// Range 120-129 is reserved for key-value store commands.
2121
const (
22-
Get dispatcher.CommandID = 120 // Get value by key
23-
Set dispatcher.CommandID = 121 // Set value with key
24-
Delete dispatcher.CommandID = 122 // Delete key
25-
Has dispatcher.CommandID = 123 // Check if key exists
22+
Get dispatcher.CommandID = 120 // Get value by key
23+
Set dispatcher.CommandID = 121 // Set value with key
24+
Delete dispatcher.CommandID = 122 // Delete key
25+
Has dispatcher.CommandID = 123 // Check if key exists
26+
EntryCommand dispatcher.CommandID = 124 // Get value with metadata
27+
ListCommand dispatcher.CommandID = 125 // List keys deterministically
28+
PutCommand dispatcher.CommandID = 126 // Rich put with conditions
2629
)
2730

2831
// GetCmd retrieves a value from the store.
@@ -93,6 +96,61 @@ func (c *HasCmd) Release() {
9396
hasCmdPool.Put(c)
9497
}
9598

99+
// EntryCmd retrieves a value with store metadata.
100+
type EntryCmd struct {
101+
Store Store
102+
Key registry.ID
103+
}
104+
105+
var entryCmdPool = sync.Pool{New: func() any { return &EntryCmd{} }}
106+
107+
// AcquireEntryCmd returns a pooled EntryCmd.
108+
func AcquireEntryCmd() *EntryCmd { return entryCmdPool.Get().(*EntryCmd) }
109+
func (c *EntryCmd) CmdID() dispatcher.CommandID { return EntryCommand }
110+
func (c *EntryCmd) Release() {
111+
c.Store = nil
112+
c.Key = registry.ID{}
113+
entryCmdPool.Put(c)
114+
}
115+
116+
// ListCmd lists store entries.
117+
type ListCmd struct {
118+
Store Store
119+
Opts ListOptions
120+
}
121+
122+
var listCmdPool = sync.Pool{New: func() any { return &ListCmd{} }}
123+
124+
// AcquireListCmd returns a pooled ListCmd.
125+
func AcquireListCmd() *ListCmd { return listCmdPool.Get().(*ListCmd) }
126+
func (c *ListCmd) CmdID() dispatcher.CommandID { return ListCommand }
127+
func (c *ListCmd) Release() {
128+
c.Store = nil
129+
c.Opts = ListOptions{}
130+
listCmdPool.Put(c)
131+
}
132+
133+
// PutCmd writes a value with optional preconditions.
134+
type PutCmd struct {
135+
Store Store
136+
Value payload.Payload
137+
Key registry.ID
138+
Opts PutOptions
139+
}
140+
141+
var putCmdPool = sync.Pool{New: func() any { return &PutCmd{} }}
142+
143+
// AcquirePutCmd returns a pooled PutCmd.
144+
func AcquirePutCmd() *PutCmd { return putCmdPool.Get().(*PutCmd) }
145+
func (c *PutCmd) CmdID() dispatcher.CommandID { return PutCommand }
146+
func (c *PutCmd) Release() {
147+
c.Store = nil
148+
c.Key = registry.ID{}
149+
c.Value = nil
150+
c.Opts = PutOptions{}
151+
putCmdPool.Put(c)
152+
}
153+
96154
// GetResponse contains the result of a get operation.
97155
type GetResponse struct {
98156
Value payload.Payload
@@ -115,3 +173,21 @@ type HasResponse struct {
115173
Error error
116174
Exists bool
117175
}
176+
177+
// EntryResponse contains an entry plus metadata.
178+
type EntryResponse struct {
179+
Error error
180+
Entry VersionedEntry
181+
}
182+
183+
// ListResponse contains a deterministic list page.
184+
type ListResponse struct {
185+
Error error
186+
Page Page
187+
}
188+
189+
// PutResponse contains the stored entry.
190+
type PutResponse struct {
191+
Error error
192+
Entry VersionedEntry
193+
}

api/store/errors.go

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,10 @@ import (
88

99
// Sentinel errors.
1010
var (
11-
ErrKeyNotFound = apierror.New(apierror.NotFound, "key not found").WithRetryable(apierror.False)
12-
ErrKeyExists = apierror.New(apierror.AlreadyExists, "key already exists").WithRetryable(apierror.False)
13-
ErrInvalidKey = apierror.New(apierror.Invalid, "invalid key format").WithRetryable(apierror.False)
11+
ErrKeyNotFound = apierror.New(apierror.NotFound, "key not found").WithRetryable(apierror.False)
12+
ErrKeyExists = apierror.New(apierror.AlreadyExists, "key already exists").WithRetryable(apierror.False)
13+
ErrInvalidKey = apierror.New(apierror.Invalid, "invalid key format").WithRetryable(apierror.False)
14+
ErrInvalidOptions = apierror.New(apierror.Invalid, "invalid store options").WithRetryable(apierror.False)
15+
ErrUnsupported = apierror.New(apierror.Invalid, "operation not supported by this store").WithRetryable(apierror.False)
16+
ErrVersionMismatch = apierror.New(apierror.Conflict, "version mismatch").WithRetryable(apierror.True)
1417
)

api/store/kv/errors.go

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
// SPDX-License-Identifier: MPL-2.0
2+
3+
package kv
4+
5+
import apierror "github.com/wippyai/runtime/api/error"
6+
7+
// Engine sentinel errors.
8+
var (
9+
ErrKeyNotFound = apierror.New(apierror.NotFound, "key not found").WithRetryable(apierror.False)
10+
ErrLeaseNotFound = apierror.New(apierror.NotFound, "lease not found").WithRetryable(apierror.False)
11+
ErrLeaseExpired = apierror.New(apierror.Invalid, "lease has expired").WithRetryable(apierror.False)
12+
ErrVersionMismatch = apierror.New(apierror.Invalid, "version mismatch").WithRetryable(apierror.True)
13+
ErrKVClosed = apierror.New(apierror.Unavailable, "kv is closed").WithRetryable(apierror.False)
14+
ErrUnsupported = apierror.New(apierror.Invalid, "operation not supported by this backend").WithRetryable(apierror.False)
15+
)

0 commit comments

Comments
 (0)