Commit 700b59a
feat(store): KV stores on the shared raft (store.kv.raft + store.kv.crdt) (#307)
* feat(raft): multiplex FSM router + optional fs-durable storage
Foundation for store.kv.raft riding the single cluster raft.
- cluster/raft/multiplex: domain-tagged FSM router (untagged -> global
registry, 0xB2 -> kv). Combined snapshot framing with legacy-bare
fallback for in-place upgrade; downgrade is one-way.
- cluster/raft/storage: openStores selects diskless in-mem (default,
unchanged) or fs-durable raft-wal log + bbolt stable store + file
snapshots under Config.DataDir.
- raft.Node Start/Stop use the strategy and close durable handles.
- boot wires the router transparently (nil kv FSM) so behavior is
unchanged until the kv FSM is plugged in.
Deps: hashicorp/raft-wal, hashicorp/raft-boltdb/v2.
* feat(kv): re-port coordination KV engine
Port the wip in-memory KV engine onto its own api surface:
- api/store/kv: Engine interface (string keys / byte values, versions,
leases, watch) + sentinel errors. The api/store wrappers will transcode
registry.ID / payload onto this.
- system/kv: in-memory Service implementing Engine (single-goroutine event
loop + atomic snapshot reads), plus the state/lease/watcher primitives the
raft and crdt backends build on.
Fixes latent lint from the wip branch (fieldalignment, unused) and ports the
engine test suite.
* feat(store): add store.kv.raft backend on the shared raft
store.kv.raft is a raft-replicated, namespace-isolated, fs-durable KV store
riding the single cluster raft via the multiplex router.
- system/kv: raft command codec, RaftFSM (deterministic apply + gob
snapshot/restore over the shared state), RaftEngine (local reads, leader
proposes; leader-side lease expiry sweeper). Follower writes return a
retryable ErrNotLeader (leader-forwarding is a planned follow-up).
- service/store/kv: api/store.Store/Scanner/Atomic wrapper with namespace
prefixing (apps see bare keys; @sys/* reserved) and payload<->JSON codec,
plus the registry.EntryListener manager.
- api/service/store/kv: store.kv.raft / store.kv.crdt kinds + config with
namespace validation.
- boot: raft component builds the kv FSM into the router and exposes the
shared engine; store component registers the store.kv.raft kind.
Tested: command round-trip, FSM apply/snapshot/restore, engine CRUD/CAS/
SetIfAbsent/scan/lease-auto-expiry, namespace isolation, and 3-node raft
replication through the router over real consensus.
* feat(store): add store.kv.crdt gossip backend
store.kv.crdt is a gossip-CRDT, namespace-isolated KV store with optional
fs durability, over the existing system/crdt substrate.
- system/kv: CRDTEngine implementing kvapi.Engine (epidemic delta broadcast +
full-state push/pull anti-entropy, local-reaper leases). A per-node wall
tiebreak gives crdt.State.Apply a total order so concurrent same-millisecond
cross-node writes converge instead of diverging. Per-namespace fs snapshots
when durable. Plus the membership user-delegate (kind 0xD2).
- service/store/kv: CRDTManager reusing the shared Store wrapper; durable
entries mark their namespace for persistence.
- boot: kv-crdt system component builds the engine + registers the gossip
delegate; store component registers the store.kv.crdt kind.
CAS/SetIfAbsent are best-effort under eventual consistency (use store.kv.raft
for linearizable conditional writes).
Tested: ops, two-replica convergence incl. concurrent conflicts, full-state
anti-entropy, per-namespace durable restart, and lease auto-expiry.
* chore(raft): field-align Config after DataDir addition
Reorder Config fields so govet fieldalignment is clean with the new DataDir
string field. No behavior change (JSON tags preserved).
* fix(store): bring store.kv.* to spec — durable-by-default raft, ":" keys, _sys
- Raft is now fs-durable by default: node data_dir resolves to ~/.wippy/store
(or cluster.raft.data_dir), shared raft + kv under <data_dir>/_sys/raft.
Satisfies "raft = durable, period" (the global registry, sharing this raft,
is now fs-persisted too — intended under the unified-KV direction).
- Physical key separator "/" -> ":" to match the spec: <namespace>:<id>.
- Reserve the _sys namespace for system coordination (app namespaces already
excluded by the leading-[a-z] rule); _sys/kvcrdt for durable crdt snapshots.
- Example store.kv.* config (docs).
fs.directory-entry resolution for data_dir is a follow-up; explicit path and
the ~/.wippy/store default are in.
* feat(store): kv leader-forwarding for store.kv.raft writes
Follower writes to store.kv.raft previously returned a retryable ErrNotLeader.
The engine now forwards a write to the raft leader over the relay (host
"storekv", req/resp topics), awaits the applied result, and re-resolves the
leader on a mid-flight election. Error sentinels (ErrKeyNotFound, etc.) are
preserved across the wire via kind codes so errors.Is still works.
This makes store.kv.raft writes work from any node and is the prerequisite for
migrating registry/locks/leases onto the shared kv (Part B).
Tested: follower->leader forward applies on the leader and returns the version;
forwarded delete-missing surfaces ErrKeyNotFound; follower with no router still
returns ErrNotLeader.
* test(store): real-stack multi-node cluster harness + E2E proofs
cluster/clustertest boots N real raft nodes over an in-process internode mesh
(async FIFO delivery + partition/kill/restart injection), each with the
multiplex(primary+kv) FSM, the kv RaftEngine registered as a relay host for
leader-forwarding, and a durable per-node DataDir. Faithfully models gossip
NodeLeft by tearing down raft mesh sessions on partition/heal/kill/restart so
they rebuild.
E2E proofs (real consensus + real relay forwarding, -race):
- follower write forwarded to leader, committed, replicated to all nodes
- leader hard-kill failover; writes survive and continue
- partition a follower, write on quorum, heal -> isolated node catches up
- single-node durable kill+restart recovers from raft-wal+bbolt on disk
- one shared raft multiplexes a non-kv command alongside kv commands
* fix(crdt): deterministic total-order merge tiebreak + property tests
Property testing found a real store.kv.crdt non-convergence bug: on an
equal-wall, cross-origin conflict, State.Apply kept whatever entry the replica
saw first (MergeNoop) — not a total order — so replicas could diverge on a key
under multi-writer churn.
Fix: when walls tie, resolve deterministically — delete beats live write
(symmetric), else by global origin string then counter (comparing the origin
STRING, not the replica-local interned id, so the order is identical on every
replica). Idempotent for identical dots. The registry shares crdt.State; its
eventual + global suites stay green.
Adds property/fuzz tests: command codec round-trip, decode-never-panics on
arbitrary bytes, namespace round-trip + cross-namespace isolation, and the
CRDT convergence property (random ops/order converge on all replicas).
* test(store): crdt real-gossip E2E + raft concurrency/linearizability stress
- crdt_gossip: 3-node real memberlist cluster with the crdt delegate; a write
converges to all nodes via delta + anti-entropy, including a node that joins
after the write (full-state backfill). Proves the gossip delegate wiring.
- stress (real raft, -race): concurrent CAS-increments of a shared key from
random nodes (forwarded to the leader) are exactly-once — final value equals
total increments, no lost updates (linearizability + forwarding under load);
and many concurrent distinct-key writes from all nodes replicate everywhere.
* feat(store): B1 — distributed locks on the shared kv (_sys:lock)
system.lock now lives in the one shared KV instead of a separate global.Strong
registration:
- system/kv LockService: Acquire = SetIfAbsent(_sys:lock:<name>, holderPID)
(linearizable via raft + leader-forwarding); Release verifies the holder;
Holder reads the owner. Auto-release on holder process exit (topology Monitor
-> the "syslock" relay host -> ReapPID) and on node leave (cluster.NodeLeft ->
ReapNode); reaps are idempotent across nodes. PIDs compared by String() (the
struct carries an unexported cache).
- Lua system.lock module retargeted to the LockService (same acquire/release
contract); global.Strong stays available until the B5 cutover.
- boot registers the lock relay host, reaps on NodeLeft, and exposes the service
in context.
Proof: unit (acquire/release/holder/ReapPID/ReapNode), retargeted Lua module
tests, and a 3-node E2E (mutual exclusion + replication + auto-release after the
holder's node is killed — no deadlock).
* fix(store): B2 — leases auto-expire cluster-wide regardless of grant node
The lease sweeper only re-armed deadlines on a leadership change, so a lease
granted on a follower (e.g. a TTL'd store.Set forwarded to the leader) was never
scheduled by the current leader and would not expire. The leader now re-arms
from replicated state every sweep tick (idempotent), so any lease — granted on
any node — is swept and its revoke replicated.
Leases are the substrate for _sys:lease (reserved system namespace) and TTL'd
store.Set; there are no separate external lease consumers to migrate.
Proof: 3-node E2E grants a lease + set-with-lease on a FOLLOWER and asserts the
key auto-expires on every node.
* feat(kv): transaction primitives for the registry on kv
Add compare-and-delete, per-key epoch (= raft log index), atomic
multi-key txn (two-pass all-or-nothing), and barriered GetLinearizable
/ ScanAtIndex on a LinearizableEngine. CRDT backend rejects Txn.
Snapshot gains an additive gob Epoch field (no snapshotVersion bump;
legacy snapshots still restore). LockService.Release now uses
CompareAndDelete, closing a read-then-delete race.
Tests: txn codec, CAD, all-or-nothing + atomic-promote, epoch stamping,
scan-at-index, snapshot+legacy-restore; cluster E2E for txn/CAD
replication from a follower and cross-node epoch agreement.
* feat(registry): kv-backed CONSISTENT name registry
New system/topology/namereg/kvbacked package implementing the
globalapi.Registry facade on the shared kv under _sys:registry:* —
register/lookup/unregister/remove/removenode via atomic kv txns with
reverse PID/node indexes, first-write-wins conflict resolution with an
override-resolve CAS swap, and ByPID lookup. Adds a forwarded
leader-read (GetViaLeader) to the kv engine for read-your-writes on a
follower after a forwarded write; in-mem Service now satisfies
LinearizableEngine.
Tests: consistent register/lookup/dedupe/conflict/override/ByPID/
unregister/remove/removenode incl. colon-containing names; cluster E2E
proves follower-register replicates with a real raft-index epoch,
conflict is rejected cluster-wide, and remove clears every node.
* feat(registry): kv-backed STRONG name registry
Strong-scope reservations on the shared kv: a pending header key, one
ack key per required node (each ack is a raft-replicated kv write — no
ack relay), atomic leader promotion via a conditional multi-key txn,
deadline expiry, and cross-scope reject. Per-node exclusions block
LOCAL/EVENTUAL shadowing during the promotion window; reconcile(name)
is the single idempotent state-machine step driven by a kv-Watch
reconciler. Conflict surfaces StrongConflictError, timeout
StrongRegistrationTimeoutError.
Tests: promote, ack-missing timeout, reject-on-conflict, reservation
window + release, unregister; 3-node E2E proves a follower-opened
reservation collects every node ack and promotes atomically,
resolvable cluster-wide.
* feat(registry): non-member dissem read-through + boot backend flag
Phase 4: kv-backed registry resolves names on non-member nodes (no raft
FSM) by reusing the 0xC1 dissem plane. A kv-Watch reconciler translates
active-binding changes into BindingEvents (dot = raft index via the new
WatchEvent.Index), Lookup falls through to the dissem cache on a local
miss, and tombstones gossip on delete.
Phase 5 (wiring): cluster.raft.registry_backend flag (fsm default | kv).
When kv, boot builds the kv-backed Service with Strong (membership +
leader + cross-scope local-conflict), dissem, the process-exit/node-
leave reaper, and the watch reconciler, registered on the same facades;
default fsm leaves the existing path unchanged. Adds a process-exit
relay host + node-leave DropNode (drop-required so promotion completes
on survivors).
Tests: non-member resolves via gossip + tombstone; reconciler
broadcasts on register. Full -race suite green (default path unchanged).
docs/store-kv-registry-traceability.md maps requirements to proofs.
* fix(registry): adversarial-review findings + watch publish-ordering
Multi-agent review (6 confirmed bugs, all fixed):
- dissem RunGC never started on the kv path -> start + ctx-stop in the
reconciler (tombstones now swept).
- forwarded read/write select race could drop a response that arrived
with the timeout -> final non-blocking drain.
- resolveConflict retry exhaustion -> more retries + retryable error.
- entryToAPI/entryToCluster dropped Epoch -> watch put-dot was 0; copy
Epoch and use the op raft index as the dissem dot.
- reconcile blocked the watch goroutine on forwarded reads -> local
reads on the watch path; register drives the just-opened pending
directly (a local read cannot see the freshly-forwarded write).
- strong deadline expiry used non-linearized reads -> barrier only on
the deadline path before giving up.
Root-cause fix exposed by the above: the kv FSM emitted watch events
before publishing the new snapshot, so a watch-driven local read missed
its own change. Buffer events and flush after snap.Store.
Full -race suite green; lint clean.
* test(registry): capstone — kv registry survives leader kill
Real-raft 3-node E2E: a CONSISTENT name registered on the leader stays
resolvable cluster-wide after the leader is killed, and registration
continues on a survivor after failover.
* test(registry): real-binary kv boot smoke + traceability
tests/capstone/kv-boot-smoke.sh boots the wippy binary with
registry_backend=kv and runs the app suite: process registry suite 46/46
and security:registry pass, no kv-path panic (7 env-only failures:
docker/hub). Documents the deferred default-flip + FSM decommission as a
deliberate one-way operational cutover.
* fix(registry): round-2 review — register fail-fast + reap owner-guard
- register(): decode the pending before installing the waiter; on an
(impossible) decode failure, fail fast + remove the bad entry instead
of blocking the waiter forever.
- deleteBinding(): always clear the dead PID reverse-index keys, but
delete the active binding only if it still belongs to that PID
(owner + version guarded) — reaping a dead PID can no longer clobber a
name already reassigned to a live PID. Regression test added.
Full -race suite green; lint clean.
* fix(registry): round-3 review — bounded wait, join barrier, leader sweep
- register(): bound the wait with a backstop deadline (grace after the
Strong deadline) so a skipped-delivery reconcile path cannot hang.
- NameReady(): gate on the reconciler having seeded (learned+latched the
cluster Strong reservations) instead of always true — join-barrier
semantics; a node cannot shadow a Strong name before learning it.
- leader sweep: periodically re-drive in-flight pendings while leader, so
a follower promoted to leader re-arms deadline timers and resumes
promotion/expiry; backstops missed watch events.
- deleteBinding(): local read (not leader-forwarded) so the reaper never
blocks the relay-receiver goroutine; CondVersion still guards.
- StrongRegistrationTimeoutError now reports MissingAcks.
Full -race suite green; lint clean.
* fix(registry): round-4 review — deterministic terminal + non-member forward
- onTerminal delivers an epoch-less Expired signal; finalize reads the
authoritative epoch/reason/missing-acks from terminal tracking, so
concurrent onTerminal calls (watch + recursive reconcile + timer) yield
a deterministic outcome. (Rejected the suggested had-guard, which would
have broken the fast reject path.)
- Lookup gains a non-member cold-miss forward-resolve via GetViaLeader
(gated by SetNonMember, default off) so a client that joined before
gossip converged still resolves an active name; members never forward.
Regression tests for both. Full -race suite green; lint clean.
* fix(registry): round-6 review — recover active Strong exclusions on seed
- seed() now also scans active Strong names and reconciles them, so a
node restart / snapshot-restore re-latches their in-memory exclusions
before NameReady flips. Without this, recovered active Strong names
reported IsStrongReserved=false, letting a LOCAL/EVENTUAL register
shadow a live Strong name. Recovery regression test added.
- handleWatchEvent surfaces a (design-violating, never-expected)
WatchExpired on a registry key as a debug log; still handled safely as
a delete.
Full -race suite green; lint clean.
* fix(registry): round-7 review — invalid-PID guard + drop-node key cleanup
- reconcile() skips an active Strong entry with an unparseable PID (debug
log) instead of latching a zero-PID exclusion.
- dropNodeFromPending() atomically deletes the departed node’s ack/reject
keys along with removing it from RequiredNodes, so they are not orphaned
(promote/expire only sweep keys for remaining required nodes).
Full -race suite green; lint clean.
* fix(registry): round-8 review — cross-scope CONSISTENT/STRONG + read bound
- registerConsistent refuses (ErrPendingConflict) when a STRONG
reservation is pending for the name; resolveConflict reports it instead
of retrying forever (cross-scope invariant: a pending owns the name).
- leaderPromote resolves a genuine conflict (active held by another
binding) to a terminal expiry instead of retrying until the deadline,
so the waiter never hangs.
- handleReadResp surfaces a truncated forwarded-read value as an error
rather than a silent empty value.
Cross-scope regression test added. Full -race suite green; lint clean.
* fix(registry,kv): round-9 review — STRONG precedence + CAS watch Previous
- resolveConflict refuses to displace a STRONG-scope active owner with a
CONSISTENT register even under a custom ResolveFunc (precedence
Strong>Consistent enforced).
- CompareAndSwap watch events now carry the correct Previous entry (was
nil in the raft FSM, post-mutation value in the in-mem service) — a
general kv watch-correctness fix.
Regression tests: cross-scope no-displace; CAS watch Previous. Full
-race suite green; lint clean.
* fix(dissem): round-10 review — deterministic same-index LWW tie-break
Dissem merge: at an equal RaftIndex (a degenerate case — distinct ops
carry distinct raft indices), a live binding deterministically wins over
a tombstone regardless of gossip arrival order. Defensive hardening of
the shared dissem plane; global + kvbacked dissem suites stay green.
Also: traceability doc updated with the 10-round review convergence
(26 bugs fixed) and the deferred operational/cross-cutting items.
* feat(registry): default to the kv-backed name registry
Flip cluster.raft.registry_backend default fsm->kv: the kv-backed
registry on the shared raft (_sys:registry) is now the live name
registry; set registry_backend=fsm to fall back to the dedicated FSM.
Verified: full go test ./... -race green (291 pkgs, 0 fail) with kv as
default; real binary boots with the kv default (capstone, process
registry suite 46/46, no kv-path panic); lint 0 issues.
* test(registry): non-member (client) resolution E2E over the real relay
Proves a registry non-member (no raft FSM, all ops forwarded to the
leader via the relay) resolves a member-registered name through the
cold-miss forward-resolve, and registers its own name (forwarded) that
members then resolve. Verifies the non-member resolution mechanism
end-to-end in-process.
* feat(kv): server-side re-forwarding so a non-member reaches the leader via any member
A registry non-member (role=client) has no raft state and cannot call
raft.Leader(), so it forwards ops to an arbitrary member. A non-leader that
receives a forwarded read/write now re-forwards it to the leader it can resolve,
hop-bounded (maxForwardHops) and off the relay Send goroutine so the re-forward's
own reply (delivered via the same Send) cannot self-deadlock.
Proven E2E under -race: a client targeting a follower has both cold-miss
forward-resolve and register-via-forward relayed to the leader, and members then
resolve the client-registered name.
* feat(boot): wire the kv name registry on role=client nodes
A node that runs no raft Node (cluster.raft.role=client / raft.enabled=false)
previously wired no global name registry at all. loadClientRegistry now builds
the forward-only kv engine (systemkv.ClientSubmitter), a non-member
kvbacked.Service, the dissem read-through delegate, and the two registry facades
-- mirroring the server path so consumers stay backend-agnostic.
The forward target is sysraft.PickForwardTarget(membership.Nodes(), self): a
pure, deterministic, eligible-only, self-excluding choice. Membership reports
only live peers, so a departed target drops out and the next call re-picks with
no failover bookkeeping; server-side re-forwarding makes any member a valid entry
point. Guards no-op (preserving prior client behavior) when the kv backend is
not selected or relay/membership/topology are absent.
Verified: PickForwardTarget, ClientSubmitter, and the no-op guards by unit tests;
the production ClientSubmitter end-to-end (resolve+register via leader and via a
re-forwarding follower) in cluster/clustertest. Full build + go test ./... -race
green; golangci-lint 0 issues. Not verified here: a real client process joining
a real cluster (the in-process harness has no gossip membership layer).
* test(capstone): real-binary boot smoke for the role=client kv registry
Boots the actual wippy binary with cluster.raft.role=client + registry_backend=kv
and asserts loadClientRegistry runs in a real process (logs the wiring line) with
no panic. Verified: the binary emits 'raft(client): kv name registry wired
(non-member, forward-resolve)' at boot. shellcheck clean.
* test(store/kv): prove reserved _sys namespace rejection
Why: the <namespace>:<key> isolation rule and the reserved _sys coordination
namespace are enforced by namespacePattern but were never exercised by a test.
What: table tests on RaftConfig/CRDTConfig.Validate asserting _sys*, colon-bearing
and malformed namespaces are rejected while deploy/sess/kb/kb_index pass;
durability is orthogonal to the name rule.
* fix(registry): complete strong reservation on survivors across a leader change
Why: leaderDrive checks completion against the stored RequiredNodes; a required
node that departed coincident with a leadership change was only pruned via the
old leader's NodeLeft, so a new leader kept the stale set and the reservation
expired at its deadline instead of promoting on the surviving acks.
What: pruneDepartedRequired drops from a pending reservation's RequiredNodes any
node absent from the live membership (reusing dropNodeFromPending), called on the
leader in reconcile before leaderDrive, so it runs on seed takeover and every
leaderSweep tick. A node still present but merely un-acked is never pruned, so
the all-live-must-ack guarantee is preserved.
Proof: unit TestStrong_PromotesOnSurvivorsWhenRequiredNodeDeparts (fails pre-fix,
passes post-fix); cluster TestE2E_KVRegistry_StrongPromotesOnSurvivorsAfterLeaderKill
(real leader kill, 3x clean) and ConsistentReapedOnNodeDrop; restart-durability
refutations TestE2E_DurableRestartPreservesEpoch (epoch never resets) and
TestCRDTEngine_DurableTombstoneSurvivesRestartNoResurrection.
* fix(raft): use a bbolt log store on Windows where raft-wal can't fsync its dir
Why: raft-wal fsyncs the WAL directory during init, which returns "Access is
denied" on Windows (directory fsync is not permitted there), so store.kv.raft
durability failed to initialize and CI Test (windows-latest) failed in
TestOpenStores_DurableRoundTrip.
What: openStores now dispatches the LogStore by OS — raft-wal on Linux/macOS (the
production platforms, higher write throughput), a bbolt LogStore (the same
cross-platform engine already used for the stable store) on Windows. Non-Windows
behavior is unchanged. openBoltLogStore is exercised by a round-trip test on
every OS so the Windows path is proven without a Windows runner; cluster/raft
cross-compiles for windows/amd64.
* test(raft): sound anti-inflation bound for the concurrent dead-streak test
Why: TestPeerStateTracker_ConcurrentTripBoundedByStreak asserted dead-streak <=
ceil(G/failureLimit)=2, but the dead window is published just after the trip is
claimed via CAS, so a scheduling-dependent number of failures already in flight
past isDead can claim a few more trips before gating engages — CI legitimately
observed 3, failing the too-tight bound.
What: assert the deterministic invariant the CAS guarantees — G goroutines'
failures collapse into strictly fewer than G trips (anti-inflation) — instead of
a fixed numeric count. Stress-verified 200x under -race.
* chore(build): module-first go.mod + gremlins mutation-testing target
Why: mutation testing was requested. gremlins reads the module name from the
FIRST line of go.mod, but the SPDX comment preceded the module directive, so it
mis-parsed the name and matched zero coverage to mutants (0% on every run).
What: move the `module` directive to go.mod line 1 (the Go convention; the SPDX
identifier is retained just below it) so gremlins — and any tool that naively
reads go.mod line 1 — resolves the module name correctly; add a `make mutation`
target (gremlins via `go run`, coverage scoped per MUTATE_DIR, workers=1 + a
generous timeout coefficient for deterministic, contention-free scores).
Verified: service/store/kv scores 18 killed / 7 lived / 72% efficacy, repeatable.
* fix(eventual): gossip-replicate registry deltas to all members
EVENTUAL deltas were emitted one-shot to GossipNodes peers and never
re-broadcast by receivers, so nodes outside that set (incl. raft-less
client nodes) converged only via slow anti-entropy (a 7-node post-join
name took ~38s to reach the tail).
- applyIncoming re-broadcasts entries that change local state (epidemic,
loop-free via CRDT idempotency).
- LocalState carries the authoritative sender id; MergeRemoteState targets
shard-pulls at it instead of a max-counter heuristic, and skips self.
- PushPullInterval 15s -> 5s backstops the rare epidemic tail-miss.
Convergence drops to sub-second (common) / a few seconds (worst). Deltas
become process.send(name,...)-resolvable on client nodes.
* test(eventual): client-publish + bidirectional reg across non-raft nodes
Why: the gossip-replicate fix proved a raft-less client can RESOLVE a
server-published eventual name, but not that it can PUBLISH a name servers
resolve. The publish direction is the other half of "eventual name reg works
on non-raft nodes" and was untested.
What: add TestEventual_ClientPublishResolvedByServers (client registers, both
servers resolve via epidemic forward + anti-entropy) and
TestEventual_ClientServerBidirectional (client and server each register, every
node resolves both), reusing the membership eventual harness. Green with -race.
* fix(store): version-guard the distributed-lock reaper (B1)
Why: the lock auto-release reaper ran on every node off a possibly-stale local
snapshot and deleted matched locks unconditionally. A reap that raced a
release+reacquire — or two nodes reaping the same departed holder — could delete
a lock a live holder had just acquired, breaking mutual exclusion (two holders).
What: reap now captures the version observed during the scan and deletes via
CompareAndDelete, so a stale victim is a no-op (OK=false) instead of a clobber —
the same guard Release already uses. Adds TestLock_ReapDoesNotClobberReacquired
Lock, which interleaves a reacquire between the reaper's scan and delete and
fails on the old unconditional path.
* fix(store): GC store.kv.crdt tombstones to bound memory (B2)
Why: every Delete (and lease expiry) on the crdt backend left a tombstone that
was never garbage-collected — State.ReapTombstones existed but had no caller for
this engine — so put/delete churn grew local state, every gossip frame, and
every durable snapshot without bound.
What: add a periodic wall-floor tombstone reaper to the crdt engine. A tombstone
is dropped only once it is older than crdtTombstoneFloor (15m, mirroring the
eventual registry's DefaultWallFloor), well past the anti-entropy window, so a
lagging peer cannot resurrect it. CV-gated (safe-counter) reaping is not used:
the store.kv.crdt delegate exchanges no per-origin digests, so safeByNode is nil
and the wall floor is the sole bound. Adds TestCRDTEngine_TombstoneGCBoundsMemory
asserting recent tombstones are retained and aged ones reaped.
* fix(store): make forwarded writes safe to retry (B3)
Why: a follower forwarding a write to the leader treated a reply TIMEOUT as a
not-leader rejection and retried it. But a timeout is ambiguous — the leader may
have committed the op — so the retry re-applied a non-idempotent write (Set/CAS/
Delete double-bumped the version or returned a spurious ErrKeyNotFound).
What: split the two outcomes. A not-leader reply means the op was provably not
applied (a leader that loses leadership mid-Apply does not commit), so it stays
retryable; a genuine timeout now returns errForwardTimeout, which the caller does
NOT retry and surfaces as an honest write error. The follower's wait is also
sized to raftApplyTimeout + grace so a slow-but-committed reply arrives instead
of spuriously timing out. Adds TestRaftEngine_ForwardTimeoutDoesNotDoubleApply
(leader commits, reply dropped → exactly one apply).
* fix(store): persist absolute lease deadline across leader change (B4)
Why: lease expiry used a leader-local now+TTL deadline with nothing replicated.
On every leadership change the new leader re-armed from TTL, discarding elapsed
time, so a lease could outlive its TTL by up to one failover — and ~N×TTL under
N failovers (a flapping leader during a partition), contradicting "leases
auto-expire cluster-wide regardless of grant node".
What: the grant/renew command now carries an absolute deadline (grantor wall +
TTL); the FSM stores it in lease state and snapshots it; leaseSnapshot returns it
and a re-arming leader honors that deadline instead of resetting to now+TTL. Adds
TestRaftEngine_LeaseDeadlineHonoredAcrossReArm (a past-deadline lease expires on
re-arm) and TestRaftFSM_LeaseDeadlineSurvivesSnapshot; command round-trip test
covers the new field.
* fix(raft): reset an absent child domain on multiplex restore (H1)
Why: raft calls FSM.Restore to REPLACE all state, but the multiplex router only
restored the child domains physically present in the snapshot. A framed snapshot
written by a kv-disabled node (or a legacy bare-primary snapshot) carries no kv
section, so restoring it over a node that already had kv state left that state
intact — the kv FSM then diverged from the cluster after an InstallSnapshot.
What: the router now resets the kv child when no kv section was seen (framed and
legacy paths). The kv FSM treats an empty Restore stream as reset-to-empty rather
than erroring. The primary section is always written, so only the kv child can be
absent. Adds router tests that restore over an existing kv FSM and a kv-FSM
empty-restore test; the previous restore tests only used fresh child FSMs.
* test(raft): kill surviving multiplex mutants (100% efficacy)
Why: gremlins left 4 live mutants in the router — the empty-entry routing guard,
the short-stream magic-read guard, and the kv-section persist error check were
unverified.
What: add tests for empty-Data Apply routing to primary, short/empty Restore
streams treated as legacy, and a Persist that must not cancel the sink on a
successful kv write. Multiplex mutation efficacy 87% -> 100%.
* test(store): cover forwarding, durable crdt + tolerant decode (mutation)
Why: gremlins left the leader-side forward/read handlers, hop re-forwarding,
durable-namespace reload, and the tolerant lease-deadline decode unexercised.
What: add forwarded-read, hop re-forward (client->member->leader), forwarded
conditional-write (incl. the version-mismatch wire path), durable-namespace
reload, OnFrame apply+emit, and a decode-without-trailing-ExpiresAtMs test.
* test(store): cover lock relay receiver + context helpers (mutation)
Why: gremlins left the LockService.Send exit-event reaping, its topic/payload
filters, and the WithLockService/GetLockService context helpers unexercised.
What: add tests for a topology exit event reaping the holder's locks (and not
others'), unrelated topics/payloads being ignored, and the context round-trip
with and without an app context.
* feat(store): expose lean Lua KV surface
---------
Co-authored-by: Wolfy-J <wolfy.jd@gmail.com>1 parent dd1dc2c commit 700b59a
109 files changed
Lines changed: 14400 additions & 382 deletions
File tree
- api
- cluster/raft
- service/store/kv
- store
- kv
- boot/components
- store
- system
- cluster
- clustertest
- membership
- raft
- multiplex
- runtime/lua/modules
- store
- system
- service/store
- kv
- memory
- sql
- system
- crdt
- kv
- topology
- namereg
- eventual
- global
- kvbacked
- tests/capstone
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
46 | 46 | | |
47 | 47 | | |
48 | 48 | | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
49 | 60 | | |
50 | 61 | | |
51 | 62 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
6 | 6 | | |
7 | 7 | | |
8 | 8 | | |
9 | | - | |
10 | | - | |
11 | | - | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
12 | 13 | | |
13 | 14 | | |
14 | 15 | | |
| |||
25 | 26 | | |
26 | 27 | | |
27 | 28 | | |
| 29 | + | |
28 | 30 | | |
29 | 31 | | |
30 | 32 | | |
31 | 33 | | |
32 | 34 | | |
33 | | - | |
34 | | - | |
35 | | - | |
36 | | - | |
37 | | - | |
38 | | - | |
39 | | - | |
40 | | - | |
41 | | - | |
42 | | - | |
43 | | - | |
44 | | - | |
45 | | - | |
46 | | - | |
47 | | - | |
48 | | - | |
49 | | - | |
50 | | - | |
51 | | - | |
52 | | - | |
53 | | - | |
54 | | - | |
55 | | - | |
56 | | - | |
57 | | - | |
58 | | - | |
59 | | - | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
60 | 40 | | |
61 | 41 | | |
62 | 42 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
12 | 12 | | |
13 | 13 | | |
14 | 14 | | |
15 | | - | |
| 15 | + | |
16 | 16 | | |
17 | 17 | | |
18 | 18 | | |
19 | 19 | | |
20 | 20 | | |
21 | 21 | | |
22 | | - | |
23 | | - | |
24 | | - | |
25 | | - | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
26 | 29 | | |
27 | 30 | | |
28 | 31 | | |
| |||
93 | 96 | | |
94 | 97 | | |
95 | 98 | | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
96 | 154 | | |
97 | 155 | | |
98 | 156 | | |
| |||
115 | 173 | | |
116 | 174 | | |
117 | 175 | | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
8 | 8 | | |
9 | 9 | | |
10 | 10 | | |
11 | | - | |
12 | | - | |
13 | | - | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
14 | 17 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
0 commit comments