fix(platform-wallet): in-band pool manifest fixes genesis-rescan flush loop; retire mirror/reconcile#3828
Conversation
…ools; contain flush blast radius On a genesis (birth_height=0) rescan, SPV can match a UTXO at a registered pool address before the live addresses_derived event for it lands. The UTXO writer's account lookup then missed, raised the fatal UtxoAddressNotDerived, and handle_flush_error dropped the WHOLE buffered changeset; core_bridge re-emitted it and the flush looped forever (173x in prod). core_derived_addresses (the per-address account_index map the UTXO writer joins) was fed only by live derive events and never from the account_address_pools snapshots that already hold every registered address. This wires the snapshot as the second source and stops one unresolvable UTXO from aborting an entire flush. C1 — apply_pools now mirrors every snapshot AddressInfo into core_derived_addresses in the same transaction, carrying the snapshot's real `used`. The row write is DRY'd into core_state::upsert_derived_address_row, called by both apply_pools and the live core_state::apply path through a shared core_state::derivation_path_label, so the two sources cannot drift. C2 — load() rehydrates core_derived_addresses from pools per wallet when the derived table is empty (already-persisted prod DBs), via core_state::rehydrate_derived_addresses_from_pools. It no-ops when rows already exist (no duplicates, no cost) and never touches ClientWalletStartState. No migration: V001 already defines both tables and the addr index. C3 — execute_upsert_utxo now SKIPS an unresolvable unspent UTXO (structured tracing::warn) instead of erroring, so the surrounding valid records, IS-locks, and sync-height still commit and the buffer drains. Funds-safe: the balance re-warms when the address later derives. The spent-only arm keeps its inert account-0 fallback. The UtxoAddressNotDerived variant and its fatal (non-transient) classification are retained for the exhaustive-match contract; it simply stops being raised. Tests: new sqlite_pool_derived_rehydration.rs covers genesis-rescan persist, pool/live row-shape parity, idempotent load rehydration, and blast-radius isolation. The structural-hardening contract test is rewritten to assert the skip semantics; the is_transient exhaustiveness gate still pins the variant as fatal. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…hydrate; review fixups The load-time reconcile fired only when core_derived_addresses was fully empty for a wallet, so a wallet with SOME live-derived rows plus a pool address that never derived was left un-repaired — its balance under-reported. Make the reconcile self-heal partial state, purely additively: - QA-003: drop the empty-only gate. The reconcile now upserts every pool-snapshot address via a dedicated INSERT ... ON CONFLICT DO NOTHING (INSERT_DERIVED_ADDRESS_IF_ABSENT_SQL), so it fills gaps without ever clobbering an existing live/mirrored row's account_index, derivation_path, or used flag. The live apply path keeps its DO UPDATE behavior unchanged. New test load_reconciles_partial_state_without_clobbering_live_rows seeds a live row (used=true, off-pool index) + a missing pool address and asserts the gap is filled while the live row is untouched (RED before, GREEN after). - PROJ-002: SCHEMA.md core_derived_addresses now documents all three population sources (live events, apply_pools mirror, load reconcile) routed through upsert_derived_address_row, and the skip (not reject) UTXO contract. - PROJ-003: rehydrate_derived_addresses_from_pools demoted to pub(crate). - QA-002: present-state comment at the upsert ON CONFLICT clause documenting write-once used (refreshes account_index/derivation_path, never used). - QA-004: reworded the C3 skip comment + warn — the balance re-warms only once the address is later derived (gap-limit dependent), not unconditionally. - QA-005: blast-radius test now mixes a live derive record with the skipped UTXO + sync-height bump and asserts the derive record committed — proving non-skipped records survive, not just sync-state. - QA-001 (deferred): TODO at the V001 core_derived_addresses PK noting it omits account_index/pool_type (ON CONFLICT collapse), practically unreachable under honest BIP32. PK unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…2-leaf PK + UNIQUE(address)
The core_derived_addresses read-index PK was (wallet_id, account_type,
address), which silently collapsed two addresses sharing that key via ON
CONFLICT. Re-key the table on the BIP32 leaf identity and demote `address`
to a UNIQUE-guarded derived attribute, so EVERY address collision —
within-pool or cross-pool — surfaces loud instead of corrupting the
address->account_index map.
PK is (wallet_id, account_type, pool_type, derivation_index): one row per
derived leaf, so a pool of N addresses persists N rows. `account_index`
stays a column (account-level context, the value the read returns) but is
not a uniqueness discriminator. UNIQUE(wallet_id, address) is the sole
arbiter of address uniqueness and its index backs
ACCOUNT_INDEX_BY_ADDRESS_SQL (the standalone idx_core_derived_addresses_addr
is dropped as redundant).
The free-text derivation_path column ("pool_type/index") is dropped: it
was a denormalised mirror of the now-typed pool_type + derivation_index
columns with no production reader. derivation_path_label is removed and
derivation_index is plumbed as a typed u32 through
upsert_derived_address_row (pub(crate)).
Authoritative upsert is ON CONFLICT(<leaf PK>) DO NOTHING — a same-leaf
re-derive is deterministic (address slot-derived, used write-once) so
there is nothing to update; a different leaf yielding the same address
trips UNIQUE(address). Reconcile stays INSERT OR IGNORE so a would-be
UNIQUE collision is skipped rather than aborting load().
V001 edited in place — no committed DB fixture exists. SCHEMA.md updated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…gaps Follow-up to the BIP32-leaf schema hardening. Test-only; no production changes. - Guard `used` write-once on the authoritative apply path: a same-leaf live re-derive (used=false) must not clear a stored used=true. The DO NOTHING clause enforces it; mutation-verified RED under DO UPDATE SET used = excluded.used, GREEN restored. - Tighten assert_unique_violation to assert SQLITE_CONSTRAINT_UNIQUE (2067), not the generic constraint bucket, matching its contract. - Anchor the new CHECK constraints directly: reject an invalid pool_type and account_type inserted into core_derived_addresses. - Exercise the FK ON DELETE CASCADE for core_derived_addresses: rows vanish when their wallet is deleted. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claudius-Maginificent
changed the title
|
✅ Review complete (commit ebb4b30) |
thepastaclaw
left a comment
thepastaclaw
left a comment
There was a problem hiding this comment.
Code Review
One in-scope blocking issue: the new core_derived_addresses PK omits account_index, so two distinct accounts that share the same account_type label (e.g. Standard { index: 0 } and Standard { index: 1 }) collide on PK at the same pool/derivation_index and the second row is silently dropped via DO NOTHING. This breaks the PR's stated BIP32-leaf invariant for multi-account wallets and re-introduces the very missing-row class of bug the PR is meant to close. One supporting nitpick on a now-stale doc comment is also kept.
🔴 1 blocking | 💬 1 nitpick(s)
1 additional finding(s) omitted (not in diff).
🤖 Prompt for all review comments with AI agents
These findings are from an automated code review. Verify each finding against the current code and only fix it if needed.
In `packages/rs-platform-wallet-storage/migrations/V001__initial.rs`:
- [BLOCKING] packages/rs-platform-wallet-storage/migrations/V001__initial.rs:146-162: `core_derived_addresses` PK omits `account_index` and silently drops rows from secondary accounts
The PR documents the PK as the BIP32-leaf identity, but the leaf identity is `(wallet_id, account_type, account_index, pool_type, derivation_index)` — `account_index` is NOT redundant. `accounts::account_type_db_label` collapses every variant carrying a numeric discriminator to a single label (e.g. both `AccountType::Standard { index: 0 }` and `AccountType::Standard { index: 1 }` map to `"standard"`), and `accounts::account_index` stores the discriminator in a companion column. With the PK as written — `(wallet_id, account_type, pool_type, derivation_index)` — two distinct standard accounts deriving the same pool slot (e.g. external, derivation_index = 0) hit the same PK. Their addresses differ (different BIP32 leaves), so `UNIQUE(wallet_id, address)` does NOT fire; instead `upsert_derived_address_row`'s `ON CONFLICT(wallet_id, account_type, pool_type, derivation_index) DO NOTHING` silently drops the second account's row. The same `INSERT OR IGNORE` on the load-time reconcile path drops it again. A later UTXO landing on that second-account address fails the address→account lookup, gets the new "skip undeclared UTXO" treatment, and is silently absent from `core_utxos` — exactly the missing-money class of bug this PR is hardening against, just relocated from single-account to multi-account wallets. The CHECK-domain / UNIQUE / row-shape regression tests don't catch it because every test fixture is a single-account wallet. Add `account_index` to the PK and to the `ON CONFLICT(...)` target in `UPSERT_DERIVED_ADDRESS_SQL` and `INSERT_DERIVED_ADDRESS_IF_ABSENT_SQL`, update the schema comment, and add a regression test with two same-label accounts (e.g. `Standard { index: 0 }` and `Standard { index: 1 }`) deriving the same pool/derivation slot to assert both rows persist and both UTXOs resolve to the correct account_index.
| wallet_id BLOB NOT NULL, | ||
| account_type TEXT NOT NULL CHECK (account_type IN {account_type_check}), | ||
| account_index INTEGER NOT NULL, | ||
| pool_type TEXT NOT NULL CHECK (pool_type IN {pool_type_check}), | ||
| derivation_index INTEGER NOT NULL, | ||
| address TEXT NOT NULL, | ||
| derivation_path TEXT NOT NULL, | ||
| used INTEGER NOT NULL, | ||
| PRIMARY KEY (wallet_id, account_type, address), | ||
| -- PK is the BIP32 leaf identity. `address` is a derived attribute, not | ||
| -- a key, so every collision (within- or cross-pool) trips | ||
| -- UNIQUE(address) loud. `account_index` is account-level context (the | ||
| -- value the read returns), not a uniqueness discriminator. The UNIQUE | ||
| -- index also backs ACCOUNT_INDEX_BY_ADDRESS_SQL. | ||
| PRIMARY KEY (wallet_id, account_type, pool_type, derivation_index), | ||
| UNIQUE (wallet_id, address), | ||
| FOREIGN KEY (wallet_id) REFERENCES wallets(wallet_id) ON DELETE CASCADE | ||
| ); |
There was a problem hiding this comment.
🔴 Blocking: core_derived_addresses PK omits account_index and silently drops rows from secondary accounts
The PR documents the PK as the BIP32-leaf identity, but the leaf identity is (wallet_id, account_type, account_index, pool_type, derivation_index) — account_index is NOT redundant. accounts::account_type_db_label collapses every variant carrying a numeric discriminator to a single label (e.g. both AccountType::Standard { index: 0 } and AccountType::Standard { index: 1 } map to "standard"), and accounts::account_index stores the discriminator in a companion column. With the PK as written — (wallet_id, account_type, pool_type, derivation_index) — two distinct standard accounts deriving the same pool slot (e.g. external, derivation_index = 0) hit the same PK. Their addresses differ (different BIP32 leaves), so UNIQUE(wallet_id, address) does NOT fire; instead upsert_derived_address_row's ON CONFLICT(wallet_id, account_type, pool_type, derivation_index) DO NOTHING silently drops the second account's row. The same INSERT OR IGNORE on the load-time reconcile path drops it again. A later UTXO landing on that second-account address fails the address→account lookup, gets the new "skip undeclared UTXO" treatment, and is silently absent from core_utxos — exactly the missing-money class of bug this PR is hardening against, just relocated from single-account to multi-account wallets. The CHECK-domain / UNIQUE / row-shape regression tests don't catch it because every test fixture is a single-account wallet. Add account_index to the PK and to the ON CONFLICT(...) target in UPSERT_DERIVED_ADDRESS_SQL and INSERT_DERIVED_ADDRESS_IF_ABSENT_SQL, update the schema comment, and add a regression test with two same-label accounts (e.g. Standard { index: 0 } and Standard { index: 1 }) deriving the same pool/derivation slot to assert both rows persist and both UTXOs resolve to the correct account_index.
source: ['codex']
…is missing from the derived index After the eager-mirror + load-time reconcile fix, every address in a wallet's persisted account_address_pools must also live in core_derived_addresses (declared => mapped). The unspent-UTXO writer used to warn-and-skip ANY address it could not resolve, which would silently swallow a regression in that mirror/reconcile and drop live money no one would notice. Discriminate the unspent miss by the pools: a declared-but-unmapped address is now a fatal DerivedIndexInvariantViolated (non-transient => Fatal at the trait boundary); a genuinely-undeclared address keeps the benign warn+skip (not-ours, or an SPV gap-limit edge). The pool-address set is decoded lazily at most once per apply, only on the first miss, and the spent-only synthetic path is left untouched (its inert account_index placeholder is excluded from reads). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…urviving UTXO in the over-fire test The skip-not-fatal test only checked the surviving declared-address UTXO's value, not its resolved account_index. Also assert the survivor is bucketed under the pool's account index (read from the derived map apply_pools mirrored), proving that skipping the undeclared row leaves the good row's attribution intact rather than mis-filing it. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
thepastaclaw
left a comment
thepastaclaw
left a comment
There was a problem hiding this comment.
Code Review
Both prior findings remain STILL VALID at head cccd217. The core_derived_addresses PK still omits account_index even though account_type_db_label (whose own docstring confirms it) collapses every indexed variant (Standard, CoinJoin, IdentityTopUp, DashpayReceivingFunds, DashpayExternalAccount, PlatformPayment) to a single label — two same-label sibling accounts deriving the same pool slot still PK-collide and silently drop via ON CONFLICT DO NOTHING. The new DerivedIndexInvariantViolated fatal guard now turns that dropped row into a fatal flush whenever a UTXO lands at the dropped address, replacing the prior emit-loop with a silent money-drop. The stale ACCOUNT_INDEX_BY_ADDRESS_SQL doc comment also still claims unreachable multi-row semantics under the new UNIQUE(wallet_id, address). No new latest-delta blocking issues; the invariant guard itself is well-shaped.
🔴 2 blocking | 🟡 1 suggestion(s) | 💬 1 nitpick(s)
1 additional finding(s) omitted (not in diff).
🤖 Prompt for all review comments with AI agents
These findings are from an automated code review. Verify each finding against the current code and only fix it if needed.
In `packages/rs-platform-wallet-storage/migrations/V001__initial.rs`:
- [BLOCKING] packages/rs-platform-wallet-storage/migrations/V001__initial.rs:146-162: STILL VALID (carried-forward): `core_derived_addresses` PK omits `account_index`; sibling same-label accounts silently drop and now fatally trip the new invariant guard
The PK at line 159 is `(wallet_id, account_type, pool_type, derivation_index)`, omitting `account_index`. `accounts::account_type_db_label` (src/sqlite/schema/accounts.rs:238) collapses every variant carrying a numeric discriminator — `Standard{index}`, `CoinJoin{index}`, `IdentityTopUp{registration_index}`, `DashpayReceivingFunds{..}`, `DashpayExternalAccount{..}`, `PlatformPayment{account,..}` — to a single TEXT label. The function's own docstring explicitly says "Variants sharing a label are distinguished by the companion `account_index` column" — so the BIP32 leaf identity is `(wallet_id, account_type, account_index, pool_type, derivation_index)` and `account_index` is load-bearing, not "account-level context" as the new PK comment claims.
Consider a wallet with two Standard accounts (index 0 and 1). They derive into the same `(account_type='standard', pool_type='external', derivation_index=N)` slot but produce different addresses A and B. Both `apply_pools` and the live `addresses_derived` path go through `upsert_derived_address_row` (src/sqlite/schema/core_state.rs:243-246) which uses `ON CONFLICT(wallet_id, account_type, pool_type, derivation_index) DO NOTHING`. The first row (account_index=0, address A) inserts; the second (account_index=1, address B) collides on PK and is silently dropped. `UNIQUE(wallet_id, address)` does NOT fire because A ≠ B. The reconcile path (line 252-254) uses `OR IGNORE`, so the row keeps being re-dropped on every load — no recovery.
Under the new latest-delta guard (core_state.rs:200-216), this now manifests as a fatal regression: a later unspent UTXO at address B is pool-declared (both account snapshots are decoded by `pool_declared_addresses`) but unmapped, so it falls into the `DerivedIndexInvariantViolated` arm. `persistence_kind()` returns `Fatal`, so the buffered changeset is wiped. The prior emit-loop is gone, but every flush touching the dropped account's address is now a silent write-off, with the pool snapshot intact so the invariant violation re-triggers on every retry. Net effect on a multi-Standard-account wallet: the second account's funds become unflushable.
No test in `sqlite_pool_derived_rehydration.rs` exercises two same-label accounts (e.g. Standard {index: 0} and Standard {index: 1}) deriving overlapping pool slots; all fixtures hardcode `index: 0`.
In `packages/rs-platform-wallet-storage/src/sqlite/schema/core_state.rs`:
- [BLOCKING] packages/rs-platform-wallet-storage/src/sqlite/schema/core_state.rs:243-246: STILL VALID (carried-forward, fix companion): `UPSERT_DERIVED_ADDRESS_SQL` conflict target must include `account_index`
Tied to the schema fix above: the authoritative writer's `ON CONFLICT(wallet_id, account_type, pool_type, derivation_index)` clause is the SQL site that silently drops the second same-label sub-account's row. The conflict target must match the corrected PK so a same-leaf re-derive remains a DO-NOTHING no-op while a different sub-account at the same pool slot persists its own row. Without aligning this target, the corrected PK would either reject the upsert (no unique index matching the target columns) or, if left at the current shape, re-introduce the silent-drop bug. The matching `INSERT_DERIVED_ADDRESS_IF_ABSENT_SQL` (line 252) uses `OR IGNORE` which auto-adapts to whatever PK the schema declares, so it follows automatically once the PK is updated.
In `packages/rs-platform-wallet-storage/tests/sqlite_pool_derived_rehydration.rs`:
- [SUGGESTION] packages/rs-platform-wallet-storage/tests/sqlite_pool_derived_rehydration.rs:38-56: Missing regression test for two same-label sub-accounts (Standard {index: 0} and {index: 1}) deriving the same pool slot
Every fixture in this test file uses a single Standard account with `index: 0`. The Design-Z thesis is that the PK encodes the BIP32-leaf identity and the schema fails loud on collisions — but neither claim is exercised against the realistic case of two same-label accounts deriving an overlapping pool slot. A test that seeds two Standard accounts ({index: 0} and {index: 1}) whose external pools both occupy derivation_index 0 should (a) persist BOTH derived rows, (b) keep each address's `account_index` lookup returning the correct sub-account, and (c) be RED today — the second row silently disappears, and the new fatal guard fires on that address when a UTXO lands. This is the load-bearing test that would have surfaced the PK bug.
| wallet_id BLOB NOT NULL, | ||
| account_type TEXT NOT NULL CHECK (account_type IN {account_type_check}), | ||
| account_index INTEGER NOT NULL, | ||
| pool_type TEXT NOT NULL CHECK (pool_type IN {pool_type_check}), | ||
| derivation_index INTEGER NOT NULL, | ||
| address TEXT NOT NULL, | ||
| derivation_path TEXT NOT NULL, | ||
| used INTEGER NOT NULL, | ||
| PRIMARY KEY (wallet_id, account_type, address), | ||
| -- PK is the BIP32 leaf identity. `address` is a derived attribute, not | ||
| -- a key, so every collision (within- or cross-pool) trips | ||
| -- UNIQUE(address) loud. `account_index` is account-level context (the | ||
| -- value the read returns), not a uniqueness discriminator. The UNIQUE | ||
| -- index also backs ACCOUNT_INDEX_BY_ADDRESS_SQL. | ||
| PRIMARY KEY (wallet_id, account_type, pool_type, derivation_index), | ||
| UNIQUE (wallet_id, address), | ||
| FOREIGN KEY (wallet_id) REFERENCES wallets(wallet_id) ON DELETE CASCADE | ||
| ); |
There was a problem hiding this comment.
🔴 Blocking: STILL VALID (carried-forward): core_derived_addresses PK omits account_index; sibling same-label accounts silently drop and now fatally trip the new invariant guard
The PK at line 159 is (wallet_id, account_type, pool_type, derivation_index), omitting account_index. accounts::account_type_db_label (src/sqlite/schema/accounts.rs:238) collapses every variant carrying a numeric discriminator — Standard{index}, CoinJoin{index}, IdentityTopUp{registration_index}, DashpayReceivingFunds{..}, DashpayExternalAccount{..}, PlatformPayment{account,..} — to a single TEXT label. The function's own docstring explicitly says "Variants sharing a label are distinguished by the companion account_index column" — so the BIP32 leaf identity is (wallet_id, account_type, account_index, pool_type, derivation_index) and account_index is load-bearing, not "account-level context" as the new PK comment claims.
Consider a wallet with two Standard accounts (index 0 and 1). They derive into the same (account_type='standard', pool_type='external', derivation_index=N) slot but produce different addresses A and B. Both apply_pools and the live addresses_derived path go through upsert_derived_address_row (src/sqlite/schema/core_state.rs:243-246) which uses ON CONFLICT(wallet_id, account_type, pool_type, derivation_index) DO NOTHING. The first row (account_index=0, address A) inserts; the second (account_index=1, address B) collides on PK and is silently dropped. UNIQUE(wallet_id, address) does NOT fire because A ≠ B. The reconcile path (line 252-254) uses OR IGNORE, so the row keeps being re-dropped on every load — no recovery.
Under the new latest-delta guard (core_state.rs:200-216), this now manifests as a fatal regression: a later unspent UTXO at address B is pool-declared (both account snapshots are decoded by pool_declared_addresses) but unmapped, so it falls into the DerivedIndexInvariantViolated arm. persistence_kind() returns Fatal, so the buffered changeset is wiped. The prior emit-loop is gone, but every flush touching the dropped account's address is now a silent write-off, with the pool snapshot intact so the invariant violation re-triggers on every retry. Net effect on a multi-Standard-account wallet: the second account's funds become unflushable.
No test in sqlite_pool_derived_rehydration.rs exercises two same-label accounts (e.g. Standard {index: 0} and Standard {index: 1}) deriving overlapping pool slots; all fixtures hardcode index: 0.
| wallet_id BLOB NOT NULL, | |
| account_type TEXT NOT NULL CHECK (account_type IN {account_type_check}), | |
| account_index INTEGER NOT NULL, | |
| pool_type TEXT NOT NULL CHECK (pool_type IN {pool_type_check}), | |
| derivation_index INTEGER NOT NULL, | |
| address TEXT NOT NULL, | |
| derivation_path TEXT NOT NULL, | |
| used INTEGER NOT NULL, | |
| PRIMARY KEY (wallet_id, account_type, address), | |
| -- PK is the BIP32 leaf identity. `address` is a derived attribute, not | |
| -- a key, so every collision (within- or cross-pool) trips | |
| -- UNIQUE(address) loud. `account_index` is account-level context (the | |
| -- value the read returns), not a uniqueness discriminator. The UNIQUE | |
| -- index also backs ACCOUNT_INDEX_BY_ADDRESS_SQL. | |
| PRIMARY KEY (wallet_id, account_type, pool_type, derivation_index), | |
| UNIQUE (wallet_id, address), | |
| FOREIGN KEY (wallet_id) REFERENCES wallets(wallet_id) ON DELETE CASCADE | |
| ); | |
| CREATE TABLE core_derived_addresses ( | |
| wallet_id BLOB NOT NULL, | |
| account_type TEXT NOT NULL CHECK (account_type IN {account_type_check}), | |
| account_index INTEGER NOT NULL, | |
| pool_type TEXT NOT NULL CHECK (pool_type IN {pool_type_check}), | |
| derivation_index INTEGER NOT NULL, | |
| address TEXT NOT NULL, | |
| used INTEGER NOT NULL, | |
| -- PK is the BIP32 leaf identity. `account_type` is a TEXT label shared | |
| -- by every variant of AccountType carrying a numeric discriminator | |
| -- (Standard{index}, CoinJoin{index}, IdentityTopUp{registration_index}, | |
| -- DashpayReceivingFunds{..}, DashpayExternalAccount{..}, | |
| -- PlatformPayment{account,..}); `account_index` is its companion | |
| -- discriminator and is LOAD-BEARING in the PK — sibling sub-accounts | |
| -- under the same label derive distinct xpubs and must persist as | |
| -- distinct rows. `address` is a derived attribute, not a key, so every | |
| -- collision (within- or cross-pool) trips UNIQUE(address) loud. The | |
| -- UNIQUE index also backs ACCOUNT_INDEX_BY_ADDRESS_SQL. | |
| PRIMARY KEY (wallet_id, account_type, account_index, pool_type, derivation_index), | |
| UNIQUE (wallet_id, address), | |
| FOREIGN KEY (wallet_id) REFERENCES wallets(wallet_id) ON DELETE CASCADE | |
| ); |
source: ['claude', 'codex']
| const UPSERT_DERIVED_ADDRESS_SQL: &str = "INSERT INTO core_derived_addresses \ | ||
| (wallet_id, account_type, account_index, pool_type, derivation_index, address, used) \ | ||
| VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7) \ | ||
| ON CONFLICT(wallet_id, account_type, pool_type, derivation_index) DO NOTHING"; |
There was a problem hiding this comment.
🔴 Blocking: STILL VALID (carried-forward, fix companion): UPSERT_DERIVED_ADDRESS_SQL conflict target must include account_index
Tied to the schema fix above: the authoritative writer's ON CONFLICT(wallet_id, account_type, pool_type, derivation_index) clause is the SQL site that silently drops the second same-label sub-account's row. The conflict target must match the corrected PK so a same-leaf re-derive remains a DO-NOTHING no-op while a different sub-account at the same pool slot persists its own row. Without aligning this target, the corrected PK would either reject the upsert (no unique index matching the target columns) or, if left at the current shape, re-introduce the silent-drop bug. The matching INSERT_DERIVED_ADDRESS_IF_ABSENT_SQL (line 252) uses OR IGNORE which auto-adapts to whatever PK the schema declares, so it follows automatically once the PK is updated.
| const UPSERT_DERIVED_ADDRESS_SQL: &str = "INSERT INTO core_derived_addresses \ | |
| (wallet_id, account_type, account_index, pool_type, derivation_index, address, used) \ | |
| VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7) \ | |
| ON CONFLICT(wallet_id, account_type, pool_type, derivation_index) DO NOTHING"; | |
| // Conflict target = the BIP32-leaf PK. A same-leaf re-derive is | |
| // deterministic — `address` is a pure function of the slot and `used` is | |
| // write-once — so there is nothing legitimate to update; DO NOTHING. A | |
| // different leaf yielding the same `address` is a UNIQUE(address) | |
| // violation, not a PK hit, so it surfaces loud. | |
| const UPSERT_DERIVED_ADDRESS_SQL: &str = "INSERT INTO core_derived_addresses \ | |
| (wallet_id, account_type, account_index, pool_type, derivation_index, address, used) \ | |
| VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7) \ | |
| ON CONFLICT(wallet_id, account_type, account_index, pool_type, derivation_index) DO NOTHING"; |
source: ['claude']
| if !matches!(account_type, AccountType::Standard { index: 0, .. }) { | ||
| continue; | ||
| } | ||
| for pool in managed.managed_account_type().address_pools() { | ||
| if pool.pool_type != AddressPoolType::External { | ||
| continue; | ||
| } | ||
| let infos: Vec<AddressInfo> = pool.addresses.values().cloned().collect(); | ||
| if infos.is_empty() { | ||
| continue; | ||
| } | ||
| return AccountAddressPoolEntry { | ||
| account_type, | ||
| pool_type: pool.pool_type, | ||
| addresses: infos, | ||
| }; | ||
| } | ||
| } | ||
| panic!("wallet must expose a non-empty Standard BIP44 external pool"); |
There was a problem hiding this comment.
🟡 Suggestion: Missing regression test for two same-label sub-accounts (Standard {index: 0} and {index: 1}) deriving the same pool slot
Every fixture in this test file uses a single Standard account with index: 0. The Design-Z thesis is that the PK encodes the BIP32-leaf identity and the schema fails loud on collisions — but neither claim is exercised against the realistic case of two same-label accounts deriving an overlapping pool slot. A test that seeds two Standard accounts ({index: 0} and {index: 1}) whose external pools both occupy derivation_index 0 should (a) persist BOTH derived rows, (b) keep each address's account_index lookup returning the correct sub-account, and (c) be RED today — the second row silently disappears, and the new fatal guard fires on that address when a UTXO lands. This is the load-bearing test that would have surfaced the PK bug.
source: ['claude']
…ount_address_pools stays complete The wallet-event adapter wrapped every post-registration CoreChangeSet with `account_address_pools` empty (`..Default::default()`), so the pool manifest was frozen at registration and never tracked gap-limit extensions. Those flowed only through the live `addresses_derived` delta, which has no `used` flag and is not a full pool — it can't rebuild the manifest. Attach a full pool snapshot in-band whenever the core delta derived new addresses: read the whole current pool for each affected account straight from the `WalletManager` the adapter holds, and ship it on the SAME changeset. The persister applies `account_address_pools` before the core UTXO delta in one tx, so a freshly-derived address is resolvable when that changeset's UTXOs are written — closing the gap-limit race in-band. Empty deltas emit nothing, so no write amplification on blocks that derive nothing. Additive only: the storage-side eager-mirror, load reconcile, and invariant guard are untouched this phase. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
… gate Adds Marvin's verification-gate scratch tests for the PR #3828 Phase-2 gate. They pin the single load-bearing claim the workaround-retirement plan rests on: a UTXO landing on a freshly-derived address resolves to the correct account_index when the pool snapshot rides the same PlatformWalletChangeSet as the UTXO, because apply_changeset_to_tx applies account_address_pools before the core UTXO delta inside one transaction. Three cases: in-band single changeset, adversarial merged-buffer arrival (UTXO stored before snapshot, Manual flush), and the non-fatal skip of a UTXO on a genuinely undeclared address. All pass on the current tree with the eager-mirror still present, demonstrating the manifest alone is a sufficient resolution source. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… manifest fallback; relocate fatal guard to emitter-contract With the Phase-1 emitter making account_address_pools a complete, in-band manifest, the three storage workarounds collapse into one resolution rule. - Remove the eager-mirror: apply_pools writes only the snapshot blob; core_derived_addresses is fed exclusively by the live addresses_derived path, acting as an indexed read-cache in front of the manifest. - Remove the load-time reconcile (rehydrate_derived_addresses_from_pools and its load-path call); resolution-on-read needs no backfill. - UTXO resolution: derived-cache hit, else fall back to the pool manifest (the in-band snapshot is applied earlier in the same tx), else warn+skip a genuinely undeclared address. The old per-UTXO fatal branch becomes the manifest-fallback success path. - Relocate the fatal guard to the emitter contract: a live addresses_derived entry absent from the manifest aborts the flush (DerivedIndexInvariantViolated) at the storage trust boundary. Lag-safe — dropped events produce no changeset, so it can only fire on an emitter bug. No migration: the new resolution is a strict superset of the old read path. Tests reframed to the new model (manifest fallback, emitter-contract guard, old-DB back-compat); SCHEMA.md updated. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
lklimek
changed the title
thepastaclaw
left a comment
thepastaclaw
left a comment
There was a problem hiding this comment.
Code Review
Prior findings reconciliation: #1 STILL VALID, #2 STILL VALID, #3 STILL VALID, #4 STILL VALID. The latest delta (manifest-fallback + apply-time guard) cleanly retires the eager-mirror/reconcile path and meaningfully reduces blast radius of the schema bug (UTXO resolution falls back to the pool manifest), but the V001 core_derived_addresses PK and its companion UPSERT conflict target still omit account_index, so the BIP32-leaf identity the schema comment claims is not the one SQLite enforces. Sibling same-label accounts (e.g. Standard{index:0} and Standard{index:1}) at the same (account_type, pool_type, derivation_index) slot silently collide; the second derived-address row is dropped and the cache becomes permanently incomplete for that sibling, even though funds are no longer at risk thanks to the manifest fallback. No new latest-delta defects identified beyond the carried-forward issues.
🔴 2 carried-forward blocking | 🟡 1 suggestion(s) | 💬 1 nitpick(s)
1 additional finding(s) omitted (not in diff).
2 carried-forward finding(s) already raised on this PR; not re-posting as new inline comments.
🤖 Prompt for all review comments with AI agents
These findings are from an automated code review. Verify each finding against the current code and only fix it if needed.
In `packages/rs-platform-wallet-storage/tests/sqlite_pool_derived_rehydration.rs`:
- [SUGGESTION] packages/rs-platform-wallet-storage/tests/sqlite_pool_derived_rehydration.rs:32-76: [carried-forward STILL VALID] No regression test for two same-label sibling accounts on the same pool slot
The fixtures (`standard_external_pool`, `wallet_with_pools`) still only exercise a single `Standard{index:0}` account, and the new rich invariant tests added in this PR (within-pool/cross-pool collisions, manifest-fallback, reload-no-reconcile) never construct two same-label sub-accounts. The load-bearing regression for the PK / ON CONFLICT items above is: seed both `Standard{index:0}` and `Standard{index:1}`, register both pools, derive both External pools at `derivation_index=0`, persist via the in-band emitter, and assert (a) both derived rows persist in `core_derived_addresses`, and (b) UTXOs at each address resolve to their respective `account_index`. Today neither claim is exercised, so the silent row-drop is invisible to the suite.
| fn standard_external_pool(info: &ManagedWalletInfo) -> AccountAddressPoolEntry { | ||
| use key_wallet::account::AccountType; | ||
| use key_wallet::managed_account::address_pool::AddressPoolType; | ||
| for managed in info.all_managed_accounts() { | ||
| let account_type = managed.managed_account_type().to_account_type(); | ||
| if !matches!(account_type, AccountType::Standard { index: 0, .. }) { | ||
| continue; | ||
| } | ||
| for pool in managed.managed_account_type().address_pools() { | ||
| if pool.pool_type != AddressPoolType::External { | ||
| continue; | ||
| } | ||
| let infos: Vec<AddressInfo> = pool.addresses.values().cloned().collect(); | ||
| if infos.is_empty() { | ||
| continue; | ||
| } | ||
| return AccountAddressPoolEntry { | ||
| account_type, | ||
| pool_type: pool.pool_type, | ||
| addresses: infos, | ||
| }; | ||
| } | ||
| } | ||
| panic!("wallet must expose a non-empty Standard BIP44 external pool"); | ||
| } | ||
|
|
||
| /// A wallet's Standard BIP44 external pool plus its first `AddressInfo` — | ||
| /// the load-bearing target the UTXO writer must resolve. | ||
| fn wallet_with_pools(seed_byte: u8) -> (Vec<AccountAddressPoolEntry>, AddressInfo) { | ||
| let seed = [seed_byte; 64]; | ||
| let wallet = Wallet::from_seed_bytes( | ||
| seed, | ||
| key_wallet::Network::Testnet, | ||
| WalletAccountCreationOptions::Default, | ||
| ) | ||
| .unwrap(); | ||
| let info = ManagedWalletInfo::from_wallet(&wallet, 0); | ||
| let pool = standard_external_pool(&info); | ||
| let target = pool | ||
| .addresses | ||
| .first() | ||
| .cloned() | ||
| .expect("non-empty external pool"); | ||
| (vec![pool], target) | ||
| } |
There was a problem hiding this comment.
🟡 Suggestion: [carried-forward STILL VALID] No regression test for two same-label sibling accounts on the same pool slot
The fixtures (standard_external_pool, wallet_with_pools) still only exercise a single Standard{index:0} account, and the new rich invariant tests added in this PR (within-pool/cross-pool collisions, manifest-fallback, reload-no-reconcile) never construct two same-label sub-accounts. The load-bearing regression for the PK / ON CONFLICT items above is: seed both Standard{index:0} and Standard{index:1}, register both pools, derive both External pools at derivation_index=0, persist via the in-band emitter, and assert (a) both derived rows persist in core_derived_addresses, and (b) UTXOs at each address resolve to their respective account_index. Today neither claim is exercised, so the silent row-drop is invisible to the suite.
source: ['claude', 'codex']
3 participants
Why this PR exists
flush failed fatally — utxo_address_not_derivedacross 7 wallets in one boot).core_derived_addresses(anaddress → account_indexindex). That index was fed only by liveaddresses_derivedevents, and theaccount_address_poolsmanifest was frozen at registration — thecore_bridgeevent adapter built every changeset with..Default::default(), never re-emitting pool snapshots on gap-limit extension. So during a rescan, SPV matched historical UTXOs at registered addresses before their derive event landed → the resolution missed →UtxoAddressNotDerived→ classified fatal → the flush-error handler dropped the entire buffered changeset (UTXOs, IS-locks, sync height) → the adapter re-emitted it → infinite loop.feat/platform-wallet-rehydration.What was done
This PR fixes the root cause and then retires the defensive workarounds an earlier iteration of this branch had added.
1. Root cause — make the manifest complete and timely (in-band emitter)
packages/rs-platform-wallet/src/changeset/core_bridge.rs: when a changeset carries newly-derived addresses, emit a fullaccount_address_poolssnapshot for the affected accounts on the same changeset as the core delta. It's sourced from the liveWalletManagerthe adapter already holds — no rust-dashcore / upstream change. The persister appliesaccount_address_poolsbefore the core UTXO delta in the same transaction, so a freshly-derived address is resolvable the instant its UTXO is written. This closes both the genesis-rescan race (registration addresses) and the gap-limit race (extension addresses), in-band.2. Retire the workarounds (storage)
With the manifest now complete + timely, the storage-side patches are redundant:
apply_poolsno longer copies pool rows intocore_derived_addresses) and the load-time reconcile (rehydrate_derived_addresses_from_poolsdeleted).core_derived_addressescache hit → use; miss → fall back to the pool manifest; miss in both (unspent) → non-fatalwarn!+ skip; spent-only → inert placeholder.DerivedIndexInvariantViolatedguard from a per-UTXO check to an apply-time emitter-contract check: everyaddresses_derivedaddress must be present in the manifest, else fatal. This validates the producer at the storage trust boundary and is lag-safe — a dropped event produces no changeset, so the guard can only fire on a genuine emitter-contract violation, never on benign event-bus lag (RecvError::Lagged).Net storage change: −90 production lines — three mechanisms collapsed into one rule.
How Has This Been Tested?
platform-wallet): in-band ordering, full-snapshot-not-delta, no-emit-on-empty-delta.cargo test -p platform-wallet --lib→ 172 passed.core_bridgeis the sole producer of persistednew_utxos; the emitter snapshots all affected account types incl. internal/external and non-ECDSA pools; every own-address UTXO is resolvable from (manifest ∪ live cache).platform-wallet-storage): manifest-fallback correctness (correctaccount_indexvia the single sharedaccounts::account_indexfn — no drift); back-compat on old-style DBs (frozen pools + live rows, no migration); the relocated guard (fatal on a real contract violation, never on production traffic).cargo test -p platform-wallet-storage→ 398 passed.cargo fmt --all --checkclean;cargo clippy -p platform-wallet -p platform-wallet-storage --all-targets --all-features -- -D warningsclean.Known follow-ups (non-blocking)
Standardmanifest for a CoinJoin-only fixture; inert today.project_derived_addressesdrops non-ECDSA addresses from the derived delta; inert because a non-ECDSA address can never be anew_utxosUTXO (an upstream classifier property, documented inSCHEMA.md, not pinnable at the storage layer).used-flag drift — ausedflip without a derivation isn't re-snapshotted; cosmetic only (does not affect address→account resolution).Breaking Changes
None — additive and corrective. No public signature removed; no migration (V001 already declares both tables and is unshipped). The behavioral changes are all corrective.
rust-dashcore dependency
Unchanged. The fix is contained to the
platform-walletandplatform-wallet-storagecrates (11 files, zeroCargo.toml/Cargo.lockchanges). The pinned rust-dashcore rev is already a published commit ondev.Checklist:
For repository code-owners and collaborators only
🤖 Co-authored by Claudius the Magnificent AI Agent