refactor: complete data.db unwire — shielded + DashPay (stacked on #860) by lklimek · Pull Request #861 · dashpay/dash-evo-tool

lklimek · 2026-05-29T05:50:29Z

Stacked on top of PR #860. Lands the two domains PR #860 left deferred: shielded (commitment_tree) and DashPay (storage + sync).

Why the deferrals turned out wrong

PR #860's 6aa9c393 recorded two deferrals with documented rationales. A fresh feasibility audit on this branch (Nagatha pass #5) found both materially incorrect:

Shielded: grovedb-commitment-tree v4.0.0 (pin 60f2968) already ships ClientPersistentCommitmentTree::open_path(Path, max_checkpoints), which opens its own SQLite file and creates schema on first use. DET only needed to call it instead of open_on_shared_connection. No upstream change required.
DashPay: the 5 prerequisites in the PR feat: platform-wallet backend rewrite (spec + implementation) #860 deferral note were 4-of-5 incorrect:
- SecretStore integration not needed — upstream stores alias / note / is_hidden / accepted_accounts as plaintext on EstablishedContact.
- AcceptContactRequest already uses Identifier, not i64 PK.
- Schema shape matches 1:1 (alias=nickname, note, is_hidden).
- State-machine vocabulary gap is real but bridgeable via a derivation table at read time.
- The 15K-LOC UI rewrite estimate was inflated ~10× under an adapter pattern that keeps Stored* types and translates upstream presence-based reads at the WalletBackend seam.

Commits (8)

#	SHA	Subject
S1	`73996e52`	shielded commitment_tree → per-network SQLite file
D1	`68b6cc35`	DashpayView adapter (read-only foundation)
D2	`3c26e3ae`	backend read paths → adapter
D3	`1f81ef24`	write paths → upstream + k/v sidecar
D4a	`a345caa0`	hoist Stored* types; migrate UI read paths
D4b	`21f58ac7`	UI writes + ContactAddressIndex sidecar primitives + atomic-increment mutex
D4c	`91917fe1`	backend_task callsites → sidecar/adapter; concurrency test
D4d	`0acb5d98`	DELETE DET tables; finalize

Key architectural pieces

Shielded migration: silent one-shot copy on first cold start — ATTACH legacy data.db and INSERT … SELECT the 4 commitment_tree_* tables into <spv_dir>/<network>/shielded-commitment-tree.sqlite. Failure mode rolls back the new file; user can retry. Existing shielded users keep their tree.
DashPay adapter pattern: WalletBackend::dashpay_view() exposes contacts() / contact_requests() / payments() / profile() returning DET-shape Stored* types built from upstream EstablishedContact / ContactRequestEntry / PaymentEntry. UI screens unchanged — they still consume the same types.
DET-local sidecar for fields upstream doesn't carry (blocked, rejected, ContactPrivateInfo, ContactAddressIndex, timestamps): bincode blobs at det:dashpay:* keys in the per-network k/v store. ContactAddressIndex send-index increments are atomic via a WalletBackend-internal Mutex (option (b) — accepted over upstream API change or best-effort semantics).
State derivation at read time:
- Contact.status = accepted ← upstream contacts_established row
- Contact.status = pending ← outgoing request, no established
- Contact.status = blocked ← k/v sidecar
- Request.status = pending ← outgoing, no matching established
- Request.status = accepted ← matching established
- Request.status = rejected ← k/v sidecar
- Request.status = expired ← pending && age > DASHPAY_REQUEST_EXPIRY_DAYS (7)
- Payment.status ← direct from upstream PaymentEntry.confirmation_state

What data.db retains after this PR

asset_lock_transaction table — dormant (no readers, no writers since 5cc6e893)
A handful of bootstrap columns + database_version

The migration tool (planned as a library in a separate PR) reads the legacy DashPay + shielded state via git history at 35eb07bf (last commit with full DET code paths).

Test plan

Deferred items (deliberate)

Alias/note via upstream EstablishedContact::set_alias/set_note setters: upstream pin 17653ba8 exposes the setters but ManagedIdentity lacks a pub &mut accessor. DET's existing alias/note flow goes through DashPay contactInfo platform documents (not DET DB writes) — unchanged. A follow-up PR can route through upstream setters once the accessor lands.

Notes for review

This PR depends on PR #860 landing first (or being merged into a base that includes it). The feat/unwire-deferred-domains branch was cut off 6aa9c393 (PR #860 tip).

🤖 Generated with Claude Code

_{🤖 Co-authored by Claudius the Magnificent AI Agent}

The C8a deferral rationale was wrong: grovedb-commitment-tree v4.0.0 already exposes ClientPersistentCommitmentTree::open_path which opens its own SQLite file. DET just had to call it. - src/context/shielded.rs: switch open_on_shared_connection to open_path at <spv_dir>/<network>/shielded-commitment-tree.sqlite (sibling to upstream's platform-wallet.sqlite). - src/database/shielded.rs: drop clear_commitment_tree_tables; replace call sites with file-unlink of the new sqlite path. - New silent one-shot migrator: on first cold start where the new file doesn't exist and data.db has commitment_tree_* tables with rows, copy them over via ATTACH. Existing users retain shielded state. Failed migrations leave data.db untouched and the new file removed, so the migrator can re-run on next launch. - src/database/initialization.rs: remove the 4 commitment_tree_* CREATE TABLE blocks; new installs never create them in data.db. data.db's load-bearing role for shielded is now retired. Only DashPay remains as a data.db dependency (will follow in D1-D5). Part of the deferred unwire (stacked PR on top of #860). 🤖 Co-authored by [Claudius the Magnificent](https://github.com/lklimek/claudius) AI Agent

D1 in the deferred-domains stack. Lays the read seam that D2-D5 will route through. No deletions, no write-path changes — just the adapter. - src/wallet_backend/dashpay.rs: new DashpayView with contacts / contact_requests / payments / profile methods. Translates upstream presence-based reads into DET-shape Stored* types. - Status derivation per Nagatha's audit (presence → DET enum): * Contact.accepted ← upstream contacts_established row * Contact.pending ← outgoing request, no established * Contact.blocked ← k/v det:dashpay:blocked:<id> (D3 writes) * Request.pending ← outgoing, no matching established * Request.accepted ← matching established * Request.rejected ← k/v det:dashpay:rejected:<id> (D3 writes) * Request.expired ← deferred to D2 (no DET threshold constant) * Payment.* ← direct from PaymentEntry.status - DET-local created_at/updated_at preserved via k/v sidecar at det:dashpay:timestamps:<id> (D3 writes). Payment timestamps keyed on tx_id. - Unit tests for the translator (12 new cases). - Bumps database::dashpay module visibility to pub(crate) so the wallet_backend adapter can reuse the existing Stored* shapes without redefining them. DET tables stay untouched; UI/backend tasks still read from Database::* methods. D2 will switch read paths over; D3 wires writes; D4 deletes the DET tables; D5 cleanup. Stacked on PR #860. 🤖 Co-authored by [Claudius the Magnificent](https://github.com/lklimek/claudius) AI Agent

D2 in the deferred-domains stack. DashPay backend read sites that sourced from DET data.db now pull from the upstream-backed adapter instead. DET tables still exist and get written to by un-migrated writers (D3 cuts those; D4 deletes the tables). - backend_task/dashpay/payments.rs: load_payment_history reads via wallet_backend.dashpay_view().payments(); DET fallback when the backend is not yet wired. - backend_task/dashpay/incoming_payments.rs: register-addresses path reads contacts via the adapter (same fallback). - backend_task/dashpay.rs LoadPaymentHistory handler: contacts read via adapter; calls wallet_backend.dashpay_sync() first to refresh upstream state on this user-initiated refresh action. Adapter additions: - DASHPAY_REQUEST_EXPIRY_DAYS = 7 — pending outgoing requests older than the threshold report as `"expired"` via derive_request_status. - WalletBackend::dashpay_sync(owner) wraps IdentityWallet::dashpay_sync for the wallet managing `owner`. Out of scope (will land in later stacks): - UI screens that read DB synchronously from ui() — the adapter is async; migrating those needs structural rework. - LoadProfile / LoadContacts / LoadContactRequests fetch from SDK directly (not DET reads), so dashpay_sync would overlap their work. - DET-only ContactAddressIndex table (DIP-0015 derivation tracking) has no upstream counterpart — not part of this seam. UI screens unchanged (Stored* types preserved via adapter). Stacked on PR #860 (D2 of 5 in the DashPay sub-stack). 🤖 Co-authored by [Claudius the Magnificent](https://github.com/lklimek/claudius) AI Agent

D3 in the deferred-domains stack. DashPay writes go upstream (for upstream-stored fields) and to k/v sidecar (for DET-local blocked / rejected / timestamps). DET DashPay tables now orphaned — readers still exist (UI sync paths from D2 deviation) but no writer populates DET data.db after this commit. - wallet_backend/dashpay.rs: 6 new WalletBackend write helpers (dashpay_set_profile / dashpay_record_payment via upstream ManagedIdentity::set_dashpay_profile + record_dashpay_payment; dashpay_mark_blocked / dashpay_unmark_blocked / dashpay_mark_rejected / dashpay_set_timestamps / dashpay_set_payment_timestamps via the k/v sidecar). - backend_task/dashpay/profile.rs: load_profile + update_profile (4 sites) → mirror_profile_to_backend pushes DashPayProfile down to upstream + timestamp sidecar. - backend_task/dashpay/payments.rs: send_payment_to_contact_impl + new mirror_sent_payment_to_backend / mirror_incoming_payment helpers route through upstream record_dashpay_payment + tx- timestamp sidecar. - backend_task/dashpay/incoming_payments.rs: incoming payment save_payment → mirror_incoming_payment_to_backend. - backend_task/dashpay/contact_requests.rs: reject_contact_request now writes det:dashpay:rejected:<sender_id> sidecar so the view surfaces the rejection precedence. - backend_task/error.rs: new DashpaySidecarStorage TaskError variant. - ContactAddressIndex / update_highest_receive_index / update_bloom_registered_count (DIP-0015 derivation, no upstream counterpart) left untouched — D4 will decide their fate. K/V sidecar keys match the D1 reader exactly: * det:dashpay:blocked:<contact_id> * det:dashpay:rejected:<counterparty_id> * det:dashpay:timestamps:<entity_id> (i64, i64) * det:dashpay:timestamps:tx:<tx_id> (i64, Option<i64>) 7 new translator tests cover the write/read contract: send→block→list yields blocked; send→reject→list yields rejected (rejected outranks expired); 7-day-old pending yields expired; plus key-encoding round-trips for blocked/rejected/timestamps/ payment-timestamps. EstablishedContact alias/note routing is a partial stop: upstream exposes set_alias/set_note on EstablishedContact but ManagedIdentity lacks a pub mutable accessor at pin 17653ba8. DET's existing alias/note plumbing flows through DashPay contactInfo platform documents (create_or_update_contact_info) which were never DET-table writes — out of D3 scope, deferred. DET dashpay tables now write-quiesced from backend_task; D4 will migrate the remaining UI sync read paths and delete the tables. Stacked on PR #860. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

D4a in the deferred-domains stack. Splits the original D4 ("UI migration + table delete") because UI sync writes are still live and the adapter's Stored* types lived inside the file slated for deletion. - Stored{Contact,ContactRequest,Payment,Profile} and ContactAddressIndex hoisted from src/database/dashpay.rs to src/model/dashpay.rs. Adapter import in wallet_backend/dashpay.rs updated; database/dashpay.rs now re-imports from model. - 13 UI sync read sites in src/ui/dashpay/ migrated: * contacts_list.rs, contact_details.rs, contact_profile_viewer.rs, send_payment.rs, profile_screen.rs, contact_requests.rs * Pattern: dropped redundant DET cache reads where async fetch already populated state; remaining sites use a `!*_loaded` flag + auto-dispatch DashPayTask::Load* at the top of render() and refresh_on_arrival. DET DashPay tables intact — UI writes still populate them. D4b will migrate writes, move ContactAddressIndex to k/v sidecar, and delete the tables. Stacked on PR #860. 🤖 Co-authored by [Claudius the Magnificent](https://github.com/lklimek/claudius) AI Agent

…ives D4b in the deferred-domains stack. Narrow scope: UI write migration + address-index k/v primitives. DET tables stay; backend_task callsites migrate in D4c; tables delete in D4d. UI writes (11 sites) cut: - contacts_list, contact_details, contact_profile_viewer, send_payment, profile_screen, contact_requests - Fields covered by D3 backend-task mirrors → UI write dropped - DET-local fields (memos, timestamps, blocked, rejected) → k/v sidecar at det:dashpay:private:<contact_id> etc. ContactAddressIndex k/v primitives added to wallet_backend/dashpay.rs: - dashpay_{get,set}_address_index - dashpay_increment_send_index (atomic via WalletBackend-internal mutex; chosen over upstream-API change or best-effort) - dashpay_{get,set}_address_mapping Key shapes: - det:dashpay:address_index:<owner>:<contact> → ContactAddressIndex - det:dashpay:addr_map:<owner>:<address> → contact_id - det:dashpay:private:<contact_id> → ContactPrivateInfo Backend_task callsites still call Database::* (D4c migrates them). DET tables remain (D4d deletes them). Stacked on PR #860. 🤖 Co-authored by [Claudius the Magnificent](https://github.com/lklimek/claudius) AI Agent

D4c in the deferred-domains stack. Cuts the remaining DET-table readers in backend_task and UI memo paths so D4d can delete the tables. Migrated: - backend_task/dashpay/incoming_payments.rs: 6 DB calls → k/v sidecar (address_index, address_mapping) + bloom counter - backend_task/dashpay/payments.rs: * get_and_increment_send_index → WalletBackend::dashpay_increment_send_index (atomic via D4b's internal mutex) * load_payment_history fallback → dashpay_view().payments() - backend_task/dashpay.rs::LoadContacts fallback → dashpay_view().contacts() - ui/dashpay/contact_profile_viewer.rs + contact_details.rs: load_contact_private_info → dashpay_get_private_info (D4b primitive) Extended dashpay_set/get_address_mapping to carry the per-address derivation index in the sidecar value, since the DET table column this replaced was the sole source of that bit for the incoming- payment detector. Existing round-trip test updated to the new schema. Added a concurrency test for dashpay_increment_send_index: 100 concurrent tokio tasks against the same key all observe distinct values in 0..=99 with no duplicates. Extracted the locked read-modify-write into a free helper (increment_send_index_locked) so the test exercises production code without standing up a full SDK + persister backend. DET DashPay tables now have ZERO readers in DET code. D4d deletes them. Stacked on PR #860. 🤖 Co-authored by [Claudius the Magnificent](https://github.com/lklimek/claudius) AI Agent

D4d closes the DashPay unwire. With S1 having retired shielded and D1-D4c having migrated all DashPay reads and writes, the DET DashPay tables have zero remaining callers in DET code. - src/database/dashpay.rs deleted (894 LOC, last full-domain module). - src/database/contacts.rs deleted (contact_private_info, 356 LOC). - src/context/wallet_lifecycle.rs::clear_network_database: new det:dashpay:* k/v prefix sweep via WalletBackend::kv() (the right home — Database has no kv handle). src/database/mod.rs:: clear_network_data: legacy dashpay/contact_private_info DELETE statements removed. - src/database/initialization.rs: CREATE TABLE removed from the fresh-install create_tables path for 7 tables — dashpay_profiles, dashpay_contacts, dashpay_contact_requests, dashpay_payments, dashpay_contact_address_indices, dashpay_address_mappings, contact_private_info. v13 / v33 ladder entries that called the deleted init helpers are now stubbed with present-state comments; the surviving column-add migrations (add_avatar_bytes_column, add_network_column_to_dashpay_*) already carry table_exists guards per the C7 / C8a pattern and stay in place for legacy installs. - 3 D4b dual-writes collapsed to single sidecar writes (contact_details.rs save_contact_private_info, contact_profile_viewer.rs save_private_info, contacts_list.rs set_contact_hidden). Bonus: a 4th contacts_list.rs load-time write path (clear_dashpay_contacts + save_dashpay_contact + save_contact_private_info per refreshed contact) is now an adapter-driven in-memory repopulate — the upstream-backed read path makes those DB writes redundant. - tests/backend-e2e/dashpay_tasks.rs::tc_041 retained as an adapter-wiring smoke check; comment now reflects the post-D4d reality (populated-state coverage lives in tc_037 / tc_044). - assert_v33_schema (initialization.rs test) no longer asserts contact_private_info / dashpay_contact_requests on fresh installs. - 3 new D4d unit tests in src/wallet_backend/dashpay.rs cover the prefix-sweep contract used by clear_network_database (all keys share det:dashpay:, sweep drains the sidecar, sweep skips unrelated overlays). - docs/ai-design/2026-05-28-migration-tool/notes.md: DashPay status flipped DONE with commit list (S1, D1, D2, D3, D4a, D4b, D4c, D4d). Migration-tool author still reads DET DashPay state at 35eb07b (pre-unwire HEAD). data.db now retains ONLY asset_lock_transaction (dormant) and a handful of bootstrap columns. With shielded retired (S1) and DashPay retired (D1-D4d), data.db's role is reduced to legacy artifact for the future migration tool. Stacked on PR #860. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

coderabbitai · 2026-05-29T05:50:36Z

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

🗂️ Base branches to auto review (2)

master
v1.0-dev

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: bc558ed6-0657-4009-b96a-62676e75d194

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch feat/unwire-deferred-domains

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

Reference table of every k/v key DET uses post-unwire — key shape, scope, backing persister (det-app.sqlite vs per-network), value type and encoding (bincode + version byte). Includes the upstream SecretStore labels DET allocates for private-key material. Stacked on PR #860 / #861. Co-authored-by: Claude <noreply@anthropic.com>

lklimek and others added 8 commits May 29, 2026 00:30

lklimek marked this pull request as ready for review May 29, 2026 07:11

lklimek merged commit b0fecac into docs/platform-wallet-migration-design May 29, 2026
6 checks passed

lklimek deleted the feat/unwire-deferred-domains branch May 29, 2026 07:11

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

refactor: complete data.db unwire — shielded + DashPay (stacked on #860)#861

refactor: complete data.db unwire — shielded + DashPay (stacked on #860)#861
lklimek merged 9 commits into
docs/platform-wallet-migration-designfrom
feat/unwire-deferred-domains

lklimek commented May 29, 2026

Uh oh!

coderabbitai Bot commented May 29, 2026 •

edited

Loading

Review skipped

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

lklimek commented May 29, 2026

Why the deferrals turned out wrong

Commits (8)

Key architectural pieces

What data.db retains after this PR

Test plan

Deferred items (deliberate)

Notes for review

Uh oh!

coderabbitai Bot commented May 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

coderabbitai Bot commented May 29, 2026 •

edited

Loading