fix(consensus): resolve cross-RPC tx freeze — deterministic ordering + derive-at-apply fees + min-shard floor [epic #21]

[tcsenpai]

Summary

Fixes the consensus instability where transactions were accepted at RPC but never committed cross-node, freezing the chain under transaction inflow (manual node restarts / shard fiddling required). Verified end-to-end on a 2-node devnet (the testnet RC config).

Epic 21. This PR is the stability mandate; the hardening mandate (state-root vote) is a scoped follow-up (see below).

Root cause

confirmTransaction (via applyGasFeeSeparation, gasFeeSeparation fork) mutated the SDK-signed tx after signing — prepended fee-distribution gcr_edits and overwrote tx.content.transaction_fee (stamped rpc_address, reshaped amounts). Both are inside the coherence hash, but tx.hash was never recomputed (can't — would break the sender signature). On gossip, every peer ran validateTxCoherence, recomputed the hash from the mutated content, got a mismatch, and rejected the tx as "not coherent". Each node then held only its own tx, forged divergent blocks, the 2/3 vote never converged → no native tx ever committed.

Changes

• P-ORDER (c831ce0c8) — deterministic (sender,nonce,hash) mempool ordering, fork-gated on nonceEnforcement. Every honest node forges a byte-identical ordered_transactions → vote convergence; same-sender txs apply in nonce order.
• derive-at-apply (b07f5e3bf) — applyGasFeeSeparation no longer mutates the signed tx (keeps only the ingress balance pre-check). Fee-distribution edits are derived at apply from each tx's SDK-shipped transaction_fee (deterministic across nodes), prepended before prepareEntities/partition. verifyGcrEditsMatch: expectFeeEdits=false at ingress, true at apply. Gossiped tx is byte-identical to the signed one → coherence passes.
• P-MINSHARD (fcf6feb6e) — isBlockValid refuses to finalize below MIN_SHARD=2 (a 1-node shard otherwise self-certifies: BFT threshold floor(2/3)+1=1 → solo-fork source). Below the floor the chain stalls (safety over liveness). + getShard peer sort localeCompare→byte comparison (locale-sensitive ordering was a latent shard-divergence source).

Verification (2-node devnet, RC config)

• ✅ single tx commits (was frozen)
• ✅ 4 sequential cross-RPC txs land in nonce order
• ✅ both nodes byte-identical (equal heights, nonce, balances)
• ✅ fees collected to treasury
• ✅ same-tx replay to both RPCs → no double-spend
• ✅ chain advances pro=2/con=0; N=2 meets the MIN_SHARD floor
• ✅ typecheck: 0 new errors (16 pre-existing on base, unrelated)

NOT in this PR — scoped follow-up (hardening mandate)

P-TRIM + P-DETECT + P-CLOCK + P-ADMIT form one coupled state-root-vote redesign (forge→apply→seal-post-apply→vote-on-post-apply-hash, + full execution determinism). High blast radius; deferred to its own branch with its own plan + adversarial pass. With P-ORDER + 204, sequential txs already commit, so these are hardening/throughput, not stability.

Notes for reviewers

• rpc_address is intentionally null at this stage → the rpc-fee share folds into treasury deterministically. Per-tx rpc-operator routing via a BFT-committed fee envelope is part of the follow-up.
• MIN_SHARD is a module constant (=2) so all nodes agree; should become a genesis/fork parameter for mainnet.
• The demosdk 4.0.9 → 4.0.12 bump in package.json/bun.lock is a separate pending change (not in this PR's commits). The fix was built/tested against 4.0.12.

Refs: docs/discoveries/nonce-multirpc-fork-2026-06-25.md

[qodo-code-review]

Qodo reviews are paused for this user.

Troubleshooting steps vary by plan Learn more →

On a Teams plan?
Reviews resume once this user has a paid seat and their Git account is linked in Qodo.
Link Git account →

Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center?
These require an Enterprise plan - Contact us
Contact us →

[coderabbitai]

Warning

Review limit reached

@tcsenpai, we couldn't start this review because you've reached your PR review rate limit.

More reviews will be available in 30 seconds. Learn how PR review limits work.

Your organization has used up its prepaid credits, and credit purchases are no longer available. Enable the review add-on in the billing tab to keep reviews running — you're only billed for reviews past your plan's rate limits ($0.25/file).

⌛ How to resolve this issue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based credits.

🚦 How do rate limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please see our Fair Usage Limits Policy for further information.

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: c58a3a7f-d5a0-42fa-94c1-101dd31a07b8

📥 Commits

Reviewing files that changed from the base of the PR and between b0a1fa6 and b17ee66.

⛔ Files ignored due to path filters (1)

• bun.lock is excluded by !**/*.lock

📒 Files selected for processing (11)

• docs/discoveries/nonce-multirpc-fork-2026-06-25.md
• package.json
• src/libs/blockchain/gcr/handleGCR.ts
• src/libs/blockchain/mempool.ts
• src/libs/blockchain/routines/applyGasFeeSeparation.ts
• src/libs/consensus/v2/PoRBFT.ts
• src/libs/consensus/v2/routines/deterministicOrder.test.ts
• src/libs/consensus/v2/routines/deterministicOrder.ts
• src/libs/consensus/v2/routines/getShard.ts
• testing/devnet/genesis.devnet.json
• testing/devnet/scripts/multi_rpc_sequential_nonce.mjs

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/consensus-deterministic-ordering-ep21

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[tcsenpai]

@greptile review

[greptile-apps]

Greptile Summary

This PR updates consensus handling for cross-RPC native transactions. The main changes are:

• Deterministic mempool ordering by sender, nonce, and hash.
• Gas-fee edits derived at apply time instead of mutating signed transactions at ingress.
• Ingress and apply-time GCR edit verification split by whether fee edits are expected.
• Minimum shard participation floor before block finalization.
• Locale-independent peer sorting for shard selection.
• Two-node devnet fixture and multi-RPC sequential nonce verification script updates.
• @kynesyslabs/demosdk dependency bump to 4.0.12.

Confidence Score: 5/5

The consensus changes are focused and include targeted coverage for deterministic transaction ordering and two-node devnet behavior.

No blocking correctness issues were identified in the reviewed changes, and the implementation aligns with the stated goal of preventing cross-RPC transaction divergence.

T-Rex Logs

What T-Rex did

• The cross-RPC stability check was attempted, and base and head captures reveal docker is missing, blocking docker-compose from starting the 2-node devnet required for cross-RPC submission.
• The confirm-time mutation experiment was performed, showing that the base state had rpc_address evolve and gcr_edits grow, with coherenceAfterConfirm invalid, while the head state kept rpc_address null, gcr_edits unchanged, and coherenceAfterConfirm valid; apply-time fee derivation also appeared with eight edits and treasury adjustments.
• The deterministic order test ran, capturing base and head runs; the base showed divergent outputs and ordered_hashes_equal=false, while the head run produced identical ordered outputs with ordered_hashes_equal=true and the scenario path deterministically applied.
• The min shard test ran, showing a base state without MIN_SHARD returning valid for pro=1,totalVotes=1, and an after state with MIN_SHARD=2 producing invalid results for some combinations and valid for pro=2,totalVotes=2; the sorting path shifted to a byte-order comparator aligned with the environment’s locale ordering.

_{Ran code and verified through T-Rex}

_{Reviews (8): Last reviewed commit: "fix: per-component fee binding + correct..." | Re-trigger Greptile}

[greptile-apps]

Greptile Summary

This PR fixes cross-RPC transaction freezing by keeping signed transactions stable and making consensus ordering deterministic. The main changes are:

• Derive gas-fee distribution edits during apply instead of mutating signed transactions at confirmation.
• Verify ingress transactions without expecting derived fee edits, while apply-time checks still bind the derived edits.
• Sort merged mempools by sender, nonce, and hash behind the nonce enforcement fork.
• Reject block finalization below a two-validator shard floor.
• Replace locale-sensitive shard peer sorting with byte-stable identity ordering.
• Add deterministic ordering tests and devnet verification assets.

Confidence Score: 4/5

The consensus changes are focused and backed by deterministic-ordering tests and devnet verification assets, with one devnet helper script issue needing cleanup.

The core transaction-stability approach is coherent and the risky mutation path is addressed, but the included multi-RPC verification script currently does not match the documented two-node devnet setup.

testing/devnet/scripts/multi_rpc_sequential_nonce.mjs

T-Rex Logs

What T-Rex did

• Imported the real deterministicOrder.ts orderDeterministically function and verified that all three permutations produced the same sender, nonce, and hash order.
• Compared the derive-at-apply-fees results before and after the change and documented that the after state shows valid transaction coherence with the expected ledger changes.
• Updated the min shard floor logic to introduce MIN_SHARD and observed the head MIN_SHARD=2; pro=1,totalVotes=1 now false, while pro=2,totalVotes=2 remains true.
• Changed the shard-byte sort to use a byte comparator instead of localeCompare in the head branch.

_{Ran code and verified through T-Rex}

_{Reviews (2): Last reviewed commit: "chore(deps): bump @kynesyslabs/demosdk 4..." | Re-trigger Greptile}

[tcsenpai]

@greptile review

[tcsenpai]

@greptile review

[tcsenpai]

@greptile review

[tcsenpai]

All Greptile findings addressed and verified on the 2-node devnet:

• P1 fee bypass / distribution — applyGasFeeSeparation now binds the shipped transaction_fee to the node-computed breakdown per-component (commit b17ee66); a skewed or underpaid split is rejected. deriveFeeEditsForApply charges from the same shipped components as verifyGcrEditsMatch regenerates.
• P2 script needs 3 RPCs — fixed: NODE_URLS is .filter(Boolean), requires >=2 (commit 6bab5cf).
• P2 sample slicing — fixed: slice(0, clients.length) / slice(clients.length) (commit b17ee66).

The latest Greptile pass re-emitted the same three comments against the new commit, but the code at those exact lines already contains the fixes (verified). Treating them as stale re-flags; threads resolved.