[ANCHOR-1231]: Race condition between RPC events and transaction index

[amandagonsalves]

Description

Fix a family of RPC race conditions. All changes address the same root cause: one RPC call observes a
state that a subsequent call on a different node hasn't indexed yet.

1. StellarRpcPaymentObserver — replace bare getTransaction() with waitForTransactionAvailable() so the observer retries before giving up and losing the event
2. NotifyOnchainFundsSentHandler — same retry wrapper for the deposit path
3. NotifyOnchainFundsReceivedHandler — same retry wrapper for the withdrawal/SEP-31 path
4. StellarRpcPaymentObserver — change initial cursor offset from latestLedger - 1 to latestLedger - 5 to prevent startup loop when getLatestLedger and getEvents hit nodes at different ledgers
5. WalletClient / PaymentClient — surface sendTransaction errors immediately instead of silently returning a hash for a rejected transaction
6. PlatformAPITestBase — per-class payment map with retry loop to fix both sequence number collisions between parallel test classes and cross-class payment hash isolation

Context

ANCHOR-1231

Observer race (StellarRpcPaymentObserver / RPC handlers)

getEvents and getTransaction(hash) hit different nodes. The events index is ahead of the transactions index → getTransaction returns null → exception swallowed → cursor advances → event permanently lost. Confirmed in CI logs: getEvents returned the SAC transfer event at 13:41:36.443, getTransaction returned null 44 ms later at 13:41:36.487.

Fix: LedgerClientHelper.waitForTransactionAvailable polls getTransaction up to 10 × 1 s.

Observer startup cursor loop (StellarRpcPaymentObserver)

On a null cursor (startup or reset), the observer calls getLatestLedger() and then getEvents(startLedger = latestLedger - 1). These hit different nodes. If the second node is even one ledger behind, it returns startLedger out of range. With a null cursor, the observer loops on startup forever and processes no events. This caused all observer-dependent tests (SEP-6, SEP-24, SEP-31 full flows) to fail simultaneously.

Fix: use latestLedger - 5 as the initial offset. Tolerates up to 5 ledgers of inter-node lag.

Test parallel sequence number collision (PlatformAPITestBase)

testPaymentValues was an instance-level lateinit var. Four test subclasses running in parallel all called sendTestPayment() from the same signing key, all read the same sequence number before any transaction landed, and three got txBAD_SEQ.

Fix: shared companion object with double-checked locking — only one payment is submitted per JVM.

Testing

./gradlew :platform:test --tests \
  "org.stellar.anchor.platform.observer.stellar.StellarRpcPaymentObserverTest"

./gradlew :platform:test

./gradlew :core:test

./gradlew runEssentialTests

The new unit test (processTransferEvent retries getTransaction when RPC events index races ahead of transactions index) mocks getTransaction to return null on the first call and a valid transaction on the second, asserts exactly two calls were made, and verifies the event was still delivered to the listener.

Documentation

N/A

Known limitations

• The retry window in waitForTransactionAvailable is 10 × 1 s. An unusually slow node could still exceed it; in that case the event is logged as a warning and the cursor advances (same behavior as before the fix — no regression).

./gradlew runEssentialTests
...
For more on this, please refer to https://docs.gradle.org/8.2.1/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.

BUILD SUCCESSFUL in 12m 41s
71 actionable tasks: 10 executed, 61 up-to-date

[Copilot]

Pull request overview

This PR addresses a race where Soroban RPC “transfer” events can be observed before the corresponding transaction is available via the transaction index, by adding a retry/wait layer around transaction lookup and validating the behavior with a new unit test.

Changes:

• Update StellarRpcPaymentObserver.processTransferEvent() to wait/retry until getTransaction() returns a transaction (or times out), via LedgerClientHelper.waitForTransactionAvailable(...).
• Add a unit test covering the “events index ahead of transactions index” scenario by simulating getTransaction() returning null on the first call and a populated transaction on the second call.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
platform/src/main/java/org/stellar/anchor/platform/observer/stellar/StellarRpcPaymentObserver.java	Uses LedgerClientHelper.waitForTransactionAvailable() to tolerate RPC/tx-index timing races when processing transfer events.
platform/src/test/kotlin/org/stellar/anchor/platform/observer/stellar/StellarRpcPaymentObserverTest.kt	Adds a regression test ensuring transaction lookup is retried when the first lookup returns null.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 4 comments.