ENG-695: make initiateCashout single-use per offerId

[islandbitcoin]

Problem

CashoutManager.executeCashout loaded the offer with a plain Redis GET and never removed, locked or marked it. Any initiateCashout with the same offerId inside the offer's one-hour TTL reached ValidOffer.execute() again. execute() drafts the ERPNext cashout before it pays IBEX, so even if IBEX refused to pay an already-paid invoice, every duplicate would still leave a second ERPNext draft.

Confirmed by test on unmodified main (374e5b9) and recorded on ENG-695. One offer executed twice in sequence, and one offer executed twice concurrently, each produced 2 ErpNext.draftCashout, 2 Ibex.payInvoice and 2 ErpNext.submitCashout calls. Both calls returned InitiatedCashout.

Change

• Typed error. OfferAlreadyExecuted is a new CacheServiceError beside OfferNotFound. error-map.ts maps it to code OFFER_ALREADY_EXECUTED with the message "This cashout offer was already submitted and can't be reused. Check your transactions before starting a new cashout.", forwarded to the client. The app shows the message as is, and the error mostly follows a retry after a lost response, when the first cashout may have paid, so the copy sends the user to their transactions before a new cashout.
• Atomic consume. The new Storage.consume(id) takes the offer record out of Redis with the existing consumeCacheKey: one DEL, true only for the caller whose DEL removed the key. It is the same gate the Plaid link-token store uses. executeCashout calls it after revalidation and strictly before execute():
  • It runs after the wallet ownership checks, so a caller cannot burn another account's offer, and after ValidOffer.from, so a transient validation failure (an ERPNext bank-account lookup, an expired invoice) does not burn the offer.
  • It runs before execute(), the only step that drafts the ERPNext cashout and pays IBEX.
  • The loser of a concurrent race gets OfferAlreadyExecuted and the ops event phase: "failed", step: "consumeOffer", error: "OfferAlreadyExecuted".
  • A Redis error on the DEL refuses the payout (fail closed) and fires the same event with the error name, for example UnknownCacheServiceError.
• Tombstone for sequential retries. Before the DEL, consume writes offer-executed:<id> with the offer's 3600 s TTL, best effort. Storage.get reads it only when the offer record is missing. A later retry then gets OfferAlreadyExecuted instead of OfferNotFound, at Storage.get, before validation, ops events or IBEX, and it emits no ops event. The DEL alone guarantees one execution; the tombstone only makes the error specific. The tombstone sits outside the offers: prefix, so no client-supplied offerId can name it, and get and consume answer any id that is not a UUID with OfferNotFound before building a key.
• Resolver. For this error, initiateCashout returns { errors: [{ code: "OFFER_ALREADY_EXECUTED", ... }], id: null } and does not record a Critical exception. Every other error still takes the existing catch-all, unchanged.
• No schema change (yarn check:sdl is clean). The offer key and value shape are unchanged, so offers created before the deploy, or by pods not yet rolled, are consumed the same way.

For ops: what changes

• A consumed offer is never executed again. This holds even when execute() fails, whether before money moves (a draftCashout error) or after (IBEX pending or failed, ERPNext submit failed after retry). The user has to request a new offer. Partial failures (paid, but the ERPNext submit failed) stay manual reconciliation, as today.
• Redis availability now gates cashouts: a DEL error refuses the payout. Each cashout adds two Redis round trips (tombstone SET, then DEL).
• A concurrent duplicate shows initiated followed by failed with step: "consumeOffer" in the ops feed. A sequential retry of a used offer is rejected silently; it moves no money.
• During the rolling deploy, old pods execute without consuming until the rollout completes. This window is accepted.

How it was tested

• Step 0: the new integration cases ran against unmodified main first (results above, recorded on ENG-695), and the fix came after.
• Integration, execute-offer.spec: real Redis and Mongo from local docker, with IBEX and ERPNext jest-mocked. cashout.skipPayment is true in the dev config the suite loads, so these cases turn it off to put the IBEX leg on the path.
  • Sequential: the second call returns OfferAlreadyExecuted. draftCashout and payInvoice are each called once.
  • Concurrent: both executions are held inside validation until both have read the offer, then released. Exactly one succeeds and the other gets OfferAlreadyExecuted. One draftCashout, one payInvoice. A 5 s safety release makes a regression fail on the arrived count instead of hanging until the jest timeout.
  • Validation failure does not burn: an ERPNext bank-lookup error fails the first attempt, the retry succeeds, and there is one payment.
  • Execution failure does burn: payInvoice returns an IBEX timeout, the retry gets OfferAlreadyExecuted, and there is one draftCashout and one payInvoice.
  • A forged offerId of executed:<id> returns OfferNotFound.
• Unit:
  • storage/redis-consume.spec (new): a won DEL returns true. A lost one returns OfferAlreadyExecuted. A DEL error is returned as is. The tombstone is written with the offer TTL before the DEL. A failed tombstone write still consumes and logs. get returns OfferAlreadyExecuted on a miss with a tombstone, OfferNotFound on a miss without one or when the tombstone read fails, and the stored offer on a hit without reading the tombstone. add keeps the offers:<id> key and the one-hour TTL. Over an in-memory Redis with ioredis-cache's JSON encoding, get("executed:<id>") after a consume returns OfferNotFound, and nothing but offer records stays under offers:. Ids that are not UUIDs never reach Redis.
  • cashout-manager-authz.spec: consume runs before execute. A lost consume or a Redis error stops execute. A validation failure, a wallet on another account, and another account's offer presented with the caller's own wallet all leave the offer unconsumed.
  • ops-events-cashout-manager.spec: a lost consume and a Redis refusal each emit one failed event with step: "consumeOffer" and the error name, and no succeeded event.
  • initiate-cash-out.spec (new): OfferAlreadyExecuted resolves to { errors: [OFFER_ALREADY_EXECUTED], id: null } with no Critical exception recorded. Success returns { errors: [], id }. Other errors keep the generic path.
  • error-map.spec: the code, the copy, forwardToClient, and OfferNotFound still mapping to NOT_FOUND.
• Mutation checks: with the gate and the resolver change reverted, 6 of the new unit cases fail. With the gate replaced by a bare consume right after Storage.get (ahead of the ownership checks and validation), 5 unit cases fail (the three that require an unconsumed offer and the two ops-event cases) and 2 integration cases fail (the concurrent case and the validation-failure case). Re-adding the offer when execute() fails fails the IBEX-timeout case (the retry pays again). The old offers:executed:<id> tombstone fails the forged-id integration case with TypeError: Cannot read properties of undefined (reading 'userAcct'), and each half of the storage fix (the key prefix and the UUID guard) fails its own unit cases when reverted alone.
• Checks run: make unit-in-ci (295 suites, 3825 passed, 3 skipped), yarn test:integration (5 suites, 37 passed), yarn tsc-check-noimplicitany, yarn tsc-check, yarn eslint-check, yarn build and yarn check:sdl (no schema diff), yarn check-yaml, yarn madge-check, prettier 3.6.2 on the changed files, and typos --config typos.toml. All pass.
• Not covered: IBEX's own handling of a second payment for an already-paid invoice. IBEX is mocked here, and the gate no longer depends on it.

Follow-ups

• ENG-716 (filed): return the original cashoutId on OfferAlreadyExecuted so a client that lost the first response can reconcile. Needs a schema change.
• The resolver catch-all still turns every other error, OfferNotFound included, into a bare InternalServerError instead of { errors, id }. It needs its own ticket. The new resolver spec pins today's behaviour, so that fix will update it.
• Mobile (ENG-688): branch on OFFER_ALREADY_EXECUTED, send the user to their transactions, then offer a new cashout.
• Exactly-once relies on Redis serializing DEL per key (single instance or sentinel today, src/services/redis/connection.ts). Re-verify if offers ever move to a cluster or a multi-primary setup.
• After merge: TEST deploy via scripts/deploy-test.sh flash. Prod apply is manual.

🤖 Generated with Claude Code

https://claude.ai/code/session_01XkM7Bm1cnzugSFi6pGnoWj

[linear]

ENG-695