Skip to content

PIPRES-796 Fix order status webhook leaving orders half-updated on database deadlock#1441

Open
TLabutis wants to merge 1 commit into
masterfrom
PIPRES-796-webhook-order-status-deadlock
Open

PIPRES-796 Fix order status webhook leaving orders half-updated on database deadlock#1441
TLabutis wants to merge 1 commit into
masterfrom
PIPRES-796-webhook-order-status-deadlock

Conversation

@TLabutis

Copy link
Copy Markdown
Collaborator

Problem

When a MySQL deadlock or lock wait timeout interrupts the order status transition mid-way (e.g. the paid webhook colliding with a scheduled ERP sync), the order is left permanently half-updated:

  • current_state is committed to paid but no history entry exists
  • the invoice is created with a number but no order lines linked ("invoice has no positions")
  • Mollie's webhook retry responds 200 but repairs nothing, because the current_state guard in OrderStatusService treats the half-applied state as already done
  • a later manual status change auto-creates a duplicate payment (false overpayment warning)

Confirmed on merchant production (8 affected orders, each matching a logged deadlock to the second) and reproduced deterministically on PS 8.2.3 by holding a row lock on the order's order_detail rows while the paid webhook arrives. Related reports: #1025 (timeout variant), #387 (duplicate payments). Bank transfer is the most exposed flow (order exists before payment), but the code path is shared by all payment methods, including refund, chargeback and Klarna shipment transitions.

Fix

  • Run the state transition (changeIdOrderState + history entry) inside a database transaction; roll back and retry up to 3 times on transient database errors (deadlock 1213, lock wait timeout 1205). A final failure rolls back cleanly so the next webhook retry finds an untouched order.
  • The current_state guard now also checks the order history: a matching current_state without a history entry is an interrupted transition, which gets logged, repaired (relink invoice order lines, link payments to the invoice, set order invoice number and date) and completed instead of skipped.
  • Status emails are sent after commit, so a mail failure can no longer break or block a transition; a failed history insert now throws instead of being silently ignored.

Validation

Tested against a reproduction harness on PS 8.2.3 with real Mollie test webhooks (bank transfer):

  1. Order broken by the old code is healed by a single webhook retry: paid history added, invoice relinked, invoice number and date set, no duplicate payment
  2. 30s lock collision: two transient failures logged, third attempt succeeds in the same request, fully consistent result
  3. 60s lock (outlasts all attempts): webhook responds 400 but the rollback is pristine, no orphan invoice, order still awaiting
  4. Subsequent webhook retry after the failure: clean full transition
  5. No collision: normal flow unchanged

Release

  • Changelog updated (6.4.5 section)

Note for reviewers

Hooks fired by changeIdOrderState now run inside the transaction: their database writes roll back together with the transition (consistent), but external side effects could fire once per retry attempt. Retries only happen on transient database errors, which previously produced a permanently broken order.

…tabase deadlock

Wrap the order status transition in a database transaction and retry
transient database errors (deadlock 1213, lock wait timeout 1205) so an
interrupted transition rolls back cleanly instead of leaving the order
with a paid current_state, no history entry and an invoice without lines.

Detect and complete previously interrupted transitions on webhook retry:
the current_state guard now checks the order history, relinks the orphan
invoice (order lines, payments, invoice number and date) and applies the
status instead of silently skipping.

Send status emails after commit and fail the transition when the history
entry cannot be added.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant