Bug: Cancelling a pending transaction causes subsequent transactions to be stuck

[KartikSoneji]

Cancelling a pending transaction doesn't reset the nonce for subsequent transactions in the queue, causing them to be stuck.

For example, the transactions with nonce 2104 and 2105 had a gas limit that was over the block gas cap, hence couldn't be submitted.
When they were cancelled, the subsequent transactions (2106 to 2114) were submitted to the mempool but remained pending because of the missing nonces.

> select
    '0x' || encode(hash, 'hex') as hash,
    status,
    gas_limit,
    nonce
from transaction
where chain_id = 4114
order by nonce desc
limit 15;

                                hash                                |  status   | gas_limit | nonce 
--------------------------------------------------------------------+-----------+-----------+-------
 0x43e350a7e40019e9e729c5782a6d46882e96123f5ee5834b6b36e1320cbff10a | INMEMPOOL |   3182131 |  2114
 0x8cdb8979c2b2d9b2f24912cc8a226cec28e3a0b49a553f80495a5bf3314a150b | INMEMPOOL |   3719913 |  2113
 0x7fb39c2bef8cd0bb31520d3f402c855f8741efd10533da9b209f5e2a92768a1b | INMEMPOOL |   4505092 |  2112
 0x332eb304e8a6f28f3d7ab8d0da71fa63022e0fe92d416161d35e186be6866532 | INMEMPOOL |   3749538 |  2111
 0xfaacac3213f8ac020fe6ba3140e88b4bf9edd3af14078a4bb2c9209d4047c415 | INMEMPOOL |   7936777 |  2110
 0x641bd627c4b8f2c9405e3f21e823b7f4badc87ad3da503623d4e7ec40cc6ec41 | INMEMPOOL |   9153385 |  2109
 0x5b6777bc9162af3ec33ebed1d2902318d05ecf9c4b42d13f77124017eeeb2458 | INMEMPOOL |   6287720 |  2108
 0x07b07df9049bbeed25ed3923a4cf057cfa902b84aaf9fd50621f2b6fb9087bb0 | INMEMPOOL |   3591698 |  2107
 0x2ad49773ba64239f5d7b0de6331362dc5585ab10bedad9255e22fc24f333bff8 | INMEMPOOL |   6044995 |  2106
 0xb99a1f055954144c8df3f5188a256d8f38e49696bd0722c7e783583686513e8f | CANCELLED |  10904455 |  2105
 0x6b66ffb69eb914c69aefc0aeb61bd9f4d3ebe83af00f00d3f13701e33458b8d6 | CANCELLED |  10952338 |  2104
 0x1ab1bc7dba57e458813374fea9602be14746acde6154e45f5135149cb73ebe50 | CONFIRMED |    603822 |  2103
 0x58cdda557238e62f1dff3b026017b928a8a6a23ed9a24e52689758073ad493f4 | CONFIRMED |    518402 |  2102
 0x4af971650fa72e0ace2199f624febe59c8187946a9ed49cd056358423718a1f0 | CONFIRMED |    731431 |  2101
 0xbb01f77847a0a0715115de70404bfda2dcb8aa1adb1d9ab10768b55d2a59a57a | CONFIRMED |    603798 |  2100

Sending two noop transactions:

• nonce 2104: https://explorer.mainnet.citrea.xyz/tx/0xd115c17321b10c2f1e78b443a32807894265c03d8304203ae148337a63f0746f
• nonce 2105: https://explorer.mainnet.citrea.xyz/tx/0xa3deeeb17edc315c5fe68f87a488dd3751428ca0471402a0b2ff1de9a5b5d562

closed the gap and allowed the rest of the transactions to go through.

> select
    '0x' || encode(hash, 'hex') as hash,
    status,
    gas_limit,
    nonce
from transaction
where chain_id = 4114
order by nonce desc
limit 15;

                                hash                                |  status   | gas_limit | nonce 
--------------------------------------------------------------------+-----------+-----------+-------
 0x43e350a7e40019e9e729c5782a6d46882e96123f5ee5834b6b36e1320cbff10a | CONFIRMED |   3182131 |  2114
 0x8cdb8979c2b2d9b2f24912cc8a226cec28e3a0b49a553f80495a5bf3314a150b | CONFIRMED |   3719913 |  2113
 0x7fb39c2bef8cd0bb31520d3f402c855f8741efd10533da9b209f5e2a92768a1b | CONFIRMED |   4505092 |  2112
 0x332eb304e8a6f28f3d7ab8d0da71fa63022e0fe92d416161d35e186be6866532 | CONFIRMED |   3749538 |  2111
 0xfaacac3213f8ac020fe6ba3140e88b4bf9edd3af14078a4bb2c9209d4047c415 | CONFIRMED |   7936777 |  2110
 0x641bd627c4b8f2c9405e3f21e823b7f4badc87ad3da503623d4e7ec40cc6ec41 | CONFIRMED |   9153385 |  2109
 0x5b6777bc9162af3ec33ebed1d2902318d05ecf9c4b42d13f77124017eeeb2458 | CONFIRMED |   6287720 |  2108
 0x07b07df9049bbeed25ed3923a4cf057cfa902b84aaf9fd50621f2b6fb9087bb0 | CONFIRMED |   3591698 |  2107
 0x2ad49773ba64239f5d7b0de6331362dc5585ab10bedad9255e22fc24f333bff8 | CONFIRMED |   6044995 |  2106
 0xb99a1f055954144c8df3f5188a256d8f38e49696bd0722c7e783583686513e8f | CANCELLED |  10904455 |  2105
 0x6b66ffb69eb914c69aefc0aeb61bd9f4d3ebe83af00f00d3f13701e33458b8d6 | CANCELLED |  10952338 |  2104
 0x1ab1bc7dba57e458813374fea9602be14746acde6154e45f5135149cb73ebe50 | CONFIRMED |    603822 |  2103
 0x58cdda557238e62f1dff3b026017b928a8a6a23ed9a24e52689758073ad493f4 | CONFIRMED |    518402 |  2102
 0x4af971650fa72e0ace2199f624febe59c8187946a9ed49cd056358423718a1f0 | CONFIRMED |    731431 |  2101
 0xbb01f77847a0a0715115de70404bfda2dcb8aa1adb1d9ab10768b55d2a59a57a | CONFIRMED |    603798 |  2100

[joshstevens19]

Ah, interesting. I spotted why it did not cancel: it was due to an attempt to increase the gas limit - #100. This will mean you can now cancel, and it will be ok, but it would be good if you managed to get the node error message, as I can also make it fail on send to make it bail out way before this (so no manual work is needed)

[KartikSoneji]

Oh I see, it doesn't adjust the nonce of the other transactions in the queue?
Interesting.

The logs are just this for hundreds of attempts

29 May - 20:56:22.492564  INFO Preparing to send transaction 4d7af6c0-d157-479d-ab75-76b8331741d4 for relayer: citrea_mainnet-pk-0 with speed FAST
29 May - 20:56:22.492572  INFO Sending transaction Transaction { id: TransactionId(4d7af6c0-d157-479d-ab75-76b8331741d4), relayer_id: RelayerId(616a2f2e-d2a8-4a92-96a2-ba37fce63846), authorization_list: None, to: EvmAddress(0x111115763723b53395308ec4c9ab9d5fb0844cae), from: EvmAddress(0x111115763723b53395308ec4c9ab9d5fb0844cae), value: TransactionValue(0), data: TransactionData(0x81da69ef00000000000000000000000000000000000000000000000000000000000000200000000000000000000000000000000000000000000000000000000000000002000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000000000000000000002e00000000000000000000000003677113f10d82acfeedd22df5c7cfd58736111ee000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000600000000000000000000000000000000000000000000000000000000000000204ce8ae915000000000000000000000000000000000000000000000000000000000000002077ba0fdd435352f6805aae84f44943267f5de7b386b1b7120f174c673dbcd17200000000000000000000000000000000000000000000000000000000000000c00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000006a1a0b7b00000000000000000000000000000000000000000000000000000000000001200000000000000000000000000000000000000000000000000000000000000160000000000000000000000000000000000000000000000000000000000000002a3078303436313665326632616161663461323230316633303764313238303766666635633635346432340000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000027b7d0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001000000000000000000000000e045e6c36cf77faa2cfb54466d71a3aef7bbe83900000000000000000000000004616e2f2aaaf4a2201f307d12807fff5c654d240000000000000000000000000000000000000000000000000000000000026d210000000000000000000000000000000000000000000000000000000000000000000000000000000004616e2f2aaaf4a2201f307d12807fff5c654d24000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000600000000000000000000000000000000000000000000000000000000000000ac40bb27723000000000000000000000000000000000000000000000000000000000000002000000000000000000000000000000000000000000000000000000000000000a000000000000000000000000000000000000000000000000000000000000000e00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000006a1a0b7b000000000000000000000000000000000000000000000000000000000000016000000000000000000000000000000000000000000000000000000000000000056563647361000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000004122edb77421f019cdafe0066ba3779ccc50d87ba2665d48981f3d4edf3b5c16884a9a3b459be7c175260a81f2165098e7e2aaf564382665c448ea8eed87394bbc1c000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000002000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000000000000000000000c03128dc93ae00bdcdc338128f6e71dc4fee9c55a0e9797dbf4c95e519894edb3400000000000000000000000000000000000000000000000000000000000010e10000000000000000000000000000000000000000000000000000000000000060000000000000000000000000000000000000000000000000000000000000000067a6a8f691e1b8c929939e66315489ae5a3dd0800ec6bb6288e27ed353911fa200000000000000000000000000000000000000000000000000000000000010120000000000000000000000000000000000000000000000000000000000000060000000000000000000000000000000000000000000000000000000000000000700000000000000000000000000000000000000000000000000000000000000e000000000000000000000000000000000000000000000000000000000000001c000000000000000000000000000000000000000000000000000000000000002a00000000000000000000000000000000000000000000000000000000000000420000000000000000000000000000000000000000000000000000000000000050000000000000000000000000000000000000000000000000000000000000005e00000000000000000000000000000000000000000000000000000000000000700000000000000000000000000e045e6c36cf77faa2cfb54466d71a3aef7bbe839000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000600000000000000000000000000000000000000000000000000000000000000044a9059cbb000000000000000000000000a4bb4c1d188b94c0cf691dad213de24169fa0a50000000000000000000000000000000000000000000000000000000000000000900000000000000000000000000000000000000000000000000000000000000000000000000000000e045e6c36cf77faa2cfb54466d71a3aef7bbe839000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000600000000000000000000000000000000000000000000000000000000000000044095ea7b3000000000000000000000000565ed3d57fe40f78a46f348c220121ae093c3cf8000000000000000000000000000000000000000000000000000000000000232800000000000000000000000000000000000000000000000000000000000000000000000000000000565ed3d57fe40f78a46f348c220121ae093c3cf80000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000006000000000000000000000000000000000000000000000000000000000000000e404e45aaf000000000000000000000000e045e6c36cf77faa2cfb54466d71a3aef7bbe83900000000000000000000000031000000000000000000000000000000000000060000000000000000000000000000000000000000000000000000000000000bb800000000000000000000000004616e2f2aaaf4a2201f307d12807fff5c654d24000000000000000000000000000000000000000000000000000000000000232800000000000000000000000000000000000000000000000000000020bbd1de15000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000e045e6c36cf77faa2cfb54466d71a3aef7bbe839000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000600000000000000000000000000000000000000000000000000000000000000044a9059cbb000000000000000000000000737680c45f88a8b1b389e354dde11419f89badd600000000000000000000000000000000000000000000000000000000000249f0000000000000000000000000000000000000000000000000000000000000000000000000000000003100000000000000000000000000000000000006000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000600000000000000000000000000000000000000000000000000000000000000044095ea7b3000000000000000000000000fb7908150b738e7db9862007c66c9eb7850706f500000000000000000000000000000000000000000000000000000020bbd1de1500000000000000000000000000000000000000000000000000000000000000000000000000000000fb7908150b738e7db9862007c66c9eb7850706f5000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000600000000000000000000000000000000000000000000000000000000000000084617ba037000000000000000000000000310000000000000000000000000000000000000600000000000000000000000000000000000000000000000000000020bbd1de1500000000000000000000000004616e2f2aaaf4a2201f307d12807fff5c654d24000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000fb7908150b738e7db9862007c66c9eb7850706f50000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000006000000000000000000000000000000000000000000000000000000000000000445a3b74b9000000000000000000000000310000000000000000000000000000000000000600000000000000000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000), nonce: TransactionNonce(2104), chain_id: ChainId(4114), gas_limit: Some(GasLimit(10952338)), status: PENDING, blobs: None, known_transaction_hash: Some(TransactionHash(0x6b66ffb69eb914c69aefc0aeb61bd9f4d3ebe83af00f00d3f13701e33458b8d6)), queued_at: 2026-05-29T20:56:19.930398263Z, expires_at: 2026-05-30T08:56:19.699650720Z, sent_at: None, confirmed_at: None, sent_with_gas: None, sent_with_blob_gas: None, mined_at: None, mined_at_block_number: None, speed: FAST, sent_with_max_priority_fee_per_gas: None, sent_with_max_fee_per_gas: None, is_noop: false, external_id: Some("739228ea-5807-4d56-a338-af380264d131"), cancelled_by_transaction_id: None } for relayer: citrea_mainnet-pk-0
29 May - 20:56:22.492743  INFO Computing gas price for transaction with speed FAST for relayer: citrea_mainnet-pk-0
29 May - 20:56:22.492752  INFO Final gas price for relayer: citrea_mainnet-pk-0 - max_fee: 1200000, max_priority_fee: 0
29 May - 20:56:22.492759  INFO Creating EIP-1559 transaction for gas estimation for relayer: citrea_mainnet-pk-0
29 May - 20:56:22.492780  INFO Creating final EIP-1559 transaction for relayer: citrea_mainnet-pk-0
29 May - 20:56:22.492805  INFO Set gas limit 10952338 for transaction 4d7af6c0-d157-479d-ab75-76b8331741d4 on relayer: citrea_mainnet-pk-0
29 May - 20:56:22.492830  INFO Sending transaction Eip1559(TxEip1559 { chain_id: 4114, nonce: 2104, gas_limit: 10952338, max_fee_per_gas: 1200000, max_priority_fee_per_gas: 0, to: Call(0x111115763723b53395308ec4c9ab9d5fb0844cae), value: 0, access_list: AccessList([]), input: 0x81da69ef00000000000000000000000000000000000000000000000000000000000000200000000000000000000000000000000000000000000000000000000000000002000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000000000000000000002e00000000000000000000000003677113f10d82acfeedd22df5c7cfd58736111ee000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000600000000000000000000000000000000000000000000000000000000000000204ce8ae915000000000000000000000000000000000000000000000000000000000000002077ba0fdd435352f6805aae84f44943267f5de7b386b1b7120f174c673dbcd17200000000000000000000000000000000000000000000000000000000000000c00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000006a1a0b7b00000000000000000000000000000000000000000000000000000000000001200000000000000000000000000000000000000000000000000000000000000160000000000000000000000000000000000000000000000000000000000000002a3078303436313665326632616161663461323230316633303764313238303766666635633635346432340000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000027b7d0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001000000000000000000000000e045e6c36cf77faa2cfb54466d71a3aef7bbe83900000000000000000000000004616e2f2aaaf4a2201f307d12807fff5c654d240000000000000000000000000000000000000000000000000000000000026d210000000000000000000000000000000000000000000000000000000000000000000000000000000004616e2f2aaaf4a2201f307d12807fff5c654d24000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000600000000000000000000000000000000000000000000000000000000000000ac40bb27723000000000000000000000000000000000000000000000000000000000000002000000000000000000000000000000000000000000000000000000000000000a000000000000000000000000000000000000000000000000000000000000000e00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000006a1a0b7b000000000000000000000000000000000000000000000000000000000000016000000000000000000000000000000000000000000000000000000000000000056563647361000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000004122edb77421f019cdafe0066ba3779ccc50d87ba2665d48981f3d4edf3b5c16884a9a3b459be7c175260a81f2165098e7e2aaf564382665c448ea8eed87394bbc1c000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000002000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000000000000000000000c03128dc93ae00bdcdc338128f6e71dc4fee9c55a0e9797dbf4c95e519894edb3400000000000000000000000000000000000000000000000000000000000010e10000000000000000000000000000000000000000000000000000000000000060000000000000000000000000000000000000000000000000000000000000000067a6a8f691e1b8c929939e66315489ae5a3dd0800ec6bb6288e27ed353911fa200000000000000000000000000000000000000000000000000000000000010120000000000000000000000000000000000000000000000000000000000000060000000000000000000000000000000000000000000000000000000000000000700000000000000000000000000000000000000000000000000000000000000e000000000000000000000000000000000000000000000000000000000000001c000000000000000000000000000000000000000000000000000000000000002a00000000000000000000000000000000000000000000000000000000000000420000000000000000000000000000000000000000000000000000000000000050000000000000000000000000000000000000000000000000000000000000005e00000000000000000000000000000000000000000000000000000000000000700000000000000000000000000e045e6c36cf77faa2cfb54466d71a3aef7bbe839000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000600000000000000000000000000000000000000000000000000000000000000044a9059cbb000000000000000000000000a4bb4c1d188b94c0cf691dad213de24169fa0a50000000000000000000000000000000000000000000000000000000000000000900000000000000000000000000000000000000000000000000000000000000000000000000000000e045e6c36cf77faa2cfb54466d71a3aef7bbe839000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000600000000000000000000000000000000000000000000000000000000000000044095ea7b3000000000000000000000000565ed3d57fe40f78a46f348c220121ae093c3cf8000000000000000000000000000000000000000000000000000000000000232800000000000000000000000000000000000000000000000000000000000000000000000000000000565ed3d57fe40f78a46f348c220121ae093c3cf80000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000006000000000000000000000000000000000000000000000000000000000000000e404e45aaf000000000000000000000000e045e6c36cf77faa2cfb54466d71a3aef7bbe83900000000000000000000000031000000000000000000000000000000000000060000000000000000000000000000000000000000000000000000000000000bb800000000000000000000000004616e2f2aaaf4a2201f307d12807fff5c654d24000000000000000000000000000000000000000000000000000000000000232800000000000000000000000000000000000000000000000000000020bbd1de15000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000e045e6c36cf77faa2cfb54466d71a3aef7bbe839000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000600000000000000000000000000000000000000000000000000000000000000044a9059cbb000000000000000000000000737680c45f88a8b1b389e354dde11419f89badd600000000000000000000000000000000000000000000000000000000000249f0000000000000000000000000000000000000000000000000000000000000000000000000000000003100000000000000000000000000000000000006000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000600000000000000000000000000000000000000000000000000000000000000044095ea7b3000000000000000000000000fb7908150b738e7db9862007c66c9eb7850706f500000000000000000000000000000000000000000000000000000020bbd1de1500000000000000000000000000000000000000000000000000000000000000000000000000000000fb7908150b738e7db9862007c66c9eb7850706f5000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000600000000000000000000000000000000000000000000000000000000000000084617ba037000000000000000000000000310000000000000000000000000000000000000600000000000000000000000000000000000000000000000000000020bbd1de1500000000000000000000000004616e2f2aaaf4a2201f307d12807fff5c654d24000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000fb7908150b738e7db9862007c66c9eb7850706f50000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000006000000000000000000000000000000000000000000000000000000000000000445a3b74b9000000000000000000000000310000000000000000000000000000000000000600000000000000000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 }) to network for relayer: citrea_mainnet-pk-0
29 May - 20:56:22.849655 ERROR Relayer id 616a2f2e-d2a8-4a92-96a2-ba37fce63846 - PENDING QUEUE ERROR: Relayer id 616a2f2e-d2a8-4a92-96a2-ba37fce63846 / address 0x111115763723b53395308ec4c9ab9d5fb0844cae - Send transaction error: 616a2f2e-d2a8-4a92-96a2-ba37fce63846

The error messages are actually incorrect, they log the tx id twice:
29 May - 20:56:22.849655 ERROR Relayer id 616a2f2e-d2a8-4a92-96a2-ba37fce63846 - PENDING QUEUE ERROR: Relayer id 616a2f2e-d2a8-4a92-96a2-ba37fce63846 / address 0x111115763723b53395308ec4c9ab9d5fb0844cae - Send transaction error: 616a2f2e-d2a8-4a92-96a2-ba37fce63846
I've opened a PR for that here: #101

Side note: does rrelayer support json logging?
Text is fine for development but structured logs allow for more powerful and precise queries.

[KartikSoneji]

as I can also make it fail on send to make it bail out way before this (so no manual work is needed)

Ideally, when rrelayer adds a 20% buffer to the gas limit, it should also cap it to block gas cap.

rrelayer/crates/core/src/transaction/queue_system/transactions_queue.rs

Line 1050 in fd5b256

let estimated_gas = estimated_gas_result * 12 / 10;

would be

let estimated_gas = (estimated_gas_result * 12 / 10).max(block_gas_cap.into());

Not sure if there is an easy way to get the block gas cap per-chain.

Of course, if the gas estimation is over the cap then the tx should fail immediately.