Test parachain runtime upgrade locally

[Kailai-Wang]

Hi, I'd like to test and simulate the runtime upgrade locally by forking off a LIVE running parachain and execute the runtime upgrade programmatically with locally built new wasm / github release.

Here're my scripts:

• https://github.com/litentry/heima/blob/p-1728-runtime-upgrade-simulation-ci-fails/parachain/scripts/runtime-upgrade.sh
• https://github.com/litentry/heima/blob/p-1728-runtime-upgrade-simulation-ci-fails/parachain/ts-tests/integration-tests/runtime-upgrade.test.ts

I don't have sudo pallet so the authorization needs to be done via democracy.
I launch the parachain + relaychain network via chopsticks xcm and follow the system.authorize => system.apply way.

Locally the script runs well until trying to apply the upgrade (= authorization was done), then I always got error:

Apply upgrade transaction sent ✅
Transaction events: 2
Event: system.ExtrinsicSuccess {
  dispatchInfo: {
    weight: { refTime: '238,427,000', proofSize: '1,493' },
    class: 'Mandatory',
    paysFee: 'Yes'
  }
}
Event: system.ExtrinsicSuccess {
  dispatchInfo: {
    weight: { refTime: '108,157,000', proofSize: '0' },
    class: 'Mandatory',
    paysFee: 'No'
  }
}
Event: system.ExtrinsicFailed {
  dispatchError: { Module: { index: '0', error: '0x02000000' } },
  dispatchInfo: {
    weight: { refTime: '99,184,662,000', proofSize: '67,035' },
    class: 'Operational',
    paysFee: 'Yes'
  }
}

So it seems to error out with 0x02000000 from system pallet, which is FailedToExtractRuntimeVersion. It happens when sp_io::misc::runtime_version fails to extract the runtime version from the wasm.

I'm not sure why it occurs - the newly built wasm is compressed (~2MB), but sp_io::misc::runtime_version should be able to deal with compressed wasm too.

I also suspect that chopsticks/smoldot expects uncompressed wasm when calling the host fn, so I try to use uncompressed wasm when applying the upgrade, but ended up having a resourceExhausted error (so it's too large and can't fit into block weight I assume).

I'm kind of stuck on this - could you help with it? Or if you have links/docs about how to simulate runtime upgrade for a LIVE parachain, I'm happy to read them too!

[ermalkaleci]

You can use dev_setStorage to modify any state, use it to set authorization hash and apply upgrade

[Kailai-Wang]

You can use dev_setStorage to modify any state, use it to set authorization hash and apply upgrade

Thank you for the reply

I tried it too (by e.g. setting the GoAhead signal), however again the error is not related to getting the approval from relaychain or storage, but when applying the update with dispatchError: { Module: { index: '0', error: '0x02000000' } }

[ermalkaleci]

GoAhead signal is managed by chopsticks if it see pending runtime. You must set authorization hash and submit enact upgrade.

What changes are you doing to the new runtime? It will be easier to debug if you can share the RPC and wasm.

[xlc]

Do you have #[sp_version::runtime_version]?
https://github.com/paritytech/polkadot-sdk/blob/efba49201f079b3630999149d1250ef4eafea124/substrate/bin/node/runtime/src/lib.rs#L173

[Kailai-Wang]

Do you have #[sp_version::runtime_version]? https://github.com/paritytech/polkadot-sdk/blob/efba49201f079b3630999149d1250ef4eafea124/substrate/bin/node/runtime/src/lib.rs#L173

Yes I do.
Subwasm could extract version correctly too 🤔

[Kailai-Wang]

GoAhead signal is managed by chopsticks if it see pending runtime. You must set authorization hash and submit enact upgrade.

What changes are you doing to the new runtime? It will be easier to debug if you can share the RPC and wasm.

Sure, the runtime only had newer polkadot-sdk deps (stable2407 -> stable2412), as well as a storage migration.

As mentioned I did follow the authorize => enact way, it's just the enact extrinsic errored out.
You could check my script in the issue description, and this is the command that I used:

./parachain/scripts/runtime-upgrade.sh heima wss://rpc.heima-parachain.heima.network:443 v0.9.25-15

[Kailai-Wang]

I tried to bypass the version extraction by injecting the scheduler storage directly (similar to what snowbridge did) - but it still failed :(

[JoeCap08055]

If you don't need to test the governance logic for applying the upgrade, but only want to test the upgrade/migrations itself, you can bypass all the complexity by simply applying the new WASM as an override in the config. When you start building blocks, the runtime will detect & run any pending/new migrations.

[Kailai-Wang]

If you don't need to test the governance logic for applying the upgrade, but only want to test the upgrade/migrations itself, you can bypass all the complexity by simply applying the new WASM as an override in the config. When you start building blocks, the runtime will detect & run any pending/new migrations.

Thanks, this is what I'm currently doing. But AFAIU it just launches the parachain with the newly built wasm so what we test is the new wasm works with the current state.

Ideally I'm looking for testing the whole upgrade flow (not that the chain is stuck somewhere during the upgrade)

[Wsteth]

@Kailai-Wang
I think you can provide the wasm file and the git branch.
runtime upgrade operations should all be executed in the runtime. I suspect there might be issues with the code, such as problems caused by polkadot SDK upgrades.

[JoeCap08055]

If you don't need to test the governance logic for applying the upgrade, but only want to test the upgrade/migrations itself, you can bypass all the complexity by simply applying the new WASM as an override in the config. When you start building blocks, the runtime will detect & run any pending/new migrations.

Thanks, this is what I'm currently doing. But AFAIU it just launches the parachain with the newly built wasm so what we test is the new wasm works with the current state.

Ideally I'm looking for testing the whole upgrade flow (not that the chain is stuck somewhere during the upgrade)

Like I said, if you need to test the Substrate upgrade mechanics (governance proposal, WASM upload, voting, acceptance, etc), then this approach wouldn't work for you. However, the approach I described essentially acts as if you've done all that & resumes the chain at essentially the block after the upgrade was applied. So any state migrations your new WASM needs to apply will get run.

As I said previously, it all depends on your use case. But, generally speaking, this approach tests the path which has the most potential for the chain to get "stuck", which would be any custom logic or storage migrations in the new runtime with actual current chain state.