Snapshot-restored op-reth diverges on CIP-64 execution despite identical pre-state

[jcortejoso]

⚠️ Root-cause superseded — see Update 5. The divergence is intermittent / concurrency-dependent (not deterministic on one "specific" tx) and is a reth in-memory state read-consistency bug, not snapshot/import-specific. Updates 1 & 3 below are retracted; the reproduction details remain valid.

Snapshot-restored op-reth diverges on CIP-64 execution despite identical pre-state

Summary

When celo-kona-reth is started from a snapshot of a previously-running canonical archive node and brought up to chain head via op-node, transaction execution deterministically intermittently diverges from the canonical chain the moment it encounters a specific CIP-64 transaction (on some CIP-64 blocks — see Update 5) — even though the pre-state (the snapshot's state root) is byte-for-byte identical to the canonical state at the same block.

Two celo-kona-reth image versions both exhibit the bug, with different symptoms:

Image	Symptom
sha-04c2a3c (pre-#184)	EVM rejects the canonical tx as invalid: "fee currency not registered: 0x0E2A3e05bc9A16F5292A6170456A710cb89C6f72". Block is dropped; chain stalls.
sha-bc50e90 (includes #184)	No outright rejection. State root computed locally differs from the block header's state root → block added to fork chain. Chain forks silently. Different runs produce different fork-block hashes.

This makes snapshot-based bootstrap (the workflow PR #191 enables) unusable for production without further fix — the snapshot mechanically reads & extracts correctly, op-reth boots, but the resulting node cannot follow canonical chain.

Reproduction

Producer side (one-time, already done)

1. Running celo-kona-reth sha-04c2a3c on /var/lib/op-reth (ZFS dataset tank/may18-fresh), in archive mode, with --proofs-history enabled.
2. podman stop -t 60 op-reth (clean shutdown, exit 0 in 4.3 s).
3. zfs snapshot tank/may18-fresh@export-2026-05-24 — captured at L2 block 67,728,745.
4. podman start op-reth (resumes normally).
5. zfs clone -o readonly=on tank/may18-fresh@export-2026-05-24 tank/snapshot-src for tooling.
6. Ran celo-reth snapshot-manifest (from PR feat(celo-reth): add download and snapshot-manifest subcommands #191, image sha-bc50e90) against the clone → produced manifest.json + 818 component archives (354 GiB), uploaded to Hetzner Object Storage at https://fsn1.your-objectstorage.com/celo/snapshots/.

Consumer side (the bug)

1. On a clean dataset, run celo-reth download --datadir /celo/data --chain celo --non-interactive --archive (image sha-bc50e90).
   • Completes successfully: emits "Snapshot download complete. Run celo-reth node to start syncing."
   • Datadir layout: db/ (100 GiB), static_files/ (487 GiB), rocksdb/ (47 GiB).
2. chown -R 10001:10001 <datadir> (download command creates files as root; op-reth runs as celo UID 10001).
3. Start celo-kona-reth node against this datadir (image sha-04c2a3c initially — the production image).
4. Start op-node (image celo-blockchain-public/op-node:celo-v2.2.1).
5. Engine API delivers payloads from snapshot point (67,728,745) forward.
6. op-reth processes blocks correctly through block 67,817,510 (verified: block hash matches forno).
7. At block 67,817,511, op-reth rejects the canonical block with:

ERROR Receipt root task received incomplete receipts, execution likely aborted
WARN Invalid block error on new payload
  invalid_hash:     0x3d7245589c3c147166ecc0b77c68ce2db263cf1af7f9a508bd7b871ad15387bc
  invalid_number:   67817511
  validation_err:   EVM reported invalid transaction (0xc0ce0d52a7594d1d35ad902786245f7665e1b254190a5218c83e8a9a98625d7f):
                    fee currency not registered: 0x0E2A3e05bc9A16F5292A6170456A710cb89C6f72

After this rejection, every subsequent payload from op-node is rejected as "links to previously rejected block". Chain stalls indefinitely.

With sha-bc50e90 (includes PR #184)

Same setup, just swap the image. The "fee currency not registered" symptom goes away, but state computation is still wrong:

WARN Changeset cache MISS, falling back to DB-based computation
       block_hash:    0x88cacb50d7e5479317e95c582b6ccf35be32cf511e59be5d9818b1cc3c8b7a72
       block_number: 67817511
INFO  State root task finished
       state_root: 0x4df15fd214e8cf7edc0207af2d13d482a47e673befa4bd0b738322f802537a15
       elapsed:    409.559µs
WARN  State root task returned incorrect state root
       state_root:        0x4df15fd214e8cf7edc0207af2d13d482a47e673befa4bd0b738322f802537a15   ← computed by op-reth
       block_state_root:  0xa1477420629fdbec1cf2c82a810a8f49d51292709b50864268d7f56e26616ec1   ← what the canonical block header says
WARN  Failed to compute state root in parallel
INFO  Block added to fork chain
INFO  Canonical chain committed

The block is added with a hash different from canonical's. Different op-reth restarts produce different fork-block hashes (e.g. 0x88cacb…, 0x0fcc36…) — strongly suggesting non-deterministic state read from an uninitialized location.

What we verified

Pre-state (block 67,817,510) matches canonical byte-for-byte

local stateRoot:  0x24276ff7b5766a423d4eace0e78bf371b0d336785ddde4059a4a1f5f8abfec02
forno stateRoot:  0x24276ff7b5766a423d4eace0e78bf371b0d336785ddde4059a4a1f5f8abfec02
                  ✓ IDENTICAL

local block hash: 0x013397ea03f01e3fb6e0d774b69909b037ecd7e1b1bd0c3e11ccb9fe285d4d97
forno block hash: 0x013397ea03f01e3fb6e0d774b69909b037ecd7e1b1bd0c3e11ccb9fe285d4d97
                  ✓ IDENTICAL

FeeCurrencyDirectory storage matches canonical at block 67,817,510

$ eth_getStorageAt 0xefb84935239dAcdecF7c5bA76d8dE40b077B7b33 0x0 0x40ad026
  local: 0x00000000000000000000000158099b74f4acd642da77b4b7966b4138ec5ba458
  forno: 0x00000000000000000000000158099b74f4acd642da77b4b7966b4138ec5ba458
         ✓ IDENTICAL

Since state root is computed over a Merkle Patricia Trie of all account states (including storage roots), an identical state root mathematically guarantees that every account's storage is identical to canonical at that block. The on-chain state at 67,817,510 is provably canonical.

Yet execution at block 67,817,511 produces different state

The only way to reconcile this is: block execution reads or modifies state that is not in the Merkle Patricia Trie.

Hypothesis

Cip64Storage (the per-block transient state that PR #184 made per-EVM instead of factory-scoped) — or something analogous — is lazily populated during normal block-by-block sync and is not captured by ZFS/file-based snapshots. From-genesis nodes accumulate this transient state as they execute every block. Snapshot-restored nodes start with it missing/empty/uninitialized, and when they hit a CIP-64 transaction whose execution depends on that state, the result diverges.

Supporting evidence for this hypothesis:

1. The same sha-04c2a3c image had been running canonical for ~2 months prior on the same physical machine (no divergence from genesis sync). After a single stop/snapshot/restart cycle, it now fails.
2. sha-bc50e90 (which post-dates PR Per evm CIP-64 storage #184) doesn't fix the issue but does change the symptom — consistent with "different transient-state handling, same core missing-state bug".
3. The fork-block hash is non-deterministic across op-reth restarts on the same input — strongly suggests the divergent state is initialized from uninitialized memory or a non-deterministic source.
4. PR Per evm CIP-64 storage #184's commit message explicitly mentions that Cip64Storage lives in CeloEvm (not in the trie) and that the proofs-history ExEx re-executes blocks through the factory — exactly the kind of side-channel state that would not survive a snapshot.

What we tried (none worked)

Attempt	Result
ZFS rollback to @export-2026-05-24 and restart with sha-04c2a3c	"static file tip behind checkpoint" + key decode failure during auto-unwind. Different bug class (interaction with tank/op-reth-proofs dataset not being rolled back).
celo-reth download to a fresh datadir + sandbox boot (no op-node, no peers)	Boots cleanly, pipeline finishes at snapshot block. False-positive validation.
celo-reth download + production op-reth + op-node + chain catchup	Fork divergence as documented above.
Restart op-reth several times	Each restart produces a different fork-block hash at 67,817,511.
Swap image from sha-04c2a3c → sha-bc50e90	Symptom changes from "invalid tx" to "wrong state root", but still divergent.
Wipe and recreate tank/op-reth-proofs (the proofs-history rocksdb)	No effect on divergence behaviour.

Environment

• Host: Hetzner EX44 in FSN1 datacenter, Debian 12, ZFS root pool, sync=standard.
• Container runtime: podman 4.x, rootful, host networking.
• Storage: 2 × 3.5 TB NVMe in ZFS stripe.
• Production image: us-west1-docker.pkg.dev/devopsre/dev-images/celo-kona-reth:sha-04c2a3c.
• Test image: us-west1-docker.pkg.dev/devopsre/dev-images/celo-kona-reth:sha-bc50e90.
• op-node: us-west1-docker.pkg.dev/devopsre/celo-blockchain-public/op-node:celo-v2.2.1.
• reth-l1: ghcr.io/paradigmxyz/reth:latest.
• Lighthouse: docker.io/sigp/lighthouse:latest.
• Chain: Celo Mainnet (--chain celo, network 42220). All hardforks active including Jovian (Mar 31 2026).
• op-reth flags include --proofs-history --proofs-history.storage-path=/celo-proofs/data --proofs-history.window=1209600.

Forensic artifacts available

The reproduction case is preserved on the affected node (celo-mainnet-archive-hetzner-fsn-1) for as long as needed:

• tank/test-download — the celo-reth download-produced datadir, byte-for-byte equivalent to the published snapshot. Mounted at /var/lib/op-reth.
• tank/may18-fresh — the previously-canonical broken state (forked at block 67,751,935 on May 24 from a separate incident). Mounted at /var/lib/op-reth.broken-2026-05-25.
• tank/op-reth-proofs.broken-2026-05-25 — the proofs-history rocksdb from the broken state.
• ZFS snapshots @backup-anchor-2026-05-17, @backup-anchor-2026-05-15, plus daily autosnaps back to early May.
• Published snapshot artifacts at https://fsn1.your-objectstorage.com/celo/snapshots/ (manifest.json + 818 chunks, 354 GiB total).
• Full op-reth + op-node journald logs from the reproduction window (2026-05-25 ~11:00–15:00 UTC).

Happy to grant access or attach raw logs to anyone who wants to reproduce or debug.

Impact on snapshot publication tooling

PR #191 added download and snapshot-manifest subcommands precisely to enable snapshot-based bootstrap. The file shuttling works correctly — the bug is downstream. Until this is resolved:

• The published snapshot at https://fsn1.your-objectstorage.com/celo/snapshots/ should be marked experimental / not for production use.
• The validation procedure for snapshots needs to extend beyond "op-reth boots cleanly" to "process N canonical blocks via engine API and verify every block's state root matches" — boot-only validation produced a false positive here.
• A separate snapshot-export tool that also captures whatever non-trie state is being missed may be needed (or, alternately, op-reth should be able to lazily reconstruct it from MDBX/RocksDB on first start after restore).

[jcortejoso]

⚠️ RETRACTED — superseded by Update 5. The proofs-history ExEx is not the cause (already walked back in Update 2). Kept below for history.

Update: Bug isolated to --proofs-history ExEx — RETRACTED

After more testing, the trigger is the proofs-history ExEx running on snapshot-restored data without prior initialization. Confirmed by experiment:

Reproduction of the fix

1. From the diverged state (op-reth at block 67,817,511 with fork hash), stopped both services.
2. Recreated op-reth without --proofs-history flags (kept all other args identical).
3. Ran celo-reth stage unwind --datadir /celo/data --chain celo to-block 67817510 to remove the fork block.
4. Restarted op-reth + op-node.
5. op-node delivered canonical block 67,817,511 via engine API and op-reth accepted it cleanly.

local block 67817511 hash: 0x3d7245589c3c147166ecc0b77c68ce2db263cf1af7f9a508bd7b871ad15387bc
forno block 67817511 hash: 0x3d7245589c3c147166ecc0b77c68ce2db263cf1af7f9a508bd7b871ad15387bc
                          ✓ MATCH

Head then advanced 178 blocks in 90 seconds with no errors, batches decoding normally, inserting new unsafe blocks via engine API. Fully on canonical chain.

Updated hypothesis

The bug is specifically the interaction between --proofs-history ExEx and snapshot-restored state. We never ran celo-reth proofs init after restoring, so the proofs storage at /var/lib/op-reth-proofs/ was empty (512 bytes). The ExEx, when re-executing blocks during normal operation, appears to read or write through Cip64Storage (or a related path) in a way that corrupts execution — even after PR #184's per-EVM fix.

Three configurations on the same hardware:

Setup	Result
From-genesis sync + --proofs-history enabled (the prior 2-month working state)	✓ Canonical execution
Snapshot-restored + --proofs-history enabled (no proofs init run)	✗ Divergence at block 67,817,511 (wrong state root)
Snapshot-restored + --proofs-history disabled	✓ Canonical execution

The proofs init step was never run on either fsn-1 or its prior from-genesis state. The from-genesis state had empty proofs storage too but worked — suggesting "from-genesis op-reth happens to keep the ExEx in a safe state by virtue of replaying every block as it goes", whereas "snapshot-restored op-reth + ExEx with no init" doesn't.

Workarounds for operators

For anyone restoring from a snapshot (including the published one at https://fsn1.your-objectstorage.com/celo/snapshots/):

• Option A (tested): drop --proofs-history* flags. Sacrifices archive proof-window support but restores correct execution.
• Option B (untested but suggested): run celo-reth proofs init --datadir <DATA_DIR> --proofs-history.storage-path <PATH> after restore, before starting the node. Theoretically initializes the ExEx's state to match the restored chain state.

Snapshot publication implications

PR #191's celo-reth download mechanism is functionally correct. The published snapshot's restored state IS canonical (state root matches forno at every block we verified including the snapshot point). Restoration works as expected.

The remaining bug is runtime: how the proofs ExEx interacts with that restored state. Once #192 is fixed (or operators know to run proofs init and/or disable --proofs-history initially), snapshot-based bootstrap is fully usable.

[jcortejoso]

Update 2: --proofs-history is NOT the only trigger

Subsequent testing showed that disabling --proofs-history postpones the failure but does NOT eliminate it.

Timeline of the second stall

After the workaround in the previous comment (drop --proofs-history* flags + stage unwind to-block 67817510), op-reth resumed canonical execution and processed 10,019 more blocks cleanly (67,817,511 → 67,827,529, ~3 hours of chain progress).

Then at 2026-05-25 17:30:27 UTC, op-reth received block 67,827,530 from the engine API and immediately marked it invalid:

{"timestamp":"2026-05-25T17:30:27.587480Z","level":"INFO","fields":{"message":"Received new payload from consensus engine","number":67827530,"hash":"0x149e5a9d5c3711eaba1f4f4b1df8792a83bfb7a90f8e3abd075ebe22845d4e4d"}}
{"timestamp":"2026-05-25T17:30:27.587622Z","level":"WARN","fields":{"message":"Encountered invalid block","number":67827530,"hash":"0x149e5a9d5c3711eaba1f4f4b1df8792a83bfb7a90f8e3abd075ebe22845d4e4d"}}

Block hash 0x149e5a9d… matches forno canonical. 725 invalid-block events in the subsequent hour, all for the same block. Same loop as before.

Crucially, no validation_err was emitted this time — sha-04c2a3c with --proofs-history gave us a clear "fee currency not registered: 0x0E2A3e05…" message; sha-bc50e90 with --proofs-history disabled emits only "Encountered invalid block" with no reason despite -vvv verbosity. The detailed reason regression is itself worth fixing for debuggability.

Updated picture

Configuration	Image	Stalls at	Blocks processed cleanly
--proofs-history enabled	sha-04c2a3c	block 67,817,511 (CIP-64 fee currency lookup fails)	~89,000
--proofs-history enabled	sha-bc50e90	block 67,817,511 (state root mismatch, fork chain)	~89,000
--proofs-history disabled	sha-bc50e90	block 67,827,530 (silent invalid)	~10,019 more (~89,000 + 10,019 = ~99,019 total)

So:

• --proofs-history is one trigger that exposes the bug more aggressively.
• A second trigger exists in the no-proofs-history config — it just shows up less frequently (~10K block intervals on Celo mainnet traffic).
• Both triggers ultimately produce the same behavior (canonical block from op-node rejected as invalid, indefinite stall).

This pattern strongly suggests the underlying bug is broader than the proofs ExEx alone. The snapshot-restored op-reth state is missing or wrong in a way that affects EVM execution of certain (probably CIP-64-adjacent) transaction patterns, and the proofs ExEx happens to be one specific code path that triggers it earliest. Disabling proofs just removes one trigger; others remain.

Status

fsn-1 has been stopped to halt the fork progression. All forensic state preserved:

• tank/test-download (mounted at /var/lib/op-reth): the snapshot-restored datadir that hit both stalls
• tank/may18-fresh (broken state from the May 24 incident, separately): preserved at /var/lib/op-reth.broken-2026-05-25
• tank/op-reth-proofs.broken-2026-05-25: the proofs storage from earlier broken state

Waiting for maintainer guidance before further experiments. Happy to provide host access or any specific log/trace extraction.

The reproducer is deterministic (we hit it on 3/3 attempts with different configurations) so should be straightforward to reproduce on a debug build with instrumented EVM tracing if anyone has that setup.

[jcortejoso]

⚠️ RETRACTED as the cause of this issue — see Update 5. The import-celo-state genesis/Registry duplicate-write is a separate import-time panic (since fixed), not the cause of the runtime CIP-64 divergence reported here. The import/migrate fixes below remain valid; the root-cause claim does not. Kept for history.

(though please take this with a pinch of salt)

Update 3: Root cause is upstream of the snapshot — import-celo-state itself — RETRACTED (see Update 5)

After several days of investigation, the bug is not in the snapshot mechanism, the proofs-history ExEx, or the snapshot-restored runtime. The true root cause lives in the offline celo-reth import-celo-state path that builds the datadir in the first place. The symptoms reported in this issue's two earlier updates are downstream artifacts of that bad import.

Validated end-to-end: with the fix branch, a fresh import-celo-state + celo-migrate-v2 produces a v2 datadir that op-reth boots cleanly and op-node drives via engine API at live tip on Celo Mainnet (block 67,932,000+ at the time of writing).

What's actually happening

import-celo-state opens the storage env via upstream EnvironmentArgs::init, which ends with init_genesis_with_settings(...). That call writes the chainspec's genesis allocation:

• Celo Mainnet's genesis alloc includes the Registry contract (0x000000000000000000000000000000000000ce10) with one storage slot at key

  0xb53127684a568b3173ae13b9f8a6016e243e63b6e8ee1178d6a717850b5d6103
  

  (value 0xE23a4c6615669526Ab58E9c37088bee4eD2b2dEE — the original L1 Registry implementation pointer).

Right after, init_from_state_dump writes the L1 final state, which contains the same Registry slot but with a different value (0xd533ca25…7972 — Governance address; the slot was overwritten somewhere in L1 history).

Both writes land in HashedStorages at the same (keccak(0xce10), keccak(0xb53127…)) pair. DbDupCursorRW::upsert on HashedStorages does not dedupe in this code path — it appends a second dup-record. The subsequent state-root computation iterates the table by (outer_key, subkey), sees the same key twice in a row, and panics in alloy_trie::hash_builder::HashBuilder::add_leaf with key == self.key. Datadir unusable.

The exact match between the panic key and keccak(0xb53127…) is the proof:

$ cast keccak 0xb53127684a568b3173ae13b9f8a6016e243e63b6e8ee1178d6a717850b5d6103
0x52df0bdf5a5f92d8037cf11e50f13d8017aefc99d20a73c826416df79570d481

which is the exact value the HashBuilder panics on.

Two co-bugs uncovered along the way

While verifying the fix end-to-end, two more issues blocked the path and had to be fixed too:

1. scripts/append_l2_allocs.py doesn't prune zero-value storage slots. The canonical L2 allocs file at storage.googleapis.com/cel2-rollup-files/celo/l2-allocs.json contains four zero-value storage entries on OP Stack implementation addresses (0xc0d3...30007/0010/0012/0014). op-geth's StateDB.Commit deletes slots stored at value 0 (SSTORE-to-0 semantics); init_from_state_dump does not. Without the prune, the computed state root after import does not match the canonical migration root 0xed980641a4bd4d2e84c6c8db980b7f05e95733c92be2e0045db3735efeb1d807 even when (1) is fixed.

2. StorageSettings::base() returns v2(), but import-celo-state should produce v1. Without an explicit StorageSettings::v1() write, the import produces a datadir with v2 metadata and v1 data layout (everything in MDBX, nothing in static files / RocksDB). celo-migrate-v2's preflight then short-circuits on "Storage is already v2, nothing to do" without performing any migration, while op-reth's startup consistency check trips on TransactionSenders static-file segment being empty vs. SenderRecovery stage checkpoint at the migration block → upstream assert_ne!(unwind_target, PipelineTarget::Unwind(0), …) panic.

3. celo-migrate-v2 itself hits the same consistency-check assertion when opening a freshly imported datadir, even before its own seed_transaction_senders step can fix the missing segment. Same root, different binary.

How this relates to the symptoms in this issue

The snapshot under test was taken from a from-genesis fsn-1 that had been running for ~2 months. We did not extract that snapshot's HashedStorages to prove the duplicate-write pattern is in there — we only know the current import-celo-state code path produces it, and that the same code path was the only known way to bootstrap fsn-1. So this is the best-fit hypothesis for the observed symptoms (deterministic divergence on specific CIP-64 transactions, no detectable diff in latest-state queries) but not a directly proven inheritance: the original snapshot may have been built via a path we don't know about. The recommendation either way is the same — regenerate from a clean import rather than keep operating on the existing snapshot.

So:

• The snapshot mechanism itself is innocent. PR feat(celo-reth): add download and snapshot-manifest subcommands #191's celo-reth download / snapshot-manifest are mechanically correct.
• The fix has to land in import-celo-state (and the corresponding script/migrate pieces) so the source datadir is clean.
• The published snapshot at https://fsn1.your-objectstorage.com/celo/snapshots/ will be deprecated and replaced from a freshly-imported + freshly-migrated datadir once fsn-1 catches up to live tip.

Fix branch

jcortejoso/celo-migrate-v2, rebased onto current main, six commits:

Commit	Bug fixed
feat(celo-reth): add celo-migrate-v2 subcommand that preserves stages	New subcommand for v1 → v2 conversion on a Celo-imported datadir, which can't go through the upstream pipeline rebuild because of the dummy pre-migration block range.
fix(celo-reth): commit TransactionSenders seed per-segment + progress	ensure_at_block(31_056_500) in one shot grew the static-file writer's in-memory state past 18 GB and deadlocked the commit on Celo Mainnet. Per-500k-block chunked commits with operator-visible ETA.
fix(celo-reth): skip genesis alloc state writes in import-celo-state	The main #192 root-cause fix. New open_env_skip_init_and_consistency_check helper that opens the env without invoking init_genesis_with_settings. Emits only the non-state side-effects of genesis init (storage settings, canonical header, stage checkpoints, segment ranges).
fix(scripts): prune zero-value storage slots from L2 allocs in append_l2_allocs	Drops the 4 zero-value slots so init_from_state_dump sees the same set op-geth's Commit would have written.
fix(celo-reth): also bypass the consistency check in celo-migrate-v2	Reuses the same helper. Unblocks the freshly imported datadir running through celo-migrate-v2 without hitting the destructive Unwind(0) assertion.
fix(celo-reth): write v1 storage settings (not base = v2) in import-celo-state	One-liner: StorageSettings::v1() instead of StorageSettings::base() (which now defaults to v2).

All 131 unit tests in cargo test -p celo-reth --lib pass.

End-to-end validation

On fsn-1, against the canonical L1 final dump (https://fsn1.your-objectstorage.com/celo/l1-final-state.json.zst) merged with the canonical L2 allocs:

1. Import: import-celo-state with image sha-70dde8a produces a 74 GiB v1 datadir. Computed state root matches the canonical migration root 0xed980641… exactly. db settings get returns storage_v2: false.

2. Migrate: celo-migrate-v2 on a copy of the v1 datadir runs to completion in ~1h45m on the ZFS NVMe array (TransactionSenders seed in 2s with the chunked-commit fix, the multi-hour part is MDBX compaction). Resulting 91 GiB v2 datadir, storage_v2: true, AccountChangeSets / StorageChangeSets / Receipts / TransactionSenders static-file segments populated to migration block, no consistency warnings, sample state queries (Registry slot 0, GoldToken slots 0/1/2/totalSupply, FeeHandler slot 0, Governance slot 0, FeeCurrencyDirectory slot 0) all match forno.

3. Run: celo-reth node (no --proofs-history) + op-node (execution-layer syncmode, identical to fsn-1 host_vars) boot cleanly on the v2 datadir. Consistency check passes (no Unwind(0) panic). Engine API accepts payloads from op-node at the live Celo Mainnet tip (block 67,932,000+ at start). EL sync pipeline is running and backfilling the 31M → 67M gap via P2P from the production trusted-peer set.

Both artifacts are preserved on the Hetzner storage box under fsn-1/state-imports/:

• celo-mainnet-l1import-v1-block31056500-sha70dde8a-2026-05-26.tar.zst (42.3 GiB)
• celo-mainnet-l1import-v2migrated-block31056500-sha70dde8a-2026-05-26.tar.zst (66.4 GiB)

Still open

• Full EL sync to live tip is in progress on fsn-1. Once it catches up cleanly, this issue can be closed.
• Republish the snapshot: the snapshot at the Hetzner Object Storage prefix is from the pre-fix import path and should be replaced. Recommended workflow: snapshot the freshly synced fsn-1 datadir once it catches up, then re-run celo-reth snapshot-manifest to publish.
• Upstream-side cleanup (deferred, not blocking): the HashedStorages upsert that silently appends a second dup-row instead of replacing is a real bug in upstream reth (the asymmetric behavior with PlainStorageState.append_dup is what makes this so subtle). Worth filing upstream once we have a minimal reproducer outside of Celo's specific genesis-collision case.