Intermittent boot hang in zfs-mount.service with txg_sync blocked in zio_wait() after successful pool import

[M-M-M-M]

System information

Type	Version/Name
Distribution Name	Rocky Linux release 8.10 (Green Obsidian)
Distribution Version	Rocky Linux release 8.10 (Green Obsidian)
Kernel Version	4.18.0-553.137.1.el8_10.x86_64
Architecture	x86_64
OpenZFS Version	zfs-2.2.10-1

Describe the problem you're observing

After replacing one disk of a Linux mdadm RAID1 array and completing a successful resync, my ZFS pool intermittently hangs during boot.

Storage layout:

2 SATA disks
-> mdadm RAID1 (/dev/md1)
-> LVM PV
-> LV vg0/zfs
-> ZFS pool (myzpool)

The pool imports successfully during boot:

Started Import ZFS pools by cache file.
Reached target ZFS pool import target.

but the boot hangs indefinitely at:

Starting Mount ZFS filesystems…
A start job is running for Mount ZFS filesystems

The machine becomes completely inaccessible (no SSH).

After approximately 120 seconds, the kernel reports a blocked task:

task: txg_sync state:D

with the following stack trace:

zio_wait()
spa_sync_frees()
spa_sync_iterate_to_convergence()
spa_sync()
txg_sync_thread()

Several hours later (or after another reboot), ZFS reports:

WARNING: Pool ‘myzpool’ has encountered an uncorrectable I/O failure and has been suspended.

The pool cannot be recovered with:

zpool clear myzpool

which returns:

cannot clear errors for myzpool: I/O error

However, after another reboot (sometimes taking several hours), the pool comes back completely healthy:

pool: myzpool
state: ONLINE
errors: No known data errors

and all datasets mount normally.

The issue is therefore intermittent.

The boot process is blocked waiting for zfs-mount.service, preventing SSH access.

From KVM screenshots :

Describe how to reproduce the problem

Unfortunately I do not have a reliable reproducer.

The issue first appeared one week after replacing one member of an mdadm RAID1 array and letting the array fully resynchronize.

The RAID is healthy:

Disks:
 /dev/sda serial=xxxxxxxxxx model=ST10000NM0156-2AA111 health=PASSED power_on_hours=50555 temp=54C reallocated=1013 pending=0 trend=stable
 /dev/sdb serial= xxxxxxxxxx model=HGST HUH721008ALE600 health=PASSED power_on_hours=45698 temp=56C reallocated=0 pending=0 trend=stable

RAID:
 /dev/md0 state=clean, resyncing (DELAYED) active=2 working=2 failed=0 raid_devices=2
 /dev/md1 state=active, checking active=2 working=2 failed=0 raid_devices=2

#cat /proc/mdstat
mdadm --detail /dev/md1
mdadm --detail /dev/md0
Personalities : [raid1]
md1 : active raid1 sdb2[1] sda2[2]
      7811795264 blocks super 1.2 [2/2] [UU]
      bitmap: 4/59 pages [16KB], 65536KB chunk

md0 : active raid1 sdb1[1] sda1[2]
      2094080 blocks super 1.2 [2/2] [UU]

unused devices: <none>
/dev/md1:
           Version : 1.2
     Creation Time : Fri Apr  1 08:45:54 2022
        Raid Level : raid1
        Array Size : 7811795264 (7.28 TiB 8.00 TB)
     Used Dev Size : 7811795264 (7.28 TiB 8.00 TB)
      Raid Devices : 2
     Total Devices : 2
       Persistence : Superblock is persistent

     Intent Bitmap : Internal

       Update Time : Wed Jul  1 08:31:57 2026
             State : clean
    Active Devices : 2
   Working Devices : 2
    Failed Devices : 0
     Spare Devices : 0

Consistency Policy : bitmap

              Name : rescue:1
              UUID : c0880b3b:0376a676:cca248aa:c7f19f26
            Events : 3682997

    Number   Major   Minor   RaidDevice State
       2       8        2        0      active sync   /dev/sda2
       1       8       18        1      active sync   /dev/sdb2
/dev/md0:
           Version : 1.2
     Creation Time : Fri Apr  1 08:45:54 2022
        Raid Level : raid1
        Array Size : 2094080 (2045.00 MiB 2144.34 MB)
     Used Dev Size : 2094080 (2045.00 MiB 2144.34 MB)
      Raid Devices : 2
     Total Devices : 2
       Persistence : Superblock is persistent

       Update Time : Wed Jul  1 07:05:14 2026
             State : clean
    Active Devices : 2
   Working Devices : 2
    Failed Devices : 0
     Spare Devices : 0

Consistency Policy : resync

              Name : rescue:0
              UUID : a1a4f0bd:a74cbcf2:daac2d1b:dea67028
            Events : 787

    Number   Major   Minor   RaidDevice State
       2       8        1        0      active sync   /dev/sda1
       1       8       17        1      active sync   /dev/sdb1

The issue has now occurred during multiple consecutive boots.

Sometimes the system boots normally.

Sometimes boot blocks indefinitely while mounting ZFS filesystems.

After another reboot the pool is again healthy.

Include any warning/errors/backtraces from the system logs

During some test ZFS crashes when I still had a SSH on the server, so I could run some commands:

#journalctl -k --since "2026-06-26 16:30" | egrep -i 'md1|md0|raid|sda|sdb|I/O|error|mismatch|resync|recovery|zfs'
Jun 26 16:30:17 localhost kernel: APIC: Switch to symmetric I/O mode setup
Jun 26 16:30:18 localhost kernel: sd 0:0:0:0: [sda] 19532873728 512-byte logical blocks: (10.0 TB/9.10 TiB)
Jun 26 16:30:18 localhost kernel: sd 1:0:0:0: [sdb] 15628053168 512-byte logical blocks: (8.00 TB/7.28 TiB)
Jun 26 16:30:18 localhost kernel: sd 0:0:0:0: [sda] 4096-byte physical blocks
Jun 26 16:30:18 localhost kernel: sd 1:0:0:0: [sdb] 4096-byte physical blocks
Jun 26 16:30:18 localhost kernel: sd 0:0:0:0: [sda] Write Protect is off
Jun 26 16:30:18 localhost kernel: sd 1:0:0:0: [sdb] Write Protect is off
Jun 26 16:30:18 localhost kernel: sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00
Jun 26 16:30:18 localhost kernel: sd 1:0:0:0: [sdb] Mode Sense: 00 3a 00 00
Jun 26 16:30:18 localhost kernel: sd 1:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
Jun 26 16:30:18 localhost kernel: sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
Jun 26 16:30:18 localhost kernel:  sdb: sdb1 sdb2 sdb3
Jun 26 16:30:18 localhost kernel: sd 1:0:0:0: [sdb] Attached SCSI disk
Jun 26 16:30:18 localhost kernel:  sda: sda1 sda2 sda3
Jun 26 16:30:18 localhost kernel: sd 0:0:0:0: [sda] Attached SCSI disk
Jun 26 16:30:18 localhost kernel: md/raid1:md0: active with 2 out of 2 mirrors
Jun 26 16:30:18 localhost kernel: md0: detected capacity change from 0 to 2144337920
Jun 26 16:30:18 localhost kernel: md/raid1:md1: active with 2 out of 2 mirrors
Jun 26 16:30:18 localhost kernel: md1: detected capacity change from 0 to 7999278350336
Jun 26 16:30:25 myserver.example.net kernel: zfs: module license 'CDDL' taints kernel.
Jun 26 16:30:25 myserver.example.net kernel: zfs: unknown parameter 'zfs_arc_meta_strategy' ignored
Jun 26 16:30:25 myserver.example.net kernel: EXT4-fs (md0): mounting ext3 file system using the ext4 subsystem
Jun 26 16:30:25 myserver.example.net kernel: ZFS: Loaded module v2.2.10-1, ZFS pool version 5000, ZFS filesystem version 5
Jun 26 16:30:25 myserver.example.net kernel: EXT4-fs (md0): mounted filesystem with ordered data mode. Opts: (null)
Jun 26 18:29:01 myserver.example.net kernel:  zio_wait+0x114/0x230 [zfs]
Jun 26 18:29:01 myserver.example.net kernel:  dsl_pool_sync_mos+0x37/0xa0 [zfs]
Jun 26 18:29:01 myserver.example.net kernel:  dsl_pool_sync+0x3d2/0x430 [zfs]
Jun 26 18:29:01 myserver.example.net kernel:  spa_sync_iterate_to_convergence+0xcf/0x1f0 [zfs]
Jun 26 18:29:01 myserver.example.net kernel:  spa_sync+0x318/0x610 [zfs]
Jun 26 18:29:01 myserver.example.net kernel:  txg_sync_thread+0x1f2/0x260 [zfs]
Jun 26 18:29:01 myserver.example.net kernel:  ? txg_dispatch_callbacks+0xf0/0xf0 [zfs]
Jun 26 18:35:10 myserver.example.net kernel:  zio_wait+0x114/0x230 [zfs]
Jun 26 18:35:10 myserver.example.net kernel:  dsl_pool_sync_mos+0x37/0xa0 [zfs]
Jun 26 18:35:10 myserver.example.net kernel:  dsl_pool_sync+0x3d2/0x430 [zfs]
Jun 26 18:35:10 myserver.example.net kernel:  spa_sync_iterate_to_convergence+0xcf/0x1f0 [zfs]
Jun 26 18:35:10 myserver.example.net kernel:  spa_sync+0x318/0x610 [zfs]
Jun 26 18:35:10 myserver.example.net kernel:  txg_sync_thread+0x1f2/0x260 [zfs]
Jun 26 18:35:10 myserver.example.net kernel:  ? txg_dispatch_callbacks+0xf0/0xf0 [zfs]
Jun 26 18:47:27 myserver.example.net kernel:  zio_wait+0x114/0x230 [zfs]
Jun 26 18:47:27 myserver.example.net kernel:  dsl_pool_sync_mos+0x37/0xa0 [zfs]
Jun 26 18:47:27 myserver.example.net kernel:  dsl_pool_sync+0x3d2/0x430 [zfs]
Jun 26 18:47:27 myserver.example.net kernel:  spa_sync_iterate_to_convergence+0xcf/0x1f0 [zfs]
Jun 26 18:47:27 myserver.example.net kernel:  spa_sync+0x318/0x610 [zfs]
Jun 26 18:47:27 myserver.example.net kernel:  txg_sync_thread+0x1f2/0x260 [zfs]
Jun 26 18:47:27 myserver.example.net kernel:  ? txg_dispatch_callbacks+0xf0/0xf0 [zfs]
Jun 26 18:55:39 myserver.example.net kernel:  zio_wait+0x114/0x230 [zfs]
Jun 26 18:55:39 myserver.example.net kernel:  dsl_pool_sync_mos+0x37/0xa0 [zfs]
Jun 26 18:55:39 myserver.example.net kernel:  dsl_pool_sync+0x3d2/0x430 [zfs]
Jun 26 18:55:39 myserver.example.net kernel:  spa_sync_iterate_to_convergence+0xcf/0x1f0 [zfs]
Jun 26 18:55:39 myserver.example.net kernel:  spa_sync+0x318/0x610 [zfs]
Jun 26 18:55:39 myserver.example.net kernel:  txg_sync_thread+0x1f2/0x260 [zfs]
Jun 26 18:55:39 myserver.example.net kernel:  ? txg_dispatch_callbacks+0xf0/0xf0 [zfs]
Jun 26 18:59:45 myserver.example.net kernel:  zio_wait+0x114/0x230 [zfs]
Jun 26 18:59:45 myserver.example.net kernel:  dsl_pool_sync+0xb7/0x430 [zfs]
Jun 26 18:59:45 myserver.example.net kernel:  ? __raw_spin_unlock+0x5/0x10 [zfs]
Jun 26 18:59:45 myserver.example.net kernel:  ? spa_errlog_sync+0x268/0x290 [zfs]
Jun 26 18:59:45 myserver.example.net kernel:  spa_sync_iterate_to_convergence+0xcf/0x1f0 [zfs]
Jun 26 18:59:45 myserver.example.net kernel:  spa_sync+0x318/0x610 [zfs]
Jun 26 18:59:45 myserver.example.net kernel:  txg_sync_thread+0x1f2/0x260 [zfs]
Jun 26 18:59:45 myserver.example.net kernel:  ? txg_dispatch_callbacks+0xf0/0xf0 [zfs]
Jun 26 19:07:56 myserver.example.net kernel:  zio_wait+0x114/0x230 [zfs]
Jun 26 19:07:56 myserver.example.net kernel:  dsl_pool_sync_mos+0x37/0xa0 [zfs]
Jun 26 19:07:56 myserver.example.net kernel:  dsl_pool_sync+0x3d2/0x430 [zfs]
Jun 26 19:07:56 myserver.example.net kernel:  spa_sync_iterate_to_convergence+0xcf/0x1f0 [zfs]
Jun 26 19:07:56 myserver.example.net kernel:  spa_sync+0x318/0x610 [zfs]
Jun 26 19:07:56 myserver.example.net kernel:  txg_sync_thread+0x1f2/0x260 [zfs]
Jun 26 19:07:56 myserver.example.net kernel:  ? txg_dispatch_callbacks+0xf0/0xf0 [zfs]
Jun 26 19:12:02 myserver.example.net kernel:  zio_wait+0x114/0x230 [zfs]
Jun 26 19:12:02 myserver.example.net kernel:  dsl_pool_sync_mos+0x37/0xa0 [zfs]
Jun 26 19:12:02 myserver.example.net kernel:  dsl_pool_sync+0x3d2/0x430 [zfs]
Jun 26 19:12:02 myserver.example.net kernel:  spa_sync_iterate_to_convergence+0xcf/0x1f0 [zfs]
Jun 26 19:12:02 myserver.example.net kernel:  spa_sync+0x318/0x610 [zfs]
Jun 26 19:12:02 myserver.example.net kernel:  txg_sync_thread+0x1f2/0x260 [zfs]
Jun 26 19:12:02 myserver.example.net kernel:  ? txg_dispatch_callbacks+0xf0/0xf0 [zfs]
Jun 26 19:14:05 myserver.example.net kernel:  zio_wait+0x114/0x230 [zfs]
Jun 26 19:14:05 myserver.example.net kernel:  dsl_pool_sync_mos+0x37/0xa0 [zfs]
Jun 26 19:14:05 myserver.example.net kernel:  dsl_pool_sync+0x3d2/0x430 [zfs]
Jun 26 19:14:05 myserver.example.net kernel:  spa_sync_iterate_to_convergence+0xcf/0x1f0 [zfs]
Jun 26 19:14:05 myserver.example.net kernel:  spa_sync+0x318/0x610 [zfs]
Jun 26 19:14:05 myserver.example.net kernel:  txg_sync_thread+0x1f2/0x260 [zfs]
Jun 26 19:14:05 myserver.example.net kernel:  ? txg_dispatch_callbacks+0xf0/0xf0 [zfs]
Jun 26 19:32:31 myserver.example.net kernel:  zio_wait+0x114/0x230 [zfs]
Jun 26 19:32:31 myserver.example.net kernel:  dsl_pool_sync_mos+0x37/0xa0 [zfs]
Jun 26 19:32:31 myserver.example.net kernel:  dsl_pool_sync+0x3d2/0x430 [zfs]
Jun 26 19:32:31 myserver.example.net kernel:  spa_sync_iterate_to_convergence+0xcf/0x1f0 [zfs]
Jun 26 19:32:31 myserver.example.net kernel:  spa_sync+0x318/0x610 [zfs]
Jun 26 19:32:31 myserver.example.net kernel:  txg_sync_thread+0x1f2/0x260 [zfs]
Jun 26 19:32:31 myserver.example.net kernel:  ? txg_dispatch_callbacks+0xf0/0xf0 [zfs]
Jun 26 19:44:48 myserver.example.net kernel:  zio_wait+0x114/0x230 [zfs]
Jun 26 19:44:48 myserver.example.net kernel:  dsl_pool_sync_mos+0x37/0xa0 [zfs]
Jun 26 19:44:48 myserver.example.net kernel:  dsl_pool_sync+0x3d2/0x430 [zfs]
Jun 26 19:44:48 myserver.example.net kernel:  spa_sync_iterate_to_convergence+0xcf/0x1f0 [zfs]
Jun 26 19:44:48 myserver.example.net kernel:  spa_sync+0x318/0x610 [zfs]
Jun 26 19:44:48 myserver.example.net kernel:  txg_sync_thread+0x1f2/0x260 [zfs]
Jun 26 19:44:48 myserver.example.net kernel:  ? txg_dispatch_callbacks+0xf0/0xf0 [zfs]
Jun 28 01:00:01 myserver.example.net kernel: md: data-check of RAID array md1
Jun 28 01:00:07 myserver.example.net kernel: md: delaying data-check of md0 until md1 has finished (they share one or more physical units)
Jun 29 17:13:42 myserver.example.net kernel: WARNING: Pool 'myzpool' has encountered an uncorrectable I/O failure and has been suspended.
Jun 30 02:39:12 myserver.example.net kernel: md: md1: data-check done.
Jun 30 02:39:12 myserver.example.net kernel: md: data-check of RAID array md0
Jun 30 02:39:25 myserver.example.net kernel: md: md0: data-check done.

Sometimes zpool status showed the error, that sometimes disappears just after a reboot :

#zpool get failmode myzpool
NAME     PROPERTY  VALUE     SOURCE
myzpool  failmode  wait      default

#zpool status -v myzpool
  pool: myzpool
 state: SUSPENDED
status: One or more devices are faulted in response to IO failures.
action: Make sure the affected devices are connected, then run 'zpool clear'.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-HC
  scan: scrub repaired 0B in 02:50:43 with 0 errors on Tue Jan  6 19:58:59 2026
config:

        NAME        STATE     READ WRITE CKSUM
        myzpool     ONLINE       0     0     0
          vg0/zfs   ONLINE       0     0     0

errors: List of errors unavailable: pool I/O is currently suspended

#zpool clear myzpool
cannot clear errors for myzpool: I/O error

After a long reboot :

#zpool status -v myzpool
  pool: myzpool
 state: ONLINE
  scan: scrub repaired 0B in 02:50:43 with 0 errors on Tue Jan  6 19:58:59 2026
config:

        NAME        STATE     READ WRITE CKSUM
        myzpool     ONLINE       0     0     0
          vg0/zfs   ONLINE       0     0     0

errors: No known data errors

#zfs list
NAME              USED  AVAIL  REFER  MOUNTPOINT
myzpool          10.9T  8.84T   157G  /myzpool
myzpool/BACKUP   9.45T  8.84T  9.45T  /myzpool/BACKUP
myzpool/storage  1.33T  8.84T  1.33T  /myzpool/storage
[root@hzr-2204][5526][~] #zfs mount
myzpool                         /myzpool
myzpool/BACKUP                  /myzpool/BACKUP
myzpool/storage                 /myzpool/storage

#zpool get health,failmode,dedupditto myzpool
NAME     PROPERTY    VALUE       SOURCE
myzpool  health      ONLINE      -
myzpool  failmode    wait        default
myzpool  dedupditto  0           default

Note on sda that below metrics are not changing since this disk has been setup :

• Reallocated_Sector_Ct = 1013
• Reported_Uncorrect = 1

[M-M-M-M]

Once I get back access to the KVM, I will try rescue mode to log in to the server without ZFS and capture some traces. And to try some manual commands.

[M-M-M-M]

On another server with similar configuration and some versions of OS, ZFS, etc. I just notice also some txg_sync errors that I can capture with some commands. See below for details. Could it be related to the June ZFS update?

# journalctl -k -b -1

[...]
juin 27 06:04:00 myserver3.example.net kernel: INFO: task txg_sync:1421 blocked for more than 120 seconds.
juin 27 06:04:00 myserver3.example.net kernel:       Tainted: P           OE  X  -------- -  - 4.18.0-553.137.1.el8_10.x86_64 #1
juin 27 06:04:00 myserver3.example.net kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
juin 27 06:04:00 myserver3.example.net kernel: task:txg_sync        state:D stack:0     pid:1421  ppid:2      flags:0x80004000
juin 27 06:04:00 myserver3.example.net kernel: Call Trace:
juin 27 06:04:00 myserver3.example.net kernel:  __schedule+0x2d1/0x870
juin 27 06:04:00 myserver3.example.net kernel:  schedule+0x55/0xf0
juin 27 06:04:00 myserver3.example.net kernel:  schedule_timeout+0x19f/0x320
juin 27 06:04:00 myserver3.example.net kernel:  ? __next_timer_interrupt+0xf0/0xf0
juin 27 06:04:00 myserver3.example.net kernel:  io_schedule_timeout+0x19/0x40
juin 27 06:04:00 myserver3.example.net kernel:  __cv_timedwait_common+0x12b/0x160 [spl]
juin 27 06:04:00 myserver3.example.net kernel:  ? finish_wait+0x80/0x80
juin 27 06:04:00 myserver3.example.net kernel:  __cv_timedwait_io+0x15/0x20 [spl]
juin 27 06:04:00 myserver3.example.net kernel:  zio_wait+0x114/0x230 [zfs]
juin 27 06:04:00 myserver3.example.net kernel:  dsl_pool_sync_mos+0x37/0xa0 [zfs]
juin 27 06:04:00 myserver3.example.net kernel:  dsl_pool_sync+0x3d2/0x430 [zfs]
juin 27 06:04:00 myserver3.example.net kernel:  spa_sync_iterate_to_convergence+0xcf/0x1f0 [zfs]
juin 27 06:04:00 myserver3.example.net kernel:  spa_sync+0x318/0x610 [zfs]
juin 27 06:04:00 myserver3.example.net kernel:  txg_sync_thread+0x1f2/0x260 [zfs]
juin 27 06:04:00 myserver3.example.net kernel:  ? txg_dispatch_callbacks+0xf0/0xf0 [zfs]
juin 27 06:04:00 myserver3.example.net kernel:  ? current_is_reclaim_thread+0x10/0x10 [spl]
juin 27 06:04:00 myserver3.example.net kernel:  thread_generic_wrapper+0x5a/0x70 [spl]
juin 27 06:04:00 myserver3.example.net kernel:  kthread+0x134/0x150
juin 27 06:04:00 myserver3.example.net kernel:  ? set_kthread_struct+0x50/0x50
juin 27 06:04:00 myserver3.example.net kernel:  ret_from_fork+0x1f/0x40
juin 27 06:06:03 myserver3.example.net kernel: INFO: task txg_sync:1421 blocked for more than 120 seconds.
juin 27 06:06:03 myserver3.example.net kernel:       Tainted: P           OE  X  -------- -  - 4.18.0-553.137.1.el8_10.x86_64 #1
juin 27 06:06:03 myserver3.example.net kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
juin 27 06:06:03 myserver3.example.net kernel: task:txg_sync        state:D stack:0     pid:1421  ppid:2      flags:0x80004000
juin 27 06:06:03 myserver3.example.net kernel: Call Trace:
juin 27 06:06:03 myserver3.example.net kernel:  __schedule+0x2d1/0x870
juin 27 06:06:03 myserver3.example.net kernel:  schedule+0x55/0xf0
juin 27 06:06:03 myserver3.example.net kernel:  schedule_timeout+0x19f/0x320
juin 27 06:06:03 myserver3.example.net kernel:  ? __next_timer_interrupt+0xf0/0xf0
juin 27 06:06:03 myserver3.example.net kernel:  io_schedule_timeout+0x19/0x40
juin 27 06:06:03 myserver3.example.net kernel:  __cv_timedwait_common+0x12b/0x160 [spl]
juin 27 06:06:03 myserver3.example.net kernel:  ? finish_wait+0x80/0x80
juin 27 06:06:03 myserver3.example.net kernel:  __cv_timedwait_io+0x15/0x20 [spl]
juin 27 06:06:03 myserver3.example.net kernel:  zio_wait+0x114/0x230 [zfs]
juin 27 06:06:03 myserver3.example.net kernel:  dsl_pool_sync_mos+0x37/0xa0 [zfs]
juin 27 06:06:03 myserver3.example.net kernel:  dsl_pool_sync+0x3d2/0x430 [zfs]
juin 27 06:06:03 myserver3.example.net kernel:  spa_sync_iterate_to_convergence+0xcf/0x1f0 [zfs]
juin 27 06:06:03 myserver3.example.net kernel:  spa_sync+0x318/0x610 [zfs]
juin 27 06:06:03 myserver3.example.net kernel:  txg_sync_thread+0x1f2/0x260 [zfs]
juin 27 06:06:03 myserver3.example.net kernel:  ? txg_dispatch_callbacks+0xf0/0xf0 [zfs]
juin 27 06:06:03 myserver3.example.net kernel:  ? current_is_reclaim_thread+0x10/0x10 [spl]
juin 27 06:06:03 myserver3.example.net kernel:  thread_generic_wrapper+0x5a/0x70 [spl]
juin 27 06:06:03 myserver3.example.net kernel:  kthread+0x134/0x150
juin 27 06:06:03 myserver3.example.net kernel:  ? set_kthread_struct+0x50/0x50
juin 27 06:06:03 myserver3.example.net kernel:  ret_from_fork+0x1f/0x40
juin 27 06:10:08 myserver3.example.net kernel: INFO: task txg_sync:1421 blocked for more than 120 seconds.
juin 27 06:10:08 myserver3.example.net kernel:       Tainted: P           OE  X  -------- -  - 4.18.0-553.137.1.el8_10.x86_64 #1
juin 27 06:10:08 myserver3.example.net kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
juin 27 06:10:08 myserver3.example.net kernel: task:txg_sync        state:D stack:0     pid:1421  ppid:2      flags:0x80004000
juin 27 06:10:08 myserver3.example.net kernel: Call Trace:
juin 27 06:10:08 myserver3.example.net kernel:  __schedule+0x2d1/0x870
juin 27 06:10:08 myserver3.example.net kernel:  schedule+0x55/0xf0
juin 27 06:10:08 myserver3.example.net kernel:  schedule_timeout+0x19f/0x320
juin 27 06:10:08 myserver3.example.net kernel:  ? __next_timer_interrupt+0xf0/0xf0
juin 27 06:10:08 myserver3.example.net kernel:  io_schedule_timeout+0x19/0x40
juin 27 06:10:08 myserver3.example.net kernel:  __cv_timedwait_common+0x12b/0x160 [spl]
juin 27 06:10:08 myserver3.example.net kernel:  ? finish_wait+0x80/0x80
juin 27 06:10:08 myserver3.example.net kernel:  __cv_timedwait_io+0x15/0x20 [spl]
juin 27 06:10:08 myserver3.example.net kernel:  zio_wait+0x114/0x230 [zfs]
juin 27 06:10:08 myserver3.example.net kernel:  dsl_pool_sync_mos+0x37/0xa0 [zfs]
juin 27 06:10:08 myserver3.example.net kernel:  dsl_pool_sync+0x3d2/0x430 [zfs]
juin 27 06:10:08 myserver3.example.net kernel:  spa_sync_iterate_to_convergence+0xcf/0x1f0 [zfs]
juin 27 06:10:08 myserver3.example.net kernel:  spa_sync+0x318/0x610 [zfs]
juin 27 06:10:08 myserver3.example.net kernel:  txg_sync_thread+0x1f2/0x260 [zfs]
juin 27 06:10:08 myserver3.example.net kernel:  ? txg_dispatch_callbacks+0xf0/0xf0 [zfs]
juin 27 06:10:08 myserver3.example.net kernel:  ? current_is_reclaim_thread+0x10/0x10 [spl]
juin 27 06:10:08 myserver3.example.net kernel:  thread_generic_wrapper+0x5a/0x70 [spl]
juin 27 06:10:08 myserver3.example.net kernel:  kthread+0x134/0x150
juin 27 06:10:08 myserver3.example.net kernel:  ? set_kthread_struct+0x50/0x50
juin 27 06:10:08 myserver3.example.net kernel:  ret_from_fork+0x1f/0x40
[...]

On this server :

#zfs get all myzpool
NAME     PROPERTY              VALUE                    SOURCE
myzpool  type                  filesystem               -
myzpool  creation              dim. mars 20 15:04 2022  -
myzpool  used                  19.9T                    -
myzpool  available             14.3T                    -
myzpool  referenced            1.45T                    -
myzpool  compressratio         1.04x                    -
myzpool  mounted               yes                      -
myzpool  quota                 none                     default
myzpool  reservation           none                     default
myzpool  recordsize            128K                     default
myzpool  mountpoint            /myzpool                 default
myzpool  sharenfs              off                      default
myzpool  checksum              on                       default
myzpool  compression           lz4                      local
myzpool  atime                 off                      local
myzpool  devices               on                       default
myzpool  exec                  on                       default
myzpool  setuid                on                       default
myzpool  readonly              off                      default
myzpool  zoned                 off                      default
myzpool  snapdir               hidden                   default
myzpool  aclmode               discard                  default
myzpool  aclinherit            restricted               default
myzpool  createtxg             1                        -
myzpool  canmount              on                       default
myzpool  xattr                 sa                       local
myzpool  copies                1                        default
myzpool  version               5                        -
myzpool  utf8only              off                      -
myzpool  normalization         none                     -
myzpool  casesensitivity       sensitive                -
myzpool  vscan                 off                      default
myzpool  nbmand                off                      default
myzpool  sharesmb              off                      default
myzpool  refquota              none                     default
myzpool  refreservation        none                     default
myzpool  guid                  14022812504280415850     -
myzpool  primarycache          metadata                 local
myzpool  secondarycache        all                      default
myzpool  usedbysnapshots       0B                       -
myzpool  usedbydataset         1.45T                    -
myzpool  usedbychildren        18.5T                    -
myzpool  usedbyrefreservation  0B                       -
myzpool  logbias               throughput               local
myzpool  objsetid              54                       -
myzpool  dedup                 on                       local
myzpool  mlslabel              none                     default
myzpool  sync                  standard                 local
myzpool  dnodesize             legacy                   default
myzpool  refcompressratio      1.01x                    -
myzpool  written               1.45T                    -
myzpool  logicalused           20.4T                    -
myzpool  logicalreferenced     1.39T                    -
myzpool  volmode               default                  default
myzpool  filesystem_limit      none                     default
myzpool  snapshot_limit        none                     default
myzpool  filesystem_count      none                     default
myzpool  snapshot_count        none                     default
myzpool  snapdev               hidden                   default
myzpool  acltype               off                      default
myzpool  context               none                     default
myzpool  fscontext             none                     default
myzpool  defcontext            none                     default
myzpool  rootcontext           none                     default
myzpool  relatime              on                       default
myzpool  redundant_metadata    all                      default
myzpool  overlay               on                       default
myzpool  encryption            off                      default
myzpool  keylocation           none                     default
myzpool  keyformat             none                     default
myzpool  pbkdf2iters           0                        default
myzpool  special_small_blocks  0                        default
myzpool  prefetch              all                      default

#zpool get all myzpool
NAME     PROPERTY                       VALUE                          SOURCE
myzpool  size                           6.29T                          -
myzpool  capacity                       93%                            -
myzpool  altroot                        -                              default
myzpool  health                         ONLINE                         -
myzpool  guid                           15758998622530687542           -
myzpool  version                        -                              default
myzpool  bootfs                         -                              default
myzpool  delegation                     on                             default
myzpool  autoreplace                    off                            default
myzpool  cachefile                      -                              default
myzpool  failmode                       wait                           default
myzpool  listsnapshots                  off                            default
myzpool  autoexpand                     off                            default
myzpool  dedupratio                     3.40x                          -
myzpool  free                           388G                           -
myzpool  allocated                      5.91T                          -
myzpool  readonly                       off                            -
myzpool  ashift                         0                              default
myzpool  comment                        -                              default
myzpool  expandsize                     -                              -
myzpool  freeing                        0                              -
myzpool  fragmentation                  87%                            -
myzpool  leaked                         0                              -
myzpool  multihost                      off                            default
myzpool  checkpoint                     -                              -
myzpool  load_guid                      17243203244383058270           -
myzpool  autotrim                       off                            default
myzpool  compatibility                  off                            default
myzpool  bcloneused                     0                              -
myzpool  bclonesaved                    0                              -
myzpool  bcloneratio                    1.00x                          -
myzpool  feature@async_destroy          enabled                        local
myzpool  feature@empty_bpobj            active                         local
myzpool  feature@lz4_compress           active                         local
myzpool  feature@multi_vdev_crash_dump  enabled                        local
myzpool  feature@spacemap_histogram     active                         local
myzpool  feature@enabled_txg            active                         local
myzpool  feature@hole_birth             active                         local
myzpool  feature@extensible_dataset     active                         local
myzpool  feature@embedded_data          active                         local
myzpool  feature@bookmarks              enabled                        local
myzpool  feature@filesystem_limits      enabled                        local
myzpool  feature@large_blocks           enabled                        local
myzpool  feature@large_dnode            enabled                        local
myzpool  feature@sha512                 enabled                        local
myzpool  feature@skein                  enabled                        local
myzpool  feature@edonr                  enabled                        local
myzpool  feature@userobj_accounting     active                         local
myzpool  feature@encryption             enabled                        local
myzpool  feature@project_quota          active                         local
myzpool  feature@device_removal         enabled                        local
myzpool  feature@obsolete_counts        enabled                        local
myzpool  feature@zpool_checkpoint       enabled                        local
myzpool  feature@spacemap_v2            active                         local
myzpool  feature@allocation_classes     enabled                        local
myzpool  feature@resilver_defer         enabled                        local
myzpool  feature@bookmark_v2            enabled                        local
myzpool  feature@redaction_bookmarks    enabled                        local
myzpool  feature@redacted_datasets      enabled                        local
myzpool  feature@bookmark_written       enabled                        local
myzpool  feature@log_spacemap           active                         local
myzpool  feature@livelist               enabled                        local
myzpool  feature@device_rebuild         enabled                        local
myzpool  feature@zstd_compress          enabled                        local
myzpool  feature@draid                  enabled                        local
myzpool  feature@zilsaxattr             active                         local
myzpool  feature@head_errlog            active                         local
myzpool  feature@blake3                 enabled                        local
myzpool  feature@block_cloning          enabled                        local
myzpool  feature@vdev_zaps_v2           active                         local

#zpool events -v

[[See output on zpool_event_long_output.txt file]]

#dmesg -T|more

[[See output on dmseg_long_output.txt]]

#zdb -C myzpool


MOS Configuration:
        version: 5000
        name: 'myzpool'
        state: 0
        txg: 24202216
        pool_guid: 15758998622530687542
        errata: 0
        min_alloc: 4096
        max_alloc: 4096
        hostname: 'myserver3.example.net'
        com.delphix:has_per_vdev_zaps
        vdev_children: 1
        vdev_tree:
            type: 'root'
            id: 0
            guid: 15758998622530687542
            create_txg: 4
            com.klarasystems:vdev_zap_root: 136
            children[0]:
                type: 'disk'
                id: 0
                guid: 5127329281454822037
                path: '/dev/vg0/zfs'
                devid: 'dm-uuid-LVM-U7m33w2C1LrM9PXWsXzYbCtF8wfEVJGsWtwaUrRCu4HqhsjxrW122fAObN5Kmgw5'
                whole_disk: 0
                metaslab_array: 256
                metaslab_shift: 33
                ashift: 12
                asize: 6919187595264
                min_alloc: 4096
                is_log: 0
                DTL: 7526
                create_txg: 4
                com.delphix:vdev_zap_leaf: 129
                com.delphix:vdev_zap_top: 130
        features_for_read:
            com.delphix:hole_birth
            com.delphix:embedded_data
            com.klarasystems:vdev_zaps_v2

dmseg_long_output.txt
zpool_event_long_output.txt

[M-M-M-M]

OpenZFS bug report update on 2026-07-02 -- Boot without ZFS and debug procedure

I finnaly got access back to the KVM.

Booting without importing/mounting ZFS

A dedicated BLS entry was added:

title NO ZFS - Rocky Linux (4.18.0-553.137.1.el8_10.x86_64) 8.10 (Green Obsidian)
version 4.18.0-553.137.1.el8_10.x86_64
linux /vmlinuz-4.18.0-553.137.1.el8_10.x86_64
initrd /initramfs-4.18.0-553.137.1.el8_10.x86_64.img $tuned_initrd
options $kernelopts $tuned_params systemd.unit=multi-user.target systemd.mask=zfs.target systemd.mask=zfs-mount.service

This entry boots into multi-user mode with SSH available while
preventing automatic ZFS import and dataset mounting.

Reproduction chronology

---

Time (approx.) Action Result

---

07:52 Boot using the NO ZFS boot System boots normally, SSH
entry available

07:55 zpool import -N myzpool Pool imports successfully,
datasets remain unmounted

09:12 Reproduced after clean boot zpool import -N myzpool
successful, scrub still
paused

09:12 zfs mount myzpool Success

09:12 zfs mount myzpool/BACKUP Success

09:12 time zfs mount myzpool/storage Command blocks indefinitely

+2 min Kernel reports hung tasks txg_sync, zfs,
z_zrele blocked in D
state

Immediately after echo w > /proc/sysrq-trigger Kernel stack traces
collected

Immediately after dmesg -T and ZFS dbgmsg saved Logs preserved for analysis

Recovery sequence

zpool import
zpool import -N myzpool
zpool status -v
zfs get mountpoint,canmount,mounted myzpool myzpool/BACKUP myzpool/storage

time zfs mount myzpool
time zfs mount myzpool/BACKUP
time zfs mount myzpool/storage

Observed behavior:

• Pool import succeeds immediately.
• Root dataset mounts normally.
• myzpool/BACKUP mounts normally.
• Only myzpool/storage hangs.

Debug commands

Before reproducing:

zpool status -v > /root/zpool-status-before.txt
zfs mount > /root/zfs-mount-before.txt
zpool events > /root/zpool-events-before.txt
zpool history myzpool > /root/zpool-history.txt
zdb -C myzpool > /root/zdb-C.txt

After the hang:

echo w > /proc/sysrq-trigger
dmesg -T > /root/dmesg-after-mount.txt
cat /proc/spl/kstat/zfs/dbgmsg > /root/zfs-dbgmsg.txt

Attached files

• zpool-status-before.txt
• zfs-mount-before.txt
• zpool-events-before.txt
• zpool-history.txt
• zdb-C.txt
• dmesg-after-mount.txt
• journalctl.txt
• zfs-dbgmsg.txt
• dmesg_after_mount_myzpool_storage.txt

dmesg_after_mount_myzpool_storage.txt
dmesg-after-mount.txt
journalctl.txt
zdb-C.txt
zfs-dbgmsg.txt
zfs-mount-before.txt
zpool-events-before.txt
zpool-history.txt
zpool-status-before.txt

These files contain the complete outputs.

Expected vs observed

Expected: all datasets mount successfully after a successful
zpool import -N.

Observed: myzpool and myzpool/BACKUP mount successfully, but
mounting myzpool/storage hangs indefinitely, leaving multiple kernel
threads (txg_sync, zfs, z_zrele) blocked in uninterruptible sleep
while replaying or cleaning filesystem state.

Preliminary analysis

Based on the observations collected so far, the issue does not
appear to be a simple pool import failure or an underlying block device
problem.

Evidence supporting this conclusion:

• zpool import -N myzpool succeeds immediately.
• zpool status reports the pool as ONLINE with no read, write
  or checksum errors.
• The underlying RAID1 (md1) is healthy (mismatch_cnt = 0).
• SMART data do not indicate any current media errors or pending
  sectors.
• myzpool and myzpool/BACKUP mount successfully.
• The problem is reproducible only when mounting myzpool/storage.
• The system remains responsive (SSH available) while only the mount
  operation hangs.

The collected kernel traces consistently show threads blocked in:

• zil_replay()
• zfs_replay_remove()
• zfs_remove()
• dmu_free_long_range()
• txg_wait_open()
• spa_sync_frees()
• txg_sync_thread()

This suggests the hang occurs during ZIL replay and/or deferred free
processing while mounting the dataset rather than during normal pool
import or device I/O.

Although an application crash or an unclean shutdown may have left
pending ZIL records, the filesystem should normally recover
automatically. Instead, the replay appears to enter a state where
multiple kernel threads wait indefinitely without making progress.

From the current evidence, this looks more like an internal OpenZFS
deadlock or replay bug than filesystem corruption or hardware failure.

[lorenzop]

I ran into this same bug nearly 2 weeks ago and I am still fighting with it. my server had more than 200 days uptime before I updated and rebooted the other day. at first all seemed ok, but it freed around 800 gigs of space which it didn't show before the reboot. I can only take a guess, I think maybe some destroyed snapshots failed to free the used disk space.

after a while, the zpool would go suspended, and the kernel would hang with 2 cores going 100% cpu wait. as days go on it became more common. I could import the zpool as read only and seemed to be stable, but the moment I import with write access, the zpool hangs.

I've ruled out hardware problems, no smart errors, it's not a problem with the backplane or HBA, power supplies, cables... I've tried the latest bleeding edge kernel and zfs, same problem. I was having all these problems with an 8 disk drive raid z3. I now have the HBA removed and the drives are disconnected, and I'm running a smaller zpool on one ssd. that is now causing the kernel hang consistently too. the disk drive zpool hangs before it can complete importing. the ssd zpool imports ok and lets me boot vm's from it, but then hangs after only minutes, or sooner.

at one point I was able to narrow down the problem to one specific dataset, but it seems the problem got progressively worse and I can no longer import the pool at all with write access.

let me know if there are any logs I should provide, or any specific commands I should try. my server is powered off and I'm looking at doing some hardware upgrades in the meantime.

I have x86_64-v2 hardware. have tried both an older HBA and onboard sata.

today I've tried zfs 2.3.6 2.3.2 and 2.2.9
and kernel versions 5.14.0-687.20.1 and 5.14.0-687.5.3

[M-M-M-M]

Additional investigation after the initial bug report

Summary

To rule out filesystem corruption, I performed a complete investigation while avoiding automatic dataset mounting.

The key findings are:

• the pool imports successfully;
• a complete scrub finishes successfully;
• 0 bytes repaired;
• 0 checksum/read/write errors;
• myzpool mounts correctly;
• myzpool/BACKUP mounts correctly;
• only myzpool/storage hangs indefinitely during mount, even after a successful scrub.

This strongly suggests that the issue is not caused by pool corruption or storage hardware, but by something specific to mounting the myzpool/storage dataset.

---

Boot procedure

To avoid the machine hanging during boot, I added a temporary GRUB entry:

systemd.unit=multi-user.target \
systemd.mask=zfs.target \
systemd.mask=zfs-mount.service

This allows Linux to boot normally without importing or mounting ZFS automatically.

I also configured this GRUB entry as the default boot option while investigating.

---

Investigation chronology

1. Boot

Boot using the NO ZFS GRUB entry.

---

2. Import pool without mounting datasets

zpool import -N myzpool

The import succeeds immediately.

---

3. Pool status before scrub

zpool status -v \
    > /root/zpool-status-before-scrub.txt

(Attached.)

---

4. zdb tests

I attempted:

time zdb myzpool \
    > /root/zdb-before-scrub.txt

This continuously enumerates every object in the dataset and produces hundreds of MB of output, so I interrupted it with Ctrl+C.

I also tried:

time zdb -b myzpool \
    > /root/zdb-b-before-scrub.txt

No useful additional information was obtained.

---

Full scrub without mounting datasets

I launched:

zpool scrub myzpool

The scrub was executed while no dataset was mounted.

Early progress:

pool: myzpool
state: ONLINE

scan: scrub in progress
437M / 1.98T scanned
73.5M issued
0B repaired
No known data errors

Near completion:

pool: myzpool
state: ONLINE

scan: scrub in progress
1.98T scanned
91.42% complete
0B repaired
No known data errors

Final result:

pool: myzpool
state: ONLINE

scan: scrub repaired 0B in 1 days 09:01:02
0 errors

Final status:

pool: myzpool
 state: ONLINE
 scan: scrub repaired 0B in 1 days 09:01:02 with 0 errors

config:

        NAME        STATE     READ WRITE CKSUM
        myzpool     ONLINE       0     0     0
          vg0/zfs   ONLINE       0     0     0

errors: No known data errors

This confirms that:

• every allocated block was successfully read;
• no checksum errors were detected;
• no repairs were required.

---

Dataset properties

I also collected:

zfs get all myzpool/storage

The complete output is included above.

Nothing appears unusual:

• canmount=on
• mounted=no
• mountpoint=/myzpool/storage
• encryption=off
• readonly=off
• sync=standard
• xattr=sa
• compression=lz4
• dedup=on

---

Dataset metadata

zdb -d myzpool/storage

returns:

Dataset myzpool/storage [ZPL], ID 515,
cr_txg 53,
1.33T,
3256714 objects

So the dataset metadata itself can be opened successfully.

---

Mount tests

Before testing the problematic dataset:

zfs mount

time zfs mount myzpool

time zfs mount myzpool/BACKUP

zfs mount

Result:

myzpool
myzpool/BACKUP

Both datasets mount immediately.

Only myzpool/storage is problematic.

---

Log reset

Before reproducing the issue:

dmesg -C

echo > /proc/spl/kstat/zfs/dbgmsg

---

Reproducing the problem

time zfs mount myzpool/storage

The command never returns.

After waiting approximately five minutes, kernel warnings began appearing in journalctl.

The mount never completed.

---

Captured diagnostics

While the mount was blocked, I collected:

echo w > /proc/sysrq-trigger
echo t > /proc/sysrq-trigger

dmesg -T \
    > /root/dmesg-after-storage-mount.txt

cat /proc/spl/kstat/zfs/dbgmsg \
    > /root/zfs-dbgmsg-after-storage-mount.txt

journalctl -k -b \
    > /root/kernel-journal-after-storage-mount.txt

ps -efL | grep zfs \
    > /root/ps-zfs.txt

cat /proc/locks \
    > /root/proc-locks.txt

zpool status -v \
    > /root/zpool-status-after-mount.txt

All of these files are attached.

dmesg-after-storage-mount.txt
kernel-journal-after-storage-mount.txt
proc-locks.txt
ps-zfs.txt
zfs-dbgmsg-after-storage-mount.txt
zpool-status-after-mount.txt

---

Conclusion

The investigation shows:

• Pool import succeeds.
• Pool scrub succeeds.
• Scrub repairs 0 bytes.
• Scrub reports 0 errors.
• Pool remains ONLINE.
• Dataset metadata is readable (zdb -d).
• Root dataset mounts correctly.
• BACKUP mounts correctly.
• Only myzpool/storage hangs during mount.
• The problem remains fully reproducible after a successful scrub.

The scrub therefore appears to rule out storage corruption or checksum errors, suggesting that the hang occurs during mounting of this specific dataset (possibly during ZIL replay or filesystem initialization).

[andriytk]

Looks like it gets stuck on I/O trying to replay ZIL when you mount the dataset. I see two possibilities here: 1) either your hardware is failing and it tries I/O again and again in the blocks where ZIL data is written; 2) hardware is fine, there are just too many records to replay, in which case you just have to wait and allow it to finish.

I'd try to check what's going on on the disk I/O level, is there any I/O activity at all. Try to check it with iostat or blktrace similar tool on your vg0/zfs device during the mount, and check if any I/O is progressing on it or not.

Btw, why to use LVM mirror and not ZFS mirror? Not sure it's an optimal configuration.

[lorenzop]

I myself am seeing zero activity after it hangs. I've left it go for hours, even over night once. it just sits with 100% IO wait until I hard reboot. I've tried multiple different HBA's and onboard sata, and a freshly created zpool on an ssd. I haven't completely ruled out hardware failure yet. I'm waiting on a replacement motherboard, cpu, and ram.

[andriytk]

Interesting...

my server had more than 200 days uptime before I updated and rebooted the other day...

Have you tried to roll back to your original SW versions yet? In particular, the kernel+ZFS version. Was it a major version upgrade or minor stable? Do you happen to remember the old version?

Also, to rule out HW issues, I'd recommend to test it on some classic filesystem, XFS or Ext4. You can try it on your SSD first.

[M-M-M-M]

Additional testing (read-only import)

Thanks for the suggestions.

Before reproducing the original issue again, I performed another test using a read-only import.

I first rebooted using a GRUB entry that prevents automatic ZFS import, then imported the pool read-only:

zpool status
zpool import -o readonly=on -N myzpool
zpool get readonly myzpool

Output:

NAME     PROPERTY  VALUE   SOURCE
myzpool  readonly  on      -

Then I mounted the datasets one by one:

zfs mount myzpool
zfs mount myzpool/BACKUP
zfs mount myzpool/storage

All three mounts completed immediately.

zpool status:

pool: myzpool
state: ONLINE
scan: scrub repaired 0B in 1 days 09:01:02 with 0 errors on Thu Jul  2 18:14:24 2026

config:

        NAME        STATE     READ WRITE CKSUM
        myzpool     ONLINE       0     0     0
          vg0/zfs   ONLINE       0     0     0

errors: No known data errors

zfs list:

NAME              USED  AVAIL  REFER  MOUNTPOINT
myzpool          10.9T  8.76T   157G  /myzpool
myzpool/BACKUP   9.36T  8.76T  9.36T  /myzpool/BACKUP
myzpool/storage  1.33T  8.76T  1.33T  /myzpool/storage

The pool is still mounted read-only, and I have now been copying data from myzpool/storage to an S3 bucket using rclone for several hours without any errors.

Once the backup is complete, I will reboot again, import the pool in read-write mode, reproduce the mount hang, and monitor I/O activity on the underlying vg0/zfs block device with iostat during the mount, as suggested.

The original issue is that, during a read-write mount, the system eventually reports hung tasks and the mount operation never completes. I would not expect ZIL replay alone to result in this behavior.

[M-M-M-M]

Interesting...

my server had more than 200 days uptime before I updated and rebooted the other day...

Have you tried to roll back to your original SW versions yet? In particular, the kernel+ZFS version. Was it a major version upgrade or minor stable? Do you happen to remember the old version?

Also, to rule out HW issues, I'd recommend to test it on some classic filesystem, XFS or Ext4. You can try it on your SSD first.

On my side, I checked the repository. Previous OpenZFS packages are still available (2.2.9-3, 2.2.9-1, 2.2.8, etc.), so I can try to downgrade if needed.

Before doing that, I’d like to isolate whether this is related to the recent kernel update or to OpenZFS itself. I’ll first boot the previous kernel (4.18.0-553.134.1) while keeping OpenZFS 2.2.10 unchanged. If the issue still reproduces, I’ll then downgrade to OpenZFS 2.2.9-3 and repeat the test.

[andriytk]

That’s a good idea to mount in read only and backup the data. In read only mode, it won’t try to replay the ZIL to avoid any changes, so you might not see the latest changes in your backup copy, maximum within txg timeout, which is 5 secs by default.

[M-M-M-M]

Additional investigation (2026-07-06 / 2026-07-07)

Following the suggestion to determine whether the issue was related to the recent kernel update or to OpenZFS itself, I performed the following tests.

Test matrix

Kernel	OpenZFS	Result
4.18.0-553.137.1	2.2.10	Hangs while mounting myzpool/storage
4.18.0-553.134.1	2.2.10	Same hang during boot (before login)
4.18.0-553.137.1	2.2.9-3	Same hang while mounting myzpool/storage

1. Previous kernel (4.18.0-553.134.1) with OpenZFS 2.2.10

The objective was to determine whether the regression was introduced by the most recent Rocky Linux kernel.

Steps:

• reboot
• boot kernel 4.18.0-553.134.1
• keep OpenZFS 2.2.10 unchanged

Result:

• first attempt lost the KVM video output and required a hard reset;
• second attempt consistently stopped after:

Reached target ZFS pool import target.

• after approximately 4 minutes, the first txg_sync hung-task backtrace appeared;
• another identical backtrace appeared approximately 10 minutes later;
• the machine never completed booting and had to be rebooted.

Conclusion

The problem is reproducible with the previous kernel as well.

---

2. Kernel 4.18.0-553.137.1 + OpenZFS 2.2.10

Booted using the NO ZFS GRUB entry to prevent automatic import.

Commands executed:

journalctl -kf
iostat -dm 1 dm-7

zpool import -N myzpool
zfs mount myzpool
zfs mount myzpool/BACKUP
zfs mount myzpool/storage

Observed behaviour:

• myzpool mounted immediately;
• myzpool/BACKUP mounted immediately;
• myzpool/storage never completed.

While the mount was blocked:

• iostat continued to report read activity on dm-7;
• repeated txg_sync hung-task backtraces appeared every two minutes;
• after about 30 minutes, additional blocked tasks (zfs, z_zrele) also appeared.

Example of iostat while blocked:

Device             tps    MB_read/s    MB_wrtn/s
dm-7            358.00         1.40         0.00

Kernel logs were collected afterwards:

journalctl -k -b \
    > /root/kernel-journal-after-storage-mount-2026-07-06_10:30:18.txt

kernel-journal-after-storage-mount-2026-07-06_10:30:18.txt

---

3. Downgrade to OpenZFS 2.2.9-3

To determine whether this was introduced by OpenZFS 2.2.10, I downgraded to the latest previous release available in the repository.

Performed from the NO ZFS boot entry.

Downgrade command:

dnf downgrade \
    zfs-2.2.9-3.el8 \
    kmod-zfs-2.2.9-3.el8 \
    libzfs5-2.2.9-3.el8 \
    libzpool5-2.2.9-3.el8 \
    libnvpair3-2.2.9-3.el8 \
    libuutil3-2.2.9-3.el8

After reboot, I repeated exactly the same test:

journalctl -kf
iostat -dm 1 dm-7

zpool import -N myzpool
zfs mount myzpool
zfs mount myzpool/BACKUP
zfs mount myzpool/storage

Observed behaviour:

• identical behaviour;
• myzpool/storage never completed mounting;
• repeated txg_sync hung-task backtraces;
• continuous read activity on dm-7;
• after approximately 30 minutes, additional blocked tasks (zfs, z_zrele) appeared.

Kernel log collected:

journalctl -k -b \
    > /root/kernel-journal-after-storage-mount-2026-07-07_08:19:59.txt

kernel-journal-after-storage-mount-2026-07-07_08:19:59.txt

---

Conclusions

At this point I have reproduced the problem under three different software combinations:

Kernel	OpenZFS	Result
4.18.0-553.137.1	2.2.10	Hangs
4.18.0-553.134.1	2.2.10	Hangs
4.18.0-553.137.1	2.2.9-3	Hangs

This suggests that the problem is neither specific to the latest kernel update nor introduced by OpenZFS 2.2.10.

The dataset mounts immediately when the pool is imported read-only, and I was able to copy the entire dataset to an S3 backend without any error. A full scrub also completed successfully with 0 repaired bytes and no known data errors.

However, mounting the dataset read-write consistently results in repeated hung-task call traces (txg_sync, and later also zfs and z_zrele) while disk read activity continues on the underlying device.

I will now leave the system running in this state for an extended period (potentially several hours or even days) to determine whether the mount eventually completes or remains permanently blocked. I will report the final outcome together with any additional observations.

[M-M-M-M]

Update

After leaving the system running for an extended period, the situation evolved further.

The kernel eventually reported:

WARNING: Pool 'myzpool' has encountered an uncorrectable I/O failure and has been suspended.

I collected another kernel log after this event:

journalctl -k -b \
    > /root/kernel-journal-after-storage-mount-2026-07-07_08:43:21.txt

kernel-journal-after-storage-mount-2026-07-07_08:43:21.txt

After the pool was suspended, iostat no longer reported any I/O activity on the underlying device:

Device             tps    MB_read/s    MB_wrtn/s    MB_read    MB_wrtn
dm-7              0.00         0.00         0.00          0          0

This indicates that the operation did not eventually complete. Instead, it ended with the pool being suspended after a prolonged period of repeated hung-task reports and read I/O activity.

The pool is now suspended:

# zpool status -v
  pool: myzpool
 state: SUSPENDED
status: One or more devices are faulted in response to IO failures.
action: Make sure the affected devices are connected, then run 'zpool clear'.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-HC
  scan: scrub repaired 0B in 1 days 09:01:02 with 0 errors on Thu Jul  2 18:14:24 2026

config:

        NAME        STATE     READ WRITE CKSUM
        myzpool     ONLINE       0     0     0
          vg0/zfs   ONLINE       0     0     0

errors: List of errors unavailable: pool I/O is currently suspended

At this point, I do not think that leaving the system running for several more hours or even days will change the outcome. The pool has now been suspended, no further I/O activity is observed, and the system does not appear to be making any further progress.

I am therefore unsure how to investigate this issue any further. Given that:

• the dataset mounts immediately in read-only mode,
• the full scrub completed successfully with no repaired data and no errors,
• the problem reproduces with both OpenZFS 2.2.10 and 2.2.9-3, and
• it also reproduces with two different kernel versions,

I would appreciate any suggestions for additional debugging or diagnostic steps that could help understand why the pool eventually reaches this suspended state.

[andriytk]

Looks like your underlying storage on LVM mirror device is really struggling to make any changes (writes) on it, making ZFS eventually to give up and enter into the suspended state.

Since it's a mirrored raid, I'd try to figure out which of the two disks is lagging and exclude it from the mirror temporary. This might allow ZFS to proceed with ZIL and finish replaying it to avoid the most recent written data loss.

And for the future, I'd make raid1 vdev on ZFS instead of LVM. It would allow ZFS to detect which disk is failing and allow you to replace it on ZFS level, without entering the pool into the suspended state.

What does smartctl -a report on each of the two mirrored drives?