Skip to content

io/disk: Added disk_cleanup module and refactor test teardown#3188

Open
maramsmurthy wants to merge 1 commit into
avocado-framework-tests:masterfrom
maramsmurthy:disk-cleanup-pep8-fixes
Open

io/disk: Added disk_cleanup module and refactor test teardown#3188
maramsmurthy wants to merge 1 commit into
avocado-framework-tests:masterfrom
maramsmurthy:disk-cleanup-pep8-fixes

Conversation

@maramsmurthy

Copy link
Copy Markdown
Contributor

Added new disk_cleanup.py module providing comprehensive disk cleanup functionality for storage test teardown. This module handles:

  • Multipath device normalization
  • LVM/RAID dependency graph building
  • Filesystem unmounting and swap disabling
  • LVM volume group and logical volume removal
  • RAID array cleanup with member device handling
  • Partition table removal and metadata wiping
  • Disk zeroing for complete cleanup

Refactor 13 test files to use the new cleanup_disks() function:

  • ltp_fsstress.py, tiobench.py, fiotest.py, lvsetup.py
  • softwareraid.py, dbench.py, iozone.py, parallel_dd.py
  • disk_info.py, bonnie.py, ltp_fs.py, disktest.py, fs_mark.py

Benefits:

  • Consistent cleanup across all disk I/O tests
  • Reduced code duplication (DRY principle)
  • Better handling of complex storage configurations
  • Improved test reliability and cleanup robustness

@maramsmurthy maramsmurthy force-pushed the disk-cleanup-pep8-fixes branch 3 times, most recently from 7d5544d to 5f629df Compare June 8, 2026 09:24
@maramsmurthy

Copy link
Copy Markdown
Contributor Author

RHEL run

TestSuite TestRun Summary

host_io_disk_cleanup_bonnie_bonnie Run Successfully executed
/home/cleanup_runs/results/job-2026-06-07T16.27-acd8fff/job.log
| PASS 8 || CANCEL 4 || ERRORS 0 || FAILURES 0 || SKIP 0 || WARN 0 || INTERRUPT 0 |

host_io_disk_cleanup_tiobench_tiobench Run Successfully executed
/home/cleanup_runs/results/job-2026-06-07T16.41-402c7f5/job.log
| PASS 24 || CANCEL 8 || ERRORS 0 || FAILURES 0 || SKIP 0 || WARN 0 || INTERRUPT 0 |

host_io_disk_cleanup_disk_info_disk_info Run Successfully executed
/home/cleanup_runs/results/job-2026-06-07T17.18-671bbd6/job.log
| PASS 4 || CANCEL 2 || ERRORS 0 || FAILURES 0 || SKIP 0 || WARN 0 || INTERRUPT 0 |

host_io_disk_cleanup_disktest_disktest Run Successfully executed
/home/cleanup_runs/results/job-2026-06-07T17.22-7b66154/job.log
| PASS 4 || CANCEL 2 || ERRORS 0 || FAILURES 0 || SKIP 0 || WARN 0 || INTERRUPT 0 |

host_io_disk_cleanup_fiotest_fio Run Successfully executed
/home/cleanup_runs/results/job-2026-06-07T17.47-a726d0f/job.log
| PASS 8 || CANCEL 4 || ERRORS 0 || FAILURES 0 || SKIP 0 || WARN 4 || INTERRUPT 0 |

host_io_disk_cleanup_softwareraid_softwareraid Run Successfully executed
/home/cleanup_runs/results/job-2026-06-07T18.12-2471752/job.log
| PASS 23 || CANCEL 0 || ERRORS 0 || FAILURES 0 || SKIP 0 || WARN 0 || INTERRUPT 0 |

host_io_disk_cleanup_ltp_fsstress_ltp_fsstress Run Successfully executed
/home/cleanup_runs/results/job-2026-06-07T20.01-d2dbbf6/job.log
| PASS 16 || CANCEL 4 || ERRORS 0 || FAILURES 0 || SKIP 0 || WARN 0 || INTERRUPT 0 |

host_io_disk_cleanup_ltp_fs_ltp_fs Run Successfully executed
/home/cleanup_runs/results/job-2026-06-07T21.03-965ae32/job.log
| PASS 64 || CANCEL 16 || ERRORS 0 || FAILURES 0 || SKIP 0 || WARN 0 || INTERRUPT 0 |

host_io_disk_cleanup_parallel_dd_parallel_dd Run Successfully executed
/home/cleanup_runs/results/job-2026-06-07T22.41-5b19c0f/job.log
| PASS 2 || CANCEL 1 || ERRORS 0 || FAILURES 0 || SKIP 0 || WARN 0 || INTERRUPT 0 |

host_io_disk_cleanup_fs_mark_fs_mark Run Successfully executed
/home/cleanup_runs/results/job-2026-06-07T22.43-7e83351/job.log
| PASS 8 || CANCEL 4 || ERRORS 0 || FAILURES 0 || SKIP 0 || WARN 0 || INTERRUPT 0 |

host_io_disk_cleanup_ioping_ioping Run Successfully executed
/home/cleanup_runs/results/job-2026-06-07T22.55-2dcefbd/job.log
| PASS 2 || CANCEL 0 || ERRORS 0 || FAILURES 0 || SKIP 0 || WARN 0 || INTERRUPT 0 |

host_io_disk_cleanup_iozone_iozone Run Successfully executed
/home/cleanup_runs/results/job-2026-06-08T12.54-df9195f/job.log
| PASS 0 || CANCEL 8 || ERRORS 11 || FAILURES 0 || SKIP 0 || WARN 13 || INTERRUPT 0 |

host_io_disk_cleanup_lvsetup_lvsetup Run Successfully executed
/home/cleanup_runs/results/job-2026-06-08T13.24-88c622d/job.log
| PASS 12 || CANCEL 3 || ERRORS 0 || FAILURES 0 || SKIP 0 || WARN 0 || INTERRUPT 0 |

SUSE bucket run
06:15:40 INFO : Summary of test results can be found below: 06:15:40 [2/1160]
TestSuite TestRun Summary

host_io_disk_cleanup_bonnie_bonnie Run Successfully executed
/home/cleanup_runs/results/job-2026-06-07T11.29-b04606c/job.log
| PASS 12 || CANCEL 0 || ERRORS 0 || FAILURES 0 || SKIP 0 || WARN 0 || INTERRUPT 0 |

host_io_disk_cleanup_tiobench_tiobench Run Successfully executed
/home/cleanup_runs/results/job-2026-06-07T11.45-29a3376/job.log
| PASS 32 || CANCEL 0 || ERRORS 0 || FAILURES 0 || SKIP 0 || WARN 0 || INTERRUPT 0 |

host_io_disk_cleanup_disk_info_disk_info Run Successfully executed
/home/cleanup_runs/results/job-2026-06-07T12.31-b517409/job.log
| PASS 3 || CANCEL 0 || ERRORS 0 || FAILURES 3 || SKIP 0 || WARN 0 || INTERRUPT 0 |

host_io_disk_cleanup_disktest_disktest Run Successfully executed
/home/cleanup_runs/results/job-2026-06-07T12.36-5469148/job.log
| PASS 6 || CANCEL 0 || ERRORS 0 || FAILURES 0 || SKIP 0 || WARN 0 || INTERRUPT 0 |

host_io_disk_cleanup_fiotest_fio Run Successfully executed
/home/cleanup_runs/results/job-2026-06-07T14.23-72e392e/job.log
| PASS 12 || CANCEL 0 || ERRORS 0 || FAILURES 0 || SKIP 0 || WARN 4 || INTERRUPT 0 |

host_io_disk_cleanup_softwareraid_softwareraid Run Successfully executed
/home/cleanup_runs/results/job-2026-06-07T15.19-f785fd4/job.log
| PASS 23 || CANCEL 0 || ERRORS 0 || FAILURES 0 || SKIP 0 || WARN 0 || INTERRUPT 0 |

host_io_disk_cleanup_ltp_fsstress_ltp_fsstress Run Successfully executed
/home/cleanup_runs/results/job-2026-06-07T16.40-317c6ad/job.log
| PASS 20 || CANCEL 0 || ERRORS 0 || FAILURES 0 || SKIP 0 || WARN 0 || INTERRUPT 0 |

host_io_disk_cleanup_ltp_fs_ltp_fs Run Successfully executed
/home/cleanup_runs/results/job-2026-06-08T00.19-60163d6/job.log
| PASS 80 || CANCEL 0 || ERRORS 0 || FAILURES 0 || SKIP 0 || WARN 0 || INTERRUPT 0 |

host_io_disk_cleanup_parallel_dd_parallel_dd Run Successfully executed
/home/cleanup_runs/results/job-2026-06-08T05.22-e691707/job.log
| PASS 3 || CANCEL 0 || ERRORS 0 || FAILURES 0 || SKIP 0 || WARN 0 || INTERRUPT 0 |

host_io_disk_cleanup_fs_mark_fs_mark Run Successfully executed
/home/cleanup_runs/results/job-2026-06-08T05.25-2ed7f78/job.log
| PASS 12 || CANCEL 0 || ERRORS 0 || FAILURES 0 || SKIP 0 || WARN 0 || INTERRUPT 0 |

host_io_disk_cleanup_ioping_ioping Run Successfully executed
/home/cleanup_runs/results/job-2026-06-08T05.40-5a8ccb7/job.log
| PASS 2 || CANCEL 0 || ERRORS 0 || FAILURES 0 || SKIP 0 || WARN 0 || INTERRUPT 0 |

host_io_disk_cleanup_iozone_iozone Run Successfully executed
/home/cleanup_runs/results/job-2026-06-08T05.42-566cc76/job.log
| PASS 0 || CANCEL 0 || ERRORS 32 || FAILURES 0 || SKIP 0 || WARN 0 || INTERRUPT 0 |

host_io_disk_cleanup_lvsetup_lvsetup Run Successfully executed
/home/cleanup_runs/results/job-2026-06-08T06.02-99b1862/job.log
| PASS 15 || CANCEL 0 || ERRORS 0 || FAILURES 0 || SKIP 0 || WARN 0 || INTERRUPT 0 |

No issues observed with the the flow with respect to io_bucket/CR flow.

@Naresh-ibm Naresh-ibm requested review from Naresh-ibm, PraveenPenguin, abdhaleegit and sacsant and removed request for sacsant June 8, 2026 11:58
@Naresh-ibm Naresh-ibm self-assigned this Jun 8, 2026
@maramsmurthy maramsmurthy force-pushed the disk-cleanup-pep8-fixes branch 2 times, most recently from 35e614e to 7aa7868 Compare June 9, 2026 03:28
@maramsmurthy

Copy link
Copy Markdown
Contributor Author

@Naresh-ibm @abdhaleegit @PraveenPenguin @sacsant

The current cleanup code PR is an advancement to the existing flow.

Advantages:

  1. Added more core cases, Which will help us to use the disk/disks without any cleanup job before starting test.
  2. No damage to current CR flow.
  3. No need to worry about disk cleanup part before writing any other new test cases.
  4. Reduced the size of all the other files where we maintained lot of cleanup code in each py file.
  5. Maintainer job will become easy as they can concentrate only on specific job.

Let me know if required any other information

@maramsmurthy maramsmurthy force-pushed the disk-cleanup-pep8-fixes branch from 7aa7868 to be707c0 Compare June 15, 2026 09:13

@PraveenPenguin PraveenPenguin left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel move cleanup disk utility on avocado it self

This commit refactors disk I/O test teardown to use centralized cleanup
utilities from the avocado framework's disk utility module instead of
local implementations.

Key changes:
- Remove local disk_cleanup.py module (functionality migrated to avocado.utils.disk)
- Update 13 test files to import cleanup_disks from avocado.utils.disk:
  * bonnie.py, dbench.py, disk_info.py, disktest.py, fiotest.py
  * fs_mark.py, iozone.py, ltp_fs.py, ltp_fsstress.py, lvsetup.py
  * parallel_dd.py, softwareraid.py, tiobench.py
- Improve RAID cleanup: Use log.info for forced stop (normal flow)
  instead of log.warning to reduce noise in test logs

Benefits:
- Centralized maintenance in avocado framework
- Consistent cleanup behavior across all disk tests
- Disk-type agnostic utilities (NVMe, SCSI, IDE, virtio, etc.)
- Dependency-aware cleanup order (mounts → swap → LVM → RAID → metadata)
- Enhanced error handling and logging

The cleanup utilities in avocado.utils.disk provide:
- Automatic multipath device detection and normalization
- Intelligent dependency graph resolution
- Auto-detection mode for 'light' or 'full' cleanup
- Orphaned device-mapper and VG directory cleanup
- Enhanced RAID array detection using /sys/block slaves
- Comprehensive metadata wiping with retry logic

Signed-off-by: Maram Srimannarayana Murthy <msmurthy@linux.vnet.ibm.com>
@maramsmurthy maramsmurthy force-pushed the disk-cleanup-pep8-fixes branch from be707c0 to e80c150 Compare June 18, 2026 09:45
@maramsmurthy

Copy link
Copy Markdown
Contributor Author

Moved all the disk cleanup utilities to avocado framework

Update all the files to use utils from avocado

CR run is also successful without any issues

TestSuite TestRun Summary

host_io_disk_cleanup_bonnie_bonnie Run Successfully executed
/home/disk_cleanup_cr/results/job-2026-06-18T07.46-502cf83/job.log
| PASS 12 || CANCEL 0 || ERRORS 0 || FAILURES 0 || SKIP 0 || WARN 0 || INTERRUPT 0 |

host_io_disk_cleanup_rawread_rawread Run Successfully executed
/home/disk_cleanup_cr/results/job-2026-06-18T07.50-85c07fb/job.log
| PASS 1 || CANCEL 1 || ERRORS 0 || FAILURES 0 || SKIP 0 || WARN 0 || INTERRUPT 0 |

host_io_disk_cleanup_tiobench_tiobench Run Successfully executed
/home/disk_cleanup_cr/results/job-2026-06-18T07.51-43a5bde/job.log
| PASS 32 || CANCEL 0 || ERRORS 0 || FAILURES 0 || SKIP 0 || WARN 0 || INTERRUPT 0 |

host_io_disk_cleanup_disktest_disktest Run Successfully executed
/home/disk_cleanup_cr/results/job-2026-06-18T08.02-8790c29/job.log
| PASS 6 || CANCEL 0 || ERRORS 0 || FAILURES 0 || SKIP 0 || WARN 0 || INTERRUPT 0 |

host_io_disk_cleanup_fiotest_fio Run Successfully executed
/home/disk_cleanup_cr/results/job-2026-06-18T08.09-5ac6b60/job.log
| PASS 12 || CANCEL 0 || ERRORS 0 || FAILURES 0 || SKIP 0 || WARN 4 || INTERRUPT 0 |

host_io_disk_cleanup_softwareraid_softwareraid Run Successfully executed
/home/disk_cleanup_cr/results/job-2026-06-18T08.19-76b23b2/job.log
| PASS 23 || CANCEL 0 || ERRORS 0 || FAILURES 0 || SKIP 0 || WARN 0 || INTERRUPT 0 |

host_io_disk_cleanup_ltp_fsstress_ltp_fsstress Run Successfully executed
/home/disk_cleanup_cr/results/job-2026-06-18T08.46-663797c/job.log
| PASS 20 || CANCEL 0 || ERRORS 0 || FAILURES 0 || SKIP 0 || WARN 0 || INTERRUPT 0 |

host_io_disk_cleanup_ltp_fs_ltp_fs Run Successfully executed
/home/disk_cleanup_cr/results/job-2026-06-18T09.53-d7c0b36/job.log
| PASS 80 || CANCEL 0 || ERRORS 0 || FAILURES 0 || SKIP 0 || WARN 0 || INTERRUPT 0 |

host_io_disk_cleanup_parallel_dd_parallel_dd Run Successfully executed
/home/disk_cleanup_cr/results/job-2026-06-18T10.35-301a08d/job.log
| PASS 3 || CANCEL 0 || ERRORS 0 || FAILURES 0 || SKIP 0 || WARN 0 || INTERRUPT 0 |

host_io_disk_cleanup_fs_mark_fs_mark Run Successfully executed
/home/disk_cleanup_cr/results/job-2026-06-18T10.36-0ab5718/job.log
| PASS 12 || CANCEL 0 || ERRORS 0 || FAILURES 0 || SKIP 0 || WARN 0 || INTERRUPT 0 |

host_io_disk_cleanup_ioping_ioping Run Successfully executed
/home/disk_cleanup_cr/results/job-2026-06-18T10.38-8bb43a9/job.log
| PASS 2 || CANCEL 0 || ERRORS 0 || FAILURES 0 || SKIP 0 || WARN 0 || INTERRUPT 0 |

host_io_disk_cleanup_iozone_iozone Run Successfully executed
/home/disk_cleanup_cr/results/job-2026-06-18T10.39-8d6cac0/job.log
| PASS 0 || CANCEL 0 || ERRORS 32 || FAILURES 0 || SKIP 0 || WARN 0 || INTERRUPT 0 |

host_io_disk_cleanup_lvsetup_lvsetup Run Successfully executed
/home/disk_cleanup_cr/results/job-2026-06-18T10.53-2d9e3d1/job.log
| PASS 15 || CANCEL 0 || ERRORS 0 || FAILURES 0 || SKIP 0 || WARN 0 || INTERRUPT 0 |
11:00:11 INFO : Removing temporary mux dir

@maramsmurthy

Copy link
Copy Markdown
Contributor Author

Just for the Information As this PR depends on utils PR avocado-framework/avocado#6312

It should be merged only after successful merge on utils PR

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants