Following my previous contributions (implementing iterate and visit), I've identified that h5fortran currently lacks high-level abstractions for SWMR (Single Writer Multiple Reader).
In scenarios where a simulation (writer) and a visualizer (reader, e.g., via h5py) access the same file, the default HDF5 locking mechanism often leads to crashes or file access errors. While this can be solved using low-level use hdf5 calls, it breaks the high-level philosophy of this library.
I am currently facing this exact issue: my visualization script (running in parallel with the main simulation) frequently crashes because of file locking/metadata contention. Providing a high-level SWMR interface would solve this real-world stability problem for users doing live data monitoring.
I'm opening this issue to keep track of the requirements for native SWMR support. Key components would include:
- File Access: Adding a swmr logical flag to %open() or %create(). This would internally handle H5Pset_libver_bounds(fapl, H5F_LIBVER_LATEST, H5F_LIBVER_LATEST) and the corresponding SWMR access flags.
- Writer Side: Exposing a %flush() method for datasets (wrapping h5dflush_f) to ensure metadata/data consistency is pushed to disk so readers can see updates.
- Reader Side: Implementing a %refresh() method (wrapping h5drefresh_f) to update the object's view without reopening the file.
Technical Context:
For the writer, the sequence usually requires:
- Setting libver = latest in FAPL.
- Creating the file and datasets.
- Calling h5fstart_swmr_write_f.
For the reader:
- Opening with H5F_ACC_RDONLY + H5F_ACC_SWMR_READ.
- Periodic calls to h5drefresh_f.
I believe adding this would make h5fortran much more robust for real-time data monitoring. I'm documenting this here to track the feature progress and discuss the best API design before implementation.
Following my previous contributions (implementing iterate and visit), I've identified that h5fortran currently lacks high-level abstractions for SWMR (Single Writer Multiple Reader).
In scenarios where a simulation (writer) and a visualizer (reader, e.g., via h5py) access the same file, the default HDF5 locking mechanism often leads to crashes or file access errors. While this can be solved using low-level use hdf5 calls, it breaks the high-level philosophy of this library.
I am currently facing this exact issue: my visualization script (running in parallel with the main simulation) frequently crashes because of file locking/metadata contention. Providing a high-level SWMR interface would solve this real-world stability problem for users doing live data monitoring.
I'm opening this issue to keep track of the requirements for native SWMR support. Key components would include:
Technical Context:
For the writer, the sequence usually requires:
For the reader:
I believe adding this would make h5fortran much more robust for real-time data monitoring. I'm documenting this here to track the feature progress and discuss the best API design before implementation.