Repository navigation
Split H5D_CONTIGUOUS_REF datasets into virtual chunks in getChunkDims - #88
Merged
Merged
Conversation
getChunkDims returned the full dataset shape for every layout that isn't H5D_CHUNKED*. That's right for contiguous data that a store keeps itself, but an H5D_CONTIGUOUS_REF dataset is a byte range in another file, so treating it as one chunk means that reading any element fetches, decodes and caches the entire range. HSDS has used getChunkDims for its chunk grid since 1.0, so on HSDS 1.0.x a one-element read of a linked contiguous dataset reads the whole thing. This is exposed as a massive performance regression in many circumstances. HSDS 0.9.x split these into virtual chunks itself, and this functionality was lost during the 1.0.0 move to hdf5-json. getChunkDims now returns getContiguousLayout's split for H5D_CONTIGUOUS_REF, using the module's CHUNK_MIN and CHUNK_MAX: virtual chunks of at most ~7.9 MB. The split depends only on the dataset's shape and type. Variable-length types, which validateLayout doesn't allow for this layout, keep the single-chunk behavior. Other layouts are unchanged. This exposes an existing HSDS bug. HSDS's getStorBytes zero-fills short range reads instead of padding them, so the final virtual chunk of a dataset that ends at end-of-file reads back as zeros. HSDS needs that one-line fix before depending on this release. Resolves HDFGroup/hsds#475.
mattjala
force-pushed
the
fix/contiguous-ref-chunk-dims
branch
from
October 2, 2026 17:18
36fa99e to
6e96783
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
getChunkDims returned the full dataset shape for every layout that isn't H5D_CHUNKED*. That's right for contiguous data that a store keeps itself, but an H5D_CONTIGUOUS_REF dataset is a byte range in another file, so treating it as one chunk means that reading any element fetches, decodes and caches the entire range. HSDS has used getChunkDims for its chunk grid since 1.0, so on HSDS 1.0.x a one-element read of a linked contiguous dataset reads the whole thing. This is exposed as a massive performance regression in many circumstances. HSDS 0.9.x split these into virtual chunks itself, and this functionality was lost during the 1.0.0 move to hdf5-json.
getChunkDims now returns getContiguousLayout's split for H5D_CONTIGUOUS_REF, using the module's CHUNK_MIN and CHUNK_MAX: virtual chunks of at most ~7.9 MB. The split depends only on the dataset's shape and type. Variable-length types, which validateLayout doesn't allow for this layout, keep the single-chunk behavior. Other layouts are unchanged.
This exposes an existing HSDS bug. HSDS's getStorBytes zero-fills short range reads instead of padding them, so the final virtual chunk of a dataset that ends at end-of-file reads back as zeros. HSDS needs that one-line fix before depending on this release.
Addresses the problem described in HDFGroup/hsds#475.