Skip to content

Split H5D_CONTIGUOUS_REF datasets into virtual chunks in getChunkDims - #88

Merged
mattjala merged 1 commit into
HDFGroup:masterfrom
mattjala:fix/contiguous-ref-chunk-dims
Oct 2, 2026
Merged

mattjala merged 1 commit into
HDFGroup:masterfrom
mattjala:fix/contiguous-ref-chunk-dims

Conversation

@mattjala

@mattjala mattjala commented Oct 2, 2026 •

Copy link
Copy Markdown
Collaborator

getChunkDims returned the full dataset shape for every layout that isn't H5D_CHUNKED*. That's right for contiguous data that a store keeps itself, but an H5D_CONTIGUOUS_REF dataset is a byte range in another file, so treating it as one chunk means that reading any element fetches, decodes and caches the entire range. HSDS has used getChunkDims for its chunk grid since 1.0, so on HSDS 1.0.x a one-element read of a linked contiguous dataset reads the whole thing. This is exposed as a massive performance regression in many circumstances. HSDS 0.9.x split these into virtual chunks itself, and this functionality was lost during the 1.0.0 move to hdf5-json.

getChunkDims now returns getContiguousLayout's split for H5D_CONTIGUOUS_REF, using the module's CHUNK_MIN and CHUNK_MAX: virtual chunks of at most ~7.9 MB. The split depends only on the dataset's shape and type. Variable-length types, which validateLayout doesn't allow for this layout, keep the single-chunk behavior. Other layouts are unchanged.

This exposes an existing HSDS bug. HSDS's getStorBytes zero-fills short range reads instead of padding them, so the final virtual chunk of a dataset that ends at end-of-file reads back as zeros. HSDS needs that one-line fix before depending on this release.

Addresses the problem described in HDFGroup/hsds#475.

getChunkDims returned the full dataset shape for every layout that
isn't H5D_CHUNKED*. That's right for contiguous data that a store keeps
itself, but an H5D_CONTIGUOUS_REF dataset is a byte range in another
file, so treating it as one chunk means that reading any element fetches,
decodes and caches the entire range. HSDS has used getChunkDims for its
chunk grid since 1.0, so on HSDS 1.0.x a one-element read of a linked
contiguous dataset reads the whole thing. This is exposed as a massive
performance regression in many circumstances. HSDS 0.9.x split these
into virtual chunks itself, and this functionality was lost during
the 1.0.0 move to hdf5-json.

getChunkDims now returns getContiguousLayout's split for
H5D_CONTIGUOUS_REF, using the module's CHUNK_MIN and CHUNK_MAX: virtual
chunks of at most ~7.9 MB. The split depends only on the dataset's
shape and type. Variable-length types, which validateLayout doesn't allow for
this layout, keep the single-chunk behavior. Other layouts are
unchanged.

This exposes an existing HSDS bug. HSDS's getStorBytes
zero-fills short range reads instead of padding them, so the final
virtual chunk of a dataset that ends at end-of-file reads back as
zeros. HSDS needs that one-line fix before depending on this release.

Resolves HDFGroup/hsds#475.
@mattjala
mattjala force-pushed the fix/contiguous-ref-chunk-dims branch from 36fa99e to 6e96783 Compare October 2, 2026 17:18
@mattjala
mattjala merged commit 99fdce7 into HDFGroup:master Oct 2, 2026
10 checks passed
@github-project-automation github-project-automation Bot moved this from To be triaged to Done in HSDS - TRIAGE & TRACK Oct 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Development

Successfully merging this pull request may close these issues.

1 participant