feat: added skip attribute to hdf5-derive

[viniciusdutra314]

Description

First, thank you for maintaining this amazing crate! I used hdf5-metno for my final graduation project, and it was incredibly helpful. While working on my project, one feature I found myself missing was the ability to explicitly skip fields of the produced H5Type, similar to how #[serde(skip)] works in Serde.

This PR introduces the #[hdf5(skip)] attribute to the hdf5-derive crate. This allows users to ignore specific fields (like large vectors, strings, or irrelevant internal state) when writing a struct to an HDF5 file.

Usage Example

#[derive(H5Type)]
#[repr(C)]
struct MyStruct {
    useful_data1: u8,
    #[hdf5(skip)]
    _useless_big_data: Vec<String>,
    useful_data2: u32,
}

Implementation Details

The implementation itself is relatively straightforward. I added a new is_hdf5_skip helper function in the derive macro that parses the field attributes (using parse_nested_meta). If #[hdf5(skip)] is found, the field is simply filtered out before the HDF5 TypeDescriptor is generated. In the code, it works exactly like the is_phantom_data function.

Tests added

As I described before, the behavior should be the same as a PhantomData field, but to actually assert that, I added the test_hdf5_skip_attribute module.
The test suite uses a custom check_fields! macro. For every test case, it allocates an uninitialized struct via std::mem::MaybeUninit, uses raw pointers to calculate the exact physical byte offset of each field in memory, and asserts that the Rust compiler's memory layout perfectly matches the offset registered in the generated HDF5 Compound descriptor.

The tests explicitly verify:

• #[repr(C)] Structs: Ensures standard C-padding and alignment are respected when fields are skipped.
• #[repr(packed)] Structs: Verifies that unaligned byte offsets are correctly calculated without padding when a field is skipped.
• Tuple Structs: Ensures that field indices (e.g., 0, 2) are correctly mapped and retain their true names/indices even if a sibling field is skipped.
• Generics: Verifies that skipping fields works seamlessly alongside generic type parameters.

[mulimoen]

I struggle in understanding how we can hdf5::read a type like this. The bytes returned will be uninit and unsafe

[viniciusdutra314]

You are absolutely right! I was so focused on the serialization side of things that I completely forgot the deserialization.

I think the most ergonomic and safe behavior is to require that any field marked with #[hdf5(skip)] must implement the Default trait. I don't know if it's possible to initialize it using field.default() during the reading, but this would prevent the UB... I'll take a look on what i can do... But the serialization it's ok right? (hdf5::write)

[mulimoen]

The serialization should be fine in theory, I have not examined the details though. Default is not going to be enough since we start from mem::uninitialized, don't think a Zeroded trait will do either. If you want this feature then we could add new traits for serialization and deserialization and make H5Type be the combination of those two

[viniciusdutra314]

I think splitting this into something like H5Serialize and H5Deserialize is a great idea. For the specific use case I described with #[hdf5(skip)], the struct would only derive H5Serialize. This allows the user to safely call write, but the compiler would block any attempts to call read (which would strictly require H5Deserialize).

By doing this, we avoid the undefined behavior issue and there is no need for any unnecessary zeroing or default initialization of the C HDF5 read buffer. It mimics Serde's asymmetric model but without the intermediate data representation.

I'd be happy to help if you provide me some direction, i used the HDF5 library in C before but i'm getting familiar with the code base

[mulimoen]

Such a change would likely involve breaking changes in most of the crate(s) and I can't see any other usecases than the one you suggest. I am not confident that it will be worth the extra work in maintaining such a solution, unfortunately

[viniciusdutra314]

I understand, since you'll be maintaining this crate long-term, it's not a great idea to introduce massive breaking changes.

What if we could achieve this without breaking the current API?

Looking quickly at the read_into_buf function, i think that I can find a way to make this zero-cost for existing structs (a no-op function for structs without skipped fields, that probably will be inlined), but enforce default initialization for the fields of structs that use #[hdf5(skip)].

Let me know if a non-breaking approach like that sounds great, otherwise I am happy to leave it as is and thank you for your time

[viniciusdutra314]

I have some great news! After a few tweaks, I've added tests for reading and writing skipped structs, and is working exactly as expected. The skipped fields are now safely initialized using the Default trait.

To not break backward compatibility, I added a new method to the H5Type trait:
unsafe fn initialize_skipped_fields(_ptr: *mut Self, _size: usize) {}

Because this has an empty default implementation, existing types incur zero runtime overhead and require absolutely no code changes.

I search for all instances of H5Dread and H5Aread across the crate to trace potential Undefined Behavior. The only place that i found that was vulnerable to uninitialized memory was read_into_buf. I updated this function to call T::initialize_skipped_fields(buf, n_elements) in the beginning.

Finally, when the derive macro finds #[hdf5(skip)] fields, it overrides the default initialization method by the following code

quote! {
    for i in 0..size {
        let ptr = ptr.add(i);
        #(
            ::std::ptr::write_unaligned(
                ::std::ptr::addr_of_mut!((*ptr).#fields),
                ::core::default::Default::default(),
            );
        )*
    }
}

[viniciusdutra314]

I updated the function signature in the H5Type trait from:

unsafe fn initialize_skipped_fields(_ptr: *mut Self, _size: usize) {}

to:

unsafe fn init_skipped_fields(element: &mut MaybeUninit<Self>)

I believe these semantics are clearer, it explicitly says that it partially initializes the skipped fields in a block of memory that may be uninitialized. The read_buf function now calls this N times across the entire buffer.

I was initially concerned that the loop in read_buf would introduce overhead for structs without skipped fields. However, after verifying with cargo-show-asm, I confirmed that the loop it is optimized away in the LLVM IR. Playing around with Compiler Explorer i think when the loop has no observable side effects, it's simple dead code elimination https://godbolt.org/z/WhnrYojh1

I also added a test to assert that when a struct with skipped fields is serialized, it can successfully be deserialized into a struct without the skipped fields (which i think it's the most common usage).