Skip to content

Add struct types to Halide - #9416

Open
alexreinking wants to merge 12 commits into
mainfrom
alexreinking/struct-types
Open

alexreinking wants to merge 12 commits into
mainfrom
alexreinking/struct-types

Conversation

@alexreinking

@alexreinking alexreinking commented Sep 1, 2026 •

Copy link
Copy Markdown
Member

This PR adds struct types to Halide. This allows for inputs and outputs to be defined in terms of popular wire formats, even if they aren't prepared for efficient computation. For example, you can define GGML's q5_0 and q8_0 formats like so:

Type q5_0 = Type::Struct({{"d", Float(16)}, {"qh", UInt(32)}, {"qs", UInt(8), 16}});
Type q8_0 = Type::Struct({{"d", Float(16)}, {"qs", Int(8), 32}});

Then you can write a pipeline that consumes them:

ImageParam x{q4_0_type(), 1, "x"};
ImageParam y{q8_0_type(), 1, "y"};

Var b("b"), k("k"), u("u");
RDom r(0, x.dim(0).extent(), 0, 32, "r");  // block, quant

// Dequantize one 4-bit weight to float: quant k is a nibble of the
// packed byte k % 16 -- its low nibble for k < 16, its high nibble
// otherwise -- biased by -8 to signed [-8, 7], times the block delta.
Expr nib = field(x(b), "qs")[k % 16];  // uint8
Func x_wt("x_wt");
x_wt(k, b) = cast<float>(field(x(b), "d")) * 
             (cast<int32_t>(select(k < 16, nib % 16, nib / 16)) - 8);

// Dequantize one int8 activation to float: quant k is int8 k, times the
// block delta.
Func y_wt("y_wt");
y_wt(k, b) = cast<float>(field(y(b), "d")) * 
             cast<int32_t>(field(y(b), "qs")[k]);

// Dequantize each side and sum every product.
Func qdot{"qdot"};
qdot() = 0.0f;
qdot() += x_wt(r[1], r[0]) * y_wt(r[1], r[0]);

As you can see, struct types support both individual elements, as well as fixed-size array elements. Fields are accessed with a named field(expr, "name") intrinsic. The tests include additional cases for struct-typed fields and output structs.

This adjusts the Type and halide_type_t representations in the following ways:

  1. Type: like Handle, a Struct type has a compile-time known field layout.
  2. halide_type_t: structs get their own kind, and the reserved field records the number of bytes in the struct.

All layouts are expected to be packed. No padding is inserted around or in between field elements.

Breaking changes

None—it's a new feature.

Checklist

  • Tests added or updated (not required for docs, CI config, or typo fixes)
  • Documentation updated (if public API changed)
  • Python bindings updated (if public API changed)
  • Benchmarks are included here if the change is intended to affect performance.
  • Commits include AI attribution where applicable (see Code of Conduct)

@alexreinking
alexreinking force-pushed the alexreinking/struct-types branch from 9f99483 to 069ce04 Compare September 1, 2026 17:16
@alexreinking
alexreinking changed the base branch from alexreinking/hoist-splitting to alexreinking/rfactor-hoisting September 1, 2026 17:32
@alexreinking
alexreinking force-pushed the alexreinking/struct-types branch from 069ce04 to 50683f7 Compare September 1, 2026 18:21
@alexreinking
alexreinking force-pushed the alexreinking/struct-types branch from 50683f7 to e9123a1 Compare September 1, 2026 19:06
@codecov

codecov Bot commented Sep 2, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 56.88705% with 313 lines in your changes missing coverage. Please review.
✅ Project coverage is 70.74%. Comparing base (3fcc674) to head (5462034).

Files with missing lines Patch % Lines
src/CodeGen_Vulkan_Dev.cpp 0.00% 125 Missing ⚠️
src/IROperator.cpp 59.23% 34 Missing and 19 partials ⚠️
src/LowerStructTypes.cpp 74.27% 35 Missing and 18 partials ⚠️
src/Type.cpp 72.80% 10 Missing and 21 partials ⚠️
src/CodeGen_D3D12Compute_Dev.cpp 0.00% 18 Missing ⚠️
src/Type.h 80.00% 2 Missing and 3 partials ⚠️
src/FuseGPUThreadLoops.cpp 0.00% 3 Missing and 1 partial ⚠️
src/Deserialization.cpp 81.25% 1 Missing and 2 partials ⚠️
src/IROperator.h 57.14% 3 Missing ⚠️
src/IRPrinter.cpp 75.00% 1 Missing and 2 partials ⚠️
... and 9 more
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #9416      +/-   ##
==========================================
- Coverage   70.92%   70.74%   -0.18%     
==========================================
  Files         261      262       +1     
  Lines       79989    80594     +605     
  Branches    19494    19653     +159     
==========================================
+ Hits        56733    57018     +285     
- Misses      17446    17637     +191     
- Partials     5810     5939     +129     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@alexreinking
alexreinking force-pushed the alexreinking/struct-types branch from e9123a1 to 425cb95 Compare September 3, 2026 17:10
@alexreinking
alexreinking force-pushed the alexreinking/struct-types branch from 425cb95 to 2740239 Compare September 4, 2026 09:09
@alexreinking
alexreinking force-pushed the alexreinking/struct-types branch from 1464bc0 to 168ce01 Compare September 4, 2026 20:16
@alexreinking
alexreinking force-pushed the alexreinking/struct-types branch from 168ce01 to e272037 Compare September 11, 2026 09:02
Base automatically changed from alexreinking/rfactor-hoisting to main September 13, 2026 01:03
@alexreinking
alexreinking force-pushed the alexreinking/struct-types branch 2 times, most recently from 166d1d5 to 1444b20 Compare September 15, 2026 15:06
@alexreinking
alexreinking force-pushed the alexreinking/struct-types branch from 1444b20 to 91c252b Compare September 20, 2026 17:23
@alexreinking
alexreinking marked this pull request as ready for review September 20, 2026 17:23
@alexreinking
alexreinking force-pushed the alexreinking/struct-types branch 2 times, most recently from 129cd6a to 53dcddc Compare September 21, 2026 17:39
Comment thread src/LowerStructTypes.cpp Outdated
Comment thread src/LowerStructTypes.cpp Outdated
Comment thread src/LowerStructTypes.cpp Outdated
Comment thread src/Type.h
Comment thread src/runtime/HalideRuntime.h
Comment thread src/Type.h Outdated
Comment thread test/correctness/struct_type.cpp
Comment thread test/correctness/struct_type_dot_product.cpp Outdated
@alexreinking
alexreinking force-pushed the alexreinking/struct-types branch 2 times, most recently from cf3799e to 011f44d Compare September 29, 2026 19:08
@alexreinking
alexreinking removed this pull request from stack #9420 September 29, 2026 20:30
@alexreinking
alexreinking added this pull request to stack #9485 September 29, 2026 20:32
@alexreinking
alexreinking removed this pull request from stack #9485 September 29, 2026 21:35
@alexreinking
alexreinking added this pull request to stack #9487 September 29, 2026 21:36
alexreinking and others added 12 commits September 30, 2026 15:39
Introduce first-class packed struct types (Type::Struct) modeled
faithfully in the type system:

- A dedicated ABI type code halide_type_struct=5; a struct's packed byte
  size rides in the halide_type_t reserved field, so struct-typed buffers
  have correct element size/strides. StructTypeInfo (field layout) is
  interned like handle metadata, keeping Type at 8 bytes. is_uint()/etc.
  are honestly false for structs, so no numeric special-casing is needed.

- field()/pack_struct() intrinsics with byte-addressed lowering
  (LowerStructTypes), plus per-field pack_struct ergonomics: an array
  field is filled by a gather() packet, a gather(extent, gen) generator,
  a single expression with one swept `_` placeholder (index arithmetic
  allowed), or a field() copy of a whole same-typed field.

- Python bindings for the type, field/pack_struct/gather, and the
  per-field forms.

- Tests: correctness (CPU + GPU), error cases, Python, and an ARM codegen
  test implementing ggml's q4_0/q8_0 dot product that verifies the packed
  qs arrays lower to dense 128-bit vector loads.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- ConstantBounds.cpp/PyType.cpp: drop two unnecessary-copy clang-tidy findings.
- struct_type_dot_product.cpp: back the test buffers with 16-byte-aligned
  storage instead of std::vector, whose allocator doesn't guarantee that on
  32-bit ABIs.
- FuseGPUThreadLoops.cpp: relax the GPU shared/global allocation clustering
  assert from "both types are powers of two bytes" to the actual requirement
  ("widest type is a whole multiple of the cluster's byte-granularity type"),
  so a non-power-of-two struct size (e.g. 12 bytes) can share GPU memory with
  other types.
- LowerStructTypes.cpp: build a packed float field (e.g. a struct's fp16
  delta) via a scalar shift/or chain instead of concat_bits's
  vector-shuffle-then-reinterpret lowering. Some AArch64 backends (LLVM < 23)
  can't legalize a bitcast straight from a vector to a scalar half, and any
  chain of pure bitcasts collapses back to that illegal form during
  optimization -- only avoiding the vector shape in the first place works.
- struct_type.cpp: move constant_bounds_test() (dead code -- declared and
  defined but never called since Halide's old internal-test runner was
  removed) into a proper correctness test.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- CodeGen_D3D12Compute_Dev: struct-backed groupshared/local arrays are
  stored as raw bytes, but Load/Store still consulted the original
  Type::Struct(...) for cast/promotion bookkeeping, crashing print_cast's
  internal_assert(source_type.is_uint()) since a struct is neither int,
  uint, nor float. Treat the storage element as Int(8) (matching what
  print_type_maybe_storage actually emits) and scale the groupshared
  array's declared element count by the struct's byte size.
- CodeGen_Vulkan_Dev: visit(Allocate) passed Type::Struct(...) directly
  to SpvBuilder::declare_type, which has no notion of struct types
  ("SPIRV: Unsupported type"). Use UInt(8) as the element type for
  declaration and later Load/Store bookkeeping instead.
- Serialization/Deserialization: add a Struct TypeCode and StructField
  table to the flatbuffers schema so struct types (and their field
  layout) round-trip instead of being rejected outright. This is needed
  because CI runs JIT compiles through a serialize/deserialize
  round-trip for regression testing.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Struct-backed allocations are declared as byte-addressable (UInt(8))
arrays in the Vulkan backend, since SPIR-V natively supports 8-bit
storage via SPV_KHR_8bit_storage. However, load_from_scalar_index()
and store_at_scalar_index() only ever accessed a single storage
element and relied on cast_type() (a numeric-conversion helper meant
for genuine Halide Cast nodes) to bridge to the field's real type.
For fields wider than one byte (e.g. a float32 struct field), this
silently read/wrote only 1 of the needed 4 bytes, and converted the
value numerically instead of bit-reinterpreting it, producing wrong
results (e.g. gpu_struct_type correctness test failures on
host-vulkan targets).

Add load_from_narrow_storage()/store_at_narrow_storage(), which detect
when a value's type is wider than the array's storage type and
combine/split the value across the correct number of consecutive
byte-sized storage elements via shift+bitwise-or (load) or
shift+truncate (store), reinterpreting (bitcasting) the combined bits
rather than numerically converting them. Narrower fields (e.g. int8)
are unaffected and continue to use the existing single-element path.

Verified via spirv-val/spirv-dis on the SPIR-V generated by the
gpu_struct_type correctness test (dumped via HL_SPIRV_DUMP_FILE) for
both the Register and GPUShared memory-space variants: the generated
module is valid and now uses OpBitcast to reinterpret the reassembled
bytes instead of OpConvertFToU/OpConvertUToF. Not yet validated by
running on real Vulkan hardware.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
HLSL has no native 8-bit integer type, so print_type_maybe_storage()
always emits a 32-bit `int` for any 8-bit-typed storage slot. This
means struct-backed arrays (used to back Halide struct types) are
declared with 32-bit-capacity "byte slots" in HLSL, even though each
slot conceptually represents one byte.

The scalar Store path performed a plain C-style assignment into these
slots, and the scalar Load path used print_cast() (a numeric-cast
helper intended for genuine Halide Cast nodes) to bridge back to the
field's real type. For fields whose "kind" differs from a plain
integer (notably float32 struct fields), this triggered numeric value
conversion rather than bit-level reinterpretation: storing 0.5f
truncated to 0, then converting 0 back to 0.0f -- exactly matching the
`result(1, 0) = 0.000000 instead of 0.500000` failures observed on the
host-d3d12compute and host-d3d12compute-hlsl_sm65 buildbot targets.

An existing reinterpret mechanism (shared_promotion_required /
promotion_str, using asint/asuint/asfloat via hlsl_reinterpret_name())
already existed for groupshared allocations when sm < 62, but was
gated too narrowly: it didn't apply to private/register-space
struct-backed allocations at all, and even for groupshared arrays the
Load side computed the promoted type from op->type instead of the
allocation's actual (struct) type, so it never triggered for a
Float(32) field load either.

Extend both the Store and Load paths so struct-backed allocations
(is_struct_backed) always use bit-level reinterpretation (forcing
promoted_type to Int(8), matching the emulated storage element) in any
memory space, not just groupshared with sm < 62.

Verified via HL_DEBUG_CODEGEN dumps of the generated HLSL for the
gpu_struct_type correctness test (host-d3d12compute and
host-d3d12compute-hlsl_sm65 targets, both the Register and GPUShared
memory-space variants, the latter via a temporary git-reverted reorder
of the test to reach it): stores now emit e.g. `_producer[0] =
asint(_3);` and loads emit `float(asfloat(_producer[0]))`, correctly
reinterpreting bits instead of converting values. Not yet validated by
running on real D3D12/HLSL hardware (Windows buildbot CI only).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@alexreinking
alexreinking force-pushed the alexreinking/struct-types branch from 011f44d to 5462034 Compare September 30, 2026 19:41
@abadams

abadams commented Oct 1, 2026

Copy link
Copy Markdown
Member

Nice to haves:

  • user-error earlier that lowering if you do something like add two structs
  • project field looks through tags so you can say select(x < 10, one_struct, likely(a_different_struct))

Far future:

  • The ability to user-define the semantics of various math ops on struct types (e.g. complex multiplication)

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants