feat: Implement se/dser method for HashTable by JkSelf · Pull Request #2160 · IBM/velox

JkSelf · 2026-06-19T00:14:56Z

Problem

HashTable currently does not support native serialization/deserialization.
To enable driver-side broadcast hash join in Gluten (similar to Spark), we need to serialize the pre-built hash table on the driver and deserialize it on executors.

Context

A HashTable cannot be restored correctly by serializing only the bucket contents. Its runtime behavior also depends on:

RowContainer row layout and stored row payloads,
VectorHasher state used by non-kHash probe paths,
optional bloom filter state,
hash-mode-specific rebuild behavior (kHash, kArray, kNormalizedKey).

Changes

Added native binary serde helpers in velox/common/serialization/NativeSerdeIO.h.
Added HashTable::serializedSize(), HashTable::serializeTo(...), and HashTable::deserializeFrom(...).
Serialized and restored:
- hash table metadata and versioning,
- key/dependent type information,
- row data and stored hashes,
- columnHasNulls_,
- VectorHasher state,
- normalized keys when present.
Added RowContainer::storeSerializedRow(std::string_view, char*) to restore rows directly from serialized row bytes.
Added VectorHasher state serialization/deserialization to preserve value-id/range state across processes.
Added binary serialization/deserialization for BigintValuesUsingBloomFilter.
Rebuilt the table during deserialization based on hash mode:
- reused serialized hashes for kHash / kNormalizedKey,
- recomputed value IDs for kArray to match restored VectorHasher state.
Added unit tests covering:
- round-trip serialization/deserialization,
- join probe after restore,
- multiple data types, nulls, long strings, duplicates, and large inputs,
- corrupted/invalid payload handling,
- cross-process serialization scenarios,
- VectorHasher state round-trip cases.

Testing

Added HashTableSerializationTest
Added VectorHasher serialization round-trip tests

JkSelf · 2026-06-19T00:20:17Z

alchemy merge

prestodb-ci · 2026-06-19T00:20:19Z

alchemy link ce74ff8

prestodb-ci · 2026-06-19T00:20:22Z

Added new rebase item:

ID 1651 velox/ibm/2160 — "feat: Implement se/dser method for HashTable"
- Group: default
- First introduced: 2026-06-19T00:00:06Z by @JkSelf
- Commits: ce74ff8
- Added: 2026-06-19T00:00:06Z by @prestodb-ci via feat: Implement se/dser method for HashTable #2160 (comment)
- Status: still effective

prestodb-ci · 2026-06-19T17:25:57Z

Failed to cherry-pick commit ce74ff8 in rebase request #2165:

exit status 1
error: could not apply ce74ff81d7... feat: Implement se/dser method for HashTable

Auto-merging velox/exec/CMakeLists.txt
Auto-merging velox/exec/tests/CMakeLists.txt
CONFLICT (content): Merge conflict in velox/exec/tests/CMakeLists.txt

Please:

Rebase your branch with staging/staging-rebase and fix the conflict. If the rebase item is a PR, you can change the base branch to this staging branch.

Comment on this issue with the updated rebase item.

For a PR (copy and edit):

alchemy merge @2026-06-19T11:32:13Z

For an issue (copy and edit):

alchemy link [updated comma-separated commit SHAs for this issue] @2026-06-19T11:32:13Z

Re-open Rebase branch staging-rebase (a93c9e8) with staging-rebase-head (1d05891) #2165 to retry the cherry-pick.

JkSelf · 2026-06-22T10:29:39Z

alchemy merge @2026-06-19T11:32:13Z

prestodb-ci · 2026-06-22T10:29:41Z

alchemy link ce74ff8 @2026-06-19T11:32:13Z

prestodb-ci · 2026-06-22T10:29:45Z

The following unexpired item was removed at 2026-06-19T11:32:13Z by @prestodb-ci via #2160 (comment):

ID 1651 velox/ibm/2160 — "feat: Implement se/dser method for HashTable"
- Group: default
- First introduced: 2026-06-19T00:00:06Z by @JkSelf
- Commits: ce74ff8
- Added: 2026-06-19T00:00:06Z by @prestodb-ci via feat: Implement se/dser method for HashTable #2160 (comment)
- Status: still effective

Added new rebase item:

ID 1661 velox/ibm/2160 — "feat: Implement se/dser method for HashTable"
- Group: default
- First introduced: 2026-06-19T00:00:06Z by @JkSelf
- Commits: ce74ff8
- Added: 2026-06-19T11:32:13Z by @prestodb-ci via feat: Implement se/dser method for HashTable #2160 (comment)
- Status: still effective

prestodb-ci · 2026-06-22T12:15:16Z

Failed to cherry-pick commit ce74ff8 in rebase request #2168:

exit status 1
error: could not apply ce74ff81d7... feat: Implement se/dser method for HashTable

Auto-merging velox/exec/CMakeLists.txt
Auto-merging velox/exec/tests/CMakeLists.txt
CONFLICT (content): Merge conflict in velox/exec/tests/CMakeLists.txt

Please:

Rebase your branch with staging/staging-rebase and fix the conflict. If the rebase item is a PR, you can change the base branch to this staging branch.

Comment on this issue with the updated rebase item.

For a PR (copy and edit):

alchemy merge @2026-06-20T05:43:02Z

For an issue (copy and edit):

alchemy link [updated comma-separated commit SHAs for this issue] @2026-06-20T05:43:02Z

Re-open Rebase branch staging-rebase (a93c9e8) with staging-rebase-head (7096ab4) #2168 to retry the cherry-pick.

JkSelf · 2026-06-22T13:52:25Z

alchemy merge @2026-06-20T05:43:02Z

prestodb-ci · 2026-06-22T13:52:27Z

alchemy link 2ca1ac8 @2026-06-20T05:43:02Z

prestodb-ci · 2026-06-22T13:52:31Z

The following unexpired item was removed at 2026-06-20T05:43:02Z by @prestodb-ci via #2160 (comment):

ID 1661 velox/ibm/2160 — "feat: Implement se/dser method for HashTable"
- Group: default
- First introduced: 2026-06-19T00:00:06Z by @JkSelf
- Commits: ce74ff8
- Added: 2026-06-19T11:32:13Z by @prestodb-ci via feat: Implement se/dser method for HashTable #2160 (comment)
- Status: still effective

Added new rebase item:

ID 1665 velox/ibm/2160 — "feat: Implement se/dser method for HashTable"
- Group: default
- First introduced: 2026-06-19T00:00:06Z by @JkSelf
- Commits: 2ca1ac8
- Added: 2026-06-20T05:43:02Z by @prestodb-ci via feat: Implement se/dser method for HashTable #2160 (comment)
- Status: still effective

prestodb-ci · 2026-06-23T07:15:20Z

Failed to cherry-pick commit 2ca1ac8 in rebase request #2174:

exit status 1
error: could not apply 2ca1ac80cb... feat: Implement se/dser method for HashTable

Auto-merging velox/exec/CMakeLists.txt
Auto-merging velox/exec/tests/CMakeLists.txt
CONFLICT (content): Merge conflict in velox/exec/tests/CMakeLists.txt

Please:

Rebase your branch with staging/staging-rebase and fix the conflict. If the rebase item is a PR, you can change the base branch to this staging branch.

Comment on this issue with the updated rebase item.

For a PR (copy and edit):

alchemy merge @2026-06-23T06:49:46Z

For an issue (copy and edit):

alchemy link [updated comma-separated commit SHAs for this issue] @2026-06-23T06:49:46Z

Re-open Rebase branch staging-rebase (a93c9e8) with staging-rebase-head (5a397b2) #2174 to retry the cherry-pick.

JkSelf · 2026-06-23T07:22:27Z

alchemy merge @2026-06-23T06:49:46Z

prestodb-ci · 2026-06-23T07:22:30Z

alchemy link 5624f6e @2026-06-23T06:49:46Z

prestodb-ci · 2026-06-23T07:22:33Z

The following unexpired item was removed at 2026-06-23T06:49:46Z by @prestodb-ci via #2160 (comment):

ID 1665 velox/ibm/2160 — "feat: Implement se/dser method for HashTable"
- Group: default
- First introduced: 2026-06-19T00:00:06Z by @JkSelf
- Commits: 2ca1ac8
- Added: 2026-06-20T05:43:02Z by @prestodb-ci via feat: Implement se/dser method for HashTable #2160 (comment)
- Status: still effective

Added new rebase item:

ID 1673 velox/ibm/2160 — "feat: Implement se/dser method for HashTable"
- Group: default
- First introduced: 2026-06-19T00:00:06Z by @JkSelf
- Commits: 5624f6e
- Added: 2026-06-23T06:49:46Z by @prestodb-ci via feat: Implement se/dser method for HashTable #2160 (comment)
- Status: still effective

JkSelf requested a review from majetideepak as a code owner June 19, 2026 00:14

prestodb-ci added Merged rebase-item labels Jun 19, 2026

prestodb-ci added the rebase-conflict label Jun 19, 2026

prestodb-ci mentioned this pull request Jun 19, 2026

Rebase branch staging-rebase (a93c9e8) with staging-rebase-head (1d05891) #2165

Closed

1 task

prestodb-ci removed the rebase-conflict label Jun 22, 2026

prestodb-ci added the rebase-conflict label Jun 22, 2026

prestodb-ci mentioned this pull request Jun 22, 2026

Rebase branch staging-rebase (a93c9e8) with staging-rebase-head (7096ab4) #2168

Closed

1 task

JkSelf force-pushed the serialize-hashtable branch from ce74ff8 to 2ca1ac8 Compare June 22, 2026 13:51

prestodb-ci removed the rebase-conflict label Jun 22, 2026

prestodb-ci added the rebase-conflict label Jun 23, 2026

prestodb-ci mentioned this pull request Jun 23, 2026

Rebase branch staging-rebase (a93c9e8) with staging-rebase-head (5a397b2) #2174

Closed

1 task

feat: Implement se/dser method for HashTable

5624f6e

JkSelf force-pushed the serialize-hashtable branch from 2ca1ac8 to 5624f6e Compare June 23, 2026 07:22

prestodb-ci removed the rebase-conflict label Jun 23, 2026

Uh oh!

Conversation

JkSelf commented Jun 19, 2026

Problem

Context

Changes

Testing

Uh oh!

JkSelf commented Jun 19, 2026

Uh oh!

prestodb-ci commented Jun 19, 2026

Uh oh!

prestodb-ci commented Jun 19, 2026

Uh oh!

prestodb-ci commented Jun 19, 2026

Uh oh!

JkSelf commented Jun 22, 2026

Uh oh!

prestodb-ci commented Jun 22, 2026

Uh oh!

prestodb-ci commented Jun 22, 2026

Uh oh!

prestodb-ci commented Jun 22, 2026

Uh oh!

JkSelf commented Jun 22, 2026

Uh oh!

prestodb-ci commented Jun 22, 2026

Uh oh!

prestodb-ci commented Jun 22, 2026

Uh oh!

prestodb-ci commented Jun 23, 2026

Uh oh!

JkSelf commented Jun 23, 2026

Uh oh!

prestodb-ci commented Jun 23, 2026

Uh oh!

prestodb-ci commented Jun 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants