Optimized PartitionedOutput staging hub by yingsu00 · Pull Request #1799 · IBM/velox

yingsu00 · 2026-03-11T14:56:04Z

This PR is to track the commits for the optimized PartitionedOutput operator. Related new PRs shall be submitted to optimized_partitionedoutput branch, and then alchemy merge to oss-main.

Review history of the first two commits can be found in
feat: Introducing PartitionedVector #1596
feat: Add PartitionedRowVector implementation #1770

yingsu00 · 2026-03-13T05:47:02Z

alchemy link 86db93b

prestodb-ci · 2026-03-13T05:47:07Z

Added new rebase item:

ID 1176 [velox/ibm/1799] "Optimized PartitionedOutput staging hub" first introduced at 2026-03-10T12:09:19Z owned by @xin-zhang2 with commits 86db93b - added at 2026-03-10T12:09:19Z by @yingsu00 via Optimized PartitionedOutput staging hub #1799 (comment), still effective

yingsu00 · 2026-03-13T05:47:54Z

alchemy link 6dd3661

prestodb-ci · 2026-03-13T05:47:58Z

The following unexpired item was removed at 2026-03-10T12:01:55Z by @yingsu00 via #1799 (comment):

ID 1176 [velox/ibm/1799] "Optimized PartitionedOutput staging hub" first introduced at 2026-03-10T12:09:19Z owned by @xin-zhang2 with commits 86db93b - added at 2026-03-10T12:09:19Z by @yingsu00 via Optimized PartitionedOutput staging hub #1799 (comment), still effective

Added new rebase item:

ID 1177 [velox/ibm/1799] "Optimized PartitionedOutput staging hub" first introduced at 2026-03-10T12:09:19Z owned by @xin-zhang2 with commits 6dd3661 - added at 2026-03-10T12:01:55Z by @yingsu00 via Optimized PartitionedOutput staging hub #1799 (comment), still effective

yingsu00 · 2026-03-13T05:50:34Z

alchemy link 2706c1e

prestodb-ci · 2026-03-13T05:50:38Z

The following unexpired item was removed at 2026-03-12T13:19:10Z by @yingsu00 via #1799 (comment):

ID 1177 [velox/ibm/1799] "Optimized PartitionedOutput staging hub" first introduced at 2026-03-10T12:09:19Z owned by @xin-zhang2 with commits 6dd3661 - added at 2026-03-10T12:01:55Z by @yingsu00 via Optimized PartitionedOutput staging hub #1799 (comment), still effective

Added new rebase item:

ID 1178 [velox/ibm/1799] "Optimized PartitionedOutput staging hub" first introduced at 2026-03-10T12:09:19Z owned by @yingsu00 with commits 2706c1e - added at 2026-03-12T13:19:10Z by @yingsu00 via Optimized PartitionedOutput staging hub #1799 (comment), still effective

unidevel · 2026-03-13T06:47:41Z

alchemy link 86db93b,6dd3661c7afef52c42ed1c5ca83c9e57e21ec2b3,2706c1e80f9463bd4fdd805e296839f964437ca3

prestodb-ci · 2026-03-13T06:47:45Z

Failed to add new rebase item:

ID (unassigned) [velox/ibm/1799] "Optimized PartitionedOutput staging hub" first introduced at 2026-03-10T12:09:19Z owned by @xin-zhang2 with commits 86db93b 6dd3661 2706c1e - added at 2026-03-10T12:09:19Z by @unidevel via Optimized PartitionedOutput staging hub #1799 (comment), still effective

The new rebase item overlaps with the following existing item:

ID 1177 [velox/ibm/1799] "Optimized PartitionedOutput staging hub" first introduced at 2026-03-10T12:09:19Z owned by @xin-zhang2 with commits 6dd3661 - added at 2026-03-10T12:01:55Z by @yingsu00 via Optimized PartitionedOutput staging hub #1799 (comment), removed at 2026-03-12T13:19:10Z by @yingsu00 via Optimized PartitionedOutput staging hub #1799 (comment)

Please double check your input and retry.

unidevel · 2026-03-13T06:48:08Z

alchemy link 86db93b,6dd3661c7afef52c42ed1c5ca83c9e57e21ec2b3,2706c1e80f9463bd4fdd805e296839f964437ca3 @2026-03-10T12:01:55Z

prestodb-ci · 2026-03-13T06:48:12Z

Failed to add new rebase item:

ID (unassigned) [velox/ibm/1799] "Optimized PartitionedOutput staging hub" first introduced at 2026-03-10T12:09:19Z owned by @xin-zhang2 with commits 86db93b 6dd3661 2706c1e - added at 2026-03-10T12:01:55Z by @unidevel via Optimized PartitionedOutput staging hub #1799 (comment), still effective

The new rebase item overlaps with the following existing item:

ID 1177 [velox/ibm/1799] "Optimized PartitionedOutput staging hub" first introduced at 2026-03-10T12:09:19Z owned by @xin-zhang2 with commits 6dd3661 - added at 2026-03-10T12:01:55Z by @yingsu00 via Optimized PartitionedOutput staging hub #1799 (comment), removed at 2026-03-12T13:19:10Z by @yingsu00 via Optimized PartitionedOutput staging hub #1799 (comment)

Please double check your input and retry.

unidevel · 2026-03-13T06:48:44Z

alchemy link 86db93b,6dd3661c7afef52c42ed1c5ca83c9e57e21ec2b3,2706c1e80f9463bd4fdd805e296839f964437ca3 @2026-03-12T13:19:10Z

prestodb-ci · 2026-03-13T06:48:48Z

The following unexpired item was removed at 2026-03-12T13:19:10Z by @unidevel via #1799 (comment):

ID 1178 [velox/ibm/1799] "Optimized PartitionedOutput staging hub" first introduced at 2026-03-10T12:09:19Z owned by @yingsu00 with commits 2706c1e - added at 2026-03-12T13:19:10Z by @yingsu00 via Optimized PartitionedOutput staging hub #1799 (comment), still effective

Added new rebase item:

ID 1179 [velox/ibm/1799] "Optimized PartitionedOutput staging hub" first introduced at 2026-03-10T12:09:19Z owned by @xin-zhang2 with commits 86db93b 6dd3661 2706c1e - added at 2026-03-12T13:19:10Z by @unidevel via Optimized PartitionedOutput staging hub #1799 (comment), still effective

prestodb-ci · 2026-06-25T18:46:20Z

Failed to cherry-pick commit 546d0bd in rebase request #2189:

exit status 1
error: could not apply 546d0bdf1... feat(PartitionedOutput): Add OptimizedPartitionedOutput operator

Auto-merging velox/core/QueryConfig.h
CONFLICT (content): Merge conflict in velox/core/QueryConfig.h
Auto-merging velox/exec/CMakeLists.txt
Auto-merging velox/exec/LocalPlanner.cpp

Please:

Rebase your branch with staging/staging-rebase and fix the conflict. If the rebase item is a PR, you can change the base branch to this staging branch.

Comment on this issue with the updated rebase item.

For a PR (copy and edit):

alchemy merge @2026-06-25T18:39:48Z

For an issue (copy and edit):

alchemy link [updated comma-separated commit SHAs for this issue] @2026-06-25T18:39:48Z

Re-open Rebase branch staging-rebase (a4c9d0c) with staging-rebase-head (1a4d124) #2189 to retry the cherry-pick.

This commit introduces OptimizedPartitionedOutput, a PartitionedOutput operator backed by PrestoIterativePartitioningSerializer. Enabled via query config key "optimized_repartitioning" (default off). LocalPlanner selects it over the standard PartitionedOutput when the flag is set. TODO: replicateNullsAndAny is not yet supported and raises a user error.

…geBenchmark - Added normal vs optimized PartitionedOutput comparison by running each exchange case twice with kOptimizedPartitionedOutputEnabled=false/true. - Added per-mode benchmark names: - exchange<Case>_normalPartitionedOutput - exchange<Case>_optimizedPartitionedOutput in ExchangeBenchmark.cpp. - Refactored result printing into shared helpers and fixed output consistency in ExchangeBenchmark.cpp.

xin-zhang2 · 2026-06-25T20:24:29Z

alchemy merge @2026-06-25T18:39:48Z

prestodb-ci · 2026-06-25T20:24:31Z

alchemy link 23e8d1d,7df0be41dac449670584be5336587a1d3df135bf,5719f909a173ce08e483b900b6a2811da82e112b,fad80645718f89d4b3598859c2b51c2ebfd9e43e,04fbad0a3b5d14d5dc890ae4e05cf35f715ae194,c3a52c9e9b46e6c6abb3d40b7f978bfac833d9eb,9efe82a5fb4e13a348573cb4d4f893541cea47d0,ff888e7bd593f0a909bc75b0b7213b3200e968b1,1511c01f8c5e7b36c77f52a0c8a7a558d3c99676,e567d50501a13f650c8d25cc18d045379631b1dc,ee25fa7738c87a1da498e4ff679cd2fa8abd8684 @2026-06-25T18:39:48Z

prestodb-ci · 2026-06-25T20:24:37Z

The following unexpired item was removed at 2026-06-25T18:39:48Z by @prestodb-ci via #1799 (comment):

ID 1596 velox/ibm/1799 — "Optimized PartitionedOutput staging hub"
- Group: default
- First introduced: 2026-03-10T12:09:19Z by @yingsu00
- Commits: 2c59ee6 71705c7 3d9e709 d787419 652dd0a 28c45bd 59b321a b27c492 48018b3 546d0bd 974bb09
- Added: 2026-04-17T05:56:57Z by @prestodb-ci via Optimized PartitionedOutput staging hub #1799 (comment)
- Status: still effective

Added new rebase item:

ID 1682 velox/ibm/1799 — "Optimized PartitionedOutput staging hub"
- Group: default
- First introduced: 2026-03-10T12:09:19Z by @yingsu00
- Commits: 23e8d1d 7df0be4 5719f90 fad8064 04fbad0 c3a52c9 9efe82a ff888e7 1511c01 e567d50 ee25fa7
- Added: 2026-06-25T18:39:48Z by @prestodb-ci via Optimized PartitionedOutput staging hub #1799 (comment)
- Status: still effective

yingsu00 requested a review from majetideepak as a code owner March 11, 2026 14:56

yingsu00 removed the request for review from majetideepak March 11, 2026 14:56

prestodb-ci added Merged rebase-item labels Mar 13, 2026

prestodb-ci mentioned this pull request Jun 25, 2026

Rebase branch staging-rebase (a4c9d0c) with staging-rebase-head (1a4d124) #2189

Closed

1 task

yingsu00 added 2 commits June 25, 2026 21:20

xin-zhang2 force-pushed the optimized_partitionedoutput branch from 974bb09 to ee25fa7 Compare June 25, 2026 20:22

xin-zhang2 requested review from jinchengchenghh and rui-mo as code owners June 25, 2026 20:22

github-actions Bot added the cudf label Jun 25, 2026

xin-zhang2 changed the base branch from staging/staging-rebase-20260611231810 to staging/staging-rebase-optimized_partitionedoutput June 25, 2026 20:23

xin-zhang2 removed request for jinchengchenghh and rui-mo June 25, 2026 20:24

xin-zhang2 removed the cudf label Jun 25, 2026

prestodb-ci removed the rebase-conflict label Jun 25, 2026

Uh oh!

Conversation

yingsu00 commented Mar 11, 2026 • edited by xin-zhang2 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yingsu00 commented Mar 13, 2026

Uh oh!

prestodb-ci commented Mar 13, 2026

Uh oh!

yingsu00 commented Mar 13, 2026

Uh oh!

prestodb-ci commented Mar 13, 2026

Uh oh!

yingsu00 commented Mar 13, 2026

Uh oh!

prestodb-ci commented Mar 13, 2026

Uh oh!

unidevel commented Mar 13, 2026

Uh oh!

prestodb-ci commented Mar 13, 2026

Uh oh!

unidevel commented Mar 13, 2026

Uh oh!

prestodb-ci commented Mar 13, 2026

Uh oh!

unidevel commented Mar 13, 2026

Uh oh!

prestodb-ci commented Mar 13, 2026

Uh oh!

prestodb-ci commented Jun 25, 2026

Uh oh!

xin-zhang2 commented Jun 25, 2026

Uh oh!

prestodb-ci commented Jun 25, 2026

Uh oh!

prestodb-ci commented Jun 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

yingsu00 commented Mar 11, 2026 •

edited by xin-zhang2

Loading