[VPlan] Insert VPBlendRecipes in post order. NFC#201782
Merged
lukel97 merged 6 commits intoJun 10, 2026
Merged
Conversation
|
@llvm/pr-subscribers-llvm-transforms Author: Luke Lau (lukel97) ChangesAn upcoming PR wants to optimize blend masks by peeking through the contents of other phi nodes. Currently we eagerly convert phis to blends in reverse post order, so switch it to post order so that phis at the bottom can see the phis in their uses. Full diff: https://github.com/llvm/llvm-project/pull/201782.diff 1 Files Affected:
diff --git a/llvm/lib/Transforms/Vectorize/VPlanPredicator.cpp b/llvm/lib/Transforms/Vectorize/VPlanPredicator.cpp
index 2717b80e2eeaa..b01fef556f8ed 100644
--- a/llvm/lib/Transforms/Vectorize/VPlanPredicator.cpp
+++ b/llvm/lib/Transforms/Vectorize/VPlanPredicator.cpp
@@ -69,6 +69,8 @@ class VPPredicator {
return EdgeMaskCache[{Src, Dst}] = Mask;
}
+ DenseMap<const VPBasicBlock *, VPBasicBlock::iterator> InsertPoints;
+
public:
VPPredicator(VPlan &Plan) : VPDT(Plan), VPPDT(Plan) {}
@@ -136,6 +138,10 @@ void VPPredicator::createBlockInMask(VPBasicBlock *VPBB) {
// Start inserting after the block's phis, which be replaced by blends later.
Builder.setInsertPoint(VPBB, VPBB->getFirstNonPhi());
+ // Keep track of where in VPBB we are inserting the masks into.
+ scope_exit UpdateInsertPoint(
+ [this, &VPBB]() { InsertPoints[VPBB] = Builder.getInsertPoint(); });
+
// Reuse the mask of the immediate dominator if the VPBB post-dominates the
// immediate dominator.
auto *IDom = VPDT.getNode(VPBB)->getIDom();
@@ -225,6 +231,7 @@ void VPPredicator::createSwitchEdgeMasks(const VPInstruction *SI) {
}
void VPPredicator::convertPhisToBlends(VPBasicBlock *VPBB) {
+ Builder.setInsertPoint(VPBB, InsertPoints[VPBB]);
SmallVector<VPPhi *> Phis;
for (VPRecipeBase &R : VPBB->phis())
Phis.push_back(cast<VPPhi>(&R));
@@ -276,10 +283,8 @@ void VPlanTransforms::introduceMasksAndLinearize(VPlan &Plan) {
// Introduce the mask for VPBB, which may introduce needed edge masks, and
// convert all phi recipes of VPBB to blend recipes unless VPBB is the
// header.
- if (VPBB != Header) {
+ if (VPBB != Header)
Predicator.createBlockInMask(VPBB);
- Predicator.convertPhisToBlends(VPBB);
- }
VPValue *BlockMask = Predicator.getBlockInMask(VPBB);
if (!BlockMask)
@@ -292,6 +297,10 @@ void VPlanTransforms::introduceMasksAndLinearize(VPlan &Plan) {
}
}
+ for (VPBlockBase *VPB : reverse(RPOT))
+ if (VPB != Header)
+ Predicator.convertPhisToBlends(cast<VPBasicBlock>(VPB));
+
// Linearize the blocks of the loop into one serial chain.
VPBlockBase *PrevVPBB = nullptr;
for (VPBasicBlock *VPBB : VPBlockUtils::blocksOnly<VPBasicBlock>(RPOT)) {
|
|
@llvm/pr-subscribers-vectorizers Author: Luke Lau (lukel97) ChangesAn upcoming PR wants to optimize blend masks by peeking through the contents of other phi nodes. Currently we eagerly convert phis to blends in reverse post order, so switch it to post order so that phis at the bottom can see the phis in their uses. Full diff: https://github.com/llvm/llvm-project/pull/201782.diff 1 Files Affected:
diff --git a/llvm/lib/Transforms/Vectorize/VPlanPredicator.cpp b/llvm/lib/Transforms/Vectorize/VPlanPredicator.cpp
index 2717b80e2eeaa..b01fef556f8ed 100644
--- a/llvm/lib/Transforms/Vectorize/VPlanPredicator.cpp
+++ b/llvm/lib/Transforms/Vectorize/VPlanPredicator.cpp
@@ -69,6 +69,8 @@ class VPPredicator {
return EdgeMaskCache[{Src, Dst}] = Mask;
}
+ DenseMap<const VPBasicBlock *, VPBasicBlock::iterator> InsertPoints;
+
public:
VPPredicator(VPlan &Plan) : VPDT(Plan), VPPDT(Plan) {}
@@ -136,6 +138,10 @@ void VPPredicator::createBlockInMask(VPBasicBlock *VPBB) {
// Start inserting after the block's phis, which be replaced by blends later.
Builder.setInsertPoint(VPBB, VPBB->getFirstNonPhi());
+ // Keep track of where in VPBB we are inserting the masks into.
+ scope_exit UpdateInsertPoint(
+ [this, &VPBB]() { InsertPoints[VPBB] = Builder.getInsertPoint(); });
+
// Reuse the mask of the immediate dominator if the VPBB post-dominates the
// immediate dominator.
auto *IDom = VPDT.getNode(VPBB)->getIDom();
@@ -225,6 +231,7 @@ void VPPredicator::createSwitchEdgeMasks(const VPInstruction *SI) {
}
void VPPredicator::convertPhisToBlends(VPBasicBlock *VPBB) {
+ Builder.setInsertPoint(VPBB, InsertPoints[VPBB]);
SmallVector<VPPhi *> Phis;
for (VPRecipeBase &R : VPBB->phis())
Phis.push_back(cast<VPPhi>(&R));
@@ -276,10 +283,8 @@ void VPlanTransforms::introduceMasksAndLinearize(VPlan &Plan) {
// Introduce the mask for VPBB, which may introduce needed edge masks, and
// convert all phi recipes of VPBB to blend recipes unless VPBB is the
// header.
- if (VPBB != Header) {
+ if (VPBB != Header)
Predicator.createBlockInMask(VPBB);
- Predicator.convertPhisToBlends(VPBB);
- }
VPValue *BlockMask = Predicator.getBlockInMask(VPBB);
if (!BlockMask)
@@ -292,6 +297,10 @@ void VPlanTransforms::introduceMasksAndLinearize(VPlan &Plan) {
}
}
+ for (VPBlockBase *VPB : reverse(RPOT))
+ if (VPB != Header)
+ Predicator.convertPhisToBlends(cast<VPBasicBlock>(VPB));
+
// Linearize the blocks of the loop into one serial chain.
VPBlockBase *PrevVPBB = nullptr;
for (VPBasicBlock *VPBB : VPBlockUtils::blocksOnly<VPBasicBlock>(RPOT)) {
|
🐧 Linux x64 Test Results
Failed Tests(click on a test name to see its output) FlangFlang.Driver/omp-driver-offload.f90If these failures are unrelated to your changes (for example tests are broken or flaky at HEAD), please open an issue at https://github.com/llvm/llvm-project/issues and add the |
david-arm
reviewed
Jun 5, 2026
fhahn
reviewed
Jun 7, 2026
fhahn
left a comment
fhahn
left a comment
Contributor
There was a problem hiding this comment.
I think at least on the VPlan-level, there may be some changes with the patch.
Something like below in the predicator.ll VPlan test should have the generated blends changed. For this one, everything gets folded and there's no IR change, not sure if there's any way to construct a test case with end-to-end changes, but may be god to add the test.
define void @blend_chain_non_trivial(ptr noalias %a, ptr noalias %b) {
entry:
br label %loop.header
loop.header:
%iv = phi i64 [ 0, %entry ], [ %iv.next, %loop.latch ]
%lb = load i64, ptr %b
%v1 = add i64 %iv, %lb
%v2 = mul i64 %iv, 3
%gep = getelementptr i64, ptr %a, i64 %iv
%c0 = icmp sle i64 %iv, 0
br i1 %c0, label %if.a, label %merge.a
if.a:
%ca = icmp sle i64 %iv, 8
br i1 %ca, label %if.a.inner, label %merge.a.inner
if.a.inner:
br label %merge.a.inner
merge.a.inner:
%blend.a.inner = phi i64 [ %v1, %if.a ], [ %v1, %if.a.inner ]
br label %merge.a
merge.a:
%blend.a = phi i64 [ %v1, %loop.header ], [ %blend.a.inner, %merge.a.inner ]
%d0 = icmp sgt i64 %iv, 0
br i1 %d0, label %if.b, label %merge.b
if.b:
%cb = icmp sle i64 %iv, 16
br i1 %cb, label %if.b.inner, label %merge.b.inner
if.b.inner:
br label %merge.b.inner
merge.b.inner:
%blend.b.inner = phi i64 [ %v2, %if.b ], [ %v2, %if.b.inner ]
br label %merge.b
merge.b:
%blend.b = phi i64 [ %v2, %merge.a ], [ %blend.b.inner, %merge.b.inner ]
%sum = add i64 %blend.a, %blend.b
store i64 %sum, ptr %gep
br label %loop.latch
loop.latch:
%iv.next = add nuw nsw i64 %iv, 1
%ec = icmp eq i64 %iv.next, 128
br i1 %ec, label %exit, label %loop.header
exit:
ret void
}
| scope_exit UpdateInsertPoint([this, &VPBB]() { | ||
| assert(!InsertPoints.contains(VPBB) && "InsertPoint clobbered?"); | ||
| InsertPoints[VPBB] = Builder.getInsertPoint(); | ||
| }); |
Contributor
There was a problem hiding this comment.
do we need the scope_exit? I think we don't reset the inert point, and only insert before, so it should not change
Contributor
Author
There was a problem hiding this comment.
We don't reset the insert point but it changes when new edge masks or block in-masks are created. Although I think we can just recalculate the insert point based off of the block-in mask. I've removed the InsertPoints map in 73427ee
lukel97
force-pushed
the
loop-vectorize/insert-blends-post-order
branch
from
e6431bc to
73427ee
Compare
Contributor
Author
For the cases where all the incoming values are equal, yeah there should be no end-to-end change because #201783 will restore the VPlan-level changes at it will be able to see when all incoming values are equal through nested phis. Added the test in f07305e |
Contributor
|
I guess this is fine for now, so I definitely wouldn't want to block it, but I have a feeling RPOT would be better if we ever try to preserve some "uniform" branches as that would require placing SSA-phis sometimes. |
Contributor
Author
Do you mean as in for partial control flow linearisation? I don't think we need to insert new phis, only adjust existing ones by replacing the incoming values with blends where control flow was linearized. Is there any reason why post order wouldn't work for that? |
Mel-Chen
approved these changes
Jun 9, 2026
Contributor
Into One could argue that we can go BB1->BB2->BB3, but it's dictated by the RPOT and you can simply mirror image the CFG to still have the linearization order where BB2 is processed last. |
Contributor
Author
Can we not do this processing in post order? I.e. after linearization (no new blocks are created AFAIK): We would visit MergeBB and see the original phi. E1 + E2 don't dominate MergeBB so we would decide to insert a phi in BB2, and then replace the original phi with a blend. The incoming values are the new phi and BB2's incoming value, blended. |
lukel97
force-pushed
the
loop-vectorize/insert-blends-post-order
branch
from
73427ee to
e40766f
Compare
Jianhui-Li
pushed a commit
to Jianhui-Li/llvm-project
that referenced
this pull request
Jun 11, 2026
llvm#201783 wants to optimize blend masks by peeking through the contents of other phi nodes. Currently we eagerly convert phis to blends in reverse post order, so switch it to post order so that phis at the bottom can see the phis in their uses.
carlobertolli
pushed a commit
to carlobertolli/llvm-project
that referenced
this pull request
Jun 11, 2026
llvm#201783 wants to optimize blend masks by peeking through the contents of other phi nodes. Currently we eagerly convert phis to blends in reverse post order, so switch it to post order so that phis at the bottom can see the phis in their uses.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
#201783 wants to optimize blend masks by peeking through the contents of other phi nodes. Currently we eagerly convert phis to blends in reverse post order, so switch it to post order so that phis at the bottom can see the phis in their uses.