Is your feature request related to a problem? Please describe.
Native joins need a reusable way to evaluate conditions beyond equality keys. For example:
SELECT *
FROM l JOIN r
ON l.key = r.key AND l.value < r.value;
Here, equality keys identify candidate pairs, while l.value < r.value determines which pairs actually match.
Evaluating these conditions by materializing all candidate pairs can create large intermediate arrays, especially with duplicate keys. Correct evaluation must also preserve null handling, conditional expression evaluation, and columns needed by the condition but excluded from the output.
Describe the solution you'd like
Introduce shared infrastructure for CPU-based native join condition evaluation:
- Evaluate conditions over bounded candidate batches.
- Avoid unnecessary expansion of deterministic expressions that reference only one input.
- Preserve lazy evaluation and volatile-expression semantics.
- Support column pruning while retaining condition-only input columns.
- Reuse the existing join-filter representation without changing the protobuf contract.
This issue covers the shared evaluator and supporting utilities. Integration into individual join operators will follow separately.
Describe alternatives you've considered
- Continue falling back to Spark for unsupported join conditions.
- Materialize candidate pairs before evaluating each condition, at the cost of potentially large intermediate allocations.
- Implement condition evaluation separately in each join operator, duplicating expression handling and projection logic.
Additional context
This provides the foundation for extending residual-condition support across native sort-merge, broadcast hash, shuffled hash, and broadcast nested-loop joins.
Tests should cover nulls, conditional and volatile expressions, column remapping, projection, and candidate groups spanning multiple batches.
Is your feature request related to a problem? Please describe.
Native joins need a reusable way to evaluate conditions beyond equality keys. For example:
Here, equality keys identify candidate pairs, while
l.value < r.valuedetermines which pairs actually match.Evaluating these conditions by materializing all candidate pairs can create large intermediate arrays, especially with duplicate keys. Correct evaluation must also preserve null handling, conditional expression evaluation, and columns needed by the condition but excluded from the output.
Describe the solution you'd like
Introduce shared infrastructure for CPU-based native join condition evaluation:
This issue covers the shared evaluator and supporting utilities. Integration into individual join operators will follow separately.
Describe alternatives you've considered
Additional context
This provides the foundation for extending residual-condition support across native sort-merge, broadcast hash, shuffled hash, and broadcast nested-loop joins.
Tests should cover nulls, conditional and volatile expressions, column remapping, projection, and candidate groups spanning multiple batches.