Skip to content

Add shared CPU evaluation for native non-equal join conditions #2536

Description

@Deegue

Is your feature request related to a problem? Please describe.
Native joins need a reusable way to evaluate conditions beyond equality keys. For example:

SELECT *
FROM l JOIN r
ON l.key = r.key AND l.value < r.value;

Here, equality keys identify candidate pairs, while l.value < r.value determines which pairs actually match.
Evaluating these conditions by materializing all candidate pairs can create large intermediate arrays, especially with duplicate keys. Correct evaluation must also preserve null handling, conditional expression evaluation, and columns needed by the condition but excluded from the output.

Describe the solution you'd like
Introduce shared infrastructure for CPU-based native join condition evaluation:

  • Evaluate conditions over bounded candidate batches.
  • Avoid unnecessary expansion of deterministic expressions that reference only one input.
  • Preserve lazy evaluation and volatile-expression semantics.
  • Support column pruning while retaining condition-only input columns.
  • Reuse the existing join-filter representation without changing the protobuf contract.
    This issue covers the shared evaluator and supporting utilities. Integration into individual join operators will follow separately.

Describe alternatives you've considered

  • Continue falling back to Spark for unsupported join conditions.
  • Materialize candidate pairs before evaluating each condition, at the cost of potentially large intermediate allocations.
  • Implement condition evaluation separately in each join operator, duplicating expression handling and projection logic.

Additional context
This provides the foundation for extending residual-condition support across native sort-merge, broadcast hash, shuffled hash, and broadcast nested-loop joins.
Tests should cover nulls, conditional and volatile expressions, column remapping, projection, and candidate groups spanning multiple batches.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions