Summary
The set operation types table specifies the output multiplicity for every SetRel operation except MINUS_MULTISET, leaving it ambiguous whether the output is deduplicated or retains the primary input's multiplicity.
Details
For the other variants the multiplicity rule is explicit:
- Minus (Primary) — "removing duplicates" (distinct output).
- Minus (Primary All) — output contains
max(0, m − sum(n1, n2, …, n)) copies.
- Intersection (Primary) — "removing duplicates".
- Intersection (Multiset) — "removing duplicates".
- Intersection (Multiset All) — output contains
min(m, n1, n2, …, n) copies.
- Union Distinct / Union All — distinct /
m + n1 + … + n copies respectively.
Minus (Multiset), however, only states:
Returns all records from the primary input excluding any records that are included in all secondary inputs. This operation does not have a direct SQL mapping.
with the example p:{1,2,3,4}, s1:{1,2}, s2:{1,2,3} → {3,4} (no duplicates in the primary, so the example doesn't disambiguate). There's no "removing duplicates" statement and no copy-count formula, so it's unclear what the output should be when the primary contains duplicates of a retained record — e.g. p:{3,3,4}, s1:{1,2}, s2:{1,2,3}: is the result {3,3,4} (retain primary multiplicity) or {3,4} (deduplicated)?
Why it matters
This ambiguity blocks a faithful conversion of MINUS_MULTISET in downstream producers/consumers that don't have a native equivalent. Concretely, in substrait-java's Isthmus (Calcite) integration, MINUS_MULTISET can be expressed as Minus(primary, Intersect(secondaries)), but the all flags on those Calcite relations can't be chosen without knowing the intended output multiplicity. See substrait-io/substrait-java#999.
Ask
Please specify the output multiplicity rule for SET_OP_MINUS_MULTISET in the set operation types table (either a "removing duplicates" statement or an explicit copy-count formula, consistent with the other variants).
🤖 Generated with AI
Summary
The set operation types table specifies the output multiplicity for every
SetReloperation exceptMINUS_MULTISET, leaving it ambiguous whether the output is deduplicated or retains the primary input's multiplicity.Details
For the other variants the multiplicity rule is explicit:
max(0, m − sum(n1, n2, …, n))copies.min(m, n1, n2, …, n)copies.m + n1 + … + ncopies respectively.Minus (Multiset), however, only states:
with the example
p:{1,2,3,4},s1:{1,2},s2:{1,2,3}→{3,4}(no duplicates in the primary, so the example doesn't disambiguate). There's no "removing duplicates" statement and no copy-count formula, so it's unclear what the output should be when the primary contains duplicates of a retained record — e.g.p:{3,3,4},s1:{1,2},s2:{1,2,3}: is the result{3,3,4}(retain primary multiplicity) or{3,4}(deduplicated)?Why it matters
This ambiguity blocks a faithful conversion of
MINUS_MULTISETin downstream producers/consumers that don't have a native equivalent. Concretely, insubstrait-java's Isthmus (Calcite) integration,MINUS_MULTISETcan be expressed asMinus(primary, Intersect(secondaries)), but theallflags on those Calcite relations can't be chosen without knowing the intended output multiplicity. See substrait-io/substrait-java#999.Ask
Please specify the output multiplicity rule for
SET_OP_MINUS_MULTISETin the set operation types table (either a "removing duplicates" statement or an explicit copy-count formula, consistent with the other variants).🤖 Generated with AI