You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
SQL lowering: run DataFusion's SimplifyExpressions (needs IR timestamp literals and regex operators) #650
The SQL front end lowers DataFusion's plan right after type coercion and does not run DataFusion's expression simplification (SimplifyExpressions). A DataFusion query reaches its executor simplified, and ours does not, so the lowered IR can differ from what DataFusion would execute. Today only one piece is matched by hand: #539 folds a numeric cast of a literal (latency < CAST(100 AS DOUBLE) → latency < 100.0) with DataFusion's ScalarValue::cast_to.
Running the whole rule (tried on the #646 restack, 2026-10-10) breaks the SQL corpora (bgp_analytics, bgp_jan2024_workload, netflow, tpch_deequ) in two ways:
Timestamp literals.time >= CAST('2018-05-17 13:10:01' AS TIMESTAMP) folds to TimestampMillisecond(1526562601000). The IR has no timestamp literal, so lowering fails with unsupported scalar. Nearly every BGP and NetFlow query has a time filter.
Regex.REGEXP_LIKE(col, pat) is rewritten into the operator col ~ pat (RegexMatch), which lowering does not support (unsupported feature: operator: RegexMatch).
The rule rewrites more than these two; the corpora only hit these.
Proposal
Run DataFusion's SimplifyExpressions after TypeCoercion, with an OptimizerContext without a query start time (without_query_execution_start_time()), so now() stays symbolic and still lowers to CurrentTimestamp. To make that work:
Add a timestamp literal to the IR's ScalarValue (types, planner, executor).
Lower RegexMatch / RegexIMatch / RegexNotMatch / RegexNotIMatch, and any other shape the simplifier produces, found by running the corpora.
A timestamp literal also lets summary coverage put an absolute time filter (ts >= t0 AND ts < t1) into the selection, which #573 §4.2.2 lists as "later": today it stays in the definition.
Problem
The SQL front end lowers DataFusion's plan right after type coercion and does not run DataFusion's expression simplification (
SimplifyExpressions). A DataFusion query reaches its executor simplified, and ours does not, so the lowered IR can differ from what DataFusion would execute. Today only one piece is matched by hand: #539 folds a numeric cast of a literal (latency < CAST(100 AS DOUBLE)→latency < 100.0) with DataFusion'sScalarValue::cast_to.Running the whole rule (tried on the #646 restack, 2026-10-10) breaks the SQL corpora (
bgp_analytics,bgp_jan2024_workload,netflow,tpch_deequ) in two ways:time >= CAST('2018-05-17 13:10:01' AS TIMESTAMP)folds toTimestampMillisecond(1526562601000). The IR has no timestamp literal, so lowering fails withunsupported scalar. Nearly every BGP and NetFlow query has a time filter.REGEXP_LIKE(col, pat)is rewritten into the operatorcol ~ pat(RegexMatch), which lowering does not support (unsupported feature: operator: RegexMatch).The rule rewrites more than these two; the corpora only hit these.
Proposal
Run DataFusion's
SimplifyExpressionsafterTypeCoercion, with anOptimizerContextwithout a query start time (without_query_execution_start_time()), sonow()stays symbolic and still lowers toCurrentTimestamp. To make that work:ScalarValue(types, planner, executor).RegexMatch/RegexIMatch/RegexNotMatch/RegexNotIMatch, and any other shape the simplifier produces, found by running the corpora.A timestamp literal also lets summary coverage put an absolute time filter (
ts >= t0 AND ts < t1) into the selection, which #573 §4.2.2 lists as "later": today it stays in the definition.Related
#539 (SQL lowering), #646 / #573 (summary coverage).