Skip to content

Extend SQL pushdown: multi-hop, filters, aggregation, ORDER BY, recursive CTE - #1003

Merged
adsharma merged 1 commit into
LadybugDB:mainfrom
adsharma:feature/sql-pushdown-extended
Sep 20, 2026
Merged

adsharma merged 1 commit into
LadybugDB:mainfrom
adsharma:feature/sql-pushdown-extended

Conversation

@adsharma

Copy link
Copy Markdown
Contributor

Stacked on the DuckLake pushdown series (first three commits below belong to it; this PR's change is the last commit: 83dc796ac).

SQL pushdown previously fired only for a narrow set of 1-hop star patterns (and, due to a null referencedEntry in the DuckDB catalog — fixed in LadybugDB/extensions#88 — often not even then). This PR generalizes ForeignJoinPushDownOptimizer:

  • N-hop chain matching: collects EXTEND hops, foreign node scans and FILTER predicates from hash-join subtrees regardless of nesting, emitting one multi-way SQL JOIN (BFS order; extra ON conjuncts for already-bound neighbours, so cycles/diamonds stay correct). Node/edge filters re-attach above the pushed scan and fold into SQL WHERE via filter pushdown.
  • Aggregation pushdown: supported AGGREGATEs (COUNT/SUM/AVG/MIN/MAX) fold into GROUP BY; kept projection spines are expanded so projection pushdown cannot prune grouped columns.
  • ORDER BY pushdown: sort keys fold into pushed scans; order-by pushdown now resolves keys against scan output aliases, goes all-or-nothing, and never drops an ORDER BY it could not push (previously an untranslatable key silently lost the sort — wrong results).
  • Recursive CTE: variable-length patterns matched top-down along the root spine (WALK, finite bound, directed) rewrite to WITH RECURSIVE; inline edge predicates bake into anchor + recursive member; path/shortest-path cases stay local.

Planner hardening so foreign variable-length patterns plan at all: the cardinality estimator skips non-lbug attached entries instead of downcasting them to AttachedLbugDatabase (UB/crash), and recursive rel-property scans tolerate foreign rels without an internal _ID.

Requires LadybugDB/extensions#88 for end-to-end pushdown over DuckDB-attached tables (engine changes alone are inert and covered by native suites). The submodule pointer bump follows once #88 merges.

Verification done locally: 1-hop/2-hop/filter/agg/order/recursive result matrices over an attached DuckDB graph (mirrored in sql_pushdown.test in #88); c_api_test 146 passed; e2e order/limit/aggregate (53), graph/attach/path (37) green (one pre-existing failure from a missing parquet fixture, unrelated).

CI: run the rest on CI (full e2e + extension-test trees).

…sive CTE

Generalize ForeignJoinPushDownOptimizer beyond single-hop star patterns:

- N-hop chain matching: collect EXTEND hops, foreign node scans and
  FILTER predicates from hash-join subtrees regardless of nesting, and
  emit one multi-way SQL JOIN (BFS order, extra ON conjuncts for
  already-bound neighbours). Node/edge filters re-attach above the pushed
  scan so filter pushdown folds them into SQL WHERE.
- Aggregation pushdown: fold supported AGGREGATEs (COUNT/SUM/AVG/MIN/MAX)
  into GROUP BY, dropping the operator; kept projection spines are
  expanded so later projection pushdown cannot prune grouped columns.
- ORDER BY pushdown: fold sort keys into pushed scans (table refs and
  pushed-output aliases); order-by pushdown additionally resolves keys
  against scan output aliases, goes all-or-nothing, and never drops an
  ORDER BY it could not push (previously an untranslatable key silently
  lost the sort).
- Recursive CTE: match variable-length patterns top-down along the root
  spine (WALK, finite bound, directed) and rewrite to WITH RECURSIVE,
  with inline edge predicates baked into anchor + recursive member.
  Anything unmappable (paths, rel outputs, shortest-path) stays local.

Planner hardening so foreign variable-length patterns plan at all:
cardinality estimator skips non-lbug attached entries instead of
downcasting them to AttachedLbugDatabase (UB), and recursive rel
property scans tolerate foreign rels without an internal _ID.
@adsharma
adsharma force-pushed the feature/sql-pushdown-extended branch from 83dc796 to d5d23b2 Compare September 20, 2026 04:41
@adsharma
adsharma merged commit ed9bfb8 into LadybugDB:main Sep 20, 2026
4 checks passed
@adsharma
adsharma deleted the feature/sql-pushdown-extended branch September 20, 2026 05:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant