Repository navigation
Extend SQL pushdown: multi-hop, filters, aggregation, ORDER BY, recursive CTE - #1003
Merged
Merged
Conversation
…sive CTE Generalize ForeignJoinPushDownOptimizer beyond single-hop star patterns: - N-hop chain matching: collect EXTEND hops, foreign node scans and FILTER predicates from hash-join subtrees regardless of nesting, and emit one multi-way SQL JOIN (BFS order, extra ON conjuncts for already-bound neighbours). Node/edge filters re-attach above the pushed scan so filter pushdown folds them into SQL WHERE. - Aggregation pushdown: fold supported AGGREGATEs (COUNT/SUM/AVG/MIN/MAX) into GROUP BY, dropping the operator; kept projection spines are expanded so later projection pushdown cannot prune grouped columns. - ORDER BY pushdown: fold sort keys into pushed scans (table refs and pushed-output aliases); order-by pushdown additionally resolves keys against scan output aliases, goes all-or-nothing, and never drops an ORDER BY it could not push (previously an untranslatable key silently lost the sort). - Recursive CTE: match variable-length patterns top-down along the root spine (WALK, finite bound, directed) and rewrite to WITH RECURSIVE, with inline edge predicates baked into anchor + recursive member. Anything unmappable (paths, rel outputs, shortest-path) stays local. Planner hardening so foreign variable-length patterns plan at all: cardinality estimator skips non-lbug attached entries instead of downcasting them to AttachedLbugDatabase (UB), and recursive rel property scans tolerate foreign rels without an internal _ID.
adsharma
force-pushed
the
feature/sql-pushdown-extended
branch
from
September 20, 2026 04:41
83dc796 to
d5d23b2
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on the DuckLake pushdown series (first three commits below belong to it; this PR's change is the last commit:
83dc796ac).SQL pushdown previously fired only for a narrow set of 1-hop star patterns (and, due to a null
referencedEntryin the DuckDB catalog — fixed in LadybugDB/extensions#88 — often not even then). This PR generalizesForeignJoinPushDownOptimizer:COUNT/SUM/AVG/MIN/MAX) fold intoGROUP BY; kept projection spines are expanded so projection pushdown cannot prune grouped columns.WITH RECURSIVE; inline edge predicates bake into anchor + recursive member; path/shortest-path cases stay local.Planner hardening so foreign variable-length patterns plan at all: the cardinality estimator skips non-lbug attached entries instead of downcasting them to
AttachedLbugDatabase(UB/crash), and recursive rel-property scans tolerate foreign rels without an internal_ID.Requires LadybugDB/extensions#88 for end-to-end pushdown over DuckDB-attached tables (engine changes alone are inert and covered by native suites). The submodule pointer bump follows once #88 merges.
Verification done locally: 1-hop/2-hop/filter/agg/order/recursive result matrices over an attached DuckDB graph (mirrored in
sql_pushdown.testin #88);c_api_test146 passed; e2e order/limit/aggregate (53), graph/attach/path (37) green (one pre-existing failure from a missing parquet fixture, unrelated).CI: run the rest on CI (full e2e + extension-test trees).