Fast tensor contractions inside SQL engines, for the XQL model.
einfold makes tensor computation fast inside SQL engines, on any hardware those engines run on. It is part of XQL Systems, which builds SQL access to large scientific arrays stored in Zarr.
In the XQL model, a dataset becomes a table with one row per combination of coordinates, and a tensor contraction (matrix multiplication is the simplest case) becomes a join followed by a SUM. SQL engines run that pattern badly: they materialize huge intermediate joins, sum too late, and ignore the dense structure of arrays.
einfold fixes this by rewriting query plans, not by executing them. It finds the einsums (Einstein-summation expressions) inside a relational plan and returns a plan that the engine runs faster. Engines can run it on CPU or GPU: Apache DataFusion, DuckDB, DuckDB with GPU extensions such as Sirius, or NVIDIA's GPU Query Engine. einfold never touches a device itself.
The name joins einsum with fold, functional programming's word for a reducing pass: einfold evaluates an einsum by folding its sums into the joins that feed them.
Design phase. There is no product code yet. Seventeen of twenty design spikes are done; their code and reports are in docs/spikes/.
- Design: the problem, principles, architecture, core abstractions, optimizer, and roadmap.
- Design supplement: the details behind the design: algorithms, rule tables, correctness arguments, and spike evidence. Its sections are numbered to match the design's.
- Spikes: short experiments that settled design questions, each with code and a report.
- Target demos: three end-to-end demos that double as design checks: rediscovering FlashAttention from SQL attention, NanoGPT in SQL with a recent sparsity record as a small diff, and GraphCast weather forecasts that read only the data they need.
- Layout convention: a draft XQL Systems convention that lets Zarr datasets declare how their data is ordered and summarized (curve orderings such as Z-order, chunk visit order, written-chunk records, and per-chunk min/max), so that query engines can scan less.
- xarray-sql: query Xarray datasets with SQL.
- duckdb-zarr: query Zarr stores from DuckDB (
INSTALL zarr FROM community). - ddx: automatic differentiation of SQL queries. Its gradient computations are mostly tensor contractions, which motivated einfold.
einfold is licensed under the Apache License, Version 2.0.
The project follows the REUSE specification: every file states its copyright and license with SPDX tags, and license texts live in LICENSES/. To check compliance, run:
reuse lint