Skip to content

fix: split bulk inserts that exceed SQLite's bind parameter limit - #10

Draft
wtsnz wants to merge 1 commit into
mainfrom
fix/bind-parameter-limit
Draft

wtsnz wants to merge 1 commit into
mainfrom
fix/bind-parameter-limit

Conversation

@wtsnz

@wtsnz wtsnz commented Sep 26, 2026

Copy link
Copy Markdown
Owner

The problem

SQLite takes at most 32,766 bind parameters per statement (SQLITE_MAX_VARIABLE_NUMBER). A bulk create inserts each batch in one statement, which needs rows × fields parameters. So a large batch, or a wide resource even at the default batch_size of 100, fails:

inputs = for i <- 1..15_000, do: %{title: "title#{i}"}
Ash.bulk_create(inputs, Post, :create, batch_size: 15_000, return_errors?: true)
# => error: "** (Exqlite.Error) variable number must be between ?1 and ?32766"

The fix

bulk_create/3 now inserts a batch in chunks that fit:

  • the rows per chunk are the 32,766-parameter limit, less 1,000 of headroom for the ON CONFLICT clause's own parameters, divided by the widest row;
  • the chunks' counts and returned rows are combined, in input order, since the returned rows are paired with the changesets by position.

A batch that fits is still a single statement. Ash runs each batch in a transaction, so a split batch stays atomic.

Tests

test/bulk_create_test.exs bulk creates 15,000 posts in one batch, checking that it succeeds and that every row is stored.

  • On main, it fails as above.
  • With the fix, it passes. Against AshSQL main (ASH_SQL_VERSION=local), the whole suite passes (279 tests).

Separately, AshSqlite main's own lockfile pins AshSQL 0.7.5. That predates the grouped aggregate strategy ash-project#232 relies on (ash-project/ash_sql#264). So with the locked dependencies, 89 aggregate tests fail on main itself, whatever this change does. They pass once AshSQL is released with #264 and the lock is bumped.

Found by

The data-layer conformance suite (wtsnz/ash#7), gap bind-parameter-limit, scenario large.bulk_create_parameters. With this branch, it passes. The same bug in AshPostgres, where it also drops the connection, is fixed in wtsnz/ash_postgres#8.

SQLite takes at most 32,766 bind parameters per statement. A bulk create
batch needs rows x fields parameters, so a large batch, or a wide
resource at the default batch size, failed with "variable number must be
between ?1 and ?32766".

Insert such a batch in chunks that fit, with headroom for the ON CONFLICT
clause's own parameters, and combine their results in input order. A
batch that fits is still one statement, and the chunks run inside the
batch's transaction.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant