Conversation
SQLite takes at most 32,766 bind parameters per statement. A bulk create batch needs rows x fields parameters, so a large batch, or a wide resource at the default batch size, failed with "variable number must be between ?1 and ?32766". Insert such a batch in chunks that fit, with headroom for the ON CONFLICT clause's own parameters, and combine their results in input order. A batch that fits is still one statement, and the chunks run inside the batch's transaction.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
SQLite takes at most 32,766 bind parameters per statement (
SQLITE_MAX_VARIABLE_NUMBER). A bulk create inserts each batch in one statement, which needs rows × fields parameters. So a large batch, or a wide resource even at the defaultbatch_sizeof 100, fails:The fix
bulk_create/3now inserts a batch in chunks that fit:ON CONFLICTclause's own parameters, divided by the widest row;A batch that fits is still a single statement. Ash runs each batch in a transaction, so a split batch stays atomic.
Tests
test/bulk_create_test.exsbulk creates 15,000 posts in one batch, checking that it succeeds and that every row is stored.main, it fails as above.main(ASH_SQL_VERSION=local), the whole suite passes (279 tests).Separately, AshSqlite
main's own lockfile pins AshSQL 0.7.5. That predates the grouped aggregate strategy ash-project#232 relies on (ash-project/ash_sql#264). So with the locked dependencies, 89 aggregate tests fail onmainitself, whatever this change does. They pass once AshSQL is released with #264 and the lock is bumped.Found by
The data-layer conformance suite (wtsnz/ash#7), gap
bind-parameter-limit, scenariolarge.bulk_create_parameters. With this branch, it passes. The same bug in AshPostgres, where it also drops the connection, is fixed in wtsnz/ash_postgres#8.