Repository navigation
feat(import): import the files of a table in alphabetical order - #71
Merged
Merged
Conversation
added 2 commits
October 27, 2026 13:17
process_sectors listed the table folder with readdir, which returns the filesystem's order (hash order on ext4), so the shards of a bulk insert started in an effectively random order. List it with scandir + alphasort: the files start 00, 01, ... ff (and 0000 ... ffff for mz), THREADS at a time, so after an interrupted import everything below the files in progress is done and everything above has not started. The resulting KB is byte-identical: a 1.9 GB sample of a monthly update (every table, 16 shards each; csv, wfp and mz) imported into an empty KB gives the same sha256 for every file with and without the change, and the import times are within run-to-run variation.
Five csvs created in a mixed order, imported with THREADS=1: the log must show them as 00 0a 10 80 ff. Fails without the scandir change (readdir order), passes with it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
bulk insertstarts the files of a table (00.csv…ff.csv,0000.mz…ffff.mz,*.bin) inreaddirorder, which on ext4 is hash order, so the shards of an import start in an effectively random order (e.g.55 2a d4 f0 7f 10 c3 a5 …).When an import is interrupted (here: the uat was hard-reset during the 26.10
fileimport), working out which shards were imported, which were half-imported and which were never started means matching the log against the files left in the folder. With the files started in order, everything below theTHREADSfiles in progress is done and everything above them has not started.Change
process_sectorslists the table folder withscandir(..., alphasort)instead ofopendir/readdir(sameDT_REGfilter, entries freed). Tables still run inoss.conforder,THREADSfiles at a time as before.Testing
A sample of the 26.10 monthly update (1.9 GB: 16 shards of every csv table, the first 2M records of 16 wfp
.binshards, 12–48.mzfiles of sources, notices and metadata, empty semgrep csvs, version.json), imported into an empty KB with the 26.10oss.confand the options ofldb-import.sh(KEY_SIZE=8, THREADS=4, FILE_DEL=1, SORT=0), in a private mount namespace so/var/lib/ldband/usr/local/etc/scanosspointed at a test folder:Identical KB: the sha256 of every file of the KB (368 files) is the same in all 6 runs (3 with
crc64as is, 3 with the change); therecords imported/collate completedlines are identical too.Order: the files start alphabetically, 4 at a time (e.g. wfp:
00 01 0f 10,2a 3c 55 6e,7f 80 99 a5,c3 d4 f0 ff).Time: without the change 425, 429, 424 s; with it 464, 449, 419 s. The spread is run-to-run variation (WSL2 disk); per table the times differ by at most ~6 s, in both directions. Sorting the names costs milliseconds.
Test suite:
./run_test.shpasses (31 tests: 12 md5, 19 crc64), the same as oncrc64. Newtest_17_import_files_in_alphabetical_order(test/test_kb_crc64): five csvs created in a mixed order, imported withTHREADS=1, the log must show00 0a 10 80 ff. It fails oncrc64and passes with this change.