Repository navigation
fix: csv import honors MAX_RECORD instead of a fixed 1KB line cap (#67) - #70
Merged
Merged
Conversation
bulk insert silently skipped every CSV line longer than 1,023 bytes via a fixed MAX_CSV_LINE_LEN cap applied to the whole getline() line, ignoring MAX_RECORD. Records that should fit (e.g. with MAX_RECORD=2048) were lost, and the skip was only logged at debug level. The upper bound now follows MAX_RECORD and is enforced on the record data size, consistent with collate (size > max_rec_ln). Oversized records are counted in the import summary and logged at info level instead of being dropped in silence. The base64 decode buffer (data_bin) grows with the line buffer instead of being a fixed 1 KB stack array, so binary records are no longer truncated. collate's tuple_bin decode buffer is likewise raised from the 1 KB constant to LDB_MAX_REC_LN so delete/collate can match records up to MAX_RECORD. Adds test_13 to test/test_kb, bumps version to 4.1.13, and documents the MAX_RECORD import behaviour in README. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #67 on the
master(4.x / MD5) line. Port of the same fix merged intocrc64(#69).bulk insertfrom CSV silently skipped every line longer than 1,023 bytes (a fixedMAX_CSV_LINE_LENcap applied to the wholegetline()line), ignoringMAX_RECORD. Records that should fit (e.g. withMAX_RECORD=2048) were lost, and the skip was only logged atdebuglevel.Changes
ldb_import_csv(src/import.c). The lower bound (line too short to hold a key + separators) is kept; the upper bound now followsMAX_RECORD.MAX_RECORDon the record data size (r_size), consistent with collate'ssize > max_rec_lnfilter. Hard-capped by the 2-byte record-size field (LDB_MAX_REC_LN).infolevel.data_bingrows with the line buffer instead of being a fixed 1 KB stack array, so binary (.enc/base64) records are no longer truncated; freed on every exit path.collate.c: raisetuple_binfromMAX_CSV_LINE_LENtoLDB_MAX_REC_LNso delete/collate can decode and match records up toMAX_RECORD(this buffer does not exist on thecrc64branch, which already usesLDB_MAX_REC_LN-sized buffers).4.1.13; document theMAX_RECORDimport behaviour inREADME.md.Testing
test_13_import_respects_max_record_not_fixed_line_lenintest/test_kb: lines of 1000/1100/2000/3000 bytes imported withMAX_RECORD=2048— the 1100 and 2000-byte records (dropped by the old cap) are now stored; the 3000-byte one (data >MAX_RECORD) is skipped, counted, and logged.masterHEAD (lines ≥1024 dropped) and confirmed the fix../run_test.sh) passes: Overall result: SUCCESS.🤖 Generated with Claude Code