[mscasso-scanoss]

Summary

Fixes #67. bulk insert from CSV silently skipped every line longer than 1,023 bytes (a fixed MAX_CSV_LINE_LEN cap applied to the whole getline() line), ignoring MAX_RECORD. Records that should fit (e.g. with MAX_RECORD=2048) were lost, and the skip was only logged at debug level — invisible unless VERBOSE was on.

Changes

• Remove the fixed 1 KB whole-line cap in ldb_import_csv (src/import.c). The lower bound (line too short to hold a key + separators) is kept; the upper bound now follows MAX_RECORD.
• Enforce MAX_RECORD on the record data size (r_size), consistent with collate's size > max_rec_ln filter. Hard-capped by the 2-byte record-size field (LDB_MAX_REC_LN).
• No more silent drops: an oversized record is counted in the import summary and logged at info level.
• data_bin grows with the line buffer instead of being a fixed 1 KB stack array, so binary (.enc/base64) records are no longer truncated.
• Drop the now-dead MAX_CSV_LINE_LEN define from src/import.c (kept in the public definitions.h to avoid breaking consumers).
• Bump version to 5.0.3-crc64; document the MAX_RECORD import behaviour in README.md.

Testing

• New regression test test_12_import_respects_max_record_not_fixed_line_len in test/test_kb: lines of 1000/1100/2000/3000 bytes imported with MAX_RECORD=2048 — the 1100 and 2000-byte records (dropped by the old cap) are now stored; the 3000-byte one (data > MAX_RECORD) is skipped, counted, and logged.
• Reproduced the bug against crc64 HEAD (lines ≥1024 dropped) and confirmed the fix.
• Full suite (./run_test.sh, both MD5 and CRC64 knowledge bases) passes: Overall result: SUCCESS.

Note: on crc64, collate.c already uses LDB_MAX_REC_LN-sized buffers, so (unlike master) the tuple_bin resize mentioned in the issue is not needed here.

🤖 Generated with Claude Code