fix(compaction): bound merge batch and log memory breakdown on OOM (11.0.251)#799
Merged
Merged
Conversation
merge_adjacent_files with no max_compacted_files rewrites every small parquet of a table in one CALL; on a 2GB-capped writer that working set alone exhausts memory_limit while search and flush share the instance. - Default max_compacted_files to 100 when the operator left it 0; leftovers are merged on subsequent cycles. - On Out of Memory during merge, dump duckdb_memory() so operators can see who holds the budget and whether disk spilling is engaged (temporary_storage_bytes > 0 proves temp_directory works). Bump VERSION_APPLICATION to 11.0.251.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up to #798:
CompactionService: merge_adjacent_filesstill OOMs on a 2GB-capped writer (could not allocate block of size 256.0 KiB (1.8 GiB/1.8 GiB used)).Unbounded
ducklake_merge_adjacent_filesrewrites every small parquet of a table in one CALL; that working set alone can exhaustmemory_limitwhile search and flush share the same DuckDB instance.Changes
max_compacted_files = 100whencompaction.max_compacted_filesis 0/unset — bounds each merge cycle's peak memory; leftovers are picked up next cycle. Explicit operator values still win.duckdb_memory()(tag, memory_mb, spilled_to_disk_mb).spilled_to_disk_mb > 0proves the fix(ducklake): enable disk spilling so long-range searches don't OOM (11.0.250) #798 spill directory is engaged; all-zero spill on a build with fix(ducklake): enable disk spilling so long-range searches don't OOM (11.0.250) #798 meanstemp_directorydidn't apply — please paste this output if the warning persists.VERSION_APPLICATION→11.0.251Test plan
go build ./...,go vet,go test ./writer/— passmerge_adjacent_files completedinstead of OOMduckdb memory breakdownWRN lines identify the holder