Repository navigation
15/15: Add db_stress and crash test support (#15683) - #15683
Open
xingbowang wants to merge 15 commits into
Open
xingbowang wants to merge 15 commits into
xingbowang wants to merge 15 commits into
Conversation
|
@xingbowang has exported this pull request. If you are a Meta employee, you can view the originating Diff in D129346853. |
|
| Check | Count |
|---|---|
bugprone-argument-comment |
60 |
cert-err58-cpp |
10 |
clang-analyzer-core.NonNullParamChecker |
2 |
clang-analyzer-cplusplus.NewDeleteLeaks |
1 |
cppcoreguidelines-avoid-non-const-global-variables |
2 |
cppcoreguidelines-missing-std-forward |
1 |
cppcoreguidelines-pro-type-member-init |
9 |
cppcoreguidelines-special-member-functions |
6 |
performance-inefficient-string-concatenation |
3 |
performance-inefficient-vector-operation |
20 |
performance-move-const-arg |
4 |
performance-no-automatic-move |
4 |
performance-unnecessary-value-param |
3 |
readability-isolate-declaration |
2 |
| Total | 127 |
Details
db/blob/blob_file_builder.cc (2 warning(s))
db/blob/blob_file_builder.cc:60:61: warning: the parameter #7 is copied for each invocation but only used as a const reference; consider making it a const reference [performance-unnecessary-value-param]
db/blob/blob_file_builder.cc:61:34: warning: the parameter #8 is copied for each invocation but only used as a const reference; consider making it a const reference [performance-unnecessary-value-param]
db/blob/blob_file_cache.cc (1 warning(s))
db/blob/blob_file_cache.cc:87:14: warning: constness of 'validation' prevents automatic move [performance-no-automatic-move]
db/blob/blob_file_cache_test.cc (1 warning(s))
db/blob/blob_file_cache_test.cc:427:1: warning: Potential leak of memory pointed to by field '_M_head_impl' [clang-analyzer-cplusplus.NewDeleteLeaks]
db/blob/blob_file_reader.cc (1 warning(s))
db/blob/blob_file_reader.cc:1581:5: warning: uninitialized record type: 'lookup_key_bytes' [cppcoreguidelines-pro-type-member-init]
db/blob/blob_file_reader_test.cc (14 warning(s))
db/blob/blob_file_reader_test.cc:419:7: warning: argument name 'last_original_offset' in comment does not match parameter name 'value' [bugprone-argument-comment]
db/blob/blob_file_reader_test.cc:833:5: warning: 'push_back' is called inside a loop; consider pre-allocating the container capacity before the loop [performance-inefficient-vector-operation]
db/blob/blob_file_reader_test.cc:1017:7: warning: 'push_back' is called inside a loop; consider pre-allocating the container capacity before the loop [performance-inefficient-vector-operation]
db/blob/blob_file_reader_test.cc:1303:5: warning: 'push_back' is called inside a loop; consider pre-allocating the container capacity before the loop [performance-inefficient-vector-operation]
db/blob/blob_file_reader_test.cc:1307:5: warning: 'push_back' is called inside a loop; consider pre-allocating the container capacity before the loop [performance-inefficient-vector-operation]
db/blob/blob_file_reader_test.cc:1372:9: warning: 'push_back' is called inside a loop; consider pre-allocating the container capacity before the loop [performance-inefficient-vector-operation]
db/blob/blob_file_reader_test.cc:1547:5: warning: 'push_back' is called inside a loop; consider pre-allocating the container capacity before the loop [performance-inefficient-vector-operation]
db/blob/blob_file_reader_test.cc:1570:5: warning: 'push_back' is called inside a loop; consider pre-allocating the container capacity before the loop [performance-inefficient-vector-operation]
db/blob/blob_file_reader_test.cc:1900:5: warning: 'push_back' is called inside a loop; consider pre-allocating the container capacity before the loop [performance-inefficient-vector-operation]
db/blob/blob_file_reader_test.cc:2099:9: warning: 'push_back' is called inside a loop; consider pre-allocating the container capacity before the loop [performance-inefficient-vector-operation]
db/blob/blob_file_reader_test.cc:4867:5: warning: 'push_back' is called inside a loop; consider pre-allocating the container capacity before the loop [performance-inefficient-vector-operation]
db/blob/blob_file_reader_test.cc:4944:5: warning: uninitialized record type: 'header' [cppcoreguidelines-pro-type-member-init]
db/blob/blob_file_reader_test.cc:4995:7: warning: 'push_back' is called inside a loop; consider pre-allocating the container capacity before the loop [performance-inefficient-vector-operation]
db/blob/blob_file_reader_test.cc:4998:7: warning: 'push_back' is called inside a loop; consider pre-allocating the container capacity before the loop [performance-inefficient-vector-operation]
db/blob/blob_garbage_meter_test.cc (3 warning(s))
db/blob/blob_garbage_meter_test.cc:160:45: warning: argument name 'generation' in comment does not match parameter name 'generation_file_number' [bugprone-argument-comment]
db/blob/blob_garbage_meter_test.cc:168:37: warning: argument name 'seq' in comment does not match parameter name 's' [bugprone-argument-comment]
db/blob/blob_garbage_meter_test.cc:238:37: warning: argument name 'seq' in comment does not match parameter name 's' [bugprone-argument-comment]
db/blob/blob_gen2_format.cc (2 warning(s))
db/blob/blob_gen2_format.cc:61:60: warning: argument name 'size' in comment does not match parameter name 'n' [bugprone-argument-comment]
db/blob/blob_gen2_format.cc:771:3: warning: uninitialized record type: 'trailer' [cppcoreguidelines-pro-type-member-init]
db/blob/blob_reference_census.cc (2 warning(s))
db/blob/blob_reference_census.cc:2334:9: warning: Forming reference to null pointer [clang-analyzer-core.NonNullParamChecker]
db/blob/blob_reference_census.cc:2377:9: warning: Forming reference to null pointer [clang-analyzer-core.NonNullParamChecker]
db/blob/blob_reference_metadata.cc (2 warning(s))
db/blob/blob_reference_metadata.cc:60:23: warning: variable 'unsorted_group_fallbacks' is non-const and globally accessible, consider making it const [cppcoreguidelines-avoid-non-const-global-variables]
db/blob/blob_reference_metadata.cc:910:10: warning: constness of 'release_status' prevents automatic move [performance-no-automatic-move]
db/blob/blob_relocation_manifest.cc (1 warning(s))
db/blob/blob_relocation_manifest.cc:34:19: warning: variable 'blob_file_table_plain_scan' is non-const and globally accessible, consider making it const [cppcoreguidelines-avoid-non-const-global-variables]
db/blob/blob_source.cc (2 warning(s))
db/blob/blob_source.cc:625:12: warning: constness of 'status' prevents automatic move [performance-no-automatic-move]
db/blob/blob_source.cc:752:12: warning: constness of 'status' prevents automatic move [performance-no-automatic-move]
db/blob/blob_source_test.cc (1 warning(s))
db/blob/blob_source_test.cc:2391:7: warning: uninitialized record type: 'record_of' [cppcoreguidelines-pro-type-member-init]
db/blob/db_blob_index_test.cc (13 warning(s))
db/blob/db_blob_index_test.cc:67:9: warning: class 'Handle' defines a non-default destructor but does not define a copy constructor, a copy assignment operator, a move constructor or a move assignment operator [cppcoreguidelines-special-member-functions]
db/blob/db_blob_index_test.cc:414:25: warning: argument name 'generation' in comment does not match parameter name 'value' [bugprone-argument-comment]
db/blob/db_blob_index_test.cc:415:25: warning: argument name 'origin' in comment does not match parameter name 'value' [bugprone-argument-comment]
db/blob/db_blob_index_test.cc:416:25: warning: argument name 'original_offset' in comment does not match parameter name 'value' [bugprone-argument-comment]
db/blob/db_blob_index_test.cc:417:25: warning: argument name 'size' in comment does not match parameter name 'value' [bugprone-argument-comment]
db/blob/db_blob_index_test.cc:419:24: warning: argument name 'logical_crc' in comment does not match parameter name 'value' [bugprone-argument-comment]
db/blob/db_blob_index_test.cc:420:24: warning: argument name 'stored_crc' in comment does not match parameter name 'value' [bugprone-argument-comment]
db/blob/db_blob_index_test.cc:754:45: warning: argument name 'generation' in comment does not match parameter name 'generation_file_number' [bugprone-argument-comment]
db/blob/db_blob_index_test.cc:1198:7: warning: 'emplace_back' is called inside a loop; consider pre-allocating the container capacity before the loop [performance-inefficient-vector-operation]
db/blob/db_blob_index_test.cc:1523:5: warning: multiple declarations in a single statement reduces readability [readability-isolate-declaration]
db/blob/db_blob_index_test.cc:1628:5: warning: 'push_back' is called inside a loop; consider pre-allocating the container capacity before the loop [performance-inefficient-vector-operation]
db/blob/db_blob_index_test.cc:1702:45: warning: argument name 'generation' in comment does not match parameter name 'generation_file_number' [bugprone-argument-comment]
db/blob/db_blob_index_test.cc:3675:7: warning: class 'ScopedCensusLogLimit' defines a non-default destructor but does not define a copy constructor, a copy assignment operator, a move constructor or a move assignment operator [cppcoreguidelines-special-member-functions]
db/db_impl/db_impl.cc (1 warning(s))
db/db_impl/db_impl.cc:7813:51: warning: string concatenation results in allocation of unnecessary temporary strings; consider using 'operator+=' or 'string::append()' instead [performance-inefficient-string-concatenation]
db/db_options_test.cc (1 warning(s))
db/db_options_test.cc:2233:5: warning: 'push_back' is called inside a loop; consider pre-allocating the container capacity before the loop [performance-inefficient-vector-operation]
db/internal_stats.cc (10 warning(s))
db/internal_stats.cc:342:26: warning: initialization of 'blob_relocation_lost_files' with static storage duration may throw an exception that cannot be caught [cert-err58-cpp]
db/internal_stats.cc:344:26: warning: initialization of 'indirect_blob_gc_idle_micros_with_candidates' with static storage duration may throw an exception that cannot be caught [cert-err58-cpp]
db/internal_stats.cc:346:26: warning: initialization of 'indirect_blob_gc_stats' with static storage duration may throw an exception that cannot be caught [cert-err58-cpp]
db/internal_stats.cc:347:26: warning: initialization of 'blob_reference_census_unready_micros' with static storage duration may throw an exception that cannot be caught [cert-err58-cpp]
db/internal_stats.cc:349:26: warning: initialization of 'blob_indirection_format_version' with static storage duration may throw an exception that cannot be caught [cert-err58-cpp]
db/internal_stats.cc:479:35: warning: initialization of 'kBlobRelocationLostFiles' with static storage duration may throw an exception that cannot be caught [cert-err58-cpp]
db/internal_stats.cc:481:35: warning: initialization of 'kIndirectBlobGcIdleMicrosWithCandidates' with static storage duration may throw an exception that cannot be caught [cert-err58-cpp]
db/internal_stats.cc:483:35: warning: initialization of 'kIndirectBlobGcStats' with static storage duration may throw an exception that cannot be caught [cert-err58-cpp]
db/internal_stats.cc:485:35: warning: initialization of 'kBlobReferenceCensusUnreadyMicros' with static storage duration may throw an exception that cannot be caught [cert-err58-cpp]
db/internal_stats.cc:487:35: warning: initialization of 'kBlobIndirectionFormatVersion' with static storage duration may throw an exception that cannot be caught [cert-err58-cpp]
db/memtable_list.cc (1 warning(s))
db/memtable_list.cc:1169:25: warning: argument name 'column_family_options' in comment does not match parameter name 'new_cf_options' [bugprone-argument-comment]
db/repair.cc (9 warning(s))
db/repair.cc:1268:10: warning: constructor does not initialize these fields: table [cppcoreguidelines-pro-type-member-init]
db/repair.cc:1334:10: warning: constructor does not initialize these fields: repairer [cppcoreguidelines-pro-type-member-init]
db/repair.cc:1334:10: warning: class 'RepairReferenceCursor' defines a non-default destructor but does not define a move constructor or a move assignment operator [cppcoreguidelines-special-member-functions]
db/repair.cc:2231:7: warning: 'push_back' is called inside a loop; consider pre-allocating the container capacity before the loop [performance-inefficient-vector-operation]
db/repair.cc:2543:43: warning: argument name 'log_number' in comment does not match parameter name 'log_num' [bugprone-argument-comment]
db/repair.cc:2968:12: warning: constructor does not initialize these fields: env, lognum [cppcoreguidelines-pro-type-member-init]
db/repair.cc:2968:12: warning: class 'LogReporter' defines a non-default destructor but does not define a copy constructor, a copy assignment operator, a move constructor or a move assignment operator [cppcoreguidelines-special-member-functions]
db/repair.cc:3900:15: warning: passing result of std::move() as a const reference argument; no move will actually happen [performance-move-const-arg]
db/repair.cc:4458:36: warning: passing result of std::move() as a const reference argument; no move will actually happen [performance-move-const-arg]
db/repair_test.cc (1 warning(s))
db/repair_test.cc:221:7: warning: 'push_back' is called inside a loop; consider pre-allocating the container capacity before the loop [performance-inefficient-vector-operation]
db/version_builder_test.cc (37 warning(s))
db/version_builder_test.cc:158:9: warning: passing result of std::move() as a const reference argument; no move will actually happen [performance-move-const-arg]
db/version_builder_test.cc:1216:39: warning: argument name 'file_number' in comment does not match parameter name 'file' [bugprone-argument-comment]
db/version_builder_test.cc:1239:7: warning: argument name 'source_file_number' in comment does not match parameter name 'old_file_number' [bugprone-argument-comment]
db/version_builder_test.cc:1239:35: warning: argument name 'physical_file_number' in comment does not match parameter name 'target_file_number' [bugprone-argument-comment]
db/version_builder_test.cc:1256:49: warning: argument name 'source_file_number' in comment does not match parameter name 'old_file_number' [bugprone-argument-comment]
db/version_builder_test.cc:1413:27: warning: argument name 'source_file_number' in comment does not match parameter name 'old_file_number' [bugprone-argument-comment]
db/version_builder_test.cc:1414:27: warning: argument name 'physical_file_number' in comment does not match parameter name 'target_file_number' [bugprone-argument-comment]
db/version_builder_test.cc:1503:27: warning: argument name 'source_file_number' in comment does not match parameter name 'old_file_number' [bugprone-argument-comment]
db/version_builder_test.cc:1504:27: warning: argument name 'physical_file_number' in comment does not match parameter name 'target_file_number' [bugprone-argument-comment]
db/version_builder_test.cc:1545:37: warning: argument name 'source_file_number' in comment does not match parameter name 'old_file_number' [bugprone-argument-comment]
db/version_builder_test.cc:1546:37: warning: argument name 'physical_file_number' in comment does not match parameter name 'target_file_number' [bugprone-argument-comment]
db/version_builder_test.cc:1677:7: warning: argument name 'source_file_number' in comment does not match parameter name 'old_file_number' [bugprone-argument-comment]
db/version_builder_test.cc:1677:35: warning: argument name 'physical_file_number' in comment does not match parameter name 'target_file_number' [bugprone-argument-comment]
db/version_builder_test.cc:1729:14: warning: argument name 'source_file_number' in comment does not match parameter name 'old_file_number' [bugprone-argument-comment]
db/version_builder_test.cc:1729:42: warning: argument name 'physical_file_number' in comment does not match parameter name 'target_file_number' [bugprone-argument-comment]
db/version_builder_test.cc:1735:14: warning: argument name 'source_file_number' in comment does not match parameter name 'old_file_number' [bugprone-argument-comment]
db/version_builder_test.cc:1735:42: warning: argument name 'physical_file_number' in comment does not match parameter name 'target_file_number' [bugprone-argument-comment]
db/version_builder_test.cc:1763:7: warning: argument name 'source_file_number' in comment does not match parameter name 'old_file_number' [bugprone-argument-comment]
db/version_builder_test.cc:1763:35: warning: argument name 'physical_file_number' in comment does not match parameter name 'target_file_number' [bugprone-argument-comment]
db/version_builder_test.cc:1780:9: warning: argument name 'source_file_number' in comment does not match parameter name 'old_file_number' [bugprone-argument-comment]
db/version_builder_test.cc:1780:37: warning: argument name 'physical_file_number' in comment does not match parameter name 'target_file_number' [bugprone-argument-comment]
db/version_builder_test.cc:1781:36: warning: argument name 'source_file_number' in comment does not match parameter name 'old_file_number' [bugprone-argument-comment]
db/version_builder_test.cc:1793:7: warning: argument name 'source_file_number' in comment does not match parameter name 'old_file_number' [bugprone-argument-comment]
db/version_builder_test.cc:1793:35: warning: argument name 'physical_file_number' in comment does not match parameter name 'target_file_number' [bugprone-argument-comment]
db/version_builder_test.cc:1808:7: warning: argument name 'source_file_number' in comment does not match parameter name 'old_file_number' [bugprone-argument-comment]
db/version_builder_test.cc:1808:35: warning: argument name 'physical_file_number' in comment does not match parameter name 'target_file_number' [bugprone-argument-comment]
db/version_builder_test.cc:1828:38: warning: argument name 'source_file_number' in comment does not match parameter name 'old_file_number' [bugprone-argument-comment]
db/version_builder_test.cc:1884:7: warning: argument name 'source_file_number' in comment does not match parameter name 'old_file_number' [bugprone-argument-comment]
db/version_builder_test.cc:1884:35: warning: argument name 'physical_file_number' in comment does not match parameter name 'target_file_number' [bugprone-argument-comment]
db/version_builder_test.cc:2454:27: warning: argument name 'source_file_number' in comment does not match parameter name 'old_file_number' [bugprone-argument-comment]
db/version_builder_test.cc:2455:27: warning: argument name 'physical_file_number' in comment does not match parameter name 'target_file_number' [bugprone-argument-comment]
db/version_builder_test.cc:2465:29: warning: argument name 'source_file_number' in comment does not match parameter name 'old_file_number' [bugprone-argument-comment]
db/version_builder_test.cc:2466:29: warning: argument name 'physical_file_number' in comment does not match parameter name 'target_file_number' [bugprone-argument-comment]
db/version_builder_test.cc:3892:9: warning: 'push_back' is called inside a loop; consider pre-allocating the container capacity before the loop [performance-inefficient-vector-operation]
db/version_builder_test.cc:4085:34: warning: argument name 'seq' in comment does not match parameter name 's' [bugprone-argument-comment]
db/version_builder_test.cc:4120:27: warning: argument name 'source_file_number' in comment does not match parameter name 'old_file_number' [bugprone-argument-comment]
db/version_builder_test.cc:4121:27: warning: argument name 'physical_file_number' in comment does not match parameter name 'target_file_number' [bugprone-argument-comment]
db/version_edit_test.cc (7 warning(s))
db/version_edit_test.cc:96:7: warning: passing result of std::move() as a const reference argument; no move will actually happen [performance-move-const-arg]
db/version_edit_test.cc:491:7: warning: argument name 'source_file_number' in comment does not match parameter name 'old_file_number' [bugprone-argument-comment]
db/version_edit_test.cc:492:34: warning: argument name 'source_file_number' in comment does not match parameter name 'old_file_number' [bugprone-argument-comment]
db/version_edit_test.cc:562:23: warning: argument name 'source_file_number' in comment does not match parameter name 'old_file_number' [bugprone-argument-comment]
db/version_edit_test.cc:563:23: warning: argument name 'physical_file_number' in comment does not match parameter name 'target_file_number' [bugprone-argument-comment]
db/version_edit_test.cc:566:31: warning: argument name 'source_file_number' in comment does not match parameter name 'old_file_number' [bugprone-argument-comment]
db/version_edit_test.cc:1058:45: warning: argument name 'generation' in comment does not match parameter name 'generation_file_number' [bugprone-argument-comment]
db/version_set.cc (1 warning(s))
db/version_set.cc:7350:7: warning: 'push_back' is called inside a loop; consider pre-allocating the container capacity before the loop [performance-inefficient-vector-operation]
db/wide/wide_column_serialization.cc (1 warning(s))
db/wide/wide_column_serialization.cc:352:55: warning: forwarding reference parameter 'visit' is never forwarded inside the function body [cppcoreguidelines-missing-std-forward]
db/wide/wide_column_serialization_test.cc (1 warning(s))
db/wide/wide_column_serialization_test.cc:943:51: warning: the parameter 'files' is copied for each invocation but only used as a const reference; consider making it a const reference [performance-unnecessary-value-param]
file/filename.cc (4 warning(s))
file/filename.cc:62:7: warning: class 'RepairCatalogReporter' defines a non-default destructor but does not define a copy constructor, a copy assignment operator, a move constructor or a move assignment operator [cppcoreguidelines-special-member-functions]
file/filename.cc:308:5: warning: multiple declarations in a single statement reduces readability [readability-isolate-declaration]
file/filename.cc:716:46: warning: string concatenation results in allocation of unnecessary temporary strings; consider using 'operator+=' or 'string::append()' instead [performance-inefficient-string-concatenation]
file/filename.cc:768:48: warning: string concatenation results in allocation of unnecessary temporary strings; consider using 'operator+=' or 'string::append()' instead [performance-inefficient-string-concatenation]
table/block_based/block_based_table_reader.cc (7 warning(s))
table/block_based/block_based_table_reader.cc:1662:21: warning: argument name 'need_upper_bound_check' in comment does not match parameter name 'disable_prefix_seek' [bugprone-argument-comment]
table/block_based/block_based_table_reader.cc:1672:3: warning: uninitialized record type: 'key_block' [cppcoreguidelines-pro-type-member-init]
table/block_based/block_based_table_reader.cc:2082:3: warning: uninitialized record type: 'trailer' [cppcoreguidelines-pro-type-member-init]
table/block_based/block_based_table_reader.cc:2882:35: warning: argument name 'aligned_buf' in comment does not match parameter name 'direct_io_buffer_context' [bugprone-argument-comment]
table/block_based/block_based_table_reader.cc:2969:7: warning: class 'BlockBasedBlobReferenceIterator' defines a non-default destructor but does not define a copy constructor, a copy assignment operator, a move constructor or a move assignment operator [cppcoreguidelines-special-member-functions]
table/block_based/block_based_table_reader.cc:3081:17: warning: argument name 'count' in comment does not match parameter name 'decoded_count' [bugprone-argument-comment]
table/block_based/block_based_table_reader.cc:5473:20: warning: argument name 'need_upper_bound_check' in comment does not match parameter name 'disable_prefix_seek' [bugprone-argument-comment]
tools/block_cache_analyzer/block_cache_trace_analyzer_test.cc (1 warning(s))
tools/block_cache_analyzer/block_cache_trace_analyzer_test.cc:883:7: warning: 'push_back' is called inside a loop; consider pre-allocating the container capacity before the loop [performance-inefficient-vector-operation]
xingbowang
force-pushed
the
export-D129346853
branch
2 times, most recently
from
November 15, 2026 05:07
4df86bb to
cf66f45
Compare
xingbowang
added a commit
to xingbowang/rocksdb
that referenced
this pull request
Nov 15, 2026
Summary: Pull Request resolved: meta#15683 ## Background This is diff 12 of 12. It tests the feature that diffs 1-11 add, and it uses these terms: - Standalone blob GC: with enable_blob_indirection on, a background job moves the live values out of blob files or relocation files that are at least half garbage, without rewriting any SST. Compaction blob GC (base BlobDB GC) moves a value only when compaction rewrites the SST that references it. - Relocation file: the new file that standalone blob GC writes with the moved values. - Route: a MANIFEST record that sends reads of the old blob file number to the relocation file that now holds its values. So SSTs keep their old references, and reads still find the values. - Route retirement: a background job that removes the routes that no SST and no running job needs any more, with one MANIFEST edit. - Census: the per-column-family index of which SSTs reference which blob files (BlobReferenceCensus). Route retirement uses it to pick routes that no SST needs. - Coarse SST: an SST that records only its oldest referenced blob file, not an exact list (flush output and some L0 compaction output). - Format fence: a MANIFEST record that marks a column family that may hold blob indirection data. It stays set after the option is turned off (see "MANIFEST records" in db/blob/blob_relocation_manifest.h). For the option, read the enable_blob_indirection comment in include/rocksdb/advanced_options.h. For a worked example of relocation files and routes, read the top comment of db/blob/blob_relocation_manifest.h. ## Why the change is required The Blob GC stack (diffs 1-11) had no stress coverage. Standalone blob GC, route retirement, turning enable_blob_indirection on and off, and crash recovery during a GC install were tested only by unit tests. Running this stress support during development found several stack bugs, including a data-loss bug when the option is turned off after GC has moved data. ## Parts of this diff 1. RocksDB library (small): one new public ticker, BLOB_RELOCATION_ROUTES_RETIRED, and 4 TEST_KILL_RANDOM points in standalone blob GC and route retirement. The crash test needs the kill points, and the db_stress report reads the ticker. 2. db_stress: new flags, startup checks, SetOptions choices, two reports, and two errors that db_stress now accepts. 3. db_crashtest.py: two parameter groups for blob indirection and a fourth kill mode. It uses the db_stress flags of part 2. ## How to read this diff 1. Read "Background" above. Then read the enable_blob_indirection comment in include/rocksdb/advanced_options.h and the top comment of db/blob/blob_relocation_manifest.h. 2. Library changes (they depend on nothing else here). Read the new ticker BLOB_RELOCATION_ROUTES_RETIRED in include/rocksdb/statistics.h and monitoring/statistics.cc. Then read the 4 TEST_KILL_RANDOM points and the new RecordTick() in db/db_impl/db_impl_compaction_flush.cc. 3. db_stress flags. Read the new flags in db_stress_tool/db_stress_gflags.cc (declared in db_stress_common.h). Then read where InitializeOptionsFromFlags() in db_stress_test_base.cc copies them into the options. Every later step uses these flags. 4. Startup rules and the debug hook. In db_stress_tool/db_stress_tool.cc, read the validation block and the sync point callbacks for blob_indirection_retire_routes_at_once. 5. Runtime behavior. Read the SetOptions table in StressTest::BuildOptionsTable() (db_stress_test_base.cc). Then read the two reports, ReportBlobIndirectionState() and ReportBlobIndirectionGc() (db_stress_test_base.h and .cc), and where db_stress_driver.cc calls them. 6. Accepted errors. Read IsRetryableBlobIndirectionIngestionError() in no_batched_ops_stress.cc and the local fallback in db_stress_compaction_service.cc. 7. Crash test last. In tools/db_crashtest.py, read _blob_indirection_gc_choice(), blob_indirection_params and blob_indirection_toggle_params, the blob indirection block in finalize_and_sanitize(), the blob_profiles choice in gen_cmd_params(), and kill mode 3 in whitebox_crash_main(). ## Previous behavior db_stress and db_crashtest.py could not turn on blob indirection, could not change it with SetOptions, and had no way to show whether standalone blob GC ran. When the remote worker used up its reserved blob file numbers, the db_stress compaction service returned a failure, and with local fallback off this became a background error. ## New behavior ### RocksDB library - New public ticker BLOB_RELOCATION_ROUTES_RETIRED ("rocksdb.blob.relocation.routes.retired"). It counts the routes that route retirement removed. Standalone blob GC reuses BLOB_DB_GC_NUM_FILES (input files collected) and BLOB_DB_GC_NUM_NEW_FILES (relocation files written). - New TEST_KILL_RANDOM points for whitebox crash tests: before GC renames its outputs, before and after the GC MANIFEST write, and before the route retirement MANIFEST write. ### db_stress - New flags: - enable_blob_indirection: the option, fixed for the run. - allow_setting_blob_indirection_dynamically: SetOptions turns the option on and off, mostly on. - blob_indirection_require_gc: fail the run if standalone blob GC collected no input file. It needs --statistics. - blob_indirection_max_references_per_sst: the per-SST blob reference cap of compaction outputs. - pin_blob_relocation_location_blocks: keep the location blocks of relocation files in memory. - blob_indirection_retire_routes_at_once (debug builds only): route retirement removes the routes it picked at once. By default it waits for a batch of 64 routes or for 10 minutes (kBlobRouteRetirementBatchSize and kBlobRouteRetirementMaxPendingMicros). - At startup, db_stress rejects these combinations: no blob files, blob direct write, FIFO, and a table format that is not block-based (the rules of ColumnFamilyData::ValidateOptions), and stacked BlobDB (a db_stress-only rule). It rejects FIFO even while the option is off, because a column family that has used blob indirection keeps rejecting FIFO. - When blob options are dynamic (allow_setting_blob_options_dynamically) and the option is on or may be turned on, SetOptions keeps blob files on, picks small blob file sizes, keeps blob GC on most of the time, picks the reference cap from 4096, 64K and 1M, and may change pin_blob_relocation_location_blocks. - At each start, db_stress prints the relocation files and routes that the DB holds. Whitebox crash tests kill most runs before the end-of-run report, so this line shows the GC work that earlier runs made durable. - At the end of a run, db_stress prints how many input files standalone blob GC collected, how many relocation files it wrote, and how many routes route retirement removed. For example, the line reads "Blob indirection GC: 104 input files retired, 44 relocation files written, 19 routes retired" (example counts; "retired" is the current log text). - Ingestion that returns TryAgain is dropped while the option may be on. Ingestion returns TryAgain when one of these became true after the external files were prepared: the option is on, the column family has the format fence, or a flush or compaction that was picked with the option on is still running. An ingestion that returns TryAgain ingests nothing. The check accepts any TryAgain, because that case has no status subcode of its own. - The db_stress compaction service runs a remote job locally when the worker uses up the range of blob file numbers that the primary reserved for it (Incomplete "Blob file number range exhausted"). Nothing is installed in that case. This gap exists without this stack too; small blob files only make it common. ### db_crashtest.py - Two parameter groups: fixed (the option stays the same across restarts) and toggle (SetOptions and each restart turn it on and off, on 3 of 4 times). When a run picks a blob profile, the fixed group and the toggle group are added to the choice with weights 4 and 1. --test_blob_indirection and --test_blob_indirection_toggle force them. - Both groups make GC run within a short test: - a small key space: 10000 keys 3 in 4 times, else 100000; - a write-heavy operation mix: 55% writes and 14% deletes; - small memtables, small SSTs and small blob files; - a reference cap of 4096, 64K or 1M, so that small caps often start a new compaction output. - Half of the runs run route retirement at once (blob_indirection_retire_routes_at_once). - Each db_stress start in the first 300 seconds after db_crashtest.py starts turns blob GC on. After that, each restart may start with it off. SetOptions may also turn it off for a while when blob options are dynamic. - A final sanitizing step turns the option off for combinations it does not support. - Whitebox runs with blob indirection add a fourth kill mode. It uses kill odds 20, excludes the frequent kill points (file writes, MANIFEST sync, CURRENT and LogAndApply), and runs route retirement at once. So the standalone blob GC and route retirement kill points get a chance to fire. The kill mode 3 comment in db_crashtest.py explains the choices. ## Implementation Stress options are set in InitializeOptionsFromFlags() and in the SetOptions table (StressTest::BuildOptionsTable()) in db_stress_test_base.cc. The GC report runs from the driver before statistics are printed. Data integrity is checked by the existing expected-state verification, the reopen checks and VerifyChecksum, which also reads relocation files. ## Safety boundaries and limitations - Test-only code, except the new ticker, the ticker recording in route retirement, and the TEST_KILL_RANDOM points (compiled out in release builds). - Public API: the new ticker is appended, so TICKER_ENUM_MAX changes. - No end-of-run check that every obsolete blob file or relocation file was deleted: injected MANIFEST faults keep files on purpose, and deletion can be asynchronous. - db_crashtest.py does not use blob_indirection_require_gc, because a short crash run may see no GC. - Blackbox runs are killed, so they do not print the end-of-run GC report. GC coverage was measured from the MANIFEST during development. Tracking issue: meta#15301 ## Stack context | Diff | Adds | |---|---| | 1 | Finalize blob metadata outside the DB mutex (prep) | | 2 | Indirect BlobIndex (kIndirectBlob) and exact SST reference metadata | | 3 | Relocation file format and stable read path | | 4 | MANIFEST routes and Version ownership | | 5 | Exact reverse reference census | | 6 | Standalone relocation GC (one input file) | | 7 | Pack whole input files; reclaim relocation files | | 8 | Flatten generation numbers; route retirement | | 9 | Remote compaction validation | | 10 | Dynamic options and operations | | 11 | RepairDB rebuilds relocation state; option gate removed | | 12 | db_stress and crash test support <== this diff | Differential Revision: D129346853
xingbowang
force-pushed
the
export-D129346853
branch
from
November 15, 2026 12:39
cf66f45 to
5ffa1cf
Compare
xingbowang
force-pushed
the
export-D129346853
branch
from
November 15, 2026 12:44
5ffa1cf to
49e4302
Compare
xingbowang
added a commit
to xingbowang/rocksdb
that referenced
this pull request
Nov 15, 2026
Summary: Pull Request resolved: meta#15683 ## Background This is diff 12 of 12. It tests the feature that diffs 1-11 add, and it uses these terms: - Standalone blob GC: with enable_blob_indirection on, a background job moves the live values out of blob files or relocation files that are at least half garbage, without rewriting any SST. Compaction blob GC (base BlobDB GC) moves a value only when compaction rewrites the SST that references it. - Relocation file: the new file that standalone blob GC writes with the moved values. - Route: a MANIFEST record that sends reads of the old blob file number to the relocation file that now holds its values. So SSTs keep their old references, and reads still find the values. - Route retirement: a background job that removes the routes that no SST and no running job needs any more, with one MANIFEST edit. - Census: the per-column-family index of which SSTs reference which blob files (BlobReferenceCensus). Route retirement uses it to pick routes that no SST needs. - Coarse SST: an SST that records only its oldest referenced blob file, not an exact list (flush output and some L0 compaction output). - Format fence: a MANIFEST record that marks a column family that may hold blob indirection data. It stays set after the option is turned off (see "MANIFEST records" in db/blob/blob_relocation_manifest.h). For the option, read the enable_blob_indirection comment in include/rocksdb/advanced_options.h. For a worked example of relocation files and routes, read the top comment of db/blob/blob_relocation_manifest.h. ## Why the change is required The Blob GC stack (diffs 1-11) had no stress coverage. Standalone blob GC, route retirement, turning enable_blob_indirection on and off, and crash recovery during a GC install were tested only by unit tests. Running this stress support during development found several stack bugs, including a data-loss bug when the option is turned off after GC has moved data. ## Parts of this diff 1. RocksDB library (small): one new public ticker, BLOB_RELOCATION_ROUTES_RETIRED, and 4 TEST_KILL_RANDOM points in standalone blob GC and route retirement. The crash test needs the kill points, and the db_stress report reads the ticker. 2. db_stress: new flags, startup checks, SetOptions choices, two reports, and two errors that db_stress now accepts. 3. db_crashtest.py: two parameter groups for blob indirection and a fourth kill mode. It uses the db_stress flags of part 2. ## How to read this diff 1. Read "Background" above. Then read the enable_blob_indirection comment in include/rocksdb/advanced_options.h and the top comment of db/blob/blob_relocation_manifest.h. 2. Library changes (they depend on nothing else here). Read the new ticker BLOB_RELOCATION_ROUTES_RETIRED in include/rocksdb/statistics.h and monitoring/statistics.cc. Then read the 4 TEST_KILL_RANDOM points and the new RecordTick() in db/db_impl/db_impl_compaction_flush.cc. 3. db_stress flags. Read the new flags in db_stress_tool/db_stress_gflags.cc (declared in db_stress_common.h). Then read where InitializeOptionsFromFlags() in db_stress_test_base.cc copies them into the options. Every later step uses these flags. 4. Startup rules and the debug hook. In db_stress_tool/db_stress_tool.cc, read the validation block and the sync point callbacks for blob_indirection_retire_routes_at_once. 5. Runtime behavior. Read the SetOptions table in StressTest::BuildOptionsTable() (db_stress_test_base.cc). Then read the two reports, ReportBlobIndirectionState() and ReportBlobIndirectionGc() (db_stress_test_base.h and .cc), and where db_stress_driver.cc calls them. 6. Accepted errors. Read IsRetryableBlobIndirectionIngestionError() in no_batched_ops_stress.cc and the local fallback in db_stress_compaction_service.cc. 7. Crash test last. In tools/db_crashtest.py, read _blob_indirection_gc_choice(), blob_indirection_params and blob_indirection_toggle_params, the blob indirection block in finalize_and_sanitize(), the blob_profiles choice in gen_cmd_params(), and kill mode 3 in whitebox_crash_main(). ## Previous behavior db_stress and db_crashtest.py could not turn on blob indirection, could not change it with SetOptions, and had no way to show whether standalone blob GC ran. When the remote worker used up its reserved blob file numbers, the db_stress compaction service returned a failure, and with local fallback off this became a background error. ## New behavior ### RocksDB library - New public ticker BLOB_RELOCATION_ROUTES_RETIRED ("rocksdb.blob.relocation.routes.retired"). It counts the routes that route retirement removed. Standalone blob GC reuses BLOB_DB_GC_NUM_FILES (input files collected) and BLOB_DB_GC_NUM_NEW_FILES (relocation files written). - New TEST_KILL_RANDOM points for whitebox crash tests: before GC renames its outputs, before and after the GC MANIFEST write, and before the route retirement MANIFEST write. ### db_stress - New flags: - enable_blob_indirection: the option, fixed for the run. - allow_setting_blob_indirection_dynamically: SetOptions turns the option on and off, mostly on. - blob_indirection_require_gc: fail the run if standalone blob GC collected no input file. It needs --statistics. - blob_indirection_max_references_per_sst: the per-SST blob reference cap of compaction outputs. - pin_blob_relocation_location_blocks: keep the location blocks of relocation files in memory. - blob_indirection_retire_routes_at_once (debug builds only): route retirement removes the routes it picked at once. By default it waits for a batch of 64 routes or for 10 minutes (kBlobRouteRetirementBatchSize and kBlobRouteRetirementMaxPendingMicros). - At startup, db_stress rejects these combinations: no blob files, blob direct write, FIFO, and a table format that is not block-based (the rules of ColumnFamilyData::ValidateOptions), and stacked BlobDB (a db_stress-only rule). It rejects FIFO even while the option is off, because a column family that has used blob indirection keeps rejecting FIFO. - When blob options are dynamic (allow_setting_blob_options_dynamically) and the option is on or may be turned on, SetOptions keeps blob files on, picks small blob file sizes, keeps blob GC on most of the time, picks the reference cap from 4096, 64K and 1M, and may change pin_blob_relocation_location_blocks. - At each start, db_stress prints the relocation files and routes that the DB holds. Whitebox crash tests kill most runs before the end-of-run report, so this line shows the GC work that earlier runs made durable. - At the end of a run, db_stress prints how many input files standalone blob GC collected, how many relocation files it wrote, and how many routes route retirement removed. For example, the line reads "Blob indirection GC: 104 input files retired, 44 relocation files written, 19 routes retired" (example counts; "retired" is the current log text). - Ingestion that returns TryAgain is dropped while the option may be on. Ingestion returns TryAgain when one of these became true after the external files were prepared: the option is on, the column family has the format fence, or a flush or compaction that was picked with the option on is still running. An ingestion that returns TryAgain ingests nothing. The check accepts any TryAgain, because that case has no status subcode of its own. - The db_stress compaction service runs a remote job locally when the worker uses up the range of blob file numbers that the primary reserved for it (Incomplete "Blob file number range exhausted"). Nothing is installed in that case. This gap exists without this stack too; small blob files only make it common. ### db_crashtest.py - Two parameter groups: fixed (the option stays the same across restarts) and toggle (SetOptions and each restart turn it on and off, on 3 of 4 times). When a run picks a blob profile, the fixed group and the toggle group are added to the choice with weights 4 and 1. --test_blob_indirection and --test_blob_indirection_toggle force them. - Both groups make GC run within a short test: - a small key space: 10000 keys 3 in 4 times, else 100000; - a write-heavy operation mix: 55% writes and 14% deletes; - small memtables, small SSTs and small blob files; - a reference cap of 4096, 64K or 1M, so that small caps often start a new compaction output. - Half of the runs run route retirement at once (blob_indirection_retire_routes_at_once). - Each db_stress start in the first 300 seconds after db_crashtest.py starts turns blob GC on. After that, each restart may start with it off. SetOptions may also turn it off for a while when blob options are dynamic. - A final sanitizing step turns the option off for combinations it does not support. - Whitebox runs with blob indirection add a fourth kill mode. It uses kill odds 20, excludes the frequent kill points (file writes, MANIFEST sync, CURRENT and LogAndApply), and runs route retirement at once. So the standalone blob GC and route retirement kill points get a chance to fire. The kill mode 3 comment in db_crashtest.py explains the choices. ## Implementation Stress options are set in InitializeOptionsFromFlags() and in the SetOptions table (StressTest::BuildOptionsTable()) in db_stress_test_base.cc. The GC report runs from the driver before statistics are printed. Data integrity is checked by the existing expected-state verification, the reopen checks and VerifyChecksum, which also reads relocation files. ## Safety boundaries and limitations - Test-only code, except the new ticker, the ticker recording in route retirement, and the TEST_KILL_RANDOM points (compiled out in release builds). - Public API: the new ticker is appended, so TICKER_ENUM_MAX changes. - No end-of-run check that every obsolete blob file or relocation file was deleted: injected MANIFEST faults keep files on purpose, and deletion can be asynchronous. - db_crashtest.py does not use blob_indirection_require_gc, because a short crash run may see no GC. - Blackbox runs are killed, so they do not print the end-of-run GC report. GC coverage was measured from the MANIFEST during development. Tracking issue: meta#15301 ## Stack context | Diff | Adds | |---|---| | 1 | Finalize blob metadata outside the DB mutex (prep) | | 2 | Indirect BlobIndex (kIndirectBlob) and exact SST reference metadata | | 3 | Relocation file format and stable read path | | 4 | MANIFEST routes and Version ownership | | 5 | Exact reverse reference census | | 6 | Standalone relocation GC (one input file) | | 7 | Pack whole input files; reclaim relocation files | | 8 | Flatten generation numbers; route retirement | | 9 | Remote compaction validation | | 10 | Dynamic options and operations | | 11 | RepairDB rebuilds relocation state; option gate removed | | 12 | db_stress and crash test support <== this diff | Differential Revision: D129346853
xingbowang
force-pushed
the
export-D129346853
branch
from
November 18, 2026 16:55
49e4302 to
0b52ec3
Compare
xingbowang
force-pushed
the
export-D129346853
branch
from
November 18, 2026 18:39
0b52ec3 to
c951342
Compare
xingbowang
force-pushed
the
export-D129346853
branch
from
November 18, 2026 18:44
c951342 to
65be358
Compare
xingbowang
added a commit
to xingbowang/rocksdb
that referenced
this pull request
Nov 18, 2026
Summary: Pull Request resolved: meta#15683 ## Background This is diff 15 of 15. It tests the feature that diffs 1-14 add, and it uses these terms: - Indirect blob GC: with enable_blob_indirection on, a background job moves the live values out of blob files or relocation files that are at least half garbage, without rewriting any SST. Compaction-time blob GC (base BlobDB GC) moves a value only when compaction rewrites the SST that references it. - Relocation file: the new file that indirect blob GC writes with the moved values. - Route: a MANIFEST record that sends reads of the old blob file number to the relocation file that holds its values. So SSTs keep their old references, and reads still find the values. - Route retirement: a background job that removes the routes that no SST and no running job needs any more, with one MANIFEST edit. - Census: the per-column-family index of which SSTs reference which blob files (BlobReferenceCensus). Route retirement uses it to pick routes that no SST needs. - Coarse SST: an SST that records only its oldest referenced blob file, not an exact list (flush output and some L0 compaction output). - Format fence: a MANIFEST record that marks a column family that may hold blob indirection data. It stays set after the option is turned off (see "MANIFEST records" in db/blob/blob_relocation_manifest.h). For the option, read the enable_blob_indirection comment in include/rocksdb/advanced_options.h. For a worked example of relocation files and routes, read the top comment of db/blob/blob_relocation_manifest.h. ## Why the change is required Without this diff, the Blob GC stack (diffs 1-14) has no stress coverage. Indirect blob GC, route retirement, turning enable_blob_indirection on and off, and crash recovery during a GC install are tested only by unit tests. ## Parts of this diff 1. RocksDB library (small): one new public ticker, BLOB_RELOCATION_ROUTES_RETIRED, and 4 TEST_KILL_RANDOM points in indirect blob GC and route retirement. The crash test needs the kill points, and the db_stress report reads the ticker. 2. db_stress: new flags, startup checks, SetOptions choices, two reports, and two errors that db_stress accepts. 3. db_crashtest.py: two parameter groups for blob indirection and a fourth kill mode. It uses the db_stress flags of part 2. ## How to read this diff 1. Read "Background" above. Then read the enable_blob_indirection comment in include/rocksdb/advanced_options.h and the top comment of db/blob/blob_relocation_manifest.h. 2. Library changes (they depend on nothing else here). Read the new ticker BLOB_RELOCATION_ROUTES_RETIRED in include/rocksdb/statistics.h and monitoring/statistics.cc. Then read the 4 TEST_KILL_RANDOM points and the new RecordTick() in db/db_impl/db_impl_compaction_flush.cc. 3. db_stress flags. Read the new flags in db_stress_tool/db_stress_gflags.cc (declared in db_stress_common.h). Then read where InitializeOptionsFromFlags() in db_stress_test_base.cc copies them into the options. Every later step uses these flags. 4. Startup rules and the debug hook. In db_stress_tool/db_stress_tool.cc, read the validation block and the sync point callbacks for blob_indirection_retire_routes_at_once. 5. Runtime behavior. Read the SetOptions table in StressTest::BuildOptionsTable() (db_stress_test_base.cc). Then read the two reports, ReportBlobIndirectionState() and ReportBlobIndirectionGc() (db_stress_test_base.h and .cc), and where db_stress_driver.cc calls them. 6. Accepted errors. Read IsRetryableBlobIndirectionIngestionError() in no_batched_ops_stress.cc and the local fallback in db_stress_compaction_service.cc. 7. Crash test last. In tools/db_crashtest.py, read _blob_indirection_gc_choice(), blob_indirection_params and blob_indirection_toggle_params, the blob indirection block in finalize_and_sanitize(), the blob_profiles choice in gen_cmd_params(), and kill mode 3 in whitebox_crash_main(). ## Previous behavior db_stress and db_crashtest.py could not turn on blob indirection, could not change it with SetOptions, and had no way to show whether indirect blob GC ran. When the remote worker used up its reserved blob file numbers, the db_stress compaction service returned a failure, and with local fallback off this became a background error. ## New behavior ### RocksDB library - New public ticker BLOB_RELOCATION_ROUTES_RETIRED ("rocksdb.blob.relocation.routes.retired"). It counts the routes that route retirement removed. Indirect blob GC reuses BLOB_DB_GC_NUM_FILES (input files collected) and BLOB_DB_GC_NUM_NEW_FILES (relocation files written). - New TEST_KILL_RANDOM points for whitebox crash tests: before GC renames its outputs, before and after the GC MANIFEST write, and before the route retirement MANIFEST write. ### db_stress - New flags: - enable_blob_indirection: the option, fixed for the run. - allow_setting_blob_indirection_dynamically: SetOptions turns the option on and off, mostly on. - blob_indirection_require_gc: fail the run if indirect blob GC collected no input file. It needs --statistics. - blob_indirection_max_references_per_sst: the per-SST blob reference cap of compaction outputs. - pin_blob_relocation_location_blocks: keep the location blocks of relocation files in memory. - blob_indirection_retire_routes_at_once (debug builds only): route retirement removes the routes it picked at once. By default it waits for a batch of 64 routes or for 10 minutes (kBlobRouteRetirementBatchSize and kBlobRouteRetirementMaxPendingMicros). - When enable_blob_indirection or allow_setting_blob_indirection_dynamically is set, db_stress rejects these combinations at startup: no blob files, blob direct write, FIFO, and a table format that is not block-based (the rules of ColumnFamilyData::ValidateOptions), and stacked BlobDB (a db_stress-only rule). It rejects FIFO also when only allow_setting_blob_indirection_dynamically is set and the option starts off, because a column family that has used blob indirection keeps rejecting FIFO. - When blob options are dynamic (allow_setting_blob_options_dynamically) and the option is on or may be turned on, SetOptions keeps blob files on, picks small blob file sizes, keeps enable_blob_garbage_collection on most of the time, picks the reference cap from 4096, 64K and 1M, and may change pin_blob_relocation_location_blocks. - At each start, db_stress prints the relocation files and routes that the DB holds. Whitebox crash tests kill most runs before the end-of-run report, so this line shows the GC work that earlier runs made durable. - At the end of a run, db_stress prints how many input files indirect blob GC collected, how many relocation files it wrote, and how many routes route retirement removed. For example, the line reads "Blob indirection GC: 104 input files retired, 44 relocation files written, 19 routes retired" (example counts; "retired" is the current log text). - While the option may be on, db_stress accepts TryAgain from file ingestion instead of failing the run. Ingestion returns TryAgain when one of these became true after the external files were prepared: the option is on, the column family has the format fence, or a flush or compaction that was picked with the option on is still running. An ingestion that returns TryAgain ingests nothing. The check accepts any TryAgain, because that case has no status subcode of its own. - The db_stress compaction service runs a remote job locally when the worker uses up the range of blob file numbers that the primary reserved for it (Incomplete "Blob file number range exhausted"). Nothing is installed in that case. This gap exists without this stack too; small blob files only make it common. ### db_crashtest.py - Two parameter groups: fixed (the option stays the same across restarts) and toggle (SetOptions and each restart turn it on and off, on 3 of 4 times). When a run picks a blob profile, the fixed group and the toggle group are added to the choice with weights 4 and 1. --test_blob_indirection and --test_blob_indirection_toggle force them. - Both groups make indirect blob GC run within a short test: - a small key space: 10000 keys 3 in 4 times, else 100000; - a write-heavy operation mix: 55% writes and 14% deletes; - small memtables, small SSTs and small blob files; - a reference cap of 4096, 64K or 1M, so that small caps often start a new compaction output. - Half of the runs run route retirement at once (blob_indirection_retire_routes_at_once). The fourth kill mode (see below) always does. - Each db_stress start in the first 300 seconds after the first db_stress command turns enable_blob_garbage_collection on (300 seconds is a fixed choice, not measured). After that, each restart may start with it off. SetOptions may also turn it off for a while when blob options are dynamic. - A step in finalize_and_sanitize() turns off both enable_blob_indirection and allow_setting_blob_indirection_dynamically when the run has no blob files, blob direct write, stacked BlobDB, FIFO or a full external table (the db_stress startup rules). It runs after the other sanitizing steps, because they can change these options. - Whitebox runs with blob indirection add a fourth kill mode. It sets kill_random_test to 20, so each kill point that is not excluded crashes the process about once in 20 calls. It excludes the frequent kill points (file writes, MANIFEST sync, CURRENT and LogAndApply), and runs route retirement at once. So the indirect blob GC and route retirement kill points get a chance to fire. The kill mode 3 comment in db_crashtest.py explains the choices. ## Implementation Stress options are set in InitializeOptionsFromFlags() and in the SetOptions table (StressTest::BuildOptionsTable()) in db_stress_test_base.cc. The indirect blob GC report runs from the driver before statistics are printed. Data integrity is checked by the existing expected-state verification, the reopen checks and VerifyChecksum, which also reads relocation files. ## Safety boundaries and limitations - Test-only code, except the new ticker, the ticker recording in route retirement, and the TEST_KILL_RANDOM points (compiled out in release builds). - Public API: the new ticker is appended, so TICKER_ENUM_MAX changes. - No end-of-run check that every obsolete blob file or relocation file was deleted: injected MANIFEST faults keep files on purpose, and deletion can be asynchronous. - db_crashtest.py does not use blob_indirection_require_gc, because a short crash run may see no indirect blob GC run. - Blackbox runs are killed, so they do not print the end-of-run GC report. The start-of-run line (relocation files and routes in the DB) shows the GC work of earlier runs. Tracking issue: meta#15301 ## Stack context | Diff | Adds | |---|---| | 1 | Finalize blob metadata outside the DB mutex (prep) | | 2 | Indirect BlobIndex (kIndirectBlob) and exact SST reference metadata | | 3 | Relocation file format and stable read path | | 4 | MANIFEST routes and relocation state in the Version | | 5 | Exact reverse reference census | | 6 | MANIFEST apply outside the DB mutex, with a precondition re-check | | 7 | Coarse flush metadata, per-SST reference cap, epoch runs | | 8 | Indirect blob GC building blocks | | 9 | Indirect blob GC job | | 10 | Flatten generation numbers; route retirement | | 11 | Remote compaction validation | | 12 | Dynamic options and operations | | 13 | Lost relocation files | | 14 | RepairDB rebuilds relocation state; option gate removed | | 15 | db_stress and crash test support <== this diff | Differential Revision: D129346853
xingbowang
force-pushed
the
export-D129346853
branch
from
November 18, 2026 18:55
65be358 to
f9a3440
Compare
…a#15323) Summary: Pull Request resolved: meta#15323 Background: This diff moves the consistency check of each new Version, and the saving of its blob files, out of the DB mutex. A Version is the in-memory list of the SST files and blob files of one column family (CF). VersionSet::LogAndApply builds each new Version with a VersionBuilder and installs it while it holds the DB mutex. Each VersionBuilder starts from a base Version: the current Version of its CF when the builder was made. In this diff, "finalizing the blob metadata" of a new Version means two things: adding the blob files to the new Version, and then checking the consistency of the whole new Version. Later diffs in the indirect blob GC stack add more blob state to each Version in the same step. So building a new Version now has two steps. Step 1 (under the DB mutex) adds the SST files and compact cursors. Step 2 (without the DB mutex) adds the blob files and checks the consistency of the new Version. If this work runs under the DB mutex, it delays unrelated DB operations, and the delay grows with the number of files. In this diff, the unlocked step is mostly the consistency check of the new Version. When force_consistency_checks is on (the default) or in debug builds, this check walks every SST. We estimate, without a measurement, that saving the blob metadata costs little in this diff, because it mostly copies shared pointers for the blob files. Later diffs add more blob work to the same unlocked step. Terms used in this diff: - ColumnFamilyData (CFD): the in-memory object of one CF. It is reference counted, and a dropped CF is freed when its last reference is released. - SuperVersion: the reference-counted set of memtables and the current Version that readers use. Each SuperVersion holds one CFD reference. - MANIFEST writer queue: LogAndApply puts one writer (a ManifestWriter) per CF into this queue. The writer at the head of the queue processes a batch of queued writers. The thread that owns that writer is the leader; the thread that called LogAndApply for a writer is its caller. - Writer callback: a function that a LogAndApply caller passes per writer (the `manifest_wcbs` argument, stored as ManifestWriter::manifest_write_callback). Flush uses it to remove the flushed memtables. - No-MANIFEST LogAndApply call: a call that passes dummy edits (VersionEdit::MarkNoManifestWriteDummy). It installs new Versions but writes no MANIFEST record. Only SetOptions uses it. In ProcessManifestWrites, such a batch has skip_manifest_write set. - Dummy flag: the mark set by VersionEdit::MarkNoManifestWriteDummy() on a dummy edit. A test can clear it on the SetOptions edits through the sync point DBImpl::SetOptions:dummy_edit, so that SetOptions runs through the MANIFEST path. - pre_cb: a callback that LogAndApply runs under the DB mutex when the caller's first writer reaches the head of the MANIFEST writer queue. - Options map: the option names and values that one SetOptions call gets for one column family handle. - Phasing anchor (compaction_phase_anchor_time): the time from which periodic catch-up compactions are spread out. SetOptions moves it to the current time when an options map sets periodic_compaction_seconds, even to its current value, and the call succeeds. It does not move if the call returns early because no option changes. See VersionSet::PrepareCompactionPhaseReanchor() in version_set.h. Why the change is required: - Blob metadata finalization must run without the DB mutex before later diffs add work to it. This diff moves finalization, and the consistency check of the new Version that follows it, out of the mutex. It changes no format. - SetOptions must change too. Before this diff, SetOptions installs the new options in pre_cb. It relies on LogAndApply keeping the DB mutex held until the new Version is installed. With this diff, the no-MANIFEST path of LogAndApply also releases the mutex during blob finalization. SetOptions builds its new Versions through the same ProcessManifestWrites code, so they need the same step 2 (the consistency check that walks every SST, and in later diffs the derived blob state). Holding the mutex only for SetOptions would keep that cost under the mutex, once for every CF that SetOptions touches. If SetOptions still installed the options in pre_cb, a reader could see the new options with the old Version. So SetOptions now only prepares the options in pre_cb, and LogAndApply publishes them together with the new Versions. The phasing anchor follows the same rule, so it now moves in two steps (prepare, then publish with the new Versions). - SetOptions now makes one LogAndApply call for all its CFs, so that the options and the Versions are published together. Because there is one call, the result is all-or-nothing: either every CF changes or none does. In one call, all writers of one CF share one new Version, built with the options of the first writer. So when two handles point to the same CF, SetOptions merges their options maps into one writer. Otherwise the second map would be lost. - Writer callbacks must run in the same DB mutex hold as the install, and at the end of the batch: - Existing race: before this diff, atomic flush removed the flushed memtables only after LogAndApply returned. Another thread can lead the batch and install the new Versions of this flush. Until the flush thread wakes up, takes the DB mutex and removes the memtables, other code under the DB mutex sees both the new SST files and the flushed memtables. This diff runs every callback in the mutex hold of the install, and atomic flush uses a callback too. This fixes the race. - End of the batch: as in base RocksDB, the callbacks run when the writers of the batch are marked done, with no DB mutex release in between. A Flush(wait=true) caller returns as soon as its memtables are removed. When the flush thread leads its batch, it installs its new SuperVersion right after LogAndApply() returns, without releasing the DB mutex. If more flushes finished during the MANIFEST write, MemTableList::TryInstallMemtableFlushResults() first commits them in another LogAndApply() round, which can release the DB mutex. A flush writer that another leader takes into its batch installs its SuperVersion after it takes the DB mutex again. Base RocksDB works the same way in both cases. Parts of this diff: 1. Unlocked blob finalization (the core): VersionBuilder saves a Version in two steps, and LogAndApply runs the second step without the DB mutex. 2. Consistency check of the builder's base Version once per builder. You can skip this part on a first read. It does not depend on the other parts. 3. All-or-nothing SetOptions: the options are prepared in pre_cb and published with the new Versions. The core needs it because the no-MANIFEST path now releases the DB mutex too, so that the new Versions of SetOptions get the same unlocked step 2 (see Why). 4. Writer callbacks run once per batch, when the writers are marked done, and atomic flush uses one. You can skip this part on a first read. Later diffs need it, and it does not depend on part 3. 5. Small items (see Implementation). You can skip these on a first read. - finish_batch signals every writer, because a SetOptions call now puts all its CFs in one batch, and the old search took quadratic time for that batch. - A null CFD in a batch returns Corruption. The check replaces an assert in code that this diff rewrites. How to read this diff: 1. Read the summary sections Why, Parts of this diff and "New behavior: building a Version". 2. Start with db/version_builder.h (SaveTableFilesTo, SaveBlobFilesAndCheckTo) and their bodies in db/version_builder.cc. They depend on nothing new. Then read base_checked_ and VersionBuilder::Apply in db/version_builder.cc (part 2). 3. Read ProcessManifestWrites in db/version_set.cc: the finalize_blob_files lambda, the two unlock paths (no-MANIFEST and MANIFEST), the comment and debug checks after mu->Lock() in the MANIFEST path, and the blob_finalization_failed branch. This is where step 2 of the save runs without the mutex. Only SetOptions uses the no-MANIFEST path. Releasing the mutex there is safe only because SetOptions now publishes its options at install time (see reading steps 5 to 7 below). 4. In the same function, read publish_batch_callbacks and finish_batch. Then read InstallMemtableAtomicFlushResults in db/memtable_list.cc (part 4). 5. Read the pieces that SetOptions needs: PrepareMutableCFOptions and PublishMutableCFOptions in db/column_family.h, ParamsForCfAtAnchor in db/periodic_compaction_phaser.h, and MutableCFOptionsPublication, the phasing anchor comment above PrepareCompactionPhaseReanchor, and the LogAndApply comment in db/version_set.h. 6. Read how these pieces go through LogAndApply in db/version_set.cc: the new ManifestWriter fields, how the new Version is built, the skip_manifest_write grouping, and the install block that publishes the options and the anchor. 7. Read DBImpl::SetOptions in db/db_impl/db_impl.cc. The comment at its top lists its steps. It groups the maps by CF, builds one dummy edit per CF, defines pre_cb (LogAndApply runs it under the DB mutex), makes one LogAndApply call, installs the new SuperVersions, and writes one OPTIONS file. Previous behavior: - LogAndApply called VersionBuilder::SaveTo under the DB mutex. SaveTo added the SST files, blob files and compact cursors to the new Version, and then checked its consistency. - VersionBuilder::Apply checked the consistency of the base Version on every edit. - SetOptions made one no-MANIFEST LogAndApply call per CF. Each call installed the options of its CF in pre_cb. When a map set periodic_compaction_seconds, SetOptions moved the phasing anchor to the current time before its first LogAndApply call (VersionSet::ReanchorCompactionPhase), so the anchor moved even when the call failed later. If a later CF failed, the earlier CFs kept their new options, and the next SetOptions call finished the update. - Each writer callback ran when its writer was taken off the queue at the end of the batch. Atomic flush updated its memtable lists after LogAndApply returned. New behavior: building a Version 1. Under the DB mutex, VersionBuilder::SaveTableFilesTo adds the SST files and compact cursors to each new Version. 2. LogAndApply releases the DB mutex. VersionBuilder::SaveBlobFilesAndCheckTo adds the blob files. Then it builds the list of dropped blob files (VersionStorageInfo::GetDroppedBlobFiles()). A dropped blob file is a missing blob file that an open with best_efforts_blob_db_repair recorded. Then it checks the consistency of the complete new Version (SST and blob state). The builder keeps its base Version referenced during this step. The batch stays at the head of the MANIFEST writer queue, so no other thread installs a Version for these CFs in the meantime. 3. LogAndApply takes the DB mutex again and installs the new Versions. In the no-MANIFEST path (used by SetOptions), Version::PrepareAppend (which calls VersionStorageInfo::PrepareForVersionAppend) also runs in step 2, without the mutex. The MANIFEST path already ran it without the mutex. If finalization fails, no MANIFEST record has been written. LogAndApply returns the error, deletes the unpublished Versions and keeps the current MANIFEST writer, the same as for a SaveTableFilesTo failure. It does not log "Error in committing version edit to MANIFEST" and does not force a new MANIFEST file. New behavior: consistency check of the builder's base Version VersionBuilder::Apply checks the consistency of the base Version only until the first check passes, so at most once per builder when the base Version is consistent. This is safe because the base Version never changes during the builder's life. It applies to all DBs and every builder user: LogAndApply, recovery and point-in-time recovery. Point-in-time recovery can undo the last edit with RollbackLastApply, but this does not change the base Version. With force_consistency_checks (true by default), a builder that applies many edits does not walk every SST of the base Version under the DB mutex for each edit. A failed check is not remembered, so the next Apply reports it again. New behavior: SetOptions The steps are listed in one place: the comment at the top of DBImpl::SetOptions (db/db_impl/db_impl.cc). The contract: - The call is all-or-nothing. If preparation fails for any CF, no CF's options change and no OPTIONS file is written. - Readers never see a mix of old and new options and Versions. LogAndApply publishes the options, the phasing anchor and the new Versions in one DB mutex hold, and SetOptions installs the new SuperVersions before it releases the mutex. - One call writes one OPTIONS file. A failed OPTIONS file write sets options_file_stale_, so the next call writes the file again. - Options maps given through different handles of the same CF go into one writer and are applied in map iteration order. - The batch of one SetOptions call holds only the writers of that call (see step 2 in the MutableCFOptionsPublication comment in version_set.h). Test-only MANIFEST path: a test can clear the dummy flag of the SetOptions edits (see Terms), so that SetOptions writes a real MANIFEST record. The options are then published only after that write succeeds. Grouping could then mix SetOptions writers with the writers of other callers. So a writer with prepared options is never grouped with another caller's writers, before or after: once the batch holds all writers of the leader's caller, it stops if the leader has prepared options, or before a next writer that has prepared options. New behavior: writer callbacks All writer callbacks of a batch run once, under the DB mutex, at the end of the batch, right before the writers are marked done. On success this is in the mutex hold of the install. The full contract (failure, early return, no user code or I/O) is in the comment of VersionSet::LogAndApply in version_set.h. Atomic flush passes one callback, on its first writer, that updates the memtable lists of the whole group. So the new Versions and the removal of their flushed memtables happen in one critical section. If LogAndApply returns before the batch starts (every CF of the group was dropped while its writers waited), no callback runs, and atomic flush updates the memtable lists itself after LogAndApply returns. Implementation: - VersionBuilder::SaveTo is split into SaveTableFilesTo (under the mutex) and SaveBlobFilesAndCheckTo (without the mutex). SaveTo still calls both. ProcessManifestWrites keeps the builder guards (BaseReferencedVersionBuilder) alive across the unlocked part. - ColumnFamilyData::PrepareMutableCFOptions parses and validates options without publishing them. ColumnFamilyData::PublishMutableCFOptions publishes them. - LogAndApply gets one new trailing parameter, a MutableCFOptionsPublication pointer (null by default). It holds the per-CF prepared options and the phasing anchor time. When it is set, the CFs must be distinct (checked in debug builds). The Version constructor takes optional PeriodicCompactionPhaseParams (std::nullopt by default), so a new Version can use the new anchor before it is committed. VersionSet::ReanchorCompactionPhase is replaced by PrepareCompactionPhaseReanchor (read the clock) and CommitCompactionPhaseReanchor (commit under the DB mutex). - finish_batch signals every writer of the batch. Only the first writer of each caller has a thread that waits on it, so the other signals do nothing. Before this diff, finish_batch searched the leader's own writers for each writer of the batch, so it could skip the signal for them. This search takes time quadratic in the number of writers of a large multi-CF call. Signaling every writer removes it. - A writer with a null CFD in a batch that is not a CF add or drop now returns Corruption. Before this diff, the code asserted, and release builds dereferenced the null pointer. Safety boundaries and unsupported cases: - No persisted format or public option changes. - Group commit of several writers for the same CF is unchanged. - SaveTableFilesTo still runs under the mutex, and its cost is linear in the number of SST files. So building a Version under the mutex is not constant time. - Not changed: an iterator, a GetEntityLazy result and a pinned GetMergeOperands result keep a dropped CF alive only through the CFD reference of their SuperVersion, as before. This diff adds a test for this, because lazy blob reads in later diffs depend on it. Validation and known limitations: - Unit tests cover SetOptions (all-or-nothing, two handles of the same CF, one OPTIONS file, no mixed view, phase anchor), unlocked blob finalization and its failure, the base Version consistency check, atomic flush when every column family is dropped while it waits, and reads that outlive a CF drop. See the Test Plan. - Performance: a db_bench run measured the DB mutex. With about 20K to 25K SSTs (W1 in the Test Plan) and frequent flush and compaction installs, the DB mutex hold time of one LogAndApply drops by about half (p50 6.67 ms to 3.13 ms, p99 13.5 ms to 5.85 ms). The total DB mutex wait drops by 20%, and reads per second rise by 6.7%. With 4 CFs and atomic flush (W2), the hold time drops by about a third and the read p99.9 latency drops by 24%. A small DB with about 110 SSTs at start (W3) shows no change. Most of the hold time that remains is SaveTableFilesTo, which still walks every SST under the mutex. - Most of the saving likely comes from consistency checks, which walk every SST (see Background): the check of the new Version now runs without the DB mutex, and the check of the base Version in Apply runs once per builder instead of once per edit. We estimate that saving the blob file metadata itself costs little. The benchmark did not measure these parts separately. Tracking issue: meta#15301 Differential Revision: D123408321
Summary: Pull Request resolved: meta#15540 Why the change is required: The goal of this stack is indirect blob GC, a blob GC that does not depend on SSTs: it removes garbage from blob files without rewriting any SST. Such a GC needs two things that RocksDB does not have: - A value identity that does not change when the value moves to another file. - A checked list of the blob references that each SST keeps. The list must include older internal versions of a key that the SST still keeps (for example for snapshots), and the blob columns of wide-column entities. This diff adds both. Later diffs build the census (an in-memory reverse index from generation number to the SSTs that use it), GC and repair on top of this list. This diff in one paragraph: With the new column family option enable_blob_indirection on, the blob file writer gives each new value an indirect BlobIndex (type 3, kIndirectBlob). It stores the value's stable identity (origin file, original offset) and its CRCs (an uncompressed value stores only the CRC of the uncompressed value; a compressed value also stores the CRC of its stored bytes). Also with the option on, the block-based table builder collects every blob reference that the SST keeps. It writes them as "exact reference metadata": data blocks, one reference index block, and a 72-byte descriptor table property. In this diff only the table builder writes this metadata, and only tests read it. A temporary option gate (see "option gate" under Terms) keeps the option off outside tests. Separately, wide-column entities with blob columns are now fully checked before their blob indexes are used, also with the option off. Parts of this diff: 1. Indirect BlobIndex (core): types 3 and 4 in db/blob/blob_index.h, and the blob file writer that writes type 3. 2. Exact reference metadata format and decoders (core): db/blob/blob_reference_metadata.{h,cc}. 3. Collector and table builder integration (core): BlobReferenceMetadataCollector, its block cache charge, and BlockBasedTableBuilder. 4. Option enable_blob_indirection and the option gate. Tests need it to turn the feature on. You can skip this part on a first read. 5. Wide-column validation (WideColumnSerialization::ForEachBlobIndex). The table builder uses it to give the blob columns of each entity to the collector. It also changes behavior with the option off (see New behavior). You can skip this part on a first read. 6. Small helpers: canonical varints (util/coding.h) and CacheReservationHandle::Resize(). You can skip these on a first read. How to read this diff: 1. Overview: read "Why" and "This diff in one paragraph" above, and "Terms used in this diff" right below. Then read the top comment of db/blob/blob_reference_metadata.h (terms, layout diagrams, run and entry layout, index layout). 2. Indirect BlobIndex (db/blob/blob_index.h) and canonical varints (util/coding.h GetCanonicalVarint32/64): read the overview comment and the kIndirectBlob / kIndirectFlattenedBlob code. Every later part uses these fields: origin, original offset, generation number, logical CRC and stored CRC. The new formats read varints only in the canonical form. 3. Blob file writer (db/blob/blob_file_builder.cc, blob_log_format.cc, blob_log_writer.cc): see how a new value gets a type-3 index and how its CRC is computed once. 4. Format types and decoders: the structs in db/blob/blob_reference_metadata.h, then in the .cc the constants, GapBase, DecodeEntryBody, DecodeRunEntriesAt, DecodeBlobReferenceIndex and BlobReferenceGroupDecoder. They use only BlobIndex and the canonical varints. Learn the on-disk format here, before you read the writer. 5. Collector and its cache charge: first read the new CacheEntryRole kBlobReferenceMetadataBuildingBuffer (include/rocksdb/cache.h) and CacheReservationHandle::Resize (cache/cache_reservation_manager.h). Then read BlobReferenceMetadataCollector in this order: the class comment, Add, AppendSorted, BuildAndWriteDataBlocks, EncodeIndex, FinishOnce. Then read the unsorted fallback (AppendUnsorted, SortUnsortedGroup, SortAndEncodeGroup). 6. Wide-column change (db/wide/wide_column_serialization.cc, ForEachBlobIndex and WalkV2Columns): it gives every blob index of an entity to a callback after it checks the whole entity. The table builder uses it to feed the collector. UpdateBoundaries, BlobGarbageMeter and the compaction iterator also use it, with the option on or off. 7. Option: enable_blob_indirection (include/rocksdb/advanced_options.h, options/cf_options.cc), and the option checks and gate in db/column_family.cc (ValidateOptions, TEST_SetAllowBlobIndirection), db/db_impl/db_impl_open.cc and db_impl_secondary.cc (the FIFO check for opens that skip ValidateOptions). 8. Table integration: table/block_based/block_based_table_factory.cc (the shared reservation manager), then block_based_table_builder.cc (Rep constructor, AddImpl, WriteBlobReferenceMetadata, EstimatedTailSize). See where the collector is created, fed (AddImpl(), which both Add() and TryAdd() call) and finished. block_based_table_reader.cc maps the new block name to a block type. Terms used in this diff: Stack terms: | Term | Meaning | |---|---| | origin | Blob file where the value was first written | | original offset | Value offset in the origin file | | stable identity | (origin, original offset). Never changes, even when the value moves | | generation number | The file number that a reference uses to find its value. A new value has generation number == origin. When indirect blob GC moves the value, the reference keeps its generation number, and a route sends the read to the relocation file | | indirect BlobIndex | A kIndirectBlob (type 3) or kIndirectFlattenedBlob (type 4) index. It stores the stable identity, so the value can move without an SST rewrite | | unflattened reference | Index with generation number == origin: the value was not moved, or it was moved but no compaction has rewritten its reference yet (see "flatten" in "Stack context: terms for later diffs") | | flattened reference | Index with generation number != origin (type 4). Only flatten (diff 10) writes it; this diff defines the encoding | | route | Maps an old file number (a file that GC has removed from the Version) to the relocation file that now holds its live values. One hop: a target is never itself routed. Stored in the MANIFEST (diff 4) | | relocation file | Block-based table keyed by (origin, original offset) that holds copied values. Each value is the stored bytes plus a 5-byte trailer (kSimpleGen2BlobTrailerSize) | | logical CRC | CRC32C of the uncompressed value (BlobIndex::stable_checksum()) | | stored CRC | CRC32C of the stored bytes plus the compression byte | | physical file | A file that holds value bytes: a normal blob file or a relocation file. A generation number can name a physical file, or it can be only the key of a route | | exact reference metadata | Per-SST sorted list of every blob reference the SST keeps, written when the SST is built. References with the same encoding class, generation number and origin form a group. The list is stored as data blocks, with an index of the groups. An SST that has it is an exact SST | | option gate | Until diff 14, ColumnFamilyData::ValidateOptions returns NotSupported for enable_blob_indirection on a read-write open, CreateColumnFamily and SetOptions, unless a test calls ColumnFamilyData::TEST_SetAllowBlobIndirection(true). It keeps the option off in production while the stack is not complete | Terms of this diff (all defined at the top of db/blob/blob_reference_metadata.h): | Term | Meaning | |---|---| | encoding class (class) | kDirect (a legacy kBlob or kBlobTTL index), unflattened, or flattened | | reference group (group) | All references of one SST with the same (encoding class, generation number, origin) | | run | The part of one group inside one data block | | record bytes | The bytes the value's record takes in the file named by its generation number | | gap | How far a record is from where it would be if the records were back to back. A gap of 0 means back to back | | descriptor | The 72-byte table property that points to the reference index block and holds the totals, the CF ID, the SST file number and the relocation-state fingerprint | | relocation-state fingerprint | 128-bit hash of the routes and relocation files of the Version that an SST was built from (fields route_snapshot_fingerprint_*). Written as 0 in this diff | | legacy | Written the way RocksDB writes it without blob indirection | Previous behavior: - A BlobIndex named only a physical blob file and an offset. A value could not move to another file without rewriting every SST that referenced it. - An SST had no metadata that listed its blob references. The only way to find them was to scan its user keys. - These readers get the blob indexes of a wide-column entity through ForEachBlobFileNumber: UpdateBoundaries, BlobGarbageMeter, the compaction iterator, WideColumnSerialization::RemoveBlobColumns, and the referenced_blob_bytes table property count in BlockBasedTableBuilder::Add. ForEachBlobFileNumber decoded each blob index, but it did not check the rest of the entity framing (name order, section sizes, exact input size), and it could call the callback before it found a later error. New behavior: Separate change (option on or off): - The same readers (UpdateBoundaries, BlobGarbageMeter, the compaction iterator, RemoveBlobColumns, and the referenced_blob_bytes count in BlockBasedTableBuilder::Add) use WideColumnSerialization::ForEachBlobIndex, with the option on or off. With the option on, the table builder also uses it to give the blob columns of each entity to the collector. For an entity with blob columns, it checks the full framing, then decodes and validates every blob index before any callback runs. So an entity with a malformed framing now fails the flush or compaction, even with the option off. The referenced_blob_bytes count itself does not fail the build on a malformed entity. It removes that property from the SST, as in base. Option and option gate: - This diff adds a new column family option, `enable_blob_indirection`. It is off by default, and SetOptions can change it. - The option gate (see Terms) keeps the option off outside tests until the stack is complete. ColumnFamilyData::ValidateOptions rejects the option with NotSupported on a read-write open, CreateColumnFamily and SetOptions. Read-only opens (DBImplReadOnly and CompactedDBImpl) and secondary opens (including DB::OpenAndCompact) do not run ValidateOptions; they check only the FIFO rule (see Implementation). Only tests can turn the option on, through the internal ColumnFamilyData::TEST_SetAllowBlobIndirection(). DBTestBase sets it. - When the option is on, ValidateOptions also requires enable_blob_files, the block-based table format, no blob direct write, and a compaction style other than FIFO. Blob file writer (option on): - The blob file writer encodes each new value with a stable identity (origin file, original offset) and two checksums: - the logical CRC: the CRC32C of the uncompressed value. - the stored CRC: the CRC32C of the exact stored (maybe compressed) bytes, extended with the compression byte. - For an uncompressed value the stored CRC is not written, because the reader can compute it from the logical CRC. - The checksums have no version field of their own. The BlobIndex type byte gives the format version: the two new types, 3 (kIndirectBlob) and 4 (kIndirectFlattenedBlob, see Implementation), are BlobIndex::kIndirectBlobFormatVersion (1). A later format uses new type values. - The read path does not check these checksums yet. The check starts in diff 3. Exact reference metadata (option on): 1. What: a sorted list of every blob reference that the SST keeps, in plain blob values and in the blob columns of wide-column entities. 2. Where: data blocks (target 16 KB each before compression; small blocks waste less I/O, because indirect blob GC reads a whole block to get a few groups) and one reference index block, written after the SST data blocks and before the SST tail, plus a 72-byte descriptor table property. 3. Who reads it: only tests in this diff. The census, GC and RepairDB read it in later diffs. 4. Details: the top comment of db/blob/blob_reference_metadata.h has the full format with diagrams, the record-bytes rules and the gap rule. - Not collected: SstFileWriter output (it has no DB file number) and any output whose CF is unknown (it has no CF ID). The descriptor needs both values, so these outputs stay in the legacy form. Checks on the metadata: - A repeated or overlapping reference in one group fails the SST build with Corruption. Readers check every count, order, size and total against the index and the descriptor, so corrupt metadata returns Corruption. The writer never makes a data block larger than kTargetDataBlockSize (16 KB), and readers accept up to kMaxDataBlockSize (64 KB). Details: "Readers" and "Integrity" at the top of db/blob/blob_reference_metadata.h, and BlobReferenceGroupDecoder. Collection and memory: - Each reference is encoded when it is added, in the on-disk entry encoding, into its group's chain of 512-byte chunks. Only a group that takes the unsorted fallback is encoded a second time, in Finish(). Inside a group, references normally arrive in rising offset order, so no reference is sorted; at the end only the groups are sorted. A group that gets a lower offset takes the unsorted fallback. The BlobReferenceMetadataCollector class comment explains the chunks, the chunk size and the fallback. - The collector keeps every reference of the SST in memory until the SST is finished, and charges this memory to the block cache under a new CacheEntryRole, kBlobReferenceMetadataBuildingBuffer. "Memory usage" and "Memory charge" in the BlobReferenceMetadataCollector class comment explain the parts of this memory and the charge rule. The measured amounts are under "Validation and known limitations". - There is no fixed limit on references. The collector fails with MemoryLimit above 2^20 groups or 2^20 data blocks in one SST; these are persisted format limits. It also fails with MemoryLimit when the reference index block could be larger than 64 MB (kMaxTopLevelMetadataBlockSize), which can happen from about 945K groups (64 MB / kMaxIndexEntrySize, 71 bytes per group), or from fewer groups when the SST also has many data blocks. - There is no charge when the table has no block cache or when charging for this role is disabled. Each block-based table factory holds one cache reservation manager for this role, shared by its builders and the SST summaries. If the charge fails, the SST build fails, so the flush or compaction fails with a background error. The buffers and their charge are released when the SST is finished or abandoned. Implementation: - BlobIndex adds two indirect encodings. The legacy encodings do not change. - Type 3, kIndirectBlob (unflattened, generation number == origin), stores the origin, the original offset, the stored size, the compression type, the logical CRC and, for compressed values, the stored CRC. It does not store the generation number. This saves the size of one varint file number per reference (about 3 to 4 bytes for typical file numbers; an estimate, not measured). - Type 4, kIndirectFlattenedBlob (flattened), stores the generation number first, then the same fields. Its generation number must be above the origin. The decoder rejects type 4 with generation number <= origin, so each reference has one encoding. - stored_checksum() computes the stored CRC of an uncompressed value when a caller asks for it: the logical CRC extended with the compression byte. A decode does not compute it, because most decodes (for example in compaction) never read it. - The encoders take the fields as one BlobIndex::IndirectBlobFields, set by field name, so values of the same type (file number, offset and size; the two CRCs) cannot be swapped. EncodeIndirectBlob() takes the generation number as a separate argument; EncodeUnflattenedIndirectBlob() uses the origin, so the common path cannot leave the generation number unset. - The new files blob_reference_metadata.{h,cc} define the exact reference metadata. The files also contain: - BlobReferenceMetadataCollector, the collector described above. It takes the SST identity (CF ID, SST file number and the relocation-state fingerprint) as one BlobReferenceSstIdentity, set by field name. - BlobReferenceGroupKey, the key of a group: (encoding class, generation number, origin). It has the order, the hash and the validity check that the collector map, the sort, the decoder and FindBlobReferenceGroup() use. BlobReference and BlobReferenceGroup keep their flat fields and return the key through group_key(). - The index codec and the descriptor table property. - BlobReferenceGroupDecoder, the only decoder of reference runs. It reads a group one data block at a time and holds every corruption check of a group walk. Readers that visit one group (found by a binary search of the index) use it, and so do readers that visit every block in order. - Callers in this diff: only the table builder runs the collector. The decoders (DecodeBlobReferenceIndex, FindBlobReferenceGroup, BlobReferenceGroupDecoder, ForEachBlobReference*) are used only by tests. kMaxDecodedBlobReferenceBytes is used only in compile-time checks of the count limits. The census (diff 5) and indirect blob GC (diffs 8 and 9) are their first production users. The census is also the first user of BlobReferenceSstSummary (see the last Implementation bullet). - Descriptor: the CF ID and the SST file number are the repair fields. They let RepairDB tell which column family and SST the list belongs to without a MANIFEST. The decoder rejects a descriptor whose repair fields are not set (unknown CF ID or SST file number 0). The descriptor also has a 128-bit relocation-state fingerprint (the route_snapshot_fingerprint fields). Remote compaction uses it in diff 11 to check that an SST built elsewhere used the same routes as the job that asked for it. In this diff the table builder always writes 0, and DecodeFrom() does not check it. The field is in the format now, so the descriptor keeps one format version (1) through the stack. While the option gate is on, no data in this format exists outside tests. - The collector validates each BlobIndex once, when it is added. Its Finish() writes the data blocks and then the reference index block through one block writer, and fills the descriptor, including the index handle. Destroying the collector frees its memory and then its charge, also when the build is abandoned. - BlockBasedTableBuilder creates the collector with the CF ID and SST file number from TableBuilderOptions. It passes the size of each entry's user key to the collector, so the collector can compute the exact size of each blob record. It adds the collector's EstimatedEncodedSize() to its tail size estimate: the blocks are written before the tail, but output file size planning uses this estimate. The reference blocks are written with BlockType::kUserDefinedIndex, which keeps their contents as opaque bytes. Readers read them without the block cache, so prepopulate_block_cache does not warm them into the block cache (WriteMaybeCompressedBlock() with warm_cache false). - To save CPU, compaction decodes each kTypeBlobIndex output value once, in CompactionOutputs::AddToOutput(). It passes the decode to TableBuilder::TryAdd() in the new field TableBuilderAddContext::decoded_blob_index, and to new overloads of BlobGarbageMeter::ProcessOutFlow() and FileMetaData::UpdateBoundaries(). BlockBasedTableBuilder decodes a kTypeBlobIndex value once for both the referenced_blob_bytes count and the collector, and uses the caller's decode when it has one. If the decode fails, no decode is passed, and each reader decodes the value itself and returns its usual error. This applies with the option on or off. The SST bytes are the same as when each reader decodes the value itself. - The collector converts each BlobIndex into a reference without deriving the stored CRC of an uncompressed value (ConvertBlobIndex() in db/blob/blob_reference_metadata.cc). The collector does not write that CRC, because a reader derives it again from the logical CRC. Deriving it would cost CPU for every reference of every output SST. BlobReference::FromBlobIndex() still derives it. - With the option on, the blob file builder computes the CRC32C of each stored value once. The blob log record CRC combines it with the key CRC (crc32c::Crc32cCombine). The stored CRC extends it with the compression byte. For an uncompressed value it is also the logical CRC. The bytes written do not change. - WideColumnSerialization::ForEachBlobIndex replaces ForEachBlobFileNumber. It reads the entity header and column types once. For a V1 entity (no column types and no blob columns), it returns right after the header. For a V2 entity without blob columns, it returns right after the type scan. It shares one V2 column walker (WalkV2Columns) with Deserialize(), so the entity is parsed in one pass. - Read-only opens (DBImplReadOnly, CompactedDBImpl) and secondary opens skip ValidateOptions, so they check the compaction style against the option once per column family: in DBImpl::Recover() for read-only opens, and at the start of OpenAsSecondaryImpl() for secondary opens. DB::OpenAndCompact opens through OpenAsSecondaryImpl(), so it gets the same check. - CacheReservationHandle gets Resize(), so each collector or summary can change its own part of the shared charge. - util/coding.h adds GetCanonicalVarint32/64, which reject overlong varint encodings. The new formats use them, so each value has one encoding. - BlobReferenceSstSummary is the in-memory list of an SST's groups and totals that Finish() can return. It has no caller yet: the table builder passes a null summary output. It is checked and does not change after Finish(), so the finished builder, the SST metadata and the census (its first user, in diff 5) can share it. It is in this diff because Finish() hands the group buffer and its cache charge over to the summary. Adding the summary later would mean rewriting the charge code of Finish() in diff 5, so the charge code is written and tested once, here. - sst_dump prints a "Blob reference metadata" section after the raw user collected properties of an SST that has exact metadata: the reference count, the stored value bytes, the column family ID, the SST file number when built and the index block handle. An SST without the descriptor prints nothing more. Safety boundaries and unsupported cases: - Public API: this diff adds the option `enable_blob_indirection` with the C API functions `rocksdb_options_set_enable_blob_indirection` and `rocksdb_options_get_enable_blob_indirection`. It also inserts a new CacheEntryRole value before kMisc, which changes the numeric value of kMisc and kNumCacheEntryRoles. - With the option off, new blob values and SSTs are written in the legacy form. Legacy blob indexes stay readable. - One behavior change applies with the option off too: the five readers listed under "Separate change" fully validate wide-column entities with blob columns and their blob indexes (see "Separate change" above). - Exact reference metadata has one format (version 1) with a fixed 72-byte descriptor. The decoders reject any other version as Corruption. - The metadata blocks are protected by the table's block checksum only. With checksum = kNoChecksum, corrupt metadata is detected only when it breaks the format or order checks. - The index decoder checks each decoded count against the remaining input size before it reserves memory. So a small corrupt index cannot make it allocate a large vector. - Inlined-TTL and same-file blob indexes are not collected and cannot be relocated. Encoding, count and checksum errors return an error; a bad reference is never skipped. - Option gate (see Terms): outside tests no DB writes data in the new format. The gate checks only the option value, never persisted state. Diff 14 removes it, once data written with the option stays readable through the end of the stack and the safety checks for turning the option on are in place. - Downgrading the binary after data was written with the option on is not supported. Validation and known limitations: - Collector memory, measured in a release build that drives the collector directly (bench/collector-chunk-20261111/README.md; host not recorded; 16-byte keys, 1 KB values, 62K to 1M references per SST in 1 to 16 groups): the buffers used about 8.3 to 9 bytes per reference for uncompressed values and 12.8 to 13.5 bytes for compressed values, including the chunk headers. With 1000 groups in a 62K-reference SST, they used about 14 bytes per uncompressed reference. - Chunk size, from the same measurement: for an SST with 1M references, the chunk size changed memory by less than 5% between 256 and 4096 bytes. For an SST with 62K references in 1000 groups, the partly filled chunks dominated: about 14 bytes per reference at 512 bytes, 22 at 1024 and 72 at 4096. 256 bytes used 3-4% more memory than 512 and made Finish() up to 1.6 times slower with 100 or more groups. - Reference index block size, measured in census db_bench runs (source report and host not recorded; 16-byte keys, 200-byte values, 1 MB blob files, no compression, fillrandom and then a full compaction): about 1.7 KB per SST with 32 MB write buffers and a target file size of 8 MB or 2 MB, and about 10 KB per SST with 1 MB write buffers and a target of 2 MB or 200 KB. The size is the bytes read for the index blocks divided by the number of SSTs. - Unit tests cover the BlobIndex encodings and their rejection of bad input, the metadata layout and every corruption check, the collector (sorted path, unsorted fallback, memory charge), the flush integration, the option checks and option gate, wide-column validation, and canonical varints. See the Test Plan. - The collection buffer is not spilled to disk. One SST with very many blob references needs a matching amount of charged memory: the per-reference amount given under "Collection and memory", plus up to one partly used 512-byte chunk per group. While a fallback group is sorted, it also needs 16 bytes of sort key per reference and a second copy of its entries. For a test group of 200K references with every other one out of order, the test comment estimates about 44 bytes per reference, and the test checks that the peak charge is at most 56 bytes per reference. - Flush still writes a single output file. Until diff 7, one compaction output collects every reference it gets, charged to the block cache. Diff 7 cuts compaction outputs (including L0) at a reference cap and keeps each user key's history in one output. With the option gate, a large output is reachable only through the test hook. Stack context (same in diffs 2-14, except that diffs 10-14 add the diff 10 route rule to the install line; "<== this diff" marks this diff's part): Goal: remove garbage from blob files without rewriting any SST. A value gets a stable identity when it is first written. Indirect blob GC copies the live values of whole files into a relocation file and installs routes in the MANIFEST. Readers follow the routes. SSTs are not changed. How a read finds a value (P is the relocation file that holds the value now): ``` SST entry indirect BlobIndex { generation number, origin, original offset, size, CRCs } | v Version file table: generation number -> live file, or route old file -> P | | live file route found | | v v blob file: read at original offset relocation file P (always one hop): relocation file: look up key look up key (origin, original offset) (origin, original offset) ``` Lifecycle of one value (in this example, F is the blob file where the value is first written, and P is the relocation file that GC writes): ``` write new value goes to blob file F; index: origin = F, generation number = F | census per generation number: the SSTs and references that use it; | per physical file: live bytes, ranked by garbage ratio GC job copy the live values of whole input files (stored bytes, checked | with the stored CRC) into relocation file P, keyed by | (origin, original offset) install one MANIFEST edit: add P, route each input generation number still | in use to P, re-point the routes into the inputs to P, and remove | each input (blob file retirement for a blob file, relocation | file deletion for a relocation file) flatten compaction sets generation number = P in the SSTs it rewrites | route route retirement: route deletion records remove the routes that delete no SST uses ``` Stack map: | Diff | Adds | |---|---| | 1 | Finalize blob metadata outside the DB mutex (prep) | | 2 | Indirect BlobIndex (kIndirectBlob) and exact SST reference metadata <== this diff | | 3 | Relocation file format and stable read path | | 4 | MANIFEST routes and relocation state in the Version | | 5 | Exact reverse reference census | | 6 | MANIFEST apply outside the DB mutex, with a precondition re-check | | 7 | Coarse flush metadata, per-SST reference cap, epoch runs | | 8 | Indirect blob GC building blocks | | 9 | Indirect blob GC job | | 10 | Flatten generation numbers; route retirement | | 11 | Remote compaction validation | | 12 | Dynamic options and operations | | 13 | Lost relocation files | | 14 | RepairDB rebuilds relocation state; option gate removed | | 15 | db_stress and crash test support | Stack context: terms for later diffs: | Term | Meaning | |---|---| | census | Background count, per generation number, of the SSTs and references that use it, plus live bytes per physical file (diff 5) | | indirect origin | Normal blob file written with the option on; it is the origin of unflattened references | | flatten | Compaction rewrites the generation number to the relocation file that holds the value, so old routes can go. Only flatten makes generation number != origin (diff 10) | | blob file retirement | MANIFEST record (kBlobFileRetirement) that removes a collected blob file from the Version (diff 4) | | route deletion | MANIFEST record that removes a route that no SST uses. Route retirement is the background job that writes these records (diff 10) | Tracking issue: meta#15301 Differential Revision: D124652097
Summary: Pull Request resolved: meta#15541 ## Why The terms are defined in "Terms used in this diff" below. An indirect BlobIndex (added in diff 2) must find its value after indirect blob GC moves the value. This must work without rewriting SSTs and without making the cached value invalid. This needs two things, and this diff adds both: 1. A file format that holds moved values and finds them by stable identity (origin, original offset): the relocation file. 2. A read path and a blob cache key that do not depend on the physical file. Nothing in the DB writes relocation files yet. Routes (MANIFEST records that send reads of an old file number to a relocation file) come in diff 4, and the indirect blob GC job (diff 9) first creates relocation files. No DB read reaches a relocation file yet (see "What is live in this diff"). ## Parts of this diff 1. Relocation file format and writer (core): db/blob/blob_relocation_file.{h,cc} and the relocation mode of BlockBasedTableBuilder. 2. Relocation file reader (core): BlockBasedTable::MultiGetRawEntries, BlobRelocationFileReader (db/blob/blob_file_reader.{h,cc}) and the relocation support in BlobFileCache. It needs part 1. 3. Stable read path and stable blob cache (core): StableBlobFingerprint, StableBlobContents, StableBlobCacheKeyBuilder and the indirect read path of BlobSource, plus its callers in Version and flush. It does not depend on parts 1 and 2 for normal blob files; it calls part 2 for relocation files. 4. Pinned location blocks (the data blocks of a relocation file, which hold its location entries): the column family option pin_blob_relocation_location_blocks. You can skip this part on a first read. 5. Legacy read-path changes (see "Changes visible to legacy blob users"). You can skip these on a first read. A read goes through the parts in this order: ``` Version (picks the physical file) -> BlobSource: stable blob cache, then the file -> BlobFileCache -> BlobFileReader blob log file: read the record at the original offset relocation file: BlobRelocationFileReader -> BlockBasedTable::MultiGetRawEntries (find the location entry) -> BlockBasedTable::MultiGetSameFileBlob (read the record) ``` ## How to read this diff 1. Read "Terms used in this diff" below. Then read the overview comment at the top of db/blob/blob_relocation_file.h. Together they explain the base terms, the file layout, the key and location-entry encoding, and the directory. 2. Read the record format changes in db/blob/blob_gen2_format.h: the size limit kMaxSimpleGen2BlobPayloadSize, compressed records and the batch read changes. Then read the format code in db/blob/blob_relocation_file.{h,cc}: the directory, the directory builder, the key, and the group encode and decode. The format code uses only the record format from this step. 3. Read the writer: the relocation methods and NewBlobRelocationTableBuilder in table/block_based/block_based_table_builder.{h,cc}. It uses only the code from step 2. 4. Read the table-level read helpers in table/block_based/block_based_table_reader.{h,cc}: MultiGetRawEntries (the batched lookup of location entries), RetrieveDataBlocks, and the no-BlobSource branch of MultiGetSameFileBlob (the record read). Also read the new fields in db/blob/same_file_blob_reader.h. Read PinDataBlocks and MultiGetRawEntriesPinned last (part 4). 5. Read BlobRelocationFileReader in db/blob/blob_file_reader.{h,cc}: the class comment ("How a read works"), Open, CheckEntry and MultiGetBlob. Also read BlobPhysicalFile and RelocationBlobReadRequest in db/blob/blob_read_request.h. This step combines steps 2 and 4. 6. Read BlobFileCache::GetRelocationFileReader and failed_relocation_opens_ in db/blob/blob_file_cache.{h,cc}. Also read GetBlockBasedTableWriterOptions in table/adaptive/adaptive_table_factory.{h,cc}, which gives the cache the table options of the column family. 7. Read the stable cache on its own: StableBlobFingerprint (db/blob/blob_index.h), StableBlobContents (db/blob/blob_contents.{h,cc}), StableBlobCacheKeyBuilder and ChooseStableKeyDomain (db/blob/blob_source.{h,cc}), and the "Stable blob cache" comment at stable_blob_cache_ in db/blob/blob_source.h. 8. Read BlobSource::GetIndirectBlobFromCache, GetIndirectBlob, MultiGetIndirectBlobFromOneFile and FinishIndirectBlobRead (db/blob/blob_source.cc). Finish with the callers: db/version_set.cc (stable cache first; normal files only in this diff) and flush prepopulation in db/blob/blob_file_builder.{h,cc}. Check the legacy-change table last. ## Terms used in this diff Stack terms: | Term | Meaning | |---|---| | origin | Blob file where the value was first written | | original offset | Value offset in the origin file | | stable identity | (origin, original offset). Never changes, even when the value moves | | generation number | The file number that a reference uses to find its value. A new value has generation number == origin. When indirect blob GC moves the value, the reference keeps its generation number, and a route sends the read to the relocation file | | indirect BlobIndex | A kIndirectBlob (type 3) or kIndirectFlattenedBlob (type 4) index (diff 2). It names the value by its stable identity, not by where the bytes are now | | route | Old file number -> the relocation file that now holds its live values. One hop: a target is never itself routed. Stored in the MANIFEST (diff 4) | | relocation file | Block-based table keyed by (origin, original offset) that holds copied values. Each value is the stored bytes plus a 5-byte trailer (kSimpleGen2BlobTrailerSize) | | physical file | A file that holds value bytes: a normal blob file or a relocation file. A generation number can name a physical file, or it can be only the key of a route | | logical CRC | CRC32C of the uncompressed value (BlobIndex::stable_checksum()) | | stored CRC | CRC32C of the stored bytes plus the compression byte | | option gate | Until diff 14, ColumnFamilyData::ValidateOptions returns NotSupported for enable_blob_indirection on a read-write open, CreateColumnFamily and SetOptions, unless a test calls ColumnFamilyData::TEST_SetAllowBlobIndirection(true) | Terms of this diff: | Term | Meaning | |---|---| | stable | Always means "does not change when indirect blob GC moves the value": the stable identity, the stable cache key, the stable blob cache. The code name stable_checksum is the logical CRC | | record | One stored value in a relocation file, in the SimpleGen2Blob record format (db/blob/blob_gen2_format.h): stored bytes, then a compression byte (1 byte) and a 4-byte checksum of the table's checksum type. Normal SSTs use the same format for embedded (same-file) blobs, so "embedded blob", "same-file blob" and "SimpleGen2 record" name the same record format | | record bytes | Stored value size + kSimpleGen2BlobTrailerSize, summed over records | | location entry | One data block entry of a relocation file. It locates a group of up to 32 values of one origin | | origin partition | All values in one relocation file that share one origin file | | served file | An old file number that may be routed to this relocation file. GC writes its input files and the file numbers already routed to them; RepairDB writes the file numbers it rebuilds | | directory | A table property of the relocation file: a header (counts, totals, directory digest, header CRC), then one entry per origin partition, then the served files | | directory digest | CRC32C of the directory header counts and totals and the directory body. It is the file's identity, and diff 4 stores it in the MANIFEST | | open tail | The end of the table that a reader loads at open: index, properties, metaindex and footer | | fingerprint | StableBlobFingerprint: the fixed fields of an indirect BlobIndex (stored size, compression, logical CRC, stored CRC) | | stable cache key | A blob cache key built from db_id, the session id and (origin, original offset). It does not use the generation number or the physical file, so it stays the same when the value moves | ## Previous behavior - A blob read used a physical file number and an offset. - The blob cache key was built from the physical file, so a moved value would get a new key and lose its cached copy. - There was no relocation file format. ## New behavior ### Relocation file The format is described once, in the overview at the top of db/blob/blob_relocation_file.h: the 8-byte prefix, the records, the location entries (groups of kBlobRelocationGroupSize (32) values, and why 32), the directory and its digest, the table options, and the num_entries check. The contract: - A lookup of (origin, original offset) finds the first entry at or after that key, checks its origin, decodes the group and finds the exact offset. An offset that is not there gives Corruption ("entry is missing"). - Every origin and every served file has a smaller file number than the relocation file. Open checks this, the order and totals of the directory, that the directory fits in the open tail, and that the records fit before the tail. - GetBlobRelocationFileMetadata returns the encoded directory and the size of the open tail. Diff 4 stores the tail size and the directory digest in the MANIFEST. ### Relocation reads - BlobRelocationFileReader supports every read mode: single reads, batched reads, range reads of uncompressed values (also used by lazy reads) and whole reads of compressed values. BlobFileReader owns it, so the file cache keeps one entry type. Its class comment ("How a read works") gives the steps. - BlockBasedTable::MultiGetRawEntries looks up the location entries of one batch (at most MultiGetContext::MAX_BATCH_SIZE (32) keys) together. It looks up all their data blocks in the block cache first. It reads one missing block through the normal block read path, and two or more missing blocks with one batched read (RetrieveMultipleBlocks). Then it finds each key in its block with SeekForGet. Sorted keys that fall in one location entry share one decode of it. A single Get takes the same path. It does not create a table iterator; it uses only an index iterator and a data block iterator on the stack. - With kBlockCacheTier, a key whose block is not in the cache returns Incomplete. A bad block fails only the keys in that block. - When a record batch has only one record, the reader uses a plain positional read (RandomAccessFileReader::Read). When it has two or more records, it uses MultiRead. Both paths do the same checks. - Exact duplicates in a batch are read once and counted once (see legacy change 7). Indirect reads of blob log files do the same in BlobSource::PlanReadBatches, where a duplicate must also have the same user key. - Checksums: the table under "Reads" in db/blob/blob_relocation_file.h says which checksums each read kind checks. Relocation files use format_version 6 or higher, so a record copied to another offset fails its checksum. - Failed opens: BlobFileCache remembers some failed relocation opens for kFailedRelocationOpenMicros (1 second), so reads routed to a bad file do not reopen it each time. It opens relocation files under their own kNumberOfRelocationMutexStripes (32) stripe mutexes, so that BlobFileCache::Evict() of a normal blob file, which runs under the DB mutex, never waits for a long relocation open. The comments at failed_relocation_opens_ and relocation_mutex_ in db/blob/blob_file_cache.h list the remembered errors and give the reasons. ### Pinned location blocks (option) - pin_blob_relocation_location_blocks is a new mutable column family option (default false), with a C API setter and getter. With it on, a relocation file reader reads all of its location blocks at open and keeps them until it is closed. With a block cache, the blocks are held through their cache handles, so they are charged and not evicted. A lookup then does a binary search over the last key of each block and a seek in the block, with no index lookup, no block cache lookup and no I/O. A change applies to relocation files opened after it. - If the blocks cannot be kept (the block cache cannot charge them, or a block read fails), the open still succeeds: the reader logs a warning and does normal lookups. With strict_capacity_limit, the kept blocks hold block cache capacity that eviction cannot free. The option comment in include/rocksdb/advanced_options.h has the details. ### Stable blob cache The overview is the "Stable blob cache" comment at stable_blob_cache_ in db/blob/blob_source.h. The contract: - Key: a standard 16-byte cache key built from db_id, the session and (origin, original offset). It stores a domain value (1, 2 or 3) in the top two bits of the origin. No real file number sets these bits: BlobIndex::DecodeFrom() rejects an indirect index with a file number above BlobIndex::kMaxFileNumber (2^62 - 1, equal to kFileNumberMask). So the key never equals another kind of cache key; ChooseStableKeyDomain() in db/blob/blob_source.cc has the proof. - Entry: StableBlobContents holds the value and its fingerprint (stored size, compression, logical CRC, stored CRC). A header with its own CRC is written only when the entry moves to a secondary cache, and the header CRC and the value CRC are checked when the entry is loaded back. The secondary cache admission policy is the same as in base RocksDB. - Cache first: Version::GetBlob, GetBlobRange and both MultiGet paths look up an indirect value in the stable cache before any blob file. A hit returns at once, and MultiGet groups only the misses by file. A hit is not checked against the BlobIndex. This is safe: the key includes db_id and the session, so entries from before a reopen never match, and a blob file never changes after it is written and the DB never reuses a file number, so (origin, original offset) names the same bytes for the life of the DB. Debug builds assert that the file of a hit's BlobIndex is in the Version. - Insert: a value that will be cached ends up in a buffer from the cache's allocator, and the entry adopts that buffer without another copy. A relocation record is read directly into that buffer. A normal blob file value is read into a local buffer and copied (or decompressed) into it once, as in base. The insert locks one admission mutex and inserts only on a primary-cache miss. - Flush: with enable_blob_indirection on, flush passes each value's fingerprint to BlobSource::PrepopulateFlushBlob. This fills the cache under the stable key, with the CRCs that BlobFileBuilder already computed for the BlobIndex. ### Retry of normal blob file reads A single indirect read from a normal blob file retries a Corruption once with a newly opened reader, for a whole value and for an uncompressed range. A batched read retries only whole values; a batched range read returns the Corruption. Base MultiGetBlobFromOneFile retries whole values in the same way, and base MultiGetBlobRangeFromOneFile does not retry. Relocation reads are not retried; their failed opens are remembered instead. ### What is live in this diff No DB read reaches a relocation file yet: BlobPhysicalFile::is_relocation is false on every DB read path. With the test hook (see "option gate"), DB reads use the indirect read path for normal blob files, and flush fills the stable blob cache. Tests drive the relocation file reader directly. The legacy read-path changes below apply to all users. ### Monitoring - BLOB_RELOCATION_FILE_OPEN_FAILURES (ticker) counts relocation file reader opens that fail, for example because the file is missing, corrupt or cannot be read. Opens of normal blob files still count only in NO_FILE_ERRORS. - BLOB_RELOCATION_FAILED_OPEN_CACHE_HITS (ticker) counts reads that get the remembered failure of an earlier open, without opening the file again. BlobFileCache remembers a failed open for kFailedRelocationOpenMicros (1 second) when the error comes from the file itself (Corruption, NotFound or PathNotFound). - TableReaderCaller::kBlobRelocationFileRead marks the block reads of a relocation file (location blocks, raw records and the location block pin) in block cache traces. Relocation readers get the DB's block cache tracer, the same one that SST readers get, so a block cache trace includes these reads: one record for each block cache hit and each block read. BlockCacheTraceHelper::IsUserAccess() returns false for this caller, because one caller value covers all relocation reads: user reads and background reads both use it. The block cache trace analyzer prints the caller as "BlobRelocationFileRead". The embedded-blob iterator wrapper skips this caller, so the relocation reader sees the raw values. - The Corruption message "Stable blob checksum mismatch" names the generation and origin file numbers and the original offset of the reference. - Feature off: both tickers change only when a relocation file is opened, and only relocation file reads use the new caller. A DB without relocation files never updates them. ## Changes visible to legacy blob users (option off) | # | Change | Impact | |---|---|---| | 1 | Flush prepopulation of legacy blobs goes through BlobSource. BlobSource charges its blob cache to the block cache when charging is enabled | Affects users with prepopulate_blob_cache=kFlushOnly and blob cache charging enabled (CacheEntryRole::kBlobCache charged). In base RocksDB these entries were not charged | | 2 | BlobFileCache returns Incomplete for kBlockCacheTier reads before it opens a reader | Callers already handled this case the same way | | 3 | Legacy decompression returns Corruption for unsupported compression types and for sizes above about 4 GiB, before it allocates memory | Safer on bad input | | 4 | SimpleGen2 readers accept compressed records in whole reads. A range read of a compressed record returns Corruption, as in base | Base writers never compress, so existing data is not affected | | 5 | Batched MultiGetSameFileBlob works without a BlobSource | No behavior change for existing callers | | 6 | An embedded-blob range read on a reader with no BlobSource (for example sst_dump or SstFileReader) works. In base RocksDB it returned Corruption | Readers that do such reads get the value | | 7 | Batched embedded-blob reads (MultiReadGen2) read an exact duplicate (same offset and length) once. A batch with one read uses a plain positional read instead of MultiRead | Fewer reads, same results. BLOB_DB_LAZY_READ_COUNT, BLOB_DB_BLOB_FILE_BYTES_READ, blob_read_count and blob_read_byte count one storage read for a read and its exact duplicate | | 8 | BlockBasedTable::RetrieveMultipleBlocks also accepts no MultiGetRange. Then it reads synchronously and no GetContext gets statistics. This lets the relocation lookup share it | Normal MultiGet passes its range as before, so its behavior does not change | ## Implementation - db/blob/blob_relocation_file.{h,cc} (new): the format code. The directory format version is 2 (BlobRelocationDirectory::kFormatVersion). The directory has no fixed count limit. Two checks bound its size instead: the decoder requires the size to equal the header plus the stated entries and served files, and ValidateForFileSize requires the encoded directory and the record bytes to each fit in the file. - BlockBasedTableBuilder: non-virtual relocation methods (start, add served file, add record, get metadata); the base TableBuilder is unchanged. NewBlobRelocationTableBuilder creates and starts a relocation builder. It takes the file identity and size target in one struct (BlobRelocationTableFileInfo), so values of the same type (db_id and db_session_id, file_number and target_file_size) cannot be swapped by mistake. MakeBlobRelocationTableOptions forces the table options, with a comment for each. - BlockBasedTable: MultiGetRawEntries, RetrieveDataBlocks (shared by the lookup and the pin), PinDataBlocks and MultiGetRawEntriesPinned. Embedded blob reads create their decompressor once per table reader. After a relocation reader checks the directory, it drops its copy of the directory property from its table properties. Debug builds assert that no read has started at that point (Rep::read_started), because the drop replaces the properties that readers share. - BlobRelocationFileReader keeps only the directory totals and digest, not the decoded directory, because the location entry check proves that a value is in the file. BlobPhysicalFile describes one resolved physical file (open info, size, relocation flag, tail size and directory digest). BlobFileCache owns the session id and builds the relocation table factory at construction; the signatures for normal files keep their base shape. - Block cache traces: ColumnFamilyData passes the DB's BlockCacheTracer to BlobFileCache, as the required last parameter of the constructor that opens relocation files. The base constructor opens no relocation file, so it takes no tracer. BlobFileCache passes the tracer to BlobRelocationFileReader::Open, which puts it in the TableReaderOptions of the table reader. With tracing off, the cost is the same is_tracing_enabled() check that SST readers do. BlockBasedTable::RetrieveDataBlocks (the raw lookup and the pin) writes a trace record for each block cache hit and each block of a batched read; a single missing block goes through RetrieveBlock, which writes its own record. - BlobSource builds StableBlobCacheKeyBuilder on first use, because db_id is set during recovery, after the column family is created. Its MultiGet classifies the requests in one pass and reads them in sorted batches that do not overlap, with fixed-size buffers (at most 32 requests). - Request structs: the fields that this diff adds to read requests are grouped in one caller-owned struct per layer, behind one pointer (SimpleGen2ReadExtras, SameFileBlobStandaloneRead, and the output pointer of RelocationBlobReadRequest), so base fields and base call sites do not change. - BlobFileBuilder takes an optional BlobSource: flush passes one, compaction passes none. - db_bench gets a --pin_blob_relocation_location_blocks flag. ## Safety boundaries and unsupported cases - Public API: new tickers BLOB_RELOCATION_FILE_OPEN_FAILURES and BLOB_RELOCATION_FAILED_OPEN_CACHE_HITS, added just before TICKER_ENUM_MAX, so its value changes. New TableReaderCaller::kBlobRelocationFileRead (15), so kMaxBlockCacheLookupCaller changes. The new tickers have no Java mapping, as for recent base tickers: all 256 byte values of org.rocksdb.TickerType are used. - CRC32C is for integrity, not for authentication. - Range reads follow the base rule. A range read that covers the whole value is a whole read: the public lazy API turns it into a whole read (kWholeBlobLength), and a whole read checks the CRC as the verify policy says. A range read of a strict part of an uncompressed value reads only the range bytes and checks no CRC, unless force_verify is set (then the lazy API reads and checks the whole value). So the application owns the integrity of the data it reads by range. Callers send the internal range functions only strict parts, and these functions do not turn a range into a whole read, the same as the base functions for separate blob files. This holds for relocation files and for embedded blobs. See the checksum table under "Reads" in db/blob/blob_relocation_file.h. - A range read of a compressed value is not supported, as in base: the internal range functions return Corruption ("Cannot range-read a compressed blob"), because a byte range of a compressed record cannot be decompressed alone. Callers read the whole value and slice it, as the public lazy API already does for a compressed reference. This compression check comes first in every range path, before the cache lookup and before the kBlockCacheTier check. So a compressed range also returns Corruption on a stable blob cache hit and for a kBlockCacheTier read. - Relocation files do not index user keys. This diff adds no separate blob map files. - Option gate (see Terms): behavior with the option on can be reached only through the test hook. Diff 14 removes the gate. ## Validation and known limitations - Unit tests cover the file format, the read paths, the stable cache and the legacy changes. See the Test Plan. - This diff cannot be benchmarked alone, because no DB code writes relocation files yet. The numbers in the Test Plan were measured at diff 9, the first diff that packs whole input files into relocation files, and they cover the format and read path added here. - The relocation block size (4 KB) is an estimate. No diff in this stack tunes it. ## Stack context (same in diffs 2-14, except that diffs 10-14 add the diff 10 route rule to the install line; "<== this diff" marks this diff's part): Goal: remove garbage from blob files without rewriting any SST. A value gets a stable identity when it is first written. Indirect blob GC copies the live values of whole files into a relocation file and installs routes in the MANIFEST. Readers follow the routes. SSTs are not changed. How a read finds a value (P is the relocation file that holds the value now): ``` SST entry indirect BlobIndex { generation number, origin, original offset, size, CRCs } | v Version file table: generation number -> live file, or route old file -> P | | live file route found | | v v blob file: read at original offset relocation file P (always one hop): relocation file: look up key look up key (origin, original offset) (origin, original offset) ``` Lifecycle of one value (in this example, F is the blob file where the value is first written, and P is the relocation file that GC writes): ``` write new value goes to blob file F; index: origin = F, generation number = F | census per generation number: the SSTs and references that use it; | per physical file: live bytes, ranked by garbage ratio GC job copy the live values of whole input files (stored bytes, checked | with the stored CRC) into relocation file P, keyed by | (origin, original offset) install one MANIFEST edit: add P, route each input generation number still | in use to P, re-point the routes into the inputs to P, and remove | each input (blob file retirement for a blob file, relocation | file deletion for a relocation file) flatten compaction sets generation number = P in the SSTs it rewrites | route route retirement: route deletion records remove the routes that delete no SST uses ``` Stack map: | Diff | Adds | |---|---| | 1 | Finalize blob metadata outside the DB mutex (prep) | | 2 | Indirect BlobIndex (kIndirectBlob) and exact SST reference metadata | | 3 | Relocation file format and stable read path <== this diff | | 4 | MANIFEST routes and relocation state in the Version | | 5 | Exact reverse reference census | | 6 | MANIFEST apply outside the DB mutex, with a precondition re-check | | 7 | Coarse flush metadata, per-SST reference cap, epoch runs | | 8 | Indirect blob GC building blocks | | 9 | Indirect blob GC job | | 10 | Flatten generation numbers; route retirement | | 11 | Remote compaction validation | | 12 | Dynamic options and operations | | 13 | Lost relocation files | | 14 | RepairDB rebuilds relocation state; option gate removed | | 15 | db_stress and crash test support | ## Stack context: terms for later diffs | Term | Meaning | |---|---| | Version file table | The table in a Version that maps a generation number to a live file or to a route (diff 4) | | census | Background count, per generation number, of the SSTs and references that use it, plus live bytes per physical file (diff 5) | | indirect origin | Normal blob file written with the option on; it is the origin of unflattened references | | flattened reference | Index with generation number != origin (type 4): compaction rewrote the generation number to the relocation file that holds the value (diff 10) | | flatten | Compaction rewrites the generation number to the relocation file that holds the value, so old routes can go. Only flatten makes generation number != origin (diff 10) | | blob file retirement | MANIFEST record (kBlobFileRetirement) that removes a collected blob file from the Version (diff 4) | | route deletion | MANIFEST record that removes a route that no SST uses. Route retirement is the background job that writes these records (diff 10) | Tracking issue: meta#15301 Differential Revision: D124682381
…5542) Summary: Pull Request resolved: meta#15542 ## Why Indirect blob GC (this stack) moves the live values of a blob file into a relocation file (a file that holds values GC copied, diff 3). It never rewrites the SSTs that point to them. An SST reference finds its value by a generation number: a file number that does not change when the value moves. So every reader needs a durable map from the generation number to the file that holds the bytes now. A route is one entry of this map: it sends reads of a removed file number to the relocation file that holds its values now. Files on disk cannot give this map: a relocation file on disk may be half written, not installed yet, or already replaced. So the map must be committed in the MANIFEST, in the same edit as the file it points to, and each Version must own routes and relocation files the same way it owns SST and blob files. Each file number must also belong to exactly one column family and one file kind (SST, blob file or relocation file). Otherwise two column families could delete each other's files. "Terms used in this diff" below defines all terms. Nothing in this diff creates routes or relocation files. The indirect blob GC job (diff 9) is the first writer. This diff is the durable model that GC installs into. ## Terms used in this diff Stack terms: | Term | Meaning | |---|---| | origin | Blob file where the value was first written | | original offset | Value offset in the origin file | | stable identity | (origin, original offset). Never changes, even when the value moves | | generation number | The file number that a reference uses to find its value. A new value has generation number == origin. When indirect blob GC moves the value, the reference keeps its generation number, and a route sends the read to the relocation file | | indirect BlobIndex | A kIndirectBlob (type 3) or kIndirectFlattenedBlob (type 4) index (diff 2). It names the value by its stable identity, not by where the bytes are now | | unflattened reference | Index with generation number == origin: the value was not moved, or it was moved but no compaction has rewritten its reference yet (see "flatten" in "Stack context: terms for later diffs") | | flattened reference | Index with generation number != origin (type 4). Only flatten (diff 10) writes it; diff 2 defines the encoding | | relocation file | Block-based table keyed by (origin, original offset) that holds copied values (diff 3) | | live file | A blob file or relocation file that GC has not removed | | old file | A file number whose file GC has removed. The number stays only as the key of a route, because SSTs may still store it as their generation number | | route | Maps an old file number to the relocation file that now holds its live values (the target). One hop: a target is never itself routed. This diff stores routes in the MANIFEST, and the Version owns them | | physical file | A file that exists on disk under its file number: an SST, a blob file or a relocation file. A generation number is not always a physical file: after GC removes a blob file, its number can stay only as the key of a route | | indirect origin | Normal blob file written with enable_blob_indirection on. It is the origin of unflattened references | | exact reference metadata | Per-SST sorted list of every blob reference the SST keeps, written when the SST is built. References with the same encoding class, generation number and origin form a group. The list is stored as data blocks, with an index of the groups. An SST that has it is an exact SST | | option gate | Until diff 14, ColumnFamilyData::ValidateOptions returns NotSupported for enable_blob_indirection on a read-write open, CreateColumnFamily and SetOptions, unless a test calls ColumnFamilyData::TEST_SetAllowBlobIndirection(true) | Terms of this diff: | Term | Meaning | |---|---| | target | The live relocation file that a route points to. It holds all live values of the old file | | format fence | A sticky, required MANIFEST record that says a column family (per-CF fence) or the DB (DB-global fence) may hold feature data. A binary without the feature fails to open the DB | | fenced column family | A column family whose per-CF fence is set. Only a fenced column family has a file table. Other column families are legacy column families | | feature data | Persisted state that only blob indirection writes: a format fence, a relocation file, a route, a blob file retirement or accepted-loss record, an indirect-origin blob file, or an SST with exact reference metadata | | activation | The first edit in the DB that sets a per-CF fence. From then on the VersionSet keeps the DB-global fence | | file table | BlobFileTable: per-Version table from a file number to a live blob file, a live relocation file or a route | | relocation state | BlobRelocationState: the live relocation files, the routes and the file table of one Version. Versions share it copy-on-write | | blob file retirement | A record (kBlobFileRetirement, one file number) that removes a collected indirect-origin blob file from the Version | | route deletion | A record that removes the route of an old file that no SST uses any more | | accepted loss | A record (kBlobPhysicalFileLoss, one file number) that declares a relocation file lost. It never names a normal or indirect-origin blob file; base RocksDB records a missing one as dropped (kDroppedBlobFile) when a DB is opened with best_efforts_blob_db_repair. This diff decodes it, and VersionBuilder rejects it | ## Parts of this diff Core (steps 1 to 6 of "How to read this diff"): - MANIFEST records: db/blob/blob_relocation_manifest.{h,cc}, db/version_edit.{h,cc}, and the indirect-origin flag in db/blob/blob_file_addition.{h,cc} and db/blob/blob_file_meta.{h,cc}. - File table and relocation state (BlobFileTable, BlobRelocationState): db/blob/blob_relocation_manifest.{h,cc}. - Read path (Version::ResolveBlobPhysicalFile): db/version_set.{h,cc}. BlobRelocationFileAddition::ToPhysicalFile() builds the description of a relocation file that the read path opens, and the point-in-time MANIFEST replay uses it too (VersionEditHandlerPointInTime::VerifyBlobRelocationFile). - Edit checks: db/version_builder.cc. - Fences and MANIFEST writes: db/flush_job.cc, db/compaction/compaction_job.cc, db/db_impl/db_impl_open.cc, db/version_set.cc, db/version_edit_handler.cc. Supporting (steps 7 and 8; you can skip them on a first read): - Freeing relocation state and catch-up without the DB mutex: db/version_set.{h,cc}, db/job_context.h, db/db_impl/db_impl_secondary.{h,cc}. A file table can be large, and the relocation file checks at catch-up do I/O. - Small items: relocation files in checkpoint and backup file lists (db/db_filesnapshot.cc), the purge prefix (db/db_impl/db_impl_files.cc), and a temporary RepairDB guard (db/repair.cc). A read goes through the core parts in this order: ``` Version::GetBlob / MultiGetBlob -> stable blob cache (diff 3): a hit returns at once -> Version::ResolveBlobPhysicalFile: look up the generation number in the file table (BlobFileTable) live blob file or live relocation file: read it (one lookup) route: look up its target, a live relocation file (two lookups) -> BlobSource reads the physical file (diff 3) ``` ## How to read this diff 1. Read "Terms used in this diff" above. Then read the top comment of db/blob/blob_relocation_manifest.h ("Background", "Terms", "Lifecycle of one value", "MANIFEST records"). Its example, blob file 12 and relocation file 20, shows the whole idea. 2. Records. In db/blob/blob_relocation_manifest.h, read BlobRelocationFileAddition, BlobRelocationFileMetaData, BlobRelocationRoute and BlobRelocationRouteDeletion. Then read the new tags and setters in db/version_edit.h, and the indirect-origin flag in db/blob/blob_file_addition.h and db/blob/blob_file_meta.h. These depend only on the relocation file format from diff 3. 3. File table. Read BlobFileTable in db/blob/blob_relocation_manifest.h (the comment first, then the .cc). It is a copy-on-write map from a file number to a tagged value that points to the metadata of step 2. 4. Version state and reads. Read BlobRelocationState (same files). Then read the relocation API of VersionStorageInfo in db/version_set.h, and Version::ResolveBlobPhysicalFile and GetBlob in db/version_set.cc. Now you know how a read finds a value. 5. Edit checks. In db/version_builder.cc, read the comment "How a relocation change reaches the new Version" above blob_relocation_file_changes_. Then read Apply() (blob file retirements first), ApplyBlobFileRetirement, ApplyBlobRelocationState and SaveBlobFilesAndCheckTo. 6. Fences and MANIFEST writes. Read VersionEdit::MaybeSetBlobIndirectionFormatFence and its callers (flush_job.cc, compaction_job.cc, db_impl_open.cc). Then read WriteCurrentStateToManifest and the fence part of ProcessManifestWrites in db/version_set.cc, and the replay of the new records in db/version_edit_handler.cc. The "no re-add" rule is explained once, at VersionSet::RejectReaddedBlobRelocationState in db/version_set.h. 7. Other supporting work. Read "Freeing relocation state" at VersionSet::FreeBlobRelocationState in db/version_set.h, then ~Version and db/job_context.h; catch_up_mutex_ in db/db_impl/db_impl_secondary.h; the RepairDB guard in db/repair.cc; and the live-file lists in db/db_filesnapshot.cc. ## Previous behavior - The MANIFEST tracked SST files and normal blob files only. - A kIndirectBlob reference (diff 2) was read from the blob file named by its generation number. Diff 3 added a relocation file reader, but no Version could send a read to a relocation file. - No MANIFEST record marked a DB or column family as using the new formats, so an older binary could open such a DB and misread it. ## New behavior The top comment of db/blob/blob_relocation_manifest.h describes the records, the route rules and the lifecycle of one value, and the BlobFileTable class comment gives the table layout. This section states the contract. ### MANIFEST records - A relocation file addition keeps only small fields: size, checksum method and value, tail size, total entries, total record bytes and the 4-byte directory digest (the file's identity, checked at open). The deletion is a separate record. The file stays on disk while any Version uses it, like a blob file. - There is one route per old file number. Route and relocation file numbers are at most kFileNumberMask (2^62 - 1), the same limit as SST and blob files. - The relocation records carry no format version or column family of their own. The fence gives the version, and the edit names the column family. - All new tags are required tags, so an older binary fails to open the DB instead of misreading it. ### Route rules (one hop) Route rules: see "MANIFEST records" in db/blob/blob_relocation_manifest.h. This diff does not check that a target holds a given origin. Indirect blob GC proves it at install, and the reader checks the key on every read. ### Reads Version::GetBlob, GetBlobRange and both MultiGet paths first look up an indirect value in the stable blob cache (diff 3). A hit does no route or file lookup. On a miss, Version::ResolveBlobPhysicalFile looks up the generation number in the file table. A live blob file (unmoved value) or a live relocation file (flattened reference) takes one lookup. A route takes two lookups, and its target is always a live relocation file. The relocation reader checks the key (origin, original offset). The read path copies no shared_ptr. ### File table - Each Version of a fenced column family has one file table with every live blob file, live relocation file and route. Legacy column families have none. - The table is sharded and copy-on-write. An unchanged Version shares the whole state. An edit copies the top array and rebuilds only the shards it changes; a route change also rebuilds the shard of its target, which holds the route count. - The table memory is charged to the file metadata cache reservation, best effort: a failed charge never fails an edit. An entry takes 16 bytes with 32-bit keys, or about 18 bytes with the shard headers, padding and top array (computed from the struct sizes, not measured; a unit test checks 12 to 32 bytes per entry for 10,000 entries). - Relocation files are also kept in a sorted vector for iteration. A debug build checks after each Version build that the table and the vectors agree. Versions and MANIFEST snapshots keep no deletion history. ### Edit checks Every edit is checked against the final state of its own column family. A bad edit fails with Corruption (an accepted-loss record and a fence on a FIFO column family fail with NotSupported) and has no partial effect (see the ApplyBlobRelocationState comment). These are rejected: duplicate or conflicting records; a set and a delete of the same key; a file number used by another file kind in the same column family; deleting a relocation file that is not live there, or that a route still targets; a route from a live file; a route to anything but a live relocation file. Two more rules: - The "no re-add" rule: adding back a relocation file, route or blob file that an earlier edit of the same MANIFEST deleted (or removed by blob file retirement) is rejected at replay and when LogAndApply writes, because file numbers are never reused. A new MANIFEST starts without deletion history. With reuse_manifest_on_open, the deletions replayed from the reused MANIFEST seed the write-time check. The comment at VersionSet::RejectReaddedBlobRelocationState names the three places that check it. - A repeated relocation file addition (MANIFEST replay or catch-up) must equal the file that the builder already has. Apply() compares the small records and returns Corruption on a mismatch. ### Fences - Both fences are sticky (never cleared, also after the option is turned off) and are required tags. - Every edit that adds an SST built with enable_blob_indirection on sets the per-CF fence. Flush, compaction and the WAL-recovery flush all use one helper for this, VersionEdit::MaybeSetBlobIndirectionFormatFence. Because of the option gate, only tests reach this. - A MANIFEST snapshot writes the DB-global fence first, and the per-CF fence only for column families that have their own. - FIFO compaction is rejected for a column family whose own fence is set, also after the option is turned off and after MANIFEST rotation. ### Blob file retirement - A blob file retirement record removes a live indirect-origin blob file that indirect blob GC has collected. The indirect blob GC job (diff 9) is its first writer. A file that still has live values also gets its route in the same edit; an empty file leaves without a route. - VersionBuilder applies blob file retirement records before the relocation records, so the rule "the old file of a route is not a live file" still holds. - SSTs can keep references under the number of a blob file removed by blob file retirement. Compaction writes no BlobFileMetaData garbage record for a routed generation number or for a live relocation file. VersionBuilder ignores a garbage record for a routed generation number (it can come from a compaction picked before the blob file retirement). The census of diff 5 tracks these values instead. - These fail with Corruption: a route from a live blob file without its blob file retirement record, a blob file retirement record for a file that is not a live indirect-origin blob file, and a new blob file or relocation file that reuses a route key. ### Accepted loss The accepted-loss record is decoded here. VersionBuilder fails on it with NotSupported instead of skipping it. Diff 5 applies it. No code in this diff writes it. ### File protection File numbers come from one DB-wide counter, so two column families do not get the same file number. Nothing checks this across column families. Three kinds of checks keep a wrong MANIFEST record from deleting or misreading a file: - Per column family: VersionBuilder checks each relocation, route and retirement record against the state of the edit's own column family (see "Edit checks" above). - Purge: the obsolete file purge is DB-wide. A .blob file stays while any column family's Version lists it as a blob file or a relocation file (Version::RemoveLiveFiles() and, for a full scan, Version::AddLiveFiles()). - Read: a reader checks the column family ID in the blob file header, or in the table properties of a relocation file, when it opens the file. Each read in a relocation file also checks the value's key (origin, original offset), so a wrong route cannot return another value. "File protection" in db/blob/blob_relocation_manifest.h lists every check. ### Freeing builder state inside a batch LogAndApply frees the relocation changes of the batch's version builders after the install, without the DB mutex, before the batch finishes (release_builder_guards() in VersionSet::ProcessManifestWrites()). The writer callbacks run after this free, when the writers are marked done (see the LogAndApply comment in db/version_set.h). So a Flush(wait=true) caller cannot return before the flush job installs its SuperVersion. During the free, other threads can see the new Versions while the flushed memtables are still in the immutable list. This window is safe: readers use SuperVersions, which are not installed yet; flush picking skips the flush-completed memtables; WAL obsolescence keeps more logs; the memtable overlap check of ingestion sees extra data; and compaction does not read memtables. ### Other changes - BlobFileAddition gets an indirect-origin flag, so a Version knows which normal blob files are indirect origins. It is a new custom field, written only when set. It adds about 8 bytes per blob file in memory (an estimate, with padding). - Checkpoints and backups copy relocation files: live-file lists, checksum lists and GetLiveFilesStorageInfo include them. ### Monitoring - BLOB_DB_INDIRECT_READ_ROUTED (ticker) counts reads that miss the blob cache and find their file through a route in Version::ResolveBlobPhysicalFile(): the reference names an old file whose live values moved to a relocation file. A blob cache hit resolves no file, so it does not count. A route whose target is missing fails with Corruption and does not count. - rocksdb.blob-indirection-format-version (DB::Properties::kBlobIndirectionFormatVersion, an int property per column family) returns the per-CF format fence of the current Version. 0 means that the column family has no format fence yet: no SST built with enable_blob_indirection on was installed. - The Corruption messages of ResolveBlobPhysicalFile() name the generation number and the origin of the reference, and the route target when the target is missing. - ldb manifest_dump prints one line for each column family that has the per-CF format fence, relocation files or routes: "--- blob indirection --- format version: V, relocation files: F (B bytes, N values), routes: R". When the DB-global fence is set, it also prints "db blob indirection format version V". A DB without blob indirection prints nothing more. - Feature off: the ticker changes only on the route branch, and only a fenced column family has routes. The property reads one field of the current Version when it is called. ## Changes visible to existing users (feature off) | Change | Effect | |---|---| | Freeing Version state | A fenced Version holds a file table that can be large. Its memory is freed without the DB mutex ("Freeing relocation state" at VersionSet::FreeBlobRelocationState). Versions, builders and FileMetaData reference counts still change only under the DB mutex. A legacy Version or builder has no relocation state, so nothing changes for it | | Secondary and follower catch-up | Record decoding, the relocation file checks (I/O) and blob metadata finalization run without the DB mutex. catch_up_mutex_ serializes them: a concurrent caller waits and then catches up. Builder apply, SST file saves and Version deletion stay under the mutex. In a legacy DB, record decoding and blob metadata finalization also run without the DB mutex | | RepairDB | First reads all MANIFESTs once, and the descriptor property of every SST, to look for feature data (see "Safety boundaries") | | LogAndApply | One O(1) check per edit (VersionEdit::SetsBlobIndirectionState). The legacy path does no other new work under the DB mutex | | Version build | The relocation state is one null shared pointer. The legacy validation does not scan blob files | ## Implementation | File | Change | |---|---| | db/blob/blob_relocation_manifest.{h,cc} (new) | Relocation file addition, route and route deletion records, the fence constant, the file table (BlobFileTable) and the copy-on-write state (BlobRelocationState) used by VersionStorageInfo | | db/version_edit.{h,cc} | Codecs for the new records: relocation file add and delete, route set and delete, blob file retirement, per-CF and DB-global fence, accepted loss | | db/version_builder.{h,cc} | Stages the records of an edit, checks them against the final state, then applies them to the shared state | | db/version_set.{h,cc} | Route resolution (ResolveBlobPhysicalFile). SyncBlobFileTable keeps the table's blob file entries in step with the saved blob files. MANIFEST snapshot of relocation state and fences. DB-global fence handling. FreeBlobRelocationState. The write-time check against adding back deleted relocation state | | db/version_edit_handler.{h,cc} | Replay of the new records, fence handling during recovery, and the ManifestTailer steps that run without the DB mutex (through the RunWithoutDbMutex helper; freed relocation state goes through VersionSet::FreeBlobRelocationState()) | | db/flush_job.cc, db/compaction/compaction_job.cc, db/db_impl/db_impl_open.cc | Write the per-CF fence through the shared helper | | db/db_filesnapshot.cc and live-file lists | Include relocation files | | db/repair.cc | Temporary RepairDB guard (see "Safety boundaries") | | db/db_impl/db_impl_secondary.*, db_impl_follower.cc | catch_up_mutex_ for catch-up without the DB mutex | ## Safety boundaries and unsupported cases - Public API: new ticker BLOB_DB_INDIRECT_READ_ROUTED, added just before TICKER_ENUM_MAX, and new property rocksdb.blob-indirection-format-version. The new ticker has no Java mapping, as for recent base tickers: all 256 byte values of org.rocksdb.TickerType are used. - Normal recovery fails closed when a required record names a file that is not valid. A bad edit is never partly applied. - Downgrade to a binary without the feature is not supported once a fence is written. The fence is a required tag, so the older binary fails to open. - Temporary RepairDB guard. Before any archive or NewDB step, RepairDB looks for feature data: in any MANIFEST, a fence, a relocation or route record, a blob file retirement or accepted-loss record, or an indirect-origin blob file addition; in any SST, a decodable exact reference metadata descriptor. If it finds any, it returns NotSupported and leaves all inputs untouched. Otherwise legacy repair runs as in base, and its WAL-converted output keeps enable_blob_indirection off. Diff 14 replaces the guard with full repair. - Nothing in this diff writes routes, relocation files or loss records. Diff 9 (the indirect blob GC job), diff 10 (flatten, route retirement) and later diffs do. - The deleter of a relocation file does not evict its cached reader. The deleter can run under the DB mutex: this happens when the last Version that owns the file is destroyed. A failed LogAndApply batch or ManifestTailer frees relocation state without the DB mutex. An eviction would take the stripe mutex of the file in the blob file cache, which a relocation file open can hold while it reads its location blocks (pin_blob_relocation_location_blocks). The deleter only drops the remembered failed open of the file. PurgeObsoleteFiles() deletes the file and evicts the reader with TableCache::Evict(), outside the DB mutex (the blob file cache shares its Cache with the table cache). A relocation file has no blob log footer, so the purge recognizes it by its prefix (kBlobRelocationFilePrefix) instead of keeping it as a footerless blob file that blob direct write may still be writing. - Option gate (see Terms): behavior with the option on is reachable only through the test hook. Diff 14 removes the gate. ## Validation and known limitations - Unit tests cover the record codecs, edit checks and atomic apply, the file table against a std::map model (both key scans, both key widths), blob file retirement, replay and MANIFEST snapshots, checkpoints, freeing state without the DB mutex, the RepairDB guard and secondary catch-up. See the Test Plan. - A db_bench benchmark compares the file table with a simpler lookup design (a sorted blob file vector and a std::map route table) and finds the same read and write performance (see the Test Plan). It does not compare with base RocksDB. - Known limitations: - RepairDB refuses DBs with feature data until diff 14. - The RepairDB search for feature data is best effort. It skips a MANIFEST, record or SST that it cannot read. So a DB whose feature data is all unreadable still gets legacy repair. - Only tests can turn the option on until diff 14. ## Stack context (same in diffs 2-14, except that diffs 10-14 add the diff 10 route rule to the install line; "<== this diff" marks this diff's part): Goal: remove garbage from blob files without rewriting any SST. A value gets a stable identity when it is first written. Indirect blob GC copies the live values of whole files into a relocation file and installs routes in the MANIFEST. Readers follow the routes. SSTs are not changed. How a read finds a value (P is the relocation file that holds the value now): ``` SST entry indirect BlobIndex { generation number, origin, original offset, size, CRCs } | v Version file table: generation number -> live file, or route old file -> P | | live file route found | | v v blob file: read at original offset relocation file P (always one hop): relocation file: look up key look up key (origin, original offset) (origin, original offset) ``` Lifecycle of one value (in this example, F is the blob file where the value is first written, and P is the relocation file that GC writes): ``` write new value goes to blob file F; index: origin = F, generation number = F | census per generation number: the SSTs and references that use it; | per physical file: live bytes, ranked by garbage ratio GC job copy the live values of whole input files (stored bytes, checked | with the stored CRC) into relocation file P, keyed by | (origin, original offset) install one MANIFEST edit: add P, route each input generation number still | in use to P, re-point the routes into the inputs to P, and remove | each input (blob file retirement for a blob file, relocation | file deletion for a relocation file) flatten compaction sets generation number = P in the SSTs it rewrites | route route retirement: route deletion records remove the routes that delete no SST uses ``` Stack map: | Diff | Adds | |---|---| | 1 | Finalize blob metadata outside the DB mutex (prep) | | 2 | Indirect BlobIndex (kIndirectBlob) and exact SST reference metadata | | 3 | Relocation file format and stable read path | | 4 | MANIFEST routes and relocation state in the Version <== this diff | | 5 | Exact reverse reference census | | 6 | MANIFEST apply outside the DB mutex, with a precondition re-check | | 7 | Coarse flush metadata, per-SST reference cap, epoch runs | | 8 | Indirect blob GC building blocks | | 9 | Indirect blob GC job | | 10 | Flatten generation numbers; route retirement | | 11 | Remote compaction validation | | 12 | Dynamic options and operations | | 13 | Lost relocation files | | 14 | RepairDB rebuilds relocation state; option gate removed | | 15 | db_stress and crash test support | ## Stack context: terms for later diffs | Term | Meaning | |---|---| | census | Background count, per generation number, of the SSTs and references that use it, plus live bytes per physical file (diff 5) | | flatten | Compaction rewrites the generation number to the relocation file that holds the value, so old routes can go. Only flatten makes generation number != origin (diff 10) | | route retirement | The background job that writes route deletion records for the routes that no SST uses (diff 10) | Tracking issue: meta#15301 Differential Revision: D124740371
Summary: Pull Request resolved: meta#15543 ## Why the change is required Indirect blob GC (this stack) moves the live values of a blob file into a relocation file and never rewrites the SSTs that reference them. An SST reference finds its value by its generation number: a file number that stays the same when the value moves. Before GC drops a value, it must prove that no SST still uses it. Routes (old file number -> relocation file, diff 4) point forward, so they cannot tell which SSTs use a generation number. GC needs the reverse view: for each generation number, the SSTs that use it and the bytes they keep. This diff adds that view, the census. "Terms used in this diff" below defines all terms. The census must not slow down the MANIFEST write queue. A full scan of the SSTs, or a copy of a DB-sized reverse index for each new Version, at the head of the queue would delay every Version install. So a background job builds the census, and each install hands it only a small prepared change. Nothing in this diff uses the census to collect files. The indirect blob GC job (diff 9) is the first user. ## Terms used in this diff Stack terms: | Term | Meaning | |---|---| | origin | Blob file where the value was first written | | original offset | Value offset in the origin file | | stable identity | (origin, original offset). Never changes, even when the value moves | | generation number | The file number that a reference uses to find its value. A new value has generation number == origin. When indirect blob GC moves the value, the reference keeps its generation number, and a route sends the read to the relocation file | | unflattened reference | Index with generation number == origin: the value was not moved, or it was moved but no compaction has rewritten its reference yet (see "flatten" in "Stack context: terms for later diffs") | | flattened reference | Index with generation number != origin (type 4). Only flatten (diff 10) writes it; diff 2 defines the encoding | | relocation file | Block-based table keyed by (origin, original offset) that holds copied values (diff 3) | | route | Maps an old file number (a file that GC removed) to the relocation file that now holds its live values (the target). One hop: a target is never itself routed. Stored in the MANIFEST (diff 4) | | indirect origin | Normal blob file written with enable_blob_indirection on. It is the origin of unflattened references | | exact reference metadata | Per-SST sorted list of every blob reference the SST keeps, written when the SST is built (diff 2). Its reference index holds the totals of each group | | group | All references of one SST with the same (encoding class, generation number, origin), as in the exact reference metadata (diff 2). The encoding class is kDirect, kIndirectUnflattened or kIndirectFlattened | | SST summary | BlobReferenceSstSummary (diff 2): the totals of every indirect group of one SST, in memory. The table builder makes it for a new SST, or a reader decodes it from the reference index of an existing SST. It is never written to the MANIFEST | | descriptor | The checksummed table property that points to an SST's reference index and holds its totals (diff 2) | | format fence, fenced CF | A required MANIFEST record (diff 4) that marks a column family (per-CF fence) or the whole DB (DB-global fence) as holding blob indirection data. A fenced CF is a column family whose per-CF fence is set | | relocation state | BlobRelocationState: the relocation files, routes and file table of one Version (diff 4) | | blob file retirement | MANIFEST record (kBlobFileRetirement, diff 4) that removes a collected indirect-origin blob file from the Version | | accepted loss | MANIFEST record (kBlobPhysicalFileLoss, decoded since diff 4) that declares a relocation file lost | | option gate | Until diff 14, ColumnFamilyData::ValidateOptions returns NotSupported for enable_blob_indirection on a read-write open, CreateColumnFamily and SetOptions, unless a test calls ColumnFamilyData::TEST_SetAllowBlobIndirection(true) | Terms of this diff (the top comment of db/blob/blob_reference_census.h defines them in more detail): | Term | Meaning | |---|---| | census | Background, in-memory count, per generation number, of the SSTs that reference it, plus live bytes per physical file. One per column family that needs it | | exact SST | An SST with exact reference metadata (a nonempty descriptor). The census counts only exact SSTs, and only their two indirect classes | | coarse SST | An SST built without exact reference metadata. It records one file number, its frontier, and GC must treat every indirect-origin file at or above the frontier as used (see "Coarse SST" in db/blob/blob_reference_metadata.h). No code in this diff writes a coarse SST | | physical file (in the census) | An indirect-origin blob file or a relocation file: a file that holds value bytes and can be a GC candidate | | holder | An exact SST that has at least one counted group with a given generation number | | slot | A small dense number that the census gives each exact SST, so that holder sets stay small | | delta | The census change of one Version install: SSTs added and removed (with their summaries) and relocation state changes | | change log | One ordered queue of deltas. The census reads it through one cursor, the build cursor, during the build and after it | | census job | The background job on the LOW pool that builds the census and applies deltas | | worker | The code that applies deltas and changes the census contents. The census job runs it, and so does a flush or compaction thread right after its install. Only one thread runs it at a time | | ready (census publication) | The census has applied every delta up to the current Version and stores its ranked GC candidates in that Version | | install reservation | While a Version install of the column family is being prepared (BeginInstallPreparation()), the census does not become ready, because that Version is about to be replaced | ## Parts of this diff Core (steps 1 to 7 of "How to read this diff"): - Census data structures (holder sets, candidate heap, deltas, change log, worker): db/blob/blob_reference_census.{h,cc}. - Summary sources: the table builders, TableCache::ReadBlobReferenceSstSummary and TableCache::GetBlobReferenceReader (db/table_cache.{h,cc}), and CompactionJob::ReadInputBlobReferenceSummaries (db/compaction/compaction_job.cc). - Install path: VersionSet::ReserveBlobReferenceCensusInstalls and PrepareBlobReferenceCensusInstalls (db/version_set.{h,cc}). - Census job and the apply after an install: DBImpl::ScheduleBlobReferenceCensus, BuildBlobReferenceCensus, BGWorkBlobReferenceCensus, ApplyBlobReferenceCensusAfterInstall and ApplyBlobReferenceCensusDeltas (db/db_impl/db_impl_open.cc, called from db/db_impl/db_impl_compaction_flush.cc), and DBImpl::ReleaseColumnFamilyDataRef (db/db_impl/db_impl.{h,cc}), which releases the column family references that the census holds. Supporting (step 8; you can skip them on a first read). Each one is needed by the census: - SST metadata in the MANIFEST (db/version_edit.{h,cc}): the census must know after reopen which SSTs are exact, so the descriptor is stored in the MANIFEST. - Blob file lifetime (db/version_builder.cc): indirect blob GC needs an indirect origin to stay in the Version while SSTs still use it. - Accepted loss (db/blob/blob_relocation_manifest.h, db/version_builder.cc): GC must not pick a lost relocation file that still has live bytes, so the Version marks the file lost. - Version release (db/version_set.{h,cc}): the census job pins a Version, and it must free the Version's relocation state without the DB mutex. - Cache charging (include/rocksdb/cache.h): the new CacheEntryRole::kBlobReferenceCensus. Groundwork with no production user in this diff: see "Groundwork" below. ## How to read this diff 1. Big picture. Read "Why the change is required" and "Terms used in this diff". Then read the top comment of db/blob/blob_reference_census.h (Background, Terms, Ownership, Data flow of one install, Scheduling, Apply after an install, Build, Per generation number, Readers, Threads and locks, Why this design, Census memory charge, Errors and rebuild, Self-check). It names every census part and the order in which they run. 2. Input type. Read BlobReferenceSstSummary in db/blob/blob_reference_metadata.h (diff 2). Every census change is a list of these summaries. Also read the "Coarse SST" section in the same file. 3. Holder sets and candidate heap. Read BlobCensusSlotTable and BlobCensusSlotSet in blob_reference_census.h. Then read BlobCensusSlotSet::Update(), UpdateBitmap() and Encode() in blob_reference_census.cc. They depend on no other new part, and they hold most of the census memory. Next read BlobGcPhysicalFileSummary and BlobGcCandidateHeap (header and .cc). The census uses this heap to rank physical files by garbage ratio. 4. Deltas and the change log. Read BlobReferenceCensusDelta, BlobReferenceChangeLogEntry and ChangeLogCursor in the header. Then read Install(), AppendChange(), TakeBuildDeltas() and AcknowledgeBuildDeltas() in the .cc. This is how each Version install reaches the census. 5. Worker. Read WorkerState::ApplyFiles() and WorkerState::Finalize(), then BlobReferenceCensus::ApplyWorkerDelta(), in blob_reference_census.cc. They apply summaries and deltas to the holder sets and to the heap. 6. Producers. Read where summaries and deltas come from: TableBuilder::GetBlobReferenceSstSummary (table/table_builder.h), TableCache::ReadBlobReferenceSstSummary and the reader it gets from TableCache::GetBlobReferenceReader (db/table_cache.h), CompactionJob::ReadInputBlobReferenceSummaries (db/compaction/compaction_job.cc), and the install path at BlobReferenceCensusInstallBatch in db/version_set.h (its comment lists the steps). 7. Drivers. Read the census job in db/db_impl/db_impl_open.cc: DBImpl::ScheduleBlobReferenceCensus, BuildBlobReferenceCensus, BGWorkBlobReferenceCensus and PublishBlobReferenceCensusWorker. Then read the apply after an install in the same file: ApplyBlobReferenceCensusAfterInstall and ApplyBlobReferenceCensusDeltas. BackgroundCallFlush and BackgroundCallCompaction (db/db_impl/db_impl_compaction_flush.cc) call them at their end. Last, read the debug check CheckBlobReferenceCensus (it uses BlobReferenceCensusState, the selected copy). 8. Supporting changes. Read the new NewFile tags (18, 19, 20) in db/version_edit.h, accepted loss (db/blob/blob_relocation_manifest.h, db/version_builder.cc), the blob file lifetime rules (db/version_builder.cc) and Version::UnrefAndFreeBlobRelocationState (db/version_set.h). Last, the groundwork listed under "Groundwork" below. ## Previous behavior - There was no reverse census, so nothing could find the SSTs that use a generation number. - The MANIFEST did not record an SST's reference descriptor. - Diff 4 decoded the accepted-loss record but failed replay on it. - A blob file left the Version when no SST linked it and its approximate garbage count reached its total. This also applied to indirect origins. - Blob files below the oldest linked blob file (the smallest oldest_blob_file_number over the saved SSTs) were dropped from the Version. This also applied to indirect origins. ## New behavior ### Core: census job - Each fenced CF has one census. One background job on the LOW pool serves the census of every fenced CF in turn. The job builds no census for a CF without the per-CF fence. - Once the DB has the DB-global format fence, a column family with enable_blob_indirection also gets the census object at its next install, before its own per-CF fence is set (ColumnFamilyData::blob_reference_census()), so that its installs can be reserved. The job builds that census only after the per-CF fence is set. - Open schedules the census of each CF that already has the fence. A CF that gets the fence after open (at its first flush with the option on) has its census scheduled when that Version's SuperVersion is installed. - Apply after an install: the census job shares the LOW pool with compactions, so with few background threads a delta could wait a long time. So at the end of each background flush and compaction, the thread releases the DB mutex and runs the worker for each column family with a census that the job flushed or compacted (also when the job failed). A column family dropped before the job ends is not visited. So the work is O(the job's column families), and a DB without blob indirection has nothing to visit. A column family in the retry list is visited too, so its deltas do not wait for the periodic task. For each column family, the thread applies the queued deltas up to the end of the change log. If it applied at least one delta, it then tries to publish. - This step does not wait: if another thread runs the worker of a column family, it skips that column family. It also skips a column family whose build has not started, has not finished its scan of the SSTs, or has a stored failure. - This step never fails the flush or the compaction. When it cannot go on, it leaves the rest to the census job. A delta that gets TryAgain (a registered indirect blob GC job has not yet copied its snapshot) is not changed and stays at the front of the queue. On a change log overflow, the thread stops, and the census job restarts the build, or fails it after a second overflow; if a delta failed at the same time, the thread drops the queued deltas. Any other failure may have changed the census part way, so that delta is never applied again: the thread stores the failure and logs it once, later applies after an install skip the column family, and the census job fails the build with that error and builds again. A census job run that fails stores its failure the same way before it releases the worker run lock, so no apply after an install runs between the failure and the end of the build. The census job also handles a publish that fails or must wait. - Installs on other threads (for example a user CompactFiles()) do not run this step. The census job, or the next flush or compaction of the same column family, applies their deltas. - After a failure, the census retries through a periodic task (period 1 second). The task is registered only when it is first needed. The retry delay doubles after each failure: 1 s, 2 s, and so on up to 32 s. ### Core: holder sets - For each generation number, the census keeps the set of SSTs that hold it, plus its reference count and byte totals. - The groups of one SST are summed per generation number. An SST that holds a generation number for several origins is one holder of it. - SST numbers map to slots. A new SST gets the lowest free slot. - Each holder set is a delta-varint list or a bitmap, whichever is smaller. Let U be one past the highest slot in use, so the bitmap takes U/8 bytes. A list becomes a bitmap when it is larger than U/8 bytes. A bitmap becomes a list when the list would be smaller than U/16 bytes. The gap between the two points keeps a set from switching back and forth. - A bitmap is updated in place: bits are set and cleared. It is encoded again only when it may cross the switch point. - A list is encoded again at its exact size, so no capacity slack is charged. - The census keeps no per-SST summary after it adds a file. ### Core: where SST summaries come from - Versions do not keep SST summaries. A summary is set only on new files in an edit, and Versions drop it. - A compaction reads the reference index of each input SST when its job starts. The read uses fill_cache off and the job's rate limiter priority, and its memory is charged to kBlobReferenceCensus. The compaction passes these summaries with its edit, so the census can remove the inputs. - Every compaction does this read, also a full compaction, so every compaction install updates the census with a delta. - If the read fails, the compaction still runs, and the census is rebuilt after the install. ### Core: paths that remove an SST An install marks the census for rebuild when it removes an exact SST without its summary. The census stops being ready, and the build runs again at once, without backoff. The summaries of the SSTs that the install adds are kept as seeds, so the rebuild does not read those SSTs. | Path | Census handling | |---|---| | Compaction (CompactionJob) | Supplies the summaries of the removed SSTs | | Full compaction (every base SST removed) | Like any compaction: supplies the summaries of the removed SSTs (delta) | | DeleteFile, DeleteFilesInRange(s) | Rebuild | | Ingestion with a replace range | Rebuild (ingestion into a fenced CF is rejected until diff 12) | | FIFO deletion | Not reachable: blob indirection rejects FIFO compaction | | Trivial move, ReFitLevel, PromoteL0, experimental metadata update | The same file number is removed and added; the census skips it | | FIFO temperature change (copies a file to a new file number) | Not reachable: a fenced CF rejects FIFO compaction | | RepairDB | Rejected with NotSupported for a DB that uses blob indirection (diff 14 adds support) | | Best-efforts recovery | New DB open; the census is built as at open | ### Core: change log and self-check - The build cursor can have at most kMaxLogMemory (256 MB) of queued delta weight. A delta's weight (QueueWeight()) counts all memory that the delta keeps in the change log, including the SST summaries that have their own cache charge. The change log charges only the rest (ChargeBytes()), so no summary is charged twice. Outside a build, an installing thread usually applies each delta at once. During the scan of a build, installing threads skip the column family, so deltas wait in the log between build slices. 256 MB is an estimate, not a measurement, of a size that only a long build of a very large DB under heavy write load should reach. - The limit is checked before each append, so the cursor holds at most the limit plus one delta. If the cursor is already at the limit, the build restarts at once. The restarted build, and every later build until one publishes, has a larger limit, kMaxRestartLogMemory (1 GB, 4 x kMaxLogMemory, an estimate, not tuned). The factor 4 leaves room for one slow full rebuild under write load. - If a build also overflows the larger limit, it fails and retries after the normal backoff (1 s, doubled after each failure, at most 32 s). The census job logs a WARN line and counts it in a new ticker, BLOB_DB_CENSUS_RESTART_OVERFLOW_COUNT. So overflows never restart builds in a loop without a delay, and the change log memory stays bounded. Indirect blob GC waits while the census is not ready, so no value is lost. - Debug builds compare the ready census with a fresh build from the same Version. They do this after 1 in 16 publications of the census job, chosen with util/random.h. A publish by the apply after an install is not checked. ### Core: GC candidates - A ready census stores ranked indirect blob GC candidates in the Version and schedules compaction picking. No picker uses them until diff 9. - One GC job selects at most kBlobGcMaxSelectedFiles (8) files. - A GC job checks that the census is ready for its Version, and copies the holders of only the generation numbers of the selected files, without the DB mutex. So GC never copies or scans the whole census under the DB mutex. ### Core: cache charging - Census memory is charged to a new CacheEntryRole, kBlobReferenceCensus. This covers the reverse index, its build state, the change log and the queued deltas. The change log charge leaves out the summaries that already have their own cache charge: the table builder charges the summary of a new SST (kBlobReferenceMetadataBuildingBuffer), and the compaction's input read charges the summary of a removed SST (kBlobReferenceCensus). - A change log entry that holds a newer relocation state is charged only for that state's relocation file metadata. The state's file table is already charged to the file metadata reservation (diff 4). Each shard of the table is charged once, until its last holder frees it, so it is not charged again for each entry. - The role follows cache_usage_options.options_overrides, with the same default as the other charged roles. It has its own line in the block cache entry stats. ### Supporting: SST metadata in the MANIFEST - Flush and compaction outputs record their SST descriptor in the MANIFEST. builder.cc and compaction_outputs.cc call ExtractBlobReferenceDescriptorFromTableProperties for this. - Trivial move, ReFitLevel, PromoteL0 and the experimental metadata update keep the blob reference metadata of the moved SST (VersionEdit::AddExistingFile). - The temporary RepairDB guard of diff 4 also treats a new SST record as feature evidence when its descriptor decodes or when it has a coarse frontier (CheckBlobIndirectionRepairAdmission() in db/repair.cc). - A fenced CF rejects a newly added SST that has no exact blob metadata (test VersionBuilderTest.FormatFenceRequiresExactSstMetadata). So in this diff, a flush with the option off on a fenced CF fails closed. Because of the option gate, only the test hook can reach this. ### Supporting: blob file lifetime - In a fenced CF, a blob file below the oldest linked blob file (the smallest oldest_blob_file_number over the saved SSTs) is dropped only when it is a normal blob file. An indirect origin below it stays in the Version until its blob file retirement record (diff 4) removes it. - Behavior change from base: a normal blob file leaves the Version as soon as no SST links it and its approximate garbage count reaches its total. An indirect origin stays even then, because SSTs can still use its generation number while compaction counts its values as garbage. It leaves only through its blob file retirement record. In this diff only tests write that record, so such a file stays in the Version and on disk. Later diffs in this stack add the indirect blob GC job that writes this record, also when the blob options are off or automatic compactions are disabled. - With best_efforts_blob_db_repair, base RocksDB records a missing blob file as dropped (kDroppedBlobFile) only when an SST may still use it: the file has linked SSTs, or some of its values do not count as garbage. A missing indirect origin that has no blob file retirement record is always recorded as dropped, even when all its values count as garbage. The reason is that its garbage count never decides whether it stays in the Version (see the bullet above). SaveDroppedBlobFilesTo() does not keep a dropped file whose number is smaller than the oldest blob file that any SST references. When no SST references a blob file, it keeps no dropped file at all. - An SST links only its oldest blob file. When that file is an indirect origin that blob file retirement removed, the oldest linked blob file still comes from the SSTs, so the newer normal blob files that the SST references keep their place in the Version. - Legacy forced blob GC marks nothing in two cases: while the oldest blob file is an indirect origin and the option is on, or while the oldest blob file has no linked SSTs. When the option is off, legacy forced blob GC still forces an oldest blob file that SSTs still link. ### Supporting: accepted loss - VersionBuilder applies the accepted-loss record, both at replay and at install. It marks a relocation file lost in the Version: BlobRelocationFileMetaData::CreateMarkedLost and IsMarkedLost, BlobRelocationState::HasLostFiles, and the lost bit of the file table entry. The census keeps a lost file that still has live bytes out of the candidate heap. - VersionBuilder checks, at replay and at install, that the record names a live relocation file of its own column family ("Blob relocation loss names no live target"). - A lost relocation file loses its own values and the values of every old file routed to it. - A read through a lost relocation file (directly or through a route to it) returns NotFound. ### Supporting: Version release - Background work that pins a Version (for example the census job) releases it with Version::UnrefAndFreeBlobRelocationState. The Version is destroyed under the DB mutex like every Version. Only its relocation state is freed outside the mutex, and only when the Version was its last owner. - LogAndApply frees the census installs and the old census state of a batch after the install, without the DB mutex, before the batch finishes ("Step 4 of the census install" in VersionSet::ProcessManifestWrites()). This is the same window in which diff 4 frees the relocation changes of the version builders, so the same argument holds (see the publish_batch_callbacks comment in ProcessManifestWrites()). - SetOptions on a fenced CF defers only the census payloads until its SuperVersions are installed (VersionSet::ReleaseDeferredPublicationCleanup). ### Groundwork These parts have no production user in this diff. Only tests call them. | Part | What it is | First user | |---|---|---| | Metadata-only commit (BeginMetadataCommit() / EndMetadataCommit()) | A MANIFEST commit that changes relocation state (routes, relocation files) but adds no SST. At most one runs per census at a time. An indirect blob GC job cannot register (BeginGcJob()) while one is in progress (TryAgain), or while its result is unknown (Busy) | Diff 9 | | GC job registration (BeginGcJob(), IsGcJobActive(), EndGcJob()) and the selected copy (BlobReferenceCensusState, SnapshotSelected()) | At most one GC job is registered per census. From BeginGcJob() until SnapshotSelected() has copied the selected generation numbers, the worker applies no delta, so the copy shows the census at the job's input Version. The job reads no later delta, and an install that invalidates the census does not end the registration. The debug self-check already uses the selected copy | Diff 9 | | Candidate publication | The ranked candidates stored in the Version | Diff 9 | | Indirect blob GC accessors (SuppressBlobFileForIndirectBlobGC and the others in VersionStorageInfo) | Pick and skip candidate files of a Version | Diff 9 | | Relocation state fingerprint | 128 bits that depend only on the relocation files, the routes and the format version, not on the order of the changes. It is the sum of one digest per relocation file and per route, so each change updates it in O(1) | Diff 7 (stored in compaction outputs) and diff 11 (remote compaction check) | | Coarse SST (tag 20, VersionStorageInfo::OldestCoarseIndirectOriginFileNumber(), the coarse pin in the candidate heap) | An SST without exact metadata that records only a frontier. The MANIFEST, VersionBuilder and the census already accept and honor a frontier; no code in this diff writes one | Diff 7 (coarse flush SSTs) | ### Monitoring - rocksdb.indirect-blob-gc-stats (DB::Properties::kIndirectBlobGcStats) is a map property per column family. DB::GetProperty() also returns it as text, one "key=value" line per key in key order, because GetProperty() needs a string form of every property (db_stress reads every property with it). A call computes it from relaxed atomics of the census and from the current Version, so it takes no census lock. Every *-micros key is a duration, not a timestamp. Keys in this diff: - census-state: ready, building (the census is unready and no build has failed; this includes a census whose first build has not started), failed (the census is unready and its last build failed) or none (the column family has no census); - census-unready-micros: how long the census has been unready, 0 when it is ready. A new census is unready from the moment it is created (ColumnFamilyData::GetOrCreateBlobReferenceCensus() calls NoteUnready()), so a census whose first build never starts, for example while background work is paused, shows its unready time and not 0; - census-consecutive-failures: failed builds in a row; census-builds: builds started; - census-bytes: an estimate of the census memory that stays (the worker state and the change log). It leaves out short-lived copies; rocksdb.block-cache-entry-stats has the exact total of the census cache role; - route-count, relocation-file-count and relocation-file-bytes, from the current Version; - tracked-files (the files that the census tracks for GC, including files below the garbage threshold), garbage-bytes-collectable, garbage-bytes-suppressed, coarse-pinned-files, coarse-pinned-bytes, coarse-pinned-garbage-bytes and relocation-garbage-bytes: the totals of the last census publish. BlobGcCandidateHeap keeps these totals up to date as the heap changes, and a publish copies them. - rocksdb.blob-reference-census-unready-micros (DB::Properties::kBlobReferenceCensusUnreadyMicros, an int property) returns the same value as census-unready-micros. One helper, InternalStats::BlobReferenceCensusUnreadyMicros(), computes both. - VersionEdit::DebugString() and DebugJSON() print the kind of blob reference metadata of each added SST: "exact (N references, B stored value bytes)", "coarse (frontier F)" or "legacy". An SST without such metadata prints nothing more. ldb manifest_dump also prints one line per column family with blob indirection: "--- blob reference metadata --- exact SSTs: E, coarse SSTs: C (lowest frontier F), legacy SSTs: L". - LOG lines: - When a build publishes, the census job logs INFO "Blob reference census is ready at Version N after M ms". - DBImpl::DumpStats() logs a WARN for each column family whose census has been unready for more than 10 minutes (kCensusUnreadyWarnMicros, a fixed choice), with the state and the number of failed builds in a row. The check runs in the stats dump, not in the census job, so it also fires when that job is stuck or paused. It runs once per stats dump (every stats_dump_period_sec seconds), so with stats dumps off there is no such WARN. - rocksdb.blob-stats prints "Number of blob files: N (+M relocation files)" when the column family has M > 0 relocation files. Without relocation files the text is the same as in base RocksDB. The DumpStats line "Blob file count: ..." does not change. - PauseBackgroundWork() also waits for a scheduled census job, because the job can write the MANIFEST, and the job does not start again while background work is paused. ContinueBackgroundWork() schedules it again for the column families in its queue. - Feature off: a column family without blob indirection has no census. The map then shows census-state none and zeros, and the DumpStats check skips the column family. ## Implementation - Install path: the comment at BlobReferenceCensusInstallBatch in db/version_set.h lists the steps: reserve under the DB mutex, prepare without it, install under it after the MANIFEST write, and free the garbage without it. Table builders hand off checked summaries, so a successful install links the prepared handles and does not open the tables again. BuildBlobReferenceCensusDelta() runs after Version::PrepareAppend(), which computes the new Version's coarse frontier, so the delta carries the frontier of the same Version as its SST changes. - MANIFEST: the NewFile record gets three new custom fields, all not safe to ignore: - kBlobReferenceDescriptor (tag 18). - kBlobReferenceLegacyMetadata (tag 19, the legacy marker). It marks an SST in a fenced column family that has no descriptor and no coarse frontier. VersionBuilder sets it in memory, a MANIFEST snapshot writes it, and an edit that copies the file metadata, such as a trivial move, carries it. - kOldestCoarseIndirectOriginFileNumber (tag 20, the coarse frontier). - Build: the build runs in slices from a pinned Version, and each slice returns the LOW thread. At reopen the census reads only each SST's reference index, which holds the totals of every group. It reads no reference data block, and the index decoder checks that the totals add up to the descriptor. The reads use the caller's ReadOptions. During a build the worker collects the changes of each generation number and builds each set once at the end. "Scheduling" at the top of db/blob/blob_reference_census.h lists the states of the census job. - Column family lifetime: the census holds a reference to the ColumnFamilyData of a column family (cfd->Ref()) in two places. ScheduleBlobReferenceCensus() takes one for each column family that it queues; the column family keeps it while it waits in the run queue or the retry list, and the census job releases it when the column family leaves both. Each build takes one more when it starts, for its starting Version, and the deleter of that Version releases it. So a dropped column family stays alive until the census is done with it. The new DBImpl::ReleaseColumnFamilyDataRef releases such a reference. It takes the DB mutex, so its callers (the census job, the cleanup of the run queue and the retry list at close, and the deleter of the starting Version) call it without the mutex. - Table readers for reference reads: the census build and the compaction job-start read get the SST's reader through TableCache::GetBlobReferenceReader(). It first looks for the reader in the table cache without I/O (the pinned reader of the SST, or a cache lookup). That reader already read the table tail, so the census reads only the reference index. On a miss it opens an ephemeral reader that is not inserted into the table cache, so a census build never evicts the reader of another SST. Both kinds read the reference blocks with fill_cache=false, checksum checks and no persistent cache. - Worker: the worker applies each delta in one merge pass per touched generation number. Removals free their slots before additions take the lowest free slots. Each build has a token (an id). A canceled build context no longer matches the token; it is detached and destroyed without the DB mutex and the census log mutex. - Memory: change log and snapshot memory is bounded and charged. The group buffer of a summary is charged before it grows. The selected copy (BlobReferenceCensusState) is charged at its real size. ## Safety boundaries and unsupported cases - The census is derived state that only indirect blob GC uses. It is not persisted, and it is rebuilt after reopen. - A census that is not ready, or that failed, pauses GC and route retirement. It does not affect ordinary reads or writes. - A coarse frontier conservatively keeps every indirect origin at or above it in use. - DeleteFile, DeleteFilesInRange(s) and ingestion with a replace range rebuild the whole census. GC and route retirement pause until the rebuild is ready. - A new exact SST that comes without its prepared summary is a bug, and debug builds assert. In release builds the commit still goes ahead: the census is rebuilt from disk, GC waits for the rebuild, and an ERROR is logged. - These still fail the commit: a duplicate SST addition in one commit, and a new SST in a fenced CF that has no exact metadata (no descriptor, no legacy marker and no coarse frontier). - So a flush in a fenced CF with enable_blob_indirection off fails, because its SST has no exact metadata. Diff 7 adds coarse flush SSTs. Only the test hook can create this state (see "option gate" in Terms). - With the option off, forced blob GC stays paused for a CF while its oldest blob file has no linked SSTs, and so do the files behind it. This happens for a relocated indirect origin that waits for its blob file retirement record, or for a normal blob file newer than an indirect origin that blob file retirement removed. - Public API change: CacheEntryRole::kBlobReferenceCensus is inserted before kMisc. This changes the numeric values of kMisc and kNumCacheEntryRoles. - Public API change: a new ticker, BLOB_DB_CENSUS_RESTART_OVERFLOW_COUNT ("rocksdb.blobdb.census.restart.overflow.count"), is added before TICKER_ENUM_MAX. The new ticker has no Java mapping, as for recent base tickers: all 256 byte values of org.rocksdb.TickerType are used. - Public API: new properties rocksdb.indirect-blob-gc-stats and rocksdb.blob-reference-census-unready-micros. ## Validation and known limitations Validation: unit tests cover the census worker, the slot sets, every SST removal path, the Version changes, retry, cache charging and SetOptions (see the Test Plan). Removal-path tests check after each step that the census equals a fresh build from the current Version. db_bench runs measure build time, memory and install CPU. Known limitations: - The initial build has no bound that is independent of the DB size. - Census memory is one record per generation number that some SST references, plus one entry per (generation number, SST) pair in the holder sets. The number of origins per generation number does not add to it. - Each compaction of a fenced CF, also a full compaction, reads the reference index of its inputs at job start and holds the input summaries in charged memory during the job. The Test Plan shows the cost. - Under very heavy write load during a long build, a build can overflow the restart limit too. Then the census stays not ready and retries after the backoff, and indirect blob GC pauses until a build publishes. The ticker BLOB_DB_CENSUS_RESTART_OVERFLOW_COUNT shows this case. - The apply after an install runs before the flush or compaction counts as finished, so its background slot stays busy for that time. A very large compaction delta can take tens of ms (estimate from the full compaction apply cost in the Test Plan). - Deltas from installs on user threads (for example a user CompactFiles()) wait for the census job or for the next flush or compaction of the same column family. ## Stack context (same in diffs 2-14, except that diffs 10-14 add the diff 10 route rule to the install line; "<== this diff" marks this diff's part): Goal: remove garbage from blob files without rewriting any SST. A value gets a stable identity when it is first written. Indirect blob GC copies the live values of whole files into a relocation file and installs routes in the MANIFEST. Readers follow the routes. SSTs are not changed. How a read finds a value (P is the relocation file that holds the value now): ``` SST entry indirect BlobIndex { generation number, origin, original offset, size, CRCs } | v Version file table: generation number -> live file, or route old file -> P | | live file route found | | v v blob file: read at original offset relocation file P (always one hop): relocation file: look up key look up key (origin, original offset) (origin, original offset) ``` Lifecycle of one value (in this example, F is the blob file where the value is first written, and P is the relocation file that GC writes): ``` write new value goes to blob file F; index: origin = F, generation number = F | census per generation number: the SSTs and references that use it; | per physical file: live bytes, ranked by garbage ratio GC job copy the live values of whole input files (stored bytes, checked | with the stored CRC) into relocation file P, keyed by | (origin, original offset) install one MANIFEST edit: add P, route each input generation number still | in use to P, re-point the routes into the inputs to P, and remove | each input (blob file retirement for a blob file, relocation | file deletion for a relocation file) flatten compaction sets generation number = P in the SSTs it rewrites | route route retirement: route deletion records remove the routes that delete no SST uses ``` Stack map: | Diff | Adds | |---|---| | 1 | Finalize blob metadata outside the DB mutex (prep) | | 2 | Indirect BlobIndex (kIndirectBlob) and exact SST reference metadata | | 3 | Relocation file format and stable read path | | 4 | MANIFEST routes and relocation state in the Version | | 5 | Exact reverse reference census <== this diff | | 6 | MANIFEST apply outside the DB mutex, with a precondition re-check | | 7 | Coarse flush metadata, per-SST reference cap, epoch runs | | 8 | Indirect blob GC building blocks | | 9 | Indirect blob GC job | | 10 | Flatten generation numbers; route retirement | | 11 | Remote compaction validation | | 12 | Dynamic options and operations | | 13 | Lost relocation files | | 14 | RepairDB rebuilds relocation state; option gate removed | | 15 | db_stress and crash test support | ## Stack context: terms for later diffs | Term | Meaning | |---|---| | flatten | Compaction rewrites the generation number to the relocation file that holds the value, so old routes can go. Only flatten makes generation number != origin (diff 10) | | route retirement | The background job that writes route deletion records for the routes that no SST uses (diff 10) | Tracking issue: meta#15301 Differential Revision: D124759451
…#15719) Summary: Pull Request resolved: meta#15719 ## Why the change is required Indirect blob GC (diff 9) installs its result with one LogAndApply() call. Its edit can hold one route change per old file number, so applying it with VersionBuilder::Apply() can take a long time. Base RocksDB applies every edit with the DB mutex held, so a large edit would block every thread that needs the DB mutex. The GC edit is valid only while the GC input is unchanged, so the caller checks a precondition (pre_cb) before the edit is installed. If the edit is applied without the DB mutex, that check must run again after the apply. This diff adds that mechanism to VersionSet. It does not depend on any GC code. Intended user: indirect blob GC. ## Terms used in this diff | Term | Meaning | |---|---| | MANIFEST writer queue | VersionSet::manifest_writers_. Each LogAndApply() call adds one writer per column family. Only the writer at the head of the queue (the queue head) writes to the MANIFEST | | group commit | The queue head (the leader) takes later writers of the queue into its batch and writes their edits in the same MANIFEST write. The later writers then return without reaching the queue head | | precondition (pre_cb) | A callback that a LogAndApply() caller passes. It checks that the caller's edit is still valid. It runs under the DB mutex. If it fails, the call fails and installs nothing | | precondition writer | The first writer of a LogAndApply() call that passes pre_cb. Its ManifestWriter::requires_precondition is set. Only the first writer gets it, because only the first writer of a call waits for the queue head | | batch barrier | A writer that ends grouping: no batch holds both a precondition writer and the writers of another caller | | unlocked apply | VersionBuilder::Apply() of one edit while the DB mutex is released (VersionSet::LogAndApplyHelper()) | ## Previous behavior LogAndApply() ran pre_cb once, under the DB mutex, when the caller's first writer reached the queue head. If an earlier leader took the writer into its group commit, the writer never reached the queue head, so its pre_cb never ran. Every edit was applied with VersionBuilder::Apply() under the DB mutex. In base RocksDB only SetOptions() passes pre_cb, and its dummy edit is never grouped, so base never skips a pre_cb. ## New behavior - A precondition writer is a batch barrier. An earlier leader stops its group before such a writer. So the writer always reaches the queue head, and its pre_cb runs there against every Version committed before it. When the writer leads, the batch holds only the writers of its own caller. Taking a later caller could change the state that the precondition checked. - When the leader is a precondition writer and the batch writes a MANIFEST record, that writer applies each of its edits that change a Version without the DB mutex (VersionSet::LogAndApplyHelper()). These are the edits of the call's first column family. The writer stays at the queue head meanwhile, so no other Version is installed and no other MANIFEST edit is applied. The builder holds a reference to its base Version, and ProcessManifestWrites() already ran the re-add check (RejectReaddedBlobRelocationState()) under the DB mutex. Then the DB mutex is taken again, and pre_cb runs again, because the caller's state can change while the DB mutex is released. Because the batch ends after the writers of the leader's caller, a failed second run fails only that caller: no candidate Version is written or installed, and the writers queued after it run in the next batch. This includes writers that joined the queue while the DB mutex was released. - So a call with one edit that changes a Version runs pre_cb twice: once at the queue head and once after the unlocked apply. pre_cb must be safe to run more than once. The pre_cb paragraph of VersionSet::LogAndApply() in db/version_set.h documents this. - A dummy edit (SetOptions()) writes no MANIFEST record. It is applied under the DB mutex, so its pre_cb runs once, as in base. - The other writers of the batch, and the other column families of the same call, are applied under the DB mutex, as in base. ## What changed in the implementation - db/version_set.cc: ManifestWriter gets requires_precondition, set for the first writer of a call with pre_cb (LogAndApplyImpl()). ProcessManifestWrites() takes pre_cb, walks the queue by index instead of by iterator (other callers can add writers to the queue while the DB mutex is released, and std::deque::push_back invalidates iterators), stops the group before a writer with requires_precondition, ends the batch after the leader caller's writers when the leader has a precondition, and passes the unlocked-apply flag and pre_cb to LogAndApplyHelper(). LogAndApplyHelper() returns InvalidArgument for a null DB mutex, because the unlocked apply must release the DB mutex and take it again. The sync point "VersionSet::LogAndApplyHelper:ApplyUnlocked" marks the unlocked apply. - db/version_set.h: the pre_cb paragraph of LogAndApply() and the new parameters. ## How to read this diff 1. This summary: "Terms used in this diff" and "New behavior". 2. db/version_set.h: the pre_cb paragraph in the comment of VersionSet::LogAndApply(). It is the contract for callers: no group commit with other callers, and pre_cb can run twice. 3. db/version_set.cc: ManifestWriter::requires_precondition (the three uses of the flag), and where LogAndApplyImpl() sets it. The first pre_cb run is also in LogAndApplyImpl(), at the queue head. 4. db/version_set.cc: VersionSet::LogAndApplyHelper(). This is the unlocked apply: unlock, VersionBuilder::Apply(), lock, run pre_cb again. Its comment says why the apply is safe without the DB mutex. 5. db/version_set.cc: VersionSet::ProcessManifestWrites(). Read the comment on the index walk, the call to LogAndApplyHelper(), and the list of stop rules at "Loop increment/conditions". Rules 4 and 5 are new. 6. db/version_set_test.cc: UnlockedApplyRevalidatesPrecondition, PreconditionedWriterIsNotGroupedBehindEarlierWriter, and the two tests of VersionSetPreconditionBarrierTest. ## Safety boundaries and unsupported cases - Outside tests, no caller in this diff applies an edit without the DB mutex: base RocksDB passes pre_cb only from SetOptions(), whose dummy edit is applied under the DB mutex. The first caller is indirect blob GC (diff 9). A test that turns the SetOptions() dummy edit into a normal edit (sync point DBImpl::SetOptions:dummy_edit) makes SetOptions() take the unlocked apply, and its pre_cb then runs twice. - A precondition writer costs one extra pre_cb run for each edit that changes a Version, it never joins an earlier leader's batch, and as leader it takes no writers of later callers. So a caller with a precondition gets no group commit with other callers. In this stack, indirect blob GC installs (diff 9), route retirement (diff 10) and compaction installs whose exact outputs have indirect references (diff 10) pass a precondition. So in a column family with blob indirection, most compaction installs (an estimate) get no group commit with other callers. A benchmark of compaction installs on one host (the B2 workload, on an earlier build of this stack) found no measurable cost (see the diff 10 summary). ## Validation and known limitations Four new unit tests in version_set_test cover the unlocked apply, the batch barrier, and a failed and a passing precondition with a writer queued during the unlocked apply. This diff has no benchmark: no caller uses the unlocked apply yet. ## Stack context (same in diffs 2-14, except that diffs 10-14 add the diff 10 route rule to the install line; "<== this diff" marks this diff's part): Goal: remove garbage from blob files without rewriting any SST. A value gets a stable identity when it is first written. Indirect blob GC copies the live values of whole files into a relocation file and installs routes in the MANIFEST. Readers follow the routes. SSTs are not changed. How a read finds a value (P is the relocation file that holds the value now): ``` SST entry indirect BlobIndex { generation number, origin, original offset, size, CRCs } | v Version file table: generation number -> live file, or route old file -> P | | live file route found | | v v blob file: read at original offset relocation file P (always one hop): relocation file: look up key look up key (origin, original offset) (origin, original offset) ``` Lifecycle of one value (in this example, F is the blob file where the value is first written, and P is the relocation file that GC writes): ``` write new value goes to blob file F; index: origin = F, generation number = F | census per generation number: the SSTs and references that use it; | per physical file: live bytes, ranked by garbage ratio GC job copy the live values of whole input files (stored bytes, checked | with the stored CRC) into relocation file P, keyed by | (origin, original offset) install one MANIFEST edit: add P, route each input generation number still | in use to P, re-point the routes into the inputs to P, and remove | each input (blob file retirement for a blob file, relocation | file deletion for a relocation file) flatten compaction sets generation number = P in the SSTs it rewrites | route route retirement: route deletion records remove the routes that delete no SST uses ``` Stack map: | Diff | Adds | |---|---| | 1 | Finalize blob metadata outside the DB mutex (prep) | | 2 | Indirect BlobIndex (kIndirectBlob) and exact SST reference metadata | | 3 | Relocation file format and stable read path | | 4 | MANIFEST routes and relocation state in the Version | | 5 | Exact reverse reference census | | 6 | MANIFEST apply outside the DB mutex, with a precondition re-check <== this diff | | 7 | Coarse flush metadata, per-SST reference cap, epoch runs | | 8 | Indirect blob GC building blocks | | 9 | Indirect blob GC job | | 10 | Flatten generation numbers; route retirement | | 11 | Remote compaction validation | | 12 | Dynamic options and operations | | 13 | Lost relocation files | | 14 | RepairDB rebuilds relocation state; option gate removed | | 15 | db_stress and crash test support | ## Stack context: terms for later diffs | Term | Meaning | |---|---| | flattened reference | Index with generation number != origin: compaction rewrote the generation number to the relocation file that holds the value (diff 10) | | flatten | Compaction rewrites the generation number to the relocation file that holds the value, so old routes can go. Only flatten makes generation number != origin (diff 10) | | route deletion | MANIFEST record that removes a route that no SST uses. Route retirement is the background job that writes these records (diff 10) | Tracking issue: meta#15301 Differential Revision: D132305444
…och runs together and count each run as one L0 sorted run (meta#15718) Summary: Pull Request resolved: meta#15718 ## Why the change is required Exact reference metadata (diff 2) lists every blob reference of an SST. The census (diff 5) reads it, and indirect blob GC (diffs 8 and 9) will read it. While an SST is built, the table builder keeps these references in memory until the SST is finished. This diff fixes two costs of that, and two problems that the second fix creates (an ordering problem and a counting problem): 1. Flush output does not need exact metadata. A later compaction rewrites it with exact metadata: a compaction into L1 or deeper, a bottommost compaction, or an L0 compaction with an exact input. So flush writes only a small table property, the coarse classification. It holds one file number (the frontier), so the flush needs no reference memory. 2. A compaction output with many references needs much memory. So compaction starts a new output when the output already holds blob_indirection_max_references_per_sst references (the reference cap). This bounds the memory of one output being built. 3. The cap can cut one compaction to L0 into several exact L0 files with one epoch number (an epoch run). A compaction output gets the smallest epoch number of its inputs. So if a later compaction to L0 takes a newer file and only part of a run, its output can overlap a file of the run that it left behind, with the same epoch. L0 orders overlapping files by epoch, so the two files have no order, and the Version consistency check (on by default) rejects them. So every picker takes all files of an epoch run. The comment of VersionStorageInfo::ExpandExactL0Inputs() has a worked example. 4. If each file of an epoch run counted as one L0 sorted run, the L0 count after such a compaction would be at least the number of files that the cap cut its output into (about the reference count divided by the cap). Universal compaction with num_levels = 1 would then compact the same run again and again, the leveled size-based intra-L0 pick would repeat, and writes could slow down or stop for good. So an exact epoch run counts as one sorted run. This is correct because the files of the run share one epoch and their key ranges do not overlap, so each key is in at most one of them, the same as in a single file. Intended users: the census and indirect blob GC read the exact metadata. This diff adds no GC code. ## Parts of this diff 1. Core: coarse flush output (db/blob/blob_reference_metadata.{h,cc}, table/table_builder.h, table/block_based/block_based_table_builder.cc, db/builder.cc, db/flush_job.cc, db/db_impl/db_impl_open.cc, db/compaction/compaction.{h,cc}, db/compaction/compaction_job.cc). The caller of the table builder decides if an SST gets exact metadata, only the coarse classification, or neither. 2. Coarse L0 files and L0 moves (Compaction::MustRewriteCoarseL0File() in db/compaction/compaction.cc, db/compaction/compaction_picker_level.cc, db/db_impl/db_impl_experimental.cc). A coarse L0 file with indirect references is rewritten when it leaves L0, never moved. 3. Reference cap (include/rocksdb/advanced_options.h, db/column_family.cc, db/compaction/compaction_outputs.{h,cc}, db/event_helpers.{h,cc}, include/rocksdb/statistics.h, the options and C API files). Safe to skip on a first read. 4. Epoch runs (VersionStorageInfo::ExpandExactL0Inputs() and the L0 sorted run count in db/version_set.{h,cc}, db/compaction/compaction_picker.cc, db/compaction/compaction_picker_level.cc, db/compaction/compaction_picker_universal.cc). Safe to skip on a first read. 5. Small changes (db/repair.cc, db/compaction/compaction_service_job.cc, table/sst_file_writer.cc). Safe to skip on a first read. ## How to read this diff 1. Read "Terms used in this diff" below. Then read the top overview of db/blob/blob_reference_metadata.h (the list of which SSTs get exact metadata, and the paragraph "Reference cap and epoch runs"), and its sections "Coarse SST" and "Coarse classification property". 2. The builder: in table/table_builder.h, read the TableBuilderOptions fields collect_blob_reference_metadata and track_blob_reference_classification, and TableBuilder::BlobReferenceLimitReached(). Then read table/block_based/block_based_table_builder.cc: BlockBasedTableBuilder::AddImpl() (called by Add() and TryAdd()) computes the frontier, WriteBlobReferenceMetadata() writes the property, and BlobReferenceLimitReached() checks the cap. EncodeBlobReferenceClassification() and DecodeBlobReferenceClassification() are in db/blob/blob_reference_metadata.cc. 3. The callers that set the two fields: FlushJob::WriteLevel0Table() in db/flush_job.cc, DBImpl::WriteLevel0TableForRecovery() in db/db_impl/db_impl_open.cc, and CompactionJob::OpenCompactionOutputFile() in db/compaction/compaction_job.cc with Compaction::CollectExactBlobReferences() in db/compaction/compaction.cc. Then read how the property gets into the file metadata: ExtractBlobReferenceClassificationFromTableProperties() in db/builder.cc, its callers in db/compaction/compaction_outputs.cc, and the blob_reference_legacy_metadata comment in db/version_edit.h. 4. L0 moves: Compaction::MustRewriteCoarseL0File() and IsTrivialMove() in db/compaction/compaction.cc, LevelCompactionBuilder::TryPickL0TrivialMove() in db/compaction/compaction_picker_level.cc, and DBImpl::PromoteL0() in db/db_impl/db_impl_experimental.cc. 5. The reference cap: the option comment in include/rocksdb/advanced_options.h, the range check in ColumnFamilyData::ValidateOptions() (db/column_family.cc), and CompactionOutputs::AddToOutput() and NoteBlobReferenceLimitCut() in db/compaction/compaction_outputs.cc. Then the statistics and the event log fields in CompactionJob::FinishCompactionOutputFile() and db/event_helpers.cc. 6. Epoch runs: VersionStorageInfo::ExpandExactL0Inputs() in db/version_set.h (its comment has a worked example of why a pick takes the whole run) and db/version_set.cc. Then its callers: CompactionPicker::ExpandL0ToWholeRuns() (used by ExpandInputsToCleanCut() and GetOverlappingL0Files()) and GetCompactionInputsFromFileNumbers() in db/compaction/compaction_picker.cc, SetupOtherL0FilesIfNeeded() and TryPickL0TrivialMove() in db/compaction/compaction_picker_level.cc, and PickDeleteTriggeredCompaction() in db/compaction/compaction_picker_universal.cc. Then the sorted run count: L0SortedRunEnd(), CountL0SortedRuns() and NumL0SortedRuns() in db/version_set.h, their callers CalculateBaseBytes(), ComputeCompactionScore() and EstimateCompactionBytesNeeded() in db/version_set.cc, PickIntraL0Compaction() and PickSizeBasedIntraL0Compaction() in db/compaction/compaction_picker_level.cc, and UniversalCompactionBuilder::CalculateSortedRuns(), AppendL0RunFiles() and CheckExactL0RunRange() in db/compaction/compaction_picker_universal.cc. 7. Last, skim the small changes: the classification check in db/repair.cc, the UpdateTableProperties() status in db/compaction/compaction_service_job.cc, and table/sst_file_writer.cc. ## Terms used in this diff Stack terms: | Term | Meaning | |---|---| | origin | Blob file where the value was first written | | generation number | The file number that a reference uses to find its value. A new value has generation number == origin. When indirect blob GC moves the value, the reference keeps its generation number, and a route sends the read to the relocation file | | route | Old file number -> the relocation file that now holds its live values. Stored in the MANIFEST | | indirect origin | Normal blob file written with enable_blob_indirection on; it is the origin of references with generation number == origin | | legacy | Written the way base RocksDB writes it, without blob indirection. A legacy reference is a kBlob or kBlobTTL index | | exact reference metadata | Per-SST sorted list of every blob reference the SST keeps, written when the SST is built. An SST that has it is an exact SST | | coarse SST | An SST without exact reference metadata that may hold indirect references. It records one frontier file number | | frontier | The oldest indirect origin that a coarse SST references. Frontier 0 means no indirect reference | | pin | A coarse SST pins every indirect-origin blob file at or above its frontier: the census and GC must treat that file as used. This is not the base RocksDB meaning (a Version or block that stays alive) | | census | Background count, per generation number, of the SSTs and references that use it, plus live bytes per physical file. Its GC candidate heap ranks the files by garbage ratio | | format fence | A MANIFEST record that marks a column family (CF) as possibly holding blob indirection data. It never clears. VersionStorageInfo::HasBlobIndirectionFormat() reports it | Terms of this diff: | Term | Meaning | |---|---| | coarse classification | The table property kBlobReferenceClassificationPropertyName that a coarse SST stores: format version 1, the frontier and a CRC32C (16 bytes) | | reference cap | The column family option blob_indirection_max_references_per_sst. A compaction output that already holds this many exact references takes no new user key. The code calls it the blob reference limit (max_blob_references, BlobReferenceLimitReached(), kBlobReferenceLimit, closed_by_blob_reference_limit) | | user key history | All versions of one user key that a compaction keeps, including deletions and inline values | | epoch number | The L0 order number of a file (base RocksDB). Newer L0 files have larger numbers | | epoch run | The exact L0 files that one compaction to L0 wrote. They share one epoch number | | exact epoch run | Adjacent L0 files with one known epoch number, at least one of them exact. This is how the code finds an epoch run | | sorted run | Base RocksDB term: files whose key ranges do not overlap, so a read checks them as one. Each L0 file is one sorted run, except that an exact epoch run is one sorted run (see "Epoch runs") | ## Previous behavior When enable_blob_indirection was on, every flush and compaction output got exact reference metadata, except SstFileWriter output and outputs whose column family was unknown. No SST stored a coarse classification, although the file metadata and the census could already hold a coarse frontier (diff 5). A compaction output had no limit on its number of references. Each compaction to L0 wrote one file, so no epoch number had more than one file. ## New behavior ### Which SSTs get exact metadata When enable_blob_indirection is on, or the CF has the format fence (the option was on before): - A compaction output gets exact metadata when its level is L1 or deeper, when the compaction is bottommost (at any level, including L0), or when an L0 input already has exact metadata (Compaction::CollectExactBlobReferences()). Bottommost is Compaction::bottommost_level(): no older L0 file and no file in a deeper level overlaps the key range of the compaction. - Other compaction outputs at L0 get only the coarse classification. A flush and an L0 compaction that is not bottommost stay coarse to save collector memory. An output never loses exact metadata that an input has, because a coarse output would pin files again that the census already tracks exactly. - Flush output and WAL recovery flush output get only the coarse classification. - SstFileWriter output gets neither and stays legacy. It can hold only same-file (embedded) blob references (SstFileWriter::OpenWithEmbeddedBlobs), never a reference to a blob file, so it has nothing to list. - The table builder does not check the column family. It follows TableBuilderOptions::collect_blob_reference_metadata, whose default is the value of enable_blob_indirection. SstFileDumper (sst_dump) builds tables only to measure their sizes and then throws them away, so it sets the field to false. When the option is off and the CF has no fence, an SST gets neither, as in base RocksDB. ### Coarse classification - The table builder keeps no reference list for a coarse SST. It keeps only the oldest indirect origin that it sees. A coarse SST must not hold a reference with generation number != origin: the builder returns Corruption ("Coarse SST cannot contain a flattened blob reference (generation number != origin)") for one. A flushed SST cannot hold one, because a flush writes only references to its own new blob files. - A classification with frontier 0 means the SST has no indirect reference (only legacy references, or none). Such an SST is stored as a legacy SST (blob_reference_legacy_metadata), so recovery accepts it without exact metadata, and it pins nothing. - An SST with both an exact-metadata descriptor and a classification is Corruption. - The Version keeps the lowest frontier of its SSTs (diff 5). The census keeps each indirect-origin blob file at or above it out of its GC candidate heap. So a coarse SST stops GC of those files until a compaction rewrites it with exact metadata. ### Coarse L0 files and L0 moves - A coarse L0 file with a frontier other than 0 is never trivially moved out of L0 (Compaction::MustRewriteCoarseL0File()). A move would keep its pin until some later compaction rewrites it, which can take a long time. The level picker's L0 trivial move stops before such a file; the files picked before it still move. experimental::PromoteL0 returns NotSupported for such a file. A coarse file with frontier 0 can move. - For sequential inserts, this adds extra writes of about (SST bytes per entry / value size). A db_bench run with 4 KB values and 16-byte keys measured +0.97% extra writes. This matches about 40 bytes per SST entry (40 / 4096 is about 1%). From that figure, the extra writes are about 16% at 256-byte values and about 60% at 64-byte values (estimates). Random-key workloads already rewrite at L0 to L1. - With num_levels = 1 there is no L1. Flushed data gets exact metadata when a bottommost compaction rewrites it, for example a universal compaction that includes the oldest sorted run. Until then, indirect blob GC cannot collect indirect-origin blob files at or above the coarse frontier. ### Reference cap - The table builder keeps the references of an SST in memory until the SST is finished. They use about 10 bytes per reference for uncompressed values and about 14 bytes for compressed values (estimates). A db_bench run measured 10.5 bytes per reference. So the default cap of 1M references (1048576) bounds the memory of one output being built to about 10 to 14 MB. When every value is in a blob file and user keys are about 20 bytes, an SST uses about 60 bytes per reference (an estimate), so the default cuts outputs at about 60 MB. - It is a soft limit. Compaction starts a new output only before a new user key, so a user key history is never split. An output may pass the cap by one key's history. The check costs one compare per row. - The option is mutable through SetOptions(). Its range is [4096, 2^28]. The lower bound is a fixed choice, not measured. The upper bound keeps one SST's metadata within the format limit of 2^20 data blocks of about 16 KB. A value outside the range fails option validation with InvalidArgument. - Flush output has no exact metadata, so the cap applies to compaction outputs only. - Monitoring: the ticker COMPACTION_OUTPUT_BLOB_REFERENCE_LIMIT_CUTS counts outputs that the cap closed. The histogram COMPACTION_OUTPUT_BLOB_REFERENCES records the reference count of each exact compaction output. The INFO log names the cut reason BlobReferenceLimit. The table_file_creation event of an exact compaction output has "blob_references", and "close_reason": "blob_reference_limit" when the cap closed it. ### Epoch runs - Every picker takes all files of an epoch run that has an exact file (VersionStorageInfo::ExpandExactL0Inputs()). This covers the level picker, the universal picker, CompactRange and CompactFiles. - In a column family with the format fence, an exact epoch run counts as one L0 sorted run (VersionStorageInfo::NumL0SortedRuns(), or L0SortedRunEnd() for a walk over the files). The count is used by: - the L0 compaction score, for leveled and universal compaction; - l0_delay_trigger_count(), which drives the write slowdown, the write stop and the compaction speedup; - FlushJobInfo::triggered_writes_slowdown and triggered_writes_stop (DBImpl::NotifyOnFlushBegin() and NotifyOnFlushCompleted()). They compare the same count with level0_slowdown_writes_trigger and level0_stop_writes_trigger, so an exact epoch run counts once in these flags too, as in the L0 part of the write stall; - both leveled intra-L0 triggers (PickIntraL0Compaction() and PickSizeBasedIntraL0Compaction()). The cost-based pick of PickIntraL0Compaction(), used when L0 to Lbase is blocked, also needs at least kMinFilesForIntraL0Compaction (4) sorted runs among the whole runs it would take, so one exact epoch run is never picked alone and rewritten into itself; - the L0 trigger of EstimateCompactionBytesNeeded(), which gives rocksdb.estimate-pending-compaction-bytes and drives the soft and hard pending compaction bytes stalls; - universal UniversalCompactionBuilder::CalculateSortedRuns(): one SortedRun per run, whose size is the sum of its files. Every universal pick of that sorted run takes all its files. - The size-ratio, size-amplification and periodic universal picks check their L0 sorted runs with CheckExactL0RunRange(). If the pick would hold only part of an exact epoch run, if a file of an exact run is being compacted, or if a picked file has epoch number 0 and an L0 file with that epoch is exact, the pick stops and logs a WARN line, also in release builds. - rocksdb.num-files-at-level0 and NumLevelFiles(0) stay file counts. Without the format fence, every L0 sorted run count equals the L0 file count, as in base RocksDB. - With universal compaction and num_levels = 1, a delete-triggered compaction takes the first sorted run that has a marked file, and the older sorted runs, until a sorted run that is being compacted or has a marked standalone range deletion. Each sorted run is a whole epoch run, so the pick never starts or stops inside one. As in base RocksDB, it does not start at the last sorted run, which can be a whole exact epoch run. The pick fails when a picked file has an unknown epoch number (0) and an L0 file with that epoch is exact. - A file of the run can widen the key range of the pick. So the L0 overlap search runs again on the wider range, until the set stops changing. The result is in the Version's epoch order (newest first), also when the inputs were found out of order. - The L0 trivial move drops a run that it could take only in part. - If a needed file of the run is already being compacted, the pick fails. CompactFiles then returns Aborted. A DB that never used enable_blob_indirection is not affected. - Point lookups in a run (TableCache::RangeTombstonesMayCoverUserKey()). The cap can cut an output inside a range tombstone. Then both files of the run keep the whole tombstone, and the left file's largest key is the exclusive end <cut key, kMaxSequenceNumber, kTypeRangeDeletion>. A point lookup reads a file's range tombstones untruncated. L0 orders the files of one epoch by largest sequence number, and Version::Get() can ask a file about a key that the file does not hold. FilePicker keeps the left file for a lookup at the cut key, because its key range filter compares user keys. With at most 3 files, all in L0, FilePicker does not filter by key range at all. So without a check, the tombstone of the file that does not hold the key could hide a live, newer value in the other file, and Get() would return NotFound. MultiGet() does not depend on the check: Version::MultiGet() does not end a key's search at a covering tombstone (it skips the key only for the rest of the current file), so it finds the newer value in the other file. The check still runs there, so the covering tombstone sequence number is correct for both Get() and MultiGet(). The fix: for an L0 file, a range tombstone counts for a lookup key only when the key is inside the file's key range, and not at an exclusive tombstone end. Get() and MultiGet() run the check only for an L0 file that has range tombstones, and it costs up to two user key compares per such file. L1 and deeper need no check, because FindFile() compares internal keys. Base L0 files are never cut and their key range includes their tombstones, so the check changes no result for them, also with the option off. ### Monitoring - COMPACTION_COARSE_L0_TRIVIAL_MOVE_BLOCKED (ticker) counts leveled L0 trivial moves that stopped before a coarse L0 file. The files picked before it still move. The coarse file waits for a normal compaction, which rewrites it with exact metadata. - New keys of rocksdb.indirect-blob-gc-stats: coarse-sst-count (the number of SSTs with a nonzero coarse frontier) and lowest-coarse-frontier (the lowest frontier of the Version, 0 when there is no coarse SST). - sst_dump also prints the coarse classification of an SST ("coarse classification: frontier F", or "no indirect references (legacy SST)" when the frontier is 0) in its "Blob reference metadata" section. - Feature off: only a column family with blob indirection has coarse files, so the ticker does not change and both keys are 0. ## What changed in the implementation - TableBuilderOptions gets collect_blob_reference_metadata (default: the value of enable_blob_indirection) and track_blob_reference_classification. Flush, WAL recovery flush and compaction set both. They set track_blob_reference_classification when the column family uses blob indirection now or has used it: UsesBlobIndirection() in db/version_set.h checks the option and the format fence. SstFileWriter and SstFileDumper set collect_blob_reference_metadata to false. The block-based table builder follows these fields instead of deciding by itself. - A compaction output stores the relocation-state fingerprint (a 128-bit hash of the relocation files and routes) of the compaction's input Version in its exact-metadata descriptor. Other builders store 0. The fingerprint is not checked when the descriptor is decoded, and no code at this diff reads it. Diff 11 (remote compaction validation) compares it with the input Version of the job. - CompactionOutputs::UpdateTableProperties() returns a Status, because reading the classification back can fail. If it fails during a remote compaction install, the install returns kFailure and removes the files it installed. If it fails while a resumed compaction restores its earlier outputs, the resume fails. - RepairDB returns NotSupported, with inputs untouched, when an SST has a valid classification, also with frontier 0. Such an SST shows that the DB used enable_blob_indirection, and RepairDB cannot rebuild that state at this diff. - The L0 sorted run count: CalculateBaseBytes() (run for each new Version) counts the L0 sorted runs in one pass over the L0 files, and stores the result for NumL0SortedRuns() and l0_delay_trigger_count(). EstimateCompactionBytesNeeded() counts again with CountL0SortedRuns(). ComputeCompactionScore() and CalculateSortedRuns() walk the L0 files run by run with L0SortedRunEnd(). Without the format fence, these functions take the base path: each L0 file is one sorted run. - FlushJob has a test hook for its TableBuilderOptions (sync point FlushJob::WriteLevel0Table:TableBuilderOptions), so tests can make a flush write exact metadata. ## Safety boundaries and unsupported cases - Changes visible to existing users from the sorted run counting: none. It applies only to column families with the format fence; without it, each L0 file stays one sorted run. - Public API: - New column family option blob_indirection_max_references_per_sst (experimental, mutable, valid range [4096, 268435456], default 1048576), with its C API getter and setter. - experimental::PromoteL0 returns NotSupported for a coarse L0 file with a frontier other than 0. - New tickers COMPACTION_OUTPUT_BLOB_REFERENCE_LIMIT_CUTS and COMPACTION_COARSE_L0_TRIVIAL_MOVE_BLOCKED, and histogram COMPACTION_OUTPUT_BLOB_REFERENCES. They are appended just before TICKER_ENUM_MAX and HISTOGRAM_ENUM_MAX, so those values change. The Java API maps the histogram (HistogramType and portal.h). The new tickers have no Java mapping, as for recent base tickers: all 256 byte values of org.rocksdb.TickerType are used. - Internal API: TableBuilder::BlobReferenceLimitReached() and the two TableBuilderOptions fields. - New on-disk table property kBlobReferenceClassificationPropertyName (16 bytes: format version 1, frontier, CRC32C). Old readers ignore it. A wrong size, version or CRC, or a frontier above the file number mask, is Corruption. - Under universal compaction, a coarse flushed run pins every indirect-origin blob file at or above its frontier until a compaction rewrites it with exact metadata (into L1 or deeper, in a bottommost compaction, or in an L0 compaction with an exact input). - Turning the option on while an older flush (picked with the option off) is still running can fail that flush's install with Corruption ("An SST added after the format fence has no exact blob reference metadata"). It becomes a background error, with no data loss. Diff 12 fixes it. In diffs 7 to 11 it is reachable only through the test hook (see "Option gate"). - Base RocksDB import (CreateColumnFamilyWithImport) rejects an SST whose metadata names a blob file (oldest_blob_file_number). This diff adds no check of the SST itself: ingestion of an SST with indirect references, and import of one whose metadata names no blob file, are not rejected in this diff. Diff 12 rejects both. ## Validation and known limitations Unit tests cover which SSTs get exact or coarse metadata, the coarse L0 move rules, the reference cap (including user key histories, timestamps, wide-column entities and per-key placement), the option range, the epoch run rules of every picker, and the sorted run count of an epoch run (compaction triggers and write stalls). A db_bench run measured the memory and output sizes of the reference cap (see the Test Plan). The extra writes of coarse L0 files were measured only for 4 KB values; the numbers for smaller values are estimates. ## Option gate At this diff, ColumnFamilyData::ValidateOptions rejects enable_blob_indirection outside tests (the option gate: diff 2 adds it, diff 14 removes it). The behavior described here with the option on is reachable only through the internal test hook ColumnFamilyData::TEST_SetAllowBlobIndirection, not in production. ## Stack context (same in diffs 2-14, except that diffs 10-14 add the diff 10 route rule to the install line; "<== this diff" marks this diff's part): Goal: remove garbage from blob files without rewriting any SST. A value gets a stable identity when it is first written. Indirect blob GC copies the live values of whole files into a relocation file and installs routes in the MANIFEST. Readers follow the routes. SSTs are not changed. How a read finds a value (P is the relocation file that holds the value now): ``` SST entry indirect BlobIndex { generation number, origin, original offset, size, CRCs } | v Version file table: generation number -> live file, or route old file -> P | | live file route found | | v v blob file: read at original offset relocation file P (always one hop): relocation file: look up key look up key (origin, original offset) (origin, original offset) ``` Lifecycle of one value (in this example, F is the blob file where the value is first written, and P is the relocation file that GC writes): ``` write new value goes to blob file F; index: origin = F, generation number = F | census per generation number: the SSTs and references that use it; | per physical file: live bytes, ranked by garbage ratio GC job copy the live values of whole input files (stored bytes, checked | with the stored CRC) into relocation file P, keyed by | (origin, original offset) install one MANIFEST edit: add P, route each input generation number still | in use to P, re-point the routes into the inputs to P, and remove | each input (blob file retirement for a blob file, relocation | file deletion for a relocation file) flatten compaction sets generation number = P in the SSTs it rewrites | route route retirement: route deletion records remove the routes that delete no SST uses ``` Stack map: | Diff | Adds | |---|---| | 1 | Finalize blob metadata outside the DB mutex (prep) | | 2 | Indirect BlobIndex (kIndirectBlob) and exact SST reference metadata | | 3 | Relocation file format and stable read path | | 4 | MANIFEST routes and relocation state in the Version | | 5 | Exact reverse reference census | | 6 | MANIFEST apply outside the DB mutex, with a precondition re-check | | 7 | Coarse flush metadata, per-SST reference cap, epoch runs <== this diff | | 8 | Indirect blob GC building blocks | | 9 | Indirect blob GC job | | 10 | Flatten generation numbers; route retirement | | 11 | Remote compaction validation | | 12 | Dynamic options and operations | | 13 | Lost relocation files | | 14 | RepairDB rebuilds relocation state; option gate removed | | 15 | db_stress and crash test support | ## Stack context: terms for later diffs | Term | Meaning | |---|---| | flattened reference | Index with generation number != origin: compaction rewrote the generation number to the relocation file that holds the value (diff 10) | | flatten | Compaction rewrites the generation number to the relocation file that holds the value, so old routes can go. Only flatten makes generation number != origin (diff 10) | | route deletion | MANIFEST record that removes a route that no SST uses. Route retirement is the background job that writes these records (diff 10) | Tracking issue: meta#15301 Differential Revision: D132305445
Summary: Pull Request resolved: meta#15544 ## Why the change is required Indirect blob GC (the next diff) copies the live values of mostly-garbage blob files into new relocation files, and rewrites no SST. Before it copies anything, it must know the exact live references of each input file, keep its memory bounded, decide which inputs share an output, and copy stored bytes without decompressing them. This diff adds those parts as building blocks that work without the DB, with unit tests only. Intended user: indirect blob GC (diff 9). Splitting them out lets a reviewer check each block on its own before the job that combines them. ## Parts of this diff 1. Live set, memory budget and packing plan (db/blob/blob_relocation_gc.{h,cc}): BlobRelocationLiveSet, BlobRelocationMemoryTracker and PlanBlobRelocationPacking(). The overview at the top of db/blob/blob_relocation_gc.h defines the terms of this diff. 2. Batch reference reader (table/table_reader.h, table/block_based/block_based_table_reader.{h,cc}, db/table_cache.{h,cc}): one call reads the references of many groups from one SST, and appends them to a vector or to a sink such as a live set. 3. Stored-byte reads and writes (db/blob/blob_gen2_format.{h,cc}, db/blob/blob_file_reader.{h,cc}, db/blob/blob_relocation_file.{h,cc}, db/blob/same_file_blob_reader.h, table/block_based/block_based_table_reader.cc): read the stored (maybe compressed) bytes of a value, check them with the stored CRC, merge reads of neighbor values, stream large values in chunks, and write a relocation record from a checked source. 4. Relocation file writing (table/block_based/block_based_table_builder.{h,cc}, db/blob/blob_relocation_file.cc): location blocks use the column family's compression, and relocation files use kCRC32c checksums. Safe to skip on a first read. 5. Cache role and metadata caches (include/rocksdb/cache.h, cache/, table/block_based/block_based_table_factory.cc, table/block_based/block_based_table_reader.h): the new CacheEntryRole::kIndirectBlobGcWorkingMemory, and metadata reads of an ephemeral reader that skip the persistent cache. Safe to skip on a first read. ## How to read this diff 1. Overview: read "Terms used in this diff" in this summary, then the overview at the top of db/blob/blob_relocation_gc.h. It lists the five parts and the order in which a GC job (diff 9) calls them. 2. Batch reference reader: in table/table_reader.h read BlobReferenceSink, BlobReferenceGroupRead, kBlobReferenceReadBatchBytes and the ReadBlobReferences() comment. Then read ReadBlobReferences(), ReadIndexedBlobReferenceGroups() and ReadBlobReferenceDataBlocks() in table/block_based/block_based_table_reader.cc, and TableCache::ReadBlobReferences() in db/table_cache.{h,cc}. The table reader part depends only on the exact reference metadata of diff 2. TableCache::ReadBlobReferences() also uses TableCache::GetBlobReferenceReader() from diff 5. 3. Memory budget and live set: read BlobRelocationMemoryTracker, BlobRelocationLiveValue, BlobRelocationOriginRun and BlobRelocationLiveSet in db/blob/blob_relocation_gc.{h,cc}, and CacheEntryRole::kIndirectBlobGcWorkingMemory in include/rocksdb/cache.h. A live set is a BlobReferenceSink from step 2. 4. Packing plan: read BlobRelocationPackingItem, BlobRelocationPackingBin and PlanBlobRelocationPacking() in db/blob/blob_relocation_gc.{h,cc}. It does not depend on step 3. 5. Coalesced reads: read CoalescedReadBuffers and ReadCoalescedRanges() in db/blob/blob_gen2_format.{h,cc}. 6. Stored reads: for normal blob files, read MultiGetStoredBlobsForRelocation() and StreamStoredBlobForRelocation() in db/blob/blob_file_reader.{h,cc}, and StreamAndVerifyBlobStoredPayload() in db/blob/blob_gen2_format.h. For relocation files, read BlobRelocationFileReader::MultiGetStoredBlobs() and StreamStoredBlob() in the same files, the read table at the top of db/blob/blob_relocation_file.h, and BlockBasedTable::MultiGetStoredSameFileBlob() and StreamStoredSameFileBlob() in table/block_based/block_based_table_reader.{h,cc}. 7. Streaming writes: read WriteSimpleGen2BlobRecordStream() in db/blob/blob_gen2_format.h and the streaming AddBlobRelocation() in table/block_based/block_based_table_builder.{h,cc}. 8. Rules for existing code paths (safe to skim): the NewBlobRelocationTableBuilder() comment in table/block_based/block_based_table_builder.h, the kCRC32c rule in db/blob/blob_relocation_file.cc, the verify policy in BlobRelocationFileReader::MultiGetBlob() (db/blob/blob_file_reader.cc), and PersistentCacheFor() in table/block_based/block_based_table_reader.h. ## Terms used in this diff Stack terms: | Term | Meaning | |---|---| | origin | Blob file where the value was first written | | original offset | Value offset in the origin file | | stable identity | (origin, original offset). Never changes, even when the value moves | | generation number | The file number that a reference uses to find its value. A new value has generation number == origin. When indirect blob GC moves the value, the reference keeps its generation number, and a route sends the read to the relocation file | | Version file table | The table in a Version that maps a generation number to a live file or to a route | | unflattened reference | Index with generation number == origin: the value was not moved, or it was moved but no compaction has rewritten its reference yet (see "flatten" in "Stack context: terms for later diffs") | | route | Old file number -> the relocation file that now holds its live values. One hop: a target is never itself routed. Stored in the MANIFEST | | physical file | A file that holds value bytes: a normal blob file or a relocation file. A generation number can name a physical file, or it can be only the key of a route | | indirect origin | Normal blob file written with the option on ("the option" is enable_blob_indirection); it is the origin of unflattened references | | relocation file | Block-based table keyed by (origin, original offset) that holds copied values. Each value is the stored bytes plus a 5-byte trailer (kSimpleGen2BlobTrailerSize) | | logical CRC | CRC32C of the uncompressed value (BlobReference::stable_checksum) | | exact reference metadata | Per-SST sorted list of every blob reference the SST keeps, written when the SST is built. References with the same encoding class, generation number and origin form a group. The list is stored as data blocks, with an index of the groups. An SST that has it is an exact SST | | census | Background count, per generation number, of the SSTs and references that use it, plus live bytes per physical file | Terms of this diff: the overview at the top of db/blob/blob_relocation_gc.h defines input, generation numbers of an input, output, live set, record bytes, stored CRC, sink, location block and coalesced read. The comment of BlobRelocationOriginRun in the same file defines origin run. ## Previous behavior Up to diff 7, a table reader could read only the reference index of an SST (TableReader::ReadBlobReferenceGroups(), which the census uses), not the references themselves. No code read the stored (maybe compressed) bytes of a value out of a file, and nothing planned outputs. Location blocks of relocation files were written with kNoCompression, and relocation files used the checksum type of the table options. A whole-value read of a relocation file always checked the record checksum, also with verify_checksums off. A range read of a compressed value was, and is, rejected with Corruption. A reader opened with avoid_shared_metadata_cache (a compaction's ephemeral reader, or the reader that TableCache::GetBlobReferenceReader() opens on a table cache miss) used the shared persistent cache for its index, filter and other metadata blocks; only its blob reference metadata blocks skipped it. ## New behavior ### Batch reference reader - TableReader::ReadBlobReferences() reads the references of several groups from one SST in one call. It decodes the reference index once and reads each needed data block once. Adjacent blocks are read together, in MultiRead batches of at most kBlobReferenceReadBatchBytes (256 KB, an estimate that holds many blocks per read but keeps the buffer small). - A request names one group, or (whole_generation) every group of one generation number. Requests are sorted and unique, and a batch does not mix whole-generation and single-group requests; otherwise the call returns InvalidArgument. - A request appends to a vector or to a sink. Before the read, each group is checked against the room left in its output, minus the references of earlier requests with the same output. A group that does not fit gets Aborted and appends nothing. - Per-request errors: NotFound (no such group), Aborted (no room) and Corruption (a corrupt block, or totals that do not match the index). The other requests continue. A failed request drops only the references it appended, so references that earlier requests or earlier calls appended stay. - after_index_read reports the real memory of the decoded reference index, before any data block is read, so a caller can charge it. The size grows with the number of groups, so a fixed estimate can be too small. - TableCache::ReadBlobReferences() gets the SST's reader with TableCache::GetBlobReferenceReader() (diff 5) and reads one batch. Only tests call it. ### Live set, memory budget and packing plan - BlobRelocationLiveSet holds the exact live references of one input, read for its generation numbers. Each appended reference is checked: its generation number must be one of the input's, and its encoding class must match it (unflattened if and only if generation number == origin). It is kept as a 40-byte BlobRelocationLiveValue. - Finalize() sorts by stable identity and merges copies of one identity, which can come from several SSTs or several generation numbers. Copies must agree on size, compression and both CRCs; otherwise it returns Corruption. It also builds one origin run per origin. - Reserve() charges the exact capacity first, so the owner pays for the memory before the batch reader fills it. - BlobRelocationMemoryTracker bounds the memory of many users together to 64 MB (kMaxBytes, an estimate) and charges it to the block cache. Several live sets can share one tracker. A request over the budget returns MemoryLimit. - PlanBlobRelocationPacking() puts whole inputs into outputs, first fit decreasing by estimated size, and is deterministic. An input is never split. An input larger than the target gets an output of its own. Two inputs that hold the same origin (possible after RepairDB rebuilds relocation state, diff 14) never share an output, because an output holds one entry per stable identity. More outputs than the limit returns MemoryLimit. ### Stored-byte reads - The reads return the exact stored bytes of a value and its compression type, check them with the stored CRC, and never decompress. A request with a null payload is only checked. - ReadCoalescedRanges() merges neighbor ranges into one read when the gap between them is at most coalesce_bytes, and keeps each read under a size limit. Ranges past the buffer limit get Incomplete, and the caller reads them in a later call. The first read is always issued. - Normal blob files: BlobFileReader::MultiGetStoredBlobsForRelocation() takes requests sorted by offset, and StreamStoredBlobForRelocation() streams one value in 256 KB chunks (kBlobRelocationStreamChunkSize, a choice, not a measurement: a value larger than one 1 MB read batch takes a few chunk reads instead of many), so memory does not depend on the value size. These reads count BLOB_DB_BLOB_FILE_BYTES_READ and blob_read_byte, merged gaps included. - Relocation files: BlobRelocationFileReader::MultiGetStoredBlobs() looks values up by stable identity and also checks the record checksum in the trailer, unless the file uses kNoChecksum. A request that would bring the call's total stored bytes past max_stored_value_size gets MemoryLimit. StreamStoredBlob() streams one value. These reads update no blob read ticker or perf counter. - WriteSimpleGen2BlobRecordStream() writes a record from a checked source without holding the whole payload. The source already computed the CRC32C of the stored bytes, so the writer only adds the file's checksum context. A source that fails, or delivers the wrong number of bytes, fails the write, and the partial file must not be published. ### Relocation file reads - A whole-value read of a relocation file (BlobRelocationFileReader::MultiGetBlob()) checks the record checksum only when verify_checksums is set, the same rule as for normal blob files (BlobVerifyPolicy::kVerifyIfNoAmplification). A strict range of an uncompressed value is never checked, because the check would need a read of the whole record. - The lookup iterator of a relocation file (BlobRelocationFileReader::NewLookupIterator()) uses TableReaderCaller::kBlobRelocationFileRead (diff 3). So block cache traces mark these location lookups like the other relocation file reads, and no wrapper iterator changes the location entries before FindBlobIndex() checks them. ### Relocation file writing - Location blocks use the column family's bottommost_compression if it is set, else its compression. A relocation file holds long-lived data, like bottommost data (a design choice). Values are written outside the blocks, already in stored form, and are never recompressed. Relocation files use no dictionary and no parallel compression, because both hold data blocks back, and the records must be written between the blocks in order. - Relocation files use kCRC32c for block and record checksums, unless the table options set kNoChecksum. A record trailer can then reuse the stored CRC32C of the value, so streaming a value needs no second checksum pass. ### Cache role and metadata caches - New CacheEntryRole::kIndirectBlobGcWorkingMemory for the working memory of indirect blob GC. BlockBasedTableOptions accepts a cache_usage_options.options_overrides entry for it, which indirect blob GC (diff 9) reads. It has its own line in the block cache entry stats. - A table reader opened with avoid_shared_metadata_cache (a compaction's ephemeral reader, and the reader that TableCache::GetBlobReferenceReader() opens on a table cache miss) skips the shared persistent cache for all its metadata blocks. Data blocks keep the configured persistent cache. ## What changed in the implementation "Parts of this diff" and "How to read this diff" list the files. Also: - table/block_based/block_based_table_reader_sync_and_async.h uses PersistentCacheFor(), and table/block_based/hash_index_reader.cc uses metadata_persistent_cache_options. - BUCK, CMakeLists.txt and src.mk add db/blob/blob_relocation_gc.cc. ## Safety boundaries and unsupported cases - No DB code calls the building blocks in this diff. Their first user is indirect blob GC in diff 9. Four rules of this diff do apply to live code: the location block compression and the kCRC32c rule (every relocation file that the stack writes), the checksum rule of relocation file reads, and the persistent cache rule for readers opened with avoid_shared_metadata_cache. - Public API: CacheEntryRole::kIndirectBlobGcWorkingMemory is inserted before kMisc. This changes the numeric values of kMisc and kNumCacheEntryRoles. - Relocation files with compressed location blocks are read by the read path of diff 3 with no change: block-based table readers decompress data blocks as usual. - Stored reads of relocation files update no blob read ticker or perf counter. ## Validation and known limitations Unit tests cover the batch reader (block reads, room checks, errors, index size report), the live set (checks, merges, origin runs, exact reservation), the packing plan, stored reads and streaming of both file kinds, coalesced reads, and compressed location blocks. Known limitation: the tests call each block directly. The job that combines them, and its end-to-end tests, come in diff 9. ## Option gate At this diff, ColumnFamilyData::ValidateOptions rejects enable_blob_indirection outside tests (the option gate: diff 2 adds it, diff 14 removes it). The behavior described here with the option on is reachable only through the internal test hook ColumnFamilyData::TEST_SetAllowBlobIndirection, not in production. ## Stack context (same in diffs 2-14, except that diffs 10-14 add the diff 10 route rule to the install line; "<== this diff" marks this diff's part): Goal: remove garbage from blob files without rewriting any SST. A value gets a stable identity when it is first written. Indirect blob GC copies the live values of whole files into a relocation file and installs routes in the MANIFEST. Readers follow the routes. SSTs are not changed. How a read finds a value (P is the relocation file that holds the value now): ``` SST entry indirect BlobIndex { generation number, origin, original offset, size, CRCs } | v Version file table: generation number -> live file, or route old file -> P | | live file route found | | v v blob file: read at original offset relocation file P (always one hop): relocation file: look up key look up key (origin, original offset) (origin, original offset) ``` Lifecycle of one value (in this example, F is the blob file where the value is first written, and P is the relocation file that GC writes): ``` write new value goes to blob file F; index: origin = F, generation number = F | census per generation number: the SSTs and references that use it; | per physical file: live bytes, ranked by garbage ratio GC job copy the live values of whole input files (stored bytes, checked | with the stored CRC) into relocation file P, keyed by | (origin, original offset) install one MANIFEST edit: add P, route each input generation number still | in use to P, re-point the routes into the inputs to P, and remove | each input (blob file retirement for a blob file, relocation | file deletion for a relocation file) flatten compaction sets generation number = P in the SSTs it rewrites | route route retirement: route deletion records remove the routes that delete no SST uses ``` Stack map: | Diff | Adds | |---|---| | 1 | Finalize blob metadata outside the DB mutex (prep) | | 2 | Indirect BlobIndex (kIndirectBlob) and exact SST reference metadata | | 3 | Relocation file format and stable read path | | 4 | MANIFEST routes and relocation state in the Version | | 5 | Exact reverse reference census | | 6 | MANIFEST apply outside the DB mutex, with a precondition re-check | | 7 | Coarse flush metadata, per-SST reference cap, epoch runs | | 8 | Indirect blob GC building blocks <== this diff | | 9 | Indirect blob GC job | | 10 | Flatten generation numbers; route retirement | | 11 | Remote compaction validation | | 12 | Dynamic options and operations | | 13 | Lost relocation files | | 14 | RepairDB rebuilds relocation state; option gate removed | | 15 | db_stress and crash test support | ## Stack context: terms for later diffs | Term | Meaning | |---|---| | flattened reference | Index with generation number != origin: compaction rewrote the generation number to the relocation file that holds the value (diff 10). The live set of this diff already accepts it (encoding class kIndirectFlattened) | | flatten | Compaction rewrites the generation number to the relocation file that holds the value, so old routes can go. Only flatten makes generation number != origin (diff 10) | | route deletion | MANIFEST record that removes a route that no SST uses. Route retirement is the background job that writes these records (diff 10) | Tracking issue: meta#15301 Differential Revision: D124782167
Summary: Pull Request resolved: meta#15545 ## Why the change is required A blob file can hold mostly garbage while the SSTs that reference it are cold. Compaction-time blob GC (enable_blob_garbage_collection) reclaims a blob file only when compaction rewrites the SSTs that reference it. This diff adds the indirect blob GC job: it collects up to 8 whole files of one column family, normal blob files or relocation files, into new relocation files, installs the result with one MANIFEST edit, and rewrites no SST. A relocation file also gets garbage when SSTs drop its values, so GC collects relocation files too. The job combines the building blocks of diff 8. ## Parts of this diff 1. Core: the job, DBImpl::IndirectBlobGCJob in db/db_impl/db_impl_compaction_flush.cc (Run() and one method per phase), its entry DBImpl::RunIndirectBlobGCBatch() in db/db_impl/db_impl.h, and the GC branch of BackgroundCompaction(). The "Indirect blob GC job" part of the overview at the top of db/blob/blob_relocation_gc.h lists the job steps and the suppression rules. 2. Census hooks (db/blob/blob_reference_census.{h,cc}, db/db_impl/db_impl_open.cc): the job reports a census mismatch and asks for one rebuild; the census snapshot is charged to the job; the census heap retries suppressed relocation files when the coarse frontier (the lowest frontier of all coarse SSTs in the Version) rises. 3. Scheduling (db/column_family.cc): ColumnFamilyData::NeedsCompaction() also returns true when GC has a candidate, because GC runs on the compaction workers. 4. Option (include/rocksdb/advanced_options.h, options/, db/c.cc, include/rocksdb/c.h, c_api_gen/): the new mutable option indirect_blob_gc_read_coalesce_bytes. Safe to skip on a first read. 5. Disk space and reads (file/sst_file_manager_impl.{h,cc}, db/blob/blob_file_cache.{h,cc}): a byte-based space reservation, and direct reads for normal blob file inputs. Safe to skip on a first read. 6. Cleanup and RepairDB (db/db_impl/db_impl_files.cc, db/blob/blob_file_completion_callback.h, db/repair.cc): an output that fails before its rename, an orphan output, and the RepairDB guard for relocation files. Safe to skip on a first read. ## How to read this diff 1. Overview: read "Terms used in this diff" in this summary. Then read the overview at the top of db/blob/blob_relocation_gc.h: the building blocks part (diff 8), then the "Indirect blob GC job" part (job terms, job steps, suppression rules and the table of reasons). 2. Census hooks: in db/blob/blob_reference_census.h read the census mismatch state table, ReportMismatch(), ClearMismatch() and TakeMismatchReport(), and the SnapshotSelected() charge comment. Then read the TakeMismatchReport() call in db/db_impl/db_impl_open.cc, and the frontier retry in BlobGcCandidateHeap::SetCoarseIndirectOriginFrontier() (db/blob/blob_reference_census.cc). They depend only on the census of diff 5. 3. Small support pieces (safe to skim): the byte-based EnoughRoomForCompaction() and OnCompactionCompletion(uint64_t) in file/sst_file_manager_impl.h, prefer_direct_reads in db/blob/blob_file_cache.h, OnBlobFileAbandoned() in db/blob/blob_file_completion_callback.h, and the option indirect_blob_gc_read_coalesce_bytes in include/rocksdb/advanced_options.h. 4. The entry: read the RunIndirectBlobGCBatch() comment in db/db_impl/db_impl.h (DB mutex, out parameters and return codes). 5. The job, part 1 (plan, read, decide): in db/db_impl/db_impl_compaction_flush.cc read the DBImpl::IndirectBlobGCJob class comment (job status, input outcomes, memory charges) and Run(). Then read Start(), SelectInputs(), PlanLiveSetReads(), ReadLiveSets() and DecideInputs(). 6. The job, part 2 (pack, copy, install): read PackOutputs(), OpenInputReaders(), WriteOutputs() (with its error priority rule), then Install() with its "Install protocol" comment, the two functions that Install() calls (BuildEdit() builds the MANIFEST edit, and ValidatePhysicalInputs() is the precondition of its LogAndApply() call), and Finish(). 7. Scheduling: read the GC branch and the retry code in BackgroundCompaction() (same file), and ColumnFamilyData::NeedsCompaction() in db/column_family.cc. 8. Last, read the purge rules in db/db_impl/db_impl_files.cc and the RepairDB guard in db/repair.cc. ## Terms used in this diff Stack terms: | Term | Meaning | |---|---| | origin | Blob file where the value was first written | | original offset | Value offset in the origin file | | stable identity | (origin, original offset). Never changes, even when the value moves | | generation number | The file number that a reference uses to find its value. A new value has generation number == origin. When indirect blob GC moves the value, the reference keeps its generation number, and a route sends the read to the relocation file | | Version file table | The table in a Version that maps a generation number to a live file or to a route | | unflattened reference | Index with generation number == origin: the value was not moved, or it was moved but no compaction has rewritten its reference yet (see "flatten" in "Stack context: terms for later diffs") | | route | Old file number -> the relocation file that now holds its live values. One hop: a target is never itself routed. Stored in the MANIFEST | | physical file | A file that holds value bytes: a normal blob file or a relocation file. A generation number can name a physical file, or it can be only the key of a route | | indirect origin | Normal blob file written with the option on ("the option" is enable_blob_indirection); it is the origin of unflattened references | | relocation file | Block-based table keyed by (origin, original offset) that holds copied values. Each value is the stored bytes plus a 5-byte trailer (kSimpleGen2BlobTrailerSize) | | exact reference metadata | Per-SST sorted list of every blob reference the SST keeps, written when the SST is built. References with the same encoding class, generation number and origin form a group. The list is stored as data blocks, with an index of the groups. An SST that has it is an exact SST | | coarse SST | An SST without exact reference metadata. It records one frontier file number and pins every indirect-origin blob file at or above it. Frontier 0 means no coarse frontier | | pin | A coarse SST pins every indirect-origin file number at or above its frontier: GC must treat that file number as used. This is not the base RocksDB meaning (a Version or block that stays alive) | | census | Background count, per generation number, of the SSTs and references that use it, plus live bytes per physical file | | metadata-only commit | A MANIFEST edit that adds no SST and changes only blob metadata (BlobReferenceCensus::BeginMetadataCommit()) | Terms of this diff: the overview at the top of db/blob/blob_relocation_gc.h defines input, generation numbers of an input, output, live set, record bytes and stored CRC (diff 8), and stale job, input Version, candidate-local failure, deferred input, census entry, census snapshot, census heap and blob file retirement (this diff). The IndirectBlobGCJob class comment defines GC job registration and candidate stamp. Two more: | Term | Meaning | |---|---| | suppress | Skip a file as a GC candidate, normally (until the file changes or the census rebuilds) or transiently (until a retry delay passes). The "Suppression" paragraph of the overview has the rules | | queue head | The point where GC's MANIFEST edit is first in the MANIFEST writer queue, under the DB mutex. GC checks its inputs again there | ## Previous behavior Up to diff 8, nothing collected a blob file outside compaction. The census ranked candidates, and the building blocks of diff 8 had only test callers. Compaction-time blob GC rewrote values of the oldest blob files (below blob_garbage_collection_age_cutoff) together with the SSTs that reference them. A garbage-heavy blob file whose SSTs were not compacted stayed on disk. ## New behavior In short: 1. GC runs on a LOW-pool compaction worker, only when that worker finds no LSM compaction to run. 2. One job collects up to 8 input files of one column family into up to 8 relocation files, and installs them with one MANIFEST edit. 3. GC copies only the exact live set, and only for inputs with at least 50% exact garbage. 4. A census mismatch triggers one census rebuild. A repeat mismatch of the same file suppresses that file. 5. A corrupt or unsupported input is suppressed alone, and the other inputs stay candidates. If this happens in the copy phase, the job also ends without installing any output. Output, I/O and MANIFEST errors are background errors. The overview in db/blob/blob_relocation_gc.h describes the job steps and the suppression rules. The rest of this section adds the numbers and the cases that the code comments do not hold. ### Scheduling (step 1) - A LOW-pool worker that has no pre-picked compaction and found no LSM compaction takes up to 8 candidates (kIndirectBlobGCMaxInputs, a fixed limit, not tuned; it matches the 8 candidates that the census publishes) in the Version's rank order. LSM compaction always has priority. The census heap ranks normal blob files and relocation files together. - This needs auto compaction on, enable_blob_files, enable_blob_garbage_collection, enable_blob_indirection, and a census that is ready for the current Version. ColumnFamilyData::NeedsCompaction() also returns true when these hold and the Version has a published candidate, so GC is scheduled even when no LSM compaction is needed. - Only one GC job runs per column family at a time (indirect_blob_gc_cfs_in_progress_), because a job removes whole relocation files and re-points the routes into them. - With a separate remote compaction pool, the worker that picks GC is a remote coordinator. GC then runs as local work through RunLocalCompaction(): it waits for a local compaction credit, is counted in bg_compaction_scheduled_, and runs in the LOW pool. - A stale job ends with TryAgain. BackgroundCompaction() turns it into Busy and enqueues the column family again. Busy is not a background error: BackgroundCallCompaction() waits only 10 ms (to prevent a hot loop) instead of the 1 s wait after an error. CompactionTooLarge and an aborted job also enqueue it again. ### Plan, read and check (steps 2 to 4) - GC takes a reference to the input Version, registers the job with the census (BeginGcJob()), and takes the census snapshot of the inputs' generation numbers. It finds the generation numbers of each input with one scan of the input Version's routes, without the DB mutex. - GC reserves each live set from the census reference counts before any read. All live sets and the GC buffers share one 64 MB memory budget (BlobRelocationMemoryTracker, diff 8). When the budget is used by earlier inputs, the input that does not fit and all later inputs are deferred. When no earlier input holds a reservation, GC suppresses only the input that does not fit (transient) and still plans the next ones. - GC reads each needed SST once, with the reader from TableCache::GetBlobReferenceReader() (diff 5): the table cache reader, or an ephemeral reader on a miss. The reader is charged 64 KB + 2 x kBlobReferenceReadBatchBytes (256 KB each) = 576 KB (an estimate, also charged for a table cache reader, to keep it simple), plus the real size of the decoded reference index. One batch reads, per SST, every generation number of the inputs that the SST holds. - GC compares the SSTs and totals that it found with the census snapshot. On a mismatch it calls BlobReferenceCensus::ReportMismatch(): the first report for a file makes the census stop being ready and requests a rebuild, and the job ends as stale. A second report for the same file, with no matching check in between, means the rebuild did not clear the mismatch: GC logs one ERROR line and suppresses only that file (normal suppression), with no further rebuild. When no checked input of a job mismatched, GC calls ClearMismatch(file_number) for each checked input. The census then forgets the stored report only if it names that file, so a later mismatch of the file requests a rebuild again. A match of another file keeps the report, so a match of one file does not hide a repeat mismatch of a different file. A report for another file replaces the stored file. After a report, the census worker of the same build cannot publish again (kStale) until the census job fails that build (TakeMismatchReport()) and starts a new one. ### Decide and pack (steps 5 and 6) - Each input must have at least 50% exact garbage on its own (a fixed choice, not measured). Exact garbage = record bytes of the input - live record bytes. So GC copies at most as many bytes as the input frees. blob_garbage_collection_age_cutoff and blob_garbage_collection_force_threshold do not apply to indirect blob GC. - An input with no live value gets only its removal: no output and no source read. - Packing uses PlanBlobRelocationPacking() (diff 8). The estimated size of an input is its live record bytes + 64 KB + 48 bytes per live value (estimates on the safe side). The target output size is 90% of min(blob_file_size, kIndirectBlobGCMaxOutputBytes (1 GB)); 1 GB and 90% are fixed choices. When the inputs need more than 8 outputs (kIndirectBlobGCMaxOutputs), the largest input is deferred and GC plans again. - Space: GC reserves the estimated size of all outputs, capped at the total input size, through a new byte-based SstFileManagerImpl::EnoughRoomForCompaction(estimated_output_size, output_path, bg_error), and releases it after the job. Both EnoughRoomForCompaction() overloads share one body, EnoughRoomForCompactionLocked(). If the check fails, the largest input is deferred first. If there is not enough space even for one input, the job returns CompactionTooLarge and the deferred inputs get a transient suppression. ### Copy (step 7) - GC copies the stored bytes and the compression type unchanged, keyed by (origin, original offset), and checks every copy with its stored CRC. GC never decompresses. - Small values are read in batches of up to 1 MB (kIndirectBlobGCReadBatchBytes, an estimate), with 512 bytes of fixed per-value memory (an estimate), so a batch has at most 2048 values. Larger values are streamed in 256 KB chunks (kBlobRelocationStreamChunkSize, a choice, not a measurement; see diff 8). - Read coalescing: two neighbor values share one read when the gap between them is at most indirect_blob_gc_read_coalesce_bytes (new mutable option, default 4096, one page; buffered I/O reads the whole page anyway). 0 reads each value on its own. One read stays at most 1 MB. - Direct reads: GC input is cold data. Buffered io_uring reads of coalesced ranges read about 30% more device bytes than needed (an estimate from an earlier io_uring test; the workload was not recorded). So GC opens normal blob file inputs for direct reads when mmap reads are off, and falls back to the configured open if that fails. User reads and relocation-file inputs keep their buffered readers. - The GC I/O workspace (the buffers that read and stream the input values) is charged 2 x 1 MB + 2 x 256 KB before the live sets are known. - Data reads and writes use Env::IO_LOW and do not fill the block cache. The MANIFEST write is not rate limited, so edits queued behind it do not wait for low-priority I/O. - GC writes each output as a temp file, syncs it, and renames it to its blob file name before the install. If all outputs together are not smaller than all inputs together, GC drops every output before the rename and suppresses all its eligible inputs. ### Install (step 8) - One MANIFEST edit adds every output, routes each live generation number of an input to the input's output, re-points the routes into a relocation input to the output, deletes the routes that no SST uses, writes the blob file retirement of each normal input and deletes each relocation input. Each output lists the old file numbers that it serves (its relocation directory). Diff 14 uses this list to rebuild routes in RepairDB; at this diff, RepairDB rejects a DB with a relocation file. - At the queue head, the precondition checks that the column family is alive and the job is still registered (IsGcJobActive()), that each input is unchanged, and that no coarse SST pins an input. A relocation input must have the same number of routes into it: only GC changes routes, and only one GC job runs per column family, so an equal count means the same routes. - Coarse pin: a normal input is pinned when its file number is at or above the coarse frontier. A relocation input is pinned when one of its routed generation numbers is at or above the coarse frontier; its own file number does not count. GC checks this at selection (and suppresses a pinned input) and again at the queue head. A pin can appear during the job: a flush takes its blob file number (for example 25) before GC selects normal input 28, and installs its coarse SST with frontier 25 after. The job is then a normal stale job (TryAgain), not an invariant failure. - Install protocol: the edit comes from the census snapshot, and the job does not follow the Version changes after it. This is enough because no SST added after the snapshot can hold a generation number of an input that the snapshot does not count: a compaction keeps or drops the references of its input SSTs, and a flush references only the new blob files that it writes. Ingestion and import are assumed to add no SST with kIndirectBlob references (diff 12 rejects them). So the snapshot can only count a holder too many, and the edit then keeps a route that no SST needs. Flushes and compactions do not wait for the GC job or its commit. - The precondition runs at the queue head and again after the unlocked apply (diff 6). On its first run it starts the metadata commit (BeginMetadataCommit()). Until the MANIFEST write ends, no other GC job or metadata-only commit can start. When the write result is unknown, BeginGcJob() returns Busy until recovery: a GC attempt then ends with no error and no install, and after Resume() GC runs again by itself. - If another Version was installed during the job, GC does not apply its job-start suppressions to the current Version, and enqueues the column family again. - When the coarse frontier rises, the census heap retries the suppressed relocation files above the old frontier, because their routed generation numbers may no longer be pinned. ### Errors and suppression The "Suppression" paragraph of the overview has the suppression levels, how long each lasts, the stamp rule, and the table of reasons. - Corruption in the exact SST metadata or in an input's records is candidate-local. GC leaves that input and its routes unchanged, suppresses it, and other background work continues. - Output corruption, I/O errors and MANIFEST errors use the normal background-error handling. When one read batch has both a candidate-local error and another error, the other error wins. - A stale job installs nothing; a later job tries again. At shutdown the job returns ShutdownInProgress. Cache pressure on the census snapshot gives the job's inputs a transient suppression. None of these is a background error. - If the MANIFEST write fails, the outputs are quarantined, not deleted, and the inputs stay. This is also the state after a crash between the rename and the install. - An output that fails before its rename never had its final name. Listeners see a failed creation (BlobFileCompletionCallback::OnBlobFileAbandoned()) and no deletion event, and the SstFileManager never tracks it. ### Working memory The live sets, the census snapshot, the SST readers and the GC buffers are charged to CacheEntryRole::kIndirectBlobGcWorkingMemory (diff 8) in the job's block cache. The role follows cache_usage_options.options_overrides and is charged by default (kFallback counts as enabled). Setting it to kDisabled turns the charge off. ### Obsolete-file purge and RepairDB - Obsolete-file purge keeps a temp file whose number is protected by pending outputs (number >= min_pending_output), so a full purge during GC does not delete the temp output. This rule applies to every temp file, not only to GC outputs: base RocksDB deletes such a temp file unless it is live, the pending MANIFEST or an options file. - An orphan output is a file with the relocation file prefix that no Version owns. Purge deletes it. Purge checks the blob log footer first, so a blob file with a valid footer costs no extra read. - The temporary RepairDB guard also rejects a DB with a validated relocation file (the file starts with the relocation prefix and its table properties hold a valid relocation directory). Files that cannot be read are skipped. ### Monitoring - Job outcome. Each job that begins (registers with the census) counts in INDIRECT_BLOB_GC_JOBS_STARTED. At its end, it counts in one of these tickers: - INDIRECT_BLOB_GC_JOBS_INSTALLED: the job installed its result; - INDIRECT_BLOB_GC_JOBS_STALE: an input changed, and a later job retries; - INDIRECT_BLOB_GC_JOBS_NOTHING_INSTALLED: no error and nothing installed, for example because every input was skipped, the job was deferred, the DB shut down or the column family was dropped; - INDIRECT_BLOB_GC_JOBS_FAILED: an error. Such a job also counts in the base ticker BLOB_DB_GC_FAILURES. - Work. INDIRECT_BLOB_GC_KEYS_RELOCATED and INDIRECT_BLOB_GC_BYTES_RELOCATED (stored bytes) count the values that installed jobs moved into relocation files. The base tickers BLOB_DB_GC_NUM_KEYS_RELOCATED and BLOB_DB_GC_BYTES_RELOCATED count them too, as totals of both GCs. INDIRECT_BLOB_GC_BYTES_READ counts the bytes that jobs read from their input files, normal and relocation files. INDIRECT_BLOB_GC_BYTES_RECLAIMED counts, for installed jobs, the size of the removed input files minus the size of the new relocation files. The base tickers BLOB_DB_GC_NUM_FILES and BLOB_DB_GC_NUM_NEW_FILES count the removed input files and the new relocation files. The docs of these four base tickers name indirect blob GC. - Problems. INDIRECT_BLOB_GC_CORRUPT_INPUTS counts inputs that a job skipped because it found corruption (in a blob record, a relocation file or exact SST metadata). INDIRECT_BLOB_GC_CENSUS_MISMATCHES counts the times a job found SSTs that do not match the census. - INDIRECT_BLOB_GC_JOB_MICROS (histogram) records the time of each job, from its start to its end. - Idle time. Each column family keeps the time since which it has had GC candidates and no job began (ColumnFamilyData::NoteIndirectBlobGcCandidates()). A census publish with candidates while no GC job is registered (BlobReferenceCensus::HasGcJob()), or the end of a job that leaves candidates, starts it. A job start, or a publish without candidates, clears it. So the idle time stays 0 while a job runs. rocksdb.indirect-blob-gc-idle-micros-with-candidates (an int property) and the map key idle-micros-with-candidates report this duration through one helper. A long idle time with candidates means that GC is starved, for example by LSM compactions. - LOG lines: - An ERROR for each corrupt input, with what the job found, also when other inputs of the job install. The file is skipped until it changes or the census is rebuilt. - An INFO line with the skip reason of every skipped input, also when other inputs install. - One INFO line at the end of each job: "N candidates, R inputs read, M installed, S skipped, ...", then outputs, values relocated, bytes read and written, and the status. R counts the inputs that the job read and packed. M counts the inputs that the installed edit removed, so it is 0 unless the job installed. - The WARN line of a GC job error starts with "Indirect blob GC error" instead of "Compaction error". - DBImpl::DumpStats() logs a WARN for each column family whose GC has had candidates for longer than one stats dump period (stats_dump_period_sec) without starting a job. With stats dumps off, there is no such WARN. - Listeners see the relocation files that GC writes with BlobFileCreationReason::kIndirectBlobGC. - Feature off: only a GC job updates these tickers and the histogram, and GC runs only in a column family with a census. Without a census the idle time stays 0, and the DumpStats check skips the column family. ## What changed in the implementation "Parts of this diff" and "How to read this diff" list the files. Also: - db/db_impl/db_impl_debug.cc: TEST_RunIndirectBlobGCBatch() and test hooks for the job. - db/blob/blob_relocation_gc.{h,cc}: the job part of the overview, and comments that name the job as the user of the building blocks. - db/table_cache.h: DBImpl is a friend, to get per-job SST readers outside the DB mutex. ## Safety boundaries and unsupported cases - Public API: new tickers INDIRECT_BLOB_GC_JOBS_STARTED, _INSTALLED, _STALE, _NOTHING_INSTALLED and _FAILED, INDIRECT_BLOB_GC_CORRUPT_INPUTS, INDIRECT_BLOB_GC_CENSUS_MISMATCHES, INDIRECT_BLOB_GC_KEYS_RELOCATED, INDIRECT_BLOB_GC_BYTES_RELOCATED, INDIRECT_BLOB_GC_BYTES_READ and INDIRECT_BLOB_GC_BYTES_RECLAIMED; new histogram INDIRECT_BLOB_GC_JOB_MICROS; new property rocksdb.indirect-blob-gc-idle-micros-with-candidates; new BlobFileCreationReason::kIndirectBlobGC. The new values are appended before TICKER_ENUM_MAX and HISTOGRAM_ENUM_MAX. The Java API maps the histogram (HistogramType and portal.h). The new tickers have no Java mapping, as for recent base tickers: all 256 byte values of org.rocksdb.TickerType are used. - Public API: the new column family option indirect_blob_gc_read_coalesce_bytes (mutable through SetOptions, default 4096), with its C API getter and setter. - A blob file referenced only by legacy BlobIndex entries is not an indirect origin, so it is not a candidate. A normal blob file at or above the coarse frontier is not a candidate. A relocation file is suppressed while one of its routed generation numbers is at or above the coarse frontier. In normal operation a coarse SST does not reference a routed generation number, but GC keeps this pin so that it never collects a file that a coarse SST may still read. - Indirect blob GC does not pack across column families, place values by user key, apply a temperature policy, or run remotely. Census estimates may understate garbage; only the exact live set allows deletion. Transient failures have retry delays, so a bad candidate cannot block other candidates for ever. - With a separate remote compaction pool, indirect blob GC runs as local work in the LOW pool and counts toward max_compactions. - Direct GC reads fall back to buffered reads only when the direct open fails. A file system that accepts O_DIRECT at open but fails some direct reads makes those GC jobs fail. - GC reads of relocation-file inputs do not update BLOB_DB_BLOB_FILE_BYTES_READ or blob_read_byte. INDIRECT_BLOB_GC_BYTES_READ counts them. - Ingestion and import of an SST with indirect references are not rejected in this diff (diff 12 rejects them before the feature can be used). ## Validation and known limitations Unit tests cover relocation of the exact live set, packing and reclaim of relocation files, removal of files with no live value, the census totals check and its rebuild, per-file mismatch suppression, memory limits, error locality, the install precondition and its retries, changes during the copy, MANIFEST failures with an unknown result, temp-file and orphan cleanup, the separate remote compaction pool, the cache role, mmap reads, coarse pins, and the RepairDB guard. Known limitations: - This diff does not test the SstFileManager CompactionTooLarge retry path. - This diff does not test the direct-read open of GC inputs or its buffered fallback. ## Option gate At this diff, ColumnFamilyData::ValidateOptions rejects enable_blob_indirection outside tests (the option gate: diff 2 adds it, diff 14 removes it). The behavior described here with the option on is reachable only through the internal test hook ColumnFamilyData::TEST_SetAllowBlobIndirection, not in production. ## Stack context (same in diffs 2-14, except that diffs 10-14 add the diff 10 route rule to the install line; "<== this diff" marks this diff's part): Goal: remove garbage from blob files without rewriting any SST. A value gets a stable identity when it is first written. Indirect blob GC copies the live values of whole files into a relocation file and installs routes in the MANIFEST. Readers follow the routes. SSTs are not changed. How a read finds a value (P is the relocation file that holds the value now): ``` SST entry indirect BlobIndex { generation number, origin, original offset, size, CRCs } | v Version file table: generation number -> live file, or route old file -> P | | live file route found | | v v blob file: read at original offset relocation file P (always one hop): relocation file: look up key look up key (origin, original offset) (origin, original offset) ``` Lifecycle of one value (in this example, F is the blob file where the value is first written, and P is the relocation file that GC writes): ``` write new value goes to blob file F; index: origin = F, generation number = F | census per generation number: the SSTs and references that use it; | per physical file: live bytes, ranked by garbage ratio GC job copy the live values of whole input files (stored bytes, checked | with the stored CRC) into relocation file P, keyed by | (origin, original offset) install one MANIFEST edit: add P, route each input generation number still | in use to P, re-point the routes into the inputs to P, and remove | each input (blob file retirement for a blob file, relocation | file deletion for a relocation file) flatten compaction sets generation number = P in the SSTs it rewrites | route route retirement: route deletion records remove the routes that delete no SST uses ``` Stack map: | Diff | Adds | |---|---| | 1 | Finalize blob metadata outside the DB mutex (prep) | | 2 | Indirect BlobIndex (kIndirectBlob) and exact SST reference metadata | | 3 | Relocation file format and stable read path | | 4 | MANIFEST routes and relocation state in the Version | | 5 | Exact reverse reference census | | 6 | MANIFEST apply outside the DB mutex, with a precondition re-check | | 7 | Coarse flush metadata, per-SST reference cap, epoch runs | | 8 | Indirect blob GC building blocks | | 9 | Indirect blob GC job <== this diff | | 10 | Flatten generation numbers; route retirement | | 11 | Remote compaction validation | | 12 | Dynamic options and operations | | 13 | Lost relocation files | | 14 | RepairDB rebuilds relocation state; option gate removed | | 15 | db_stress and crash test support | ## Stack context: terms for later diffs | Term | Meaning | |---|---| | flattened reference | Index with generation number != origin: compaction rewrote the generation number to the relocation file that holds the value (diff 10). The live set already accepts it (encoding class kIndirectFlattened) | | flatten | Compaction rewrites the generation number to the relocation file that holds the value, so old routes can go. Only flatten makes generation number != origin (diff 10) | | route deletion | MANIFEST record that removes a route that no SST uses. Route retirement is the background job that writes these records (diff 10) | Tracking issue: meta#15301 Differential Revision: D124855808
Summary: Pull Request resolved: meta#15547 ## Why the change is required After diff 9, every indirect blob GC install leaves routes behind: each old generation number that SSTs still use keeps a route to the relocation file that holds its values. Nothing deletes these routes, so the routes in the Version file table (and in the MANIFEST) only grow, and every read of an old generation number takes the route hop. This diff lets compaction rewrite an old generation number to the relocation file that holds the value (flatten). Then route retirement, a step of the census job, deletes the routes that no SST uses any more. Both must stay correct while GC, flush and other compactions run at the same time. This diff also adds version 2 of the blob file header. RepairDB (diff 14) must rebuild relocation state from the files alone. A version 1 header does not say which file number the file has, or whether indirect BlobIndexes reference its values. Version 2 stores both, and the footer stores a CRC of the header. ## Parts of this diff 1. Core: flatten in the compaction iterator (db/compaction/compaction_iterator.{h,cc}, BlobIndex::SetGenerationFileNumber() in db/blob/blob_index.h). 2. Compaction safety protocol (db/compaction/compaction_job.{h,cc}, Version::ValidateBlobReferenceGeneration() in db/version_set.{h,cc}): the checks that prove that every number an output writes can still be read at install. The overview comment "Compaction safety protocol for blob relocation" in compaction_job.h gives the argument and the checks. 3. Route retirement (DBImpl::RetireBlobRelocationRoutes() in db/db_impl/db_impl_compaction_flush.cc with its comment in db/db_impl/db_impl.h, its caller in the census job in db_impl_open.cc, and the retirement set in db/blob/blob_reference_census.{h,cc}): it deletes the routes that flatten made unused. 4. Indirect blob GC install and flattening (DBImpl::IndirectBlobGCJob::BuildEdit() and the "Install protocol" comment in Install(), db/db_impl/db_impl_compaction_flush.cc): a GC job keeps all routes of a relocation input that gets an output, because a compaction may flatten to it while GC runs. 5. Blob file header version 2 (db/blob/blob_log_format.{h,cc}, the writer in db/blob/blob_file_builder.{h,cc}, the checks in db/blob/blob_file_reader.{h,cc}). It depends on nothing else in this diff. 6. Remote compaction rules (CompactionJob::ShouldUseLocalCompaction() in compaction_job.cc): some feature-aware compactions run locally instead of on a CompactionService. Safe to skip on a first read. 7. Small side changes: the UpdateBoundaries() overloads in db/version_edit.{h,cc} and BlobGarbageMeter. Safe to skip on a first read. 8. Inactive hooks: hooks that later diffs use, for values whose blob file is lost and for compactions that may not create blob files (see "Code whose first users come later"). Safe to skip on a first read. ## How to read this diff 1. Read "Terms used in this diff" in this summary ("Stack context" at the end gives the goal, the diagrams and the stack map). Then read the summary sections "Flatten", "Compaction safety protocol", "Route retirement" and "Indirect blob GC install and flattening". For background, read steps 4 to 7 of "Lifecycle of one value" at the top of db/blob/blob_relocation_manifest.h (step 6 is flatten, step 7 is route deletion). This diff does not change that file. 2. Read flatten. First read BlobIndex::SetGenerationFileNumber() in db/blob/blob_index.h. It depends on nothing else in this diff. It rewrites the generation number and picks the encoding. Then read the comment on FlattenBlobIndexIfNeeded() in db/compaction/compaction_iterator.h, and FlattenBlobIndexIfNeeded(), FlattenBlobIfNeeded(), FlattenBlobIndexesIfNeeded(), FlattenEntityBlobIndexesIfNeeded() and PrepareOutput() in compaction_iterator.cc, and the rewrite table in GarbageCollectBlobIfNeeded(). Skip the branches for values whose blob file is lost and for compactions that may not create blob files; they have no user in this diff. 3. Read Version::ValidateBlobReferenceGeneration() in db/version_set.{h,cc}. Then read the overview "Compaction safety protocol for blob relocation" in db/compaction/compaction_job.h. Then read its three checks in compaction_job.cc: ValidateBlobRelocationInputGenerations(), PrepareBlobRelocationOutputDependencies() with BlobOutputDependency in compaction_job.h, and ValidateBlobRelocationOutputDependencies() with the relocation_state_precondition lambda in CompactionJob::InstallCompactionResults(). 4. Read the census retirement set in db/blob/blob_reference_census.{h,cc}: retirement_generations, its rebuild in WorkerState::Finalize(), the delta updates (MarkRetirementIfRouted()) and SelectWorkerRetirementSources(). Also read BeginGcJob(version_number, candidate_files, token) and IsGcInputFile(). 5. Read the comment on DBImpl::RetireBlobRelocationRoutes() in db/db_impl/db_impl.h, then its body in db/db_impl/db_impl_compaction_flush.cc (held_by_running_compaction is the running-compaction rule), and its caller after the census publish in DBImpl::BuildBlobReferenceCensus() (db/db_impl/db_impl_open.cc). 6. Read the GC install change in db/db_impl/db_impl_compaction_flush.cc: keeps_all_generations and the route loop in DBImpl::IndirectBlobGCJob::BuildEdit(), then the "Install protocol" comment, which explains with an example why a relocation input that gets an output keeps all of its routes. 7. Read the blob file header version 2. It is independent of steps 2 to 6. Read the format comments in db/blob/blob_log_format.h, then BlobLogHeader::EncodeTo() and DecodeFrom() in blob_log_format.cc, then BlobFileBuilder::OpenBlobFileIfNeeded() and CloseBlobFile() in blob_file_builder.cc, then BlobFileReader::ReadHeader() and ReadFooter() in blob_file_reader.cc. 8. Last, read the side changes: the remote compaction rules in compaction_job.cc, the UpdateBoundaries() overloads in db/version_edit.cc, and BlobGarbageMeter. ## Terms used in this diff Stack terms: | Term | Meaning | |---|---| | origin | Blob file where the value was first written | | original offset | Value offset in the origin file | | stable identity | (origin, original offset). Never changes, even when the value moves | | generation number | The file number that a reference uses to find its value. A new value has generation number == origin. When indirect blob GC moves the value, the reference keeps its generation number, and a route sends the read to the relocation file | | unflattened reference | Index with generation number == origin (value not relocated, or not flattened yet) | | route | Old file number -> the relocation file that now holds its live values. One hop: a target is never itself routed. Stored in the MANIFEST | | physical file | A file that holds value bytes: a normal blob file or a relocation file. A generation number can name a physical file, or it can be only the key of a route | | indirect origin | Normal blob file written with the option on; it is the origin of unflattened references | | relocation file | Block-based table keyed by (origin, original offset) that holds copied values. Each value is the stored bytes plus a 5-byte trailer (kSimpleGen2BlobTrailerSize) | | stored CRC | CRC32C of the stored bytes plus the compression byte | | logical CRC | CRC32C of the uncompressed value (BlobReference::stable_checksum) | | exact reference metadata | Per-SST sorted list of every blob reference the SST keeps, written when the SST is built. References with the same encoding class, generation number and origin form a group. The list is stored as data blocks, with an index of the groups. An SST that has it is an exact SST | | reference group | In exact reference metadata, the references of one SST that share an encoding class, a generation number and an origin (BlobReferenceGroup) | | coarse SST | An SST without exact reference metadata. It records one frontier file number and pins every indirect-origin blob file at or above it. Frontier 0 means no coarse frontier | | census | Background count, per generation number, of the SSTs and references that use it, plus live bytes per physical file | | holder | An SST with exact reference metadata that lists the generation number. A coarse SST is never a holder; its coarse frontier protects its numbers instead | | blob file retirement | MANIFEST record (kBlobFileRetirement) that removes a collected blob file from the Version | | Version file table | The table in a Version that maps a generation number to a live file or to a route | "The option" is enable_blob_indirection. Terms of this diff: | Term | Meaning | |---|---| | flatten | Compaction rewrites the generation number of a reference to the route target, the relocation file that holds the value. Only flatten makes generation number != origin | | flattened reference | Index with generation number != origin | | kept number / flattened number | For one indirect reference group, the output writes the group's generation number (kept), its route target in the compaction's input Version (flattened), or neither (compaction-time blob GC rewrites the value) | | coarse L0 output | L0 compaction output without exact reference metadata. It does not flatten | | output dependency | One physical file (a normal blob file or a relocation file) that an output number resolves to in the compaction's input Version, with the file's metadata object (BlobOutputDependency) | | install check | The check of the output dependencies against the current Version, in the LogAndApply() precondition of the compaction install (CompactionJob::ValidateBlobRelocationOutputDependencies()) | | route deletion | MANIFEST record that removes a route that no SST uses | | route retirement | DBImpl::RetireBlobRelocationRoutes(). The census job runs it after each census publish. It writes route deletion records in batches | | running-compaction rule | Route retirement does not delete the route of a generation number while that number is a live blob file or relocation file in the input Version of a running compaction of the column family (a compaction in CompactionPicker::compactions_in_progress()) | | census job | The background job that builds or updates the census (DBImpl::BuildBlobReferenceCensus()). A census publish is the step where it makes a new census ready for readers | | metadata commit | A MANIFEST write that changes only relocation metadata, between BeginMetadataCommit() and EndMetadataCommit() (diff 5): route retirement (this diff) or an indirect blob GC install (diff 9). At most one runs per column family. While one runs, a new GC job registration (BeginGcJob()) gets TryAgain. A GC install registered its job before its commit, so it does not block itself. db/blob/blob_reference_census.h also calls it a "metadata-only commit" | | unknown commit result | A metadata commit whose MANIFEST write failed with an I/O error. The edit may or may not be on disk (EndMetadataCommit(false)) | | feature-aware compaction | A compaction whose input Version has the indirection format, or that runs with the option on | | blob file header version 2 | BlobLogHeader with version kVersion2 (2). It stores the file's own number and the is_indirect_origin flag, and the file has no TTL. See "Blob file header version 2" below | ## Previous behavior Compaction kept the generation number of every indirect BlobIndex as it was. A route stayed in the MANIFEST for as long as its target file lived, even when no SST used its generation number. An indirect blob GC install routed only the live generation numbers of a relocation input to its output and deleted the routes of the others. New blob files used the version 1 header, which has no file number and no indirect-origin flag, and BlobLogHeader::DecodeFrom() accepted only version 1. ## New behavior ### Flatten - With the option on, when compaction writes an indirect BlobIndex whose generation number has a route in the compaction's input Version, it sets the generation number to the route target (BlobIndex::SetGenerationFileNumber()). Routes are one hop, so the target holds the value. The origin, the original offset and both CRCs do not change, and no value is read. - The new generation number decides the encoding: equal to the origin writes kIndirectBlob, above the origin writes kIndirectFlattenedBlob, and below the origin is InvalidArgument. - Plain values and wide-column blob columns are both flattened. - Flatten checks every indirect value, so it has two fast paths. If the input Version has no route (checked once per subcompaction, at its first indirect value), it does no route lookup. Otherwise it remembers the last generation number and its route target, because consecutive keys usually share a generation number. The input Version is fixed for the whole compaction, so a remembered answer cannot go stale. Each skipped call is one Version file table lookup. A one-time profile of a 120 s write run with 1 MB SSTs and one lookup per indirect value spent 5 to 6 s of compaction CPU in these lookups. The fast paths were not profiled. - Coarse L0 output does not flatten. It keeps compaction-time blob GC (age cutoff). A coarse SST has no reference list, so the census could not see a flattened reference in it, and indirect blob GC could remove the relocation file it points to. The coarse frontier protects only unflattened references, whose generation number is an indirect origin. - Remote compaction does not flatten (AllowBlobIndexFlattening() is false); diff 11 validates remote output. - With the option off and compaction-time blob GC on, compaction rewrites the indirect values it meets into legacy blob files or inline values. In coarse L0 output (an L0 compaction that is not bottommost and has no exact input), it rewrites only values below the age cutoff. With compaction-time blob GC off, indirect values stay as they are and stay readable. The table in CompactionIterator::GarbageCollectBlobIfNeeded() lists which values compaction-time blob GC rewrites. - For normal blob file liveness (oldest_blob_file_number), compaction output ignores flattened references and references whose generation number has a route. Neither keeps a normal blob file alive. This check runs once per output reference. To save CPU, it returns early when the generation number is at or above the output's current oldest_blob_file_number, because such a number cannot lower it, and it looks up a route only when the Version has routes. Together with the shared BlobIndex decode and the on-demand stored CRC of diff 2, this lowers the extra compaction CPU of blob indirection (see the B3 benchmark in the Test Plan). ### Compaction safety protocol The overview comment "Compaction safety protocol for blob relocation" in db/compaction/compaction_job.h has the full argument. While a compaction runs, indirect blob GC and route retirement can change routes and remove files. No compaction waits for them, and no compaction runs again because of them. Instead, the rules below make sure that every number an output writes can still be read at install. - Why a kept number stays readable: the job's input SSTs hold the number, and they stay in the current Version until the job installs. So the census counts the number as held, and route retirement deletes only routes that no SST holds. GC never removes a route into a relocation input that gets an output (see "Indirect blob GC install and flattening"). - Why a flattened number stays readable: the running-compaction rule in route retirement. See "Route retirement" for why it is needed. - Coarse L0 inputs have no reference list. Until the install, a coarse input is still in the current Version, and its coarse frontier pins every generation number at or above it: route retirement keeps those routes, and indirect blob GC does not select the indirect-origin blob files at or above the frontier. Coarse L0 output does not flatten. The job runs three checks: 1. Input check, before the run (ValidateBlobRelocationInputGenerations()): for each indirect reference group of the exact inputs, it checks the generation number in the job's own input Version with Version::ValidateBlobReferenceGeneration(). The number must be a live indirect-origin blob file, a live relocation file, or a route to a live relocation file. The check follows the route, so one check per group covers both the kept and the flattened number. It uses the per-SST reference summaries that the job read at its start for the census (diff 5), and reads a summary only for an input that has none. A number that the input Version cannot read (for example, its blob file was lost and best_efforts_recovery left it out of the Version) fails the job with Corruption and a background error. The job does not run again: a later run on the same input files cannot do better. 2. Output dependencies, after the run (PrepareBlobRelocationOutputDependencies()): for each indirect reference group of each output SST, the job records the physical file that the number resolves to in the input Version, with that file's metadata object. An output number that resolves to no file is Corruption, except a new indirect-origin blob file that this compaction wrote. The memory is charged to the block cache (CacheEntryRole::kBlobReferenceMetadataBuildingBuffer). Groups of many outputs can name the same generation number, so the job resolves each generation number once, with a set that lives only during the call. 3. Install check (ValidateBlobRelocationOutputDependencies()): it runs in the LogAndApply() precondition, against the current Version, only when the output has a flattened reference or output dependencies. The precondition can run more than once, so the check runs for each new current Version. It skips a file whose metadata object is the same as in the input Version: that file still exists, and its numbers can become unreadable only if a route is deleted, which the rules above prevent. For a changed file, it checks each number again. If a number cannot be read, the install fails with Corruption and a background error, and the compaction does not run again. This should never happen: the check guards the rules above, so that a future bug fails loudly instead of losing data. Because the install passes a precondition, the compaction install is a batch barrier (diff 6): it applies its edit without the DB mutex, runs its precondition twice, and shares no MANIFEST write with other callers. The output dependencies come only from exact outputs, so in a column family with blob indirection this applies to most compaction installs (an estimate). The C1 benchmark in the Test Plan found no measurable cost from this. ### Route retirement - The census keeps, per CF, the set of routed generation numbers that no SST holds (retirement_generations). It is rebuilt once at the end of each census build, and each delta keeps it up to date. Its memory is charged to the block cache. It normally has no more entries than the Version has routes. During a delta it can briefly hold a few more, so the charge uses the larger of the two counts. - Route retirement runs only in the census job, after the census is published (DBImpl::BuildBlobReferenceCensus()). The apply step that a flush or compaction thread runs after its install does not run it, because a MANIFEST write can be slow and can fail. A route deletion installs a new Version, so the census job runs again to catch up. - If an install publishes the census itself while the census job runs the column family, the job runs it once more (ColumnFamilyData::blob_reference_census_rerun_), so no retirement is missed. - Before each census publish, the census worker selects up to 64 such generation numbers (SelectWorkerRetirementSources()). It skips generation numbers at or above the coarse frontier, and generation numbers whose route target is an input of the running indirect blob GC job. It does not know the running compactions; route retirement checks them later. It starts after the last generation number it visited, so generation numbers that stay skipped do not starve the others. - DBImpl::RetireBlobRelocationRoutes() pins the current Version and reads a census snapshot of the selected numbers without the DB mutex. Then, under the DB mutex, it keeps a generation number G only if no SST holds it in the snapshot, it is below the coarse frontier (or there is none), it still has a route, its route target is not an input of the running GC job, and the running-compaction rule does not apply. It writes one MANIFEST edit with the route deletion records. Its comment in db_impl.h has the numbered steps. - Why the running-compaction rule is needed: a compaction may write a number that no installed SST holds yet. In this example, the file numbers are examples only. Compaction C runs with an input Version that has route 12 -> 20, so its output SSTs say 20 (flattened). Then GC moves relocation file 20 into file 30 and writes route 20 -> 30. C has not installed, so no installed SST holds 20, and the census sees route 20 -> 30 as unused. If route retirement deleted it, C would install SSTs that name 20, and 20 would have no file and no route: those values could not be read. So route retirement skips route G -> X while G is a live blob file or relocation file in the input Version of a running compaction. When C installs, its SSTs become holders of 20; a later route retirement deletes the route only after no SST holds 20. - A compaction picked after route retirement pinned its Version has that Version as its input Version (if the current Version changes, the precondition fails). There G is already a route, not a live file, and file numbers are never reused, so it cannot write G. A compaction stays in compactions_in_progress() until its output Version is installed, so the rule covers it until its SSTs are holders. - The running-compaction rule costs one Version file table lookup per selected number and per running compaction whose input Version is not the pinned Version. Route retirement runs it up to 3 times per commit, all under the DB mutex: once when it filters the selected numbers (step 3 in db_impl.h), then twice in the LogAndApply() precondition for the numbers it keeps for deletion, before and after the unlocked apply. With at most 64 selected numbers, this is small (an estimate, not measured). - It skips a route whose target is an input of the running GC job. BeginGcJob() records the job's candidate files, and IsGcInputFile() checks them. The GC install checks the route count of each relocation input, so a route deletion would make the job stale. When the input gets an output, the GC edit re-points such a route to the output, and a later route retirement deletes it if it is still unheld. When the input gets no output, the GC edit deletes the route (see "Indirect blob GC install and flattening"). - Batching: route retirement writes an edit only when one call can delete a full selection of 64 routes (kBlobRouteRetirementBatchSize), or when routes have waited 10 minutes (kBlobRouteRetirementMaxPendingMicros, checked when the census job runs after an install). 64 and 10 minutes are fixed choices, not measured. A pending route costs only Version memory: when no SST uses a relocation file, indirect blob GC removes it as an empty input, and that edit deletes the routes into it. A deferred call still reads the census snapshot, because the number of routes to delete is known only after that read. Pending routes are not written at close; the next census build finds them again after reopen. When no route can be deleted, the pending time is reset. - The edit is a metadata commit: the LogAndApply() precondition calls BeginMetadataCommit(). Before that, the precondition returns TryAgain if the current Version is not the pinned Version, if the census changed, if a route target became a GC input, or if the running-compaction rule now applies to a selected number. With TryAgain, route retirement writes nothing, and the next census publish tries again. - Route retirement is optional cleanup. When it fails (for example, the block cache cannot take the census snapshot charge, or background work is stopped), the census stays ready and the next publish tries again. Only a shutdown, a dropped column family and Corruption stop the census. ### Unknown MANIFEST result of a metadata commit - When the MANIFEST write of route retirement fails with an I/O error, the edit may or may not be on disk. Route retirement sets a background error (kManifestWrite), so MANIFEST recovery decides the result. The failed edit is not applied to the in-memory Version, so the route stays. - Until a new MANIFEST file with the full state is written (by Resume(), or by the next MANIFEST write), BeginGcJob() returns Busy, so no indirect blob GC job starts, and a new metadata commit gets TryAgain. A GC install with an unknown result causes the same state. - Compactions never wait for the commit result itself. The background error that route retirement sets stops background work, as after any failed MANIFEST write, until Resume() or automatic recovery writes a new MANIFEST. No compaction runs again because of the unknown result: both possible results keep every number that a running compaction may write. After a failed MANIFEST write, VersionSet::ProcessManifestWrites() drops the new Versions of the batch and resets the MANIFEST log, so the next write starts a new MANIFEST from the in-memory state, which does not hold the failed edit. ### Indirect blob GC install and flattening - Rule: in the GC edit, a relocation input that gets an output routes every generation number to that output, held or not. This includes the input's own file number and every number routed to it (keeps_all_generations in build_edit). An input without an output still loses its unheld routes, and a normal input still keeps only its live generation numbers. - Why: the GC job takes its census snapshot at its input Version and does not follow later Version changes (diff 9). Most changes add no holder of an input's generation number. Flattening can: a compaction rewrites a reference to the route target in its own input Version, which can be older than the GC job's input Version. For example (the file numbers are examples only), compaction C is picked while route 10 -> 400 exists. An earlier GC job then moves file 400 into file 500, and a new GC job starts on file 500. No SST holds 400, so the census snapshot of the new job shows 400 without a holder. C then rewrites generation number 10 to 400 and installs. Without the rule, the new GC edit would delete route 400 -> 500 and the value would become unreadable. With the rule, it writes route 400 -> output. - Cost: the edit can keep routes that no SST holds. Route retirement deletes them later. - An input without an output had no holder of any of its generation numbers in the snapshot. A compaction that flattens to one of them holds a generation number that routes to the same input, and the snapshot counts that holder. So nothing can flatten to such an input. - The "Install protocol" comment in DBImpl::IndirectBlobGCJob::Install() has the full argument. ### Remote compaction rules With a CompactionService, a feature-aware compaction runs locally in these cases: 1. The input Version has the indirection format, the option is off and compaction-time blob GC is on. The compaction rewrites indirect values. It runs locally as a conservative choice, because remote workers have not been validated for this rewrite. 2. The job overrides enable_blob_garbage_collection or blob_garbage_collection_age_cutoff. The current CompactionService protocol rebuilds these options from the options file. 3. The job has several subcompactions, or it is the first activation compaction (its input Version does not have the indirection format yet). Feature-aware output must be accepted or rejected as a whole, and CompactionService runs each remote subcompaction as a separate job. ### Blob file header version 2 Background: a blob file starts with a 30-byte header (BlobLogHeader::kSize) and ends with a 32-byte footer (BlobLogFooter::kSize). The header has no checksum. Version 2 reuses the two expiration-range fields, because a version 2 file has no TTL. The format version says that the new fields are there; no flag bit says it. ``` Header field (offset) Version 1 Version 2 ------------------------ ------------------------ -------------------------- version (4) 1 2 flags (12) bit 0: has_ttl bit 2: is_indirect_origin expiration field 1 (14) expiration start the file's own number (not 0) expiration field 2 (22) expiration end 0 Footer field Version 1 Version 2 ------------------------ ------------------------ -------------------------- expiration field 1 0 (no TTL) bytes between header and footer expiration field 2 0 (no TTL) masked CRC32C of the header ``` - Writers: BlobFileBuilder writes version 2 when flush, WAL recovery or RepairDB (BuildTable()) runs with the option on, and when compaction runs with the option on or the column family already has the indirection format. is_indirect_origin is set when the option is on. All other blob files keep the version 1 header. - BlobLogHeader::DecodeFrom() accepts versions 1 and 2. It returns Corruption for any other version, for a flag bit that the version does not allow (has_ttl in version 2, is_indirect_origin in version 1, bit 1 always), and, in version 2, for file number 0 or a nonzero second field. - BlobFileReader: for a version 2 file, the file number in the header must equal the number of the file that it opens. Unless footer validation is skipped, the footer must hold the byte count between header and footer and the header CRC. Otherwise it returns Corruption. For a version 1 file, the footer must have an empty expiration range (no TTL). - Nothing else in this diff reads the new fields: flatten and route retirement do not use them. RepairDB in diff 14 reads them to rebuild relocation state. - Compatibility boundary: readers accept version 1 files with the base checks. A RocksDB release without this diff rejects a version 2 header ("Unknown header version") when it opens the file. Version 2 is written only with the option on or when the column family already has the indirection format. Such a DB already has the blob indirection format tags in its MANIFEST (kBlobIndirectionFormatVersion and kDbBlobIndirectionFormatVersion, which are not safe to ignore), so older releases cannot open it, and version 2 adds no new downgrade limit. ### Garbage meter - BlobGarbageMeter skips flattened references (generation number != origin). The census tracks the live bytes of the relocation file they point to. Unflattened references still count toward legacy garbage. ### Monitoring - New key retirement-pending-micros of rocksdb.indirect-blob-gc-stats: how long routes that route retirement can delete have waited for a batch (see "Batching" under "Route retirement"), 0 when no route waits. A written batch, or a call that can delete no route, resets it. - BLOB_RELOCATION_ROUTES_RETIRED (ticker) counts the routes that route retirement removed. - Route retirement runs in the census job, so PauseBackgroundWork() stops it too, and ContinueBackgroundWork() starts it again. - Feature off: only a column family with a census has routes to retire, so the key is 0 and the ticker does not change. ## What changed in the implementation "Parts of this diff" and "How to read this diff" list the files. These changes are not named there: - db/compaction/compaction_outputs.cc: CompactionOutputs::AddToOutput() reuses the BlobIndex decode that the compaction iterator made in FlattenBlobIfNeeded() (CompactionIterator::OutputBlobIndex()), so it does not decode the value again. - db/builder.cc: BuildTable() (flush, WAL recovery, RepairDB) passes allow_blob_index_flattening = false and writes the version 2 header when the option is on. db/flush_job.cc: MemPurge() passes allow_blob_index_flattening = false. - db/compaction/subcompaction_state.h: has_flattened_blob_index tells InstallCompactionResults() that the output has a flattened reference. - db/db_impl/db_impl_compaction_flush.cc: BeginGcJob() is called under the DB mutex with the job's candidate files, so a route retirement that has not reached the MANIFEST queue head sees them in its precondition. BackgroundCompaction() has a debug-only sync point after the compaction run, for tests. ## Code whose first users come later - The loss hooks (BlobFetcher::IsBlobValueLost() and HasBlobValueLoss()) and the lost-value handling in the compaction iterator become active in diff 13. - The SST-only output policy (AllowBlobFileCreation() and SetSstOnlyBlobOutputPolicy()) gets its first override in diff 11. ## Safety boundaries and unsupported cases - Public API: new ticker BLOB_RELOCATION_ROUTES_RETIRED, added just before TICKER_ENUM_MAX. The new ticker has no Java mapping, as for recent base tickers: all 256 byte values of org.rocksdb.TickerType are used. - An indirect blob GC job cannot start while route retirement writes its MANIFEST edit: route retirement holds a metadata commit, and BeginGcJob() returns TryAgain during it. Route retirement also checks the GC inputs again at the MANIFEST queue head, so a GC job that registers later and takes a route target as input makes route retirement end with TryAgain. - Generation numbers at or above the coarse frontier keep their routes, including the file numbers of old relocation files, which a coarse SST can never hold. Such routes wait until the frontier moves past them. This only delays route retirement. - A route whose number is a live file in the input Version of a running compaction waits until that compaction ends. A long compaction only delays route retirement. - The install check and the input check fail the compaction with Corruption and a background error; there is no internal retry. The input check fails when a number is already unreadable in the job's own input Version. The install check guards the protocol invariants and should never fail. - A coarse, unknown or unready state never proves that a generation number is unused. The census includes keys hidden by newer versions and wide-column blob references. - Route retirement checks the 10-minute wait only when the census job runs after an install. So in an idle column family, waiting routes stay until the next install. They cost only Version memory, because indirect blob GC still removes a relocation file that no SST uses. - Blob files with a version 2 header cannot be opened by RocksDB releases without this diff (see "Blob file header version 2"). ## Option gate At this diff, ColumnFamilyData::ValidateOptions rejects enable_blob_indirection outside tests (the option gate: diff 2 adds it, diff 14 removes it). The behavior described here with the option on is reachable only through the internal test hook ColumnFamilyData::TEST_SetAllowBlobIndirection, not in production. ## Stack context (same in diffs 2-14, except that diffs 10-14 add the diff 10 route rule to the install line; "<== this diff" marks this diff's part): Goal: remove garbage from blob files without rewriting any SST. A value gets a stable identity when it is first written. Indirect blob GC copies the live values of whole files into a relocation file and installs routes in the MANIFEST. Readers follow the routes. SSTs are not changed. How a read finds a value (P is the relocation file that holds the value now): ``` SST entry indirect BlobIndex { generation number, origin, original offset, size, CRCs } | v Version file table: generation number -> live file, or route old file -> P | | live file route found | | v v blob file: read at original offset relocation file P (always one hop): relocation file: look up key look up key (origin, original offset) (origin, original offset) ``` Lifecycle of one value (in this example, F is the blob file where the value is first written, and P is the relocation file that GC writes): ``` write new value goes to blob file F; index: origin = F, generation number = F | census per generation number: the SSTs and references that use it; | per physical file: live bytes, ranked by garbage ratio GC job copy the live values of whole input files (stored bytes, checked | with the stored CRC) into relocation file P, keyed by | (origin, original offset) install one MANIFEST edit: add P, route each input generation number still | in use to P (a relocation input that gets an output keeps a route | for all its generation numbers), re-point the routes into the | inputs to P, and remove each input (blob file retirement for a | blob file, relocation file deletion for a relocation file) flatten compaction sets generation number = P in the SSTs it rewrites | route route retirement: route deletion records remove the routes that delete no SST uses ``` Stack map: | Diff | Adds | |---|---| | 1 | Finalize blob metadata outside the DB mutex (prep) | | 2 | Indirect BlobIndex (kIndirectBlob) and exact SST reference metadata | | 3 | Relocation file format and stable read path | | 4 | MANIFEST routes and relocation state in the Version | | 5 | Exact reverse reference census | | 6 | MANIFEST apply outside the DB mutex, with a precondition re-check | | 7 | Coarse flush metadata, per-SST reference cap, epoch runs | | 8 | Indirect blob GC building blocks | | 9 | Indirect blob GC job | | 10 | Flatten generation numbers; route retirement <== this diff | | 11 | Remote compaction validation | | 12 | Dynamic options and operations | | 13 | Lost relocation files | | 14 | RepairDB rebuilds relocation state; option gate removed | | 15 | db_stress and crash test support | ## Stack context: terms for later diffs | Term | Meaning | |---|---| | lost value | A value whose blob file is known to be lost. The hooks in this diff report no loss (diff 13) | | SST-only output policy | A compaction that may not create blob files ends with TryAgain when a value needs blob-file output (diff 11) | Tracking issue: meta#15301 Differential Revision: D124905374
Summary: Pull Request resolved: meta#15548 ## Why the change is required With CompactionService, a remote worker runs the compaction and the primary installs its output. Before the primary installs output of a column family that uses blob indirection, it must know that the worker used the same stable identities, the same routes, the same metadata layout and the same output placement rules. Otherwise the primary could install SSTs whose blob references it cannot read. ## Overview This diff adds a contract between the primary and the remote worker for compactions of a column family that uses blob indirection (feature-aware jobs). The primary sends three values that describe its options, its format versions and the relocation files and routes of the job's input Version. The worker runs the job only when its own values match and its Version has the format fence, and it never creates blob files. The primary checks the result before it imports the output SSTs, and checks each SST again after the import: it reads and checks all of the SST's exact reference metadata, and it accepts only generation numbers that the job's inputs hold or route to. When any check fails, the primary runs the job locally. Compactions of other column families (legacy jobs) keep the existing remote BlobDB protocol, with one new check: the primary declines a legacy result that holds blob indirection metadata. ## Parts of this diff 1. Core: db/compaction/compaction_service_job.cc has the primary checks (ValidateBlobGCCompactionServiceResult(), the garbage filter, CompactionJob::ValidateRemoteExactBlobMetadata() and CompactionJob::PrepareRemoteAllowedBlobGenerations()) and the options fingerprint. 2. db/compaction/compaction_job.h: the contract fields of CompactionServiceInput and CompactionServiceResult, the allowed generation numbers of a job, and the overrides of the worker job (CompactionServiceCompactionJob). 3. db/db_impl/db_impl_secondary.cc: the worker's contract check. 4. db/version_set.{h,cc}: Version::ValidateBlobReferenceGroup(), the origin rule. 5. table/table_reader.h and table/block_based/block_based_table_reader.{h,cc}: TableReader::VerifyBlobReferenceMetadata(), the full check of one SST's exact reference metadata. 6. db/table_cache.h: friend access for CompactionJob. 7. Comment updates in db/blob/blob_reference_metadata.h and db/compaction/compaction_iterator.h, and a test hook in db/compaction/compaction_outputs.cc. Safe to skip on a first read. ## How to read this diff 1. Read "Terms used in this diff" (the next section). Then read "New behavior", which starts with a diagram of the remote flow, and "Safety boundaries". 2. Read Version::ValidateBlobReferenceGroup() in db/version_set.h and db/version_set.cc. It depends on nothing else in this diff. 3. Read TableReader::VerifyBlobReferenceMetadata() in table/table_reader.h, then its implementation in table/block_based/block_based_table_reader.cc. It also depends on nothing else in this diff. 4. Read the comment at the new CompactionServiceInput fields in db/compaction/compaction_job.h, then ComputeBlobGCCompactionServiceOptionsFingerprint() in db/compaction/compaction_service_job.cc. 5. Read the worker: the contract check in DBImplSecondary::CompactWithoutInstallation() (db/db_impl/db_impl_secondary.cc), the TryAgain branch at the end of that function, then AllowBlobFileCreation() and AllowBlobIndexFlattening() of CompactionServiceCompactionJob. The SST-only decline itself comes from diff 10. 6. Read ValidateBlobGCCompactionServiceResult() in db/compaction/compaction_service_job.cc. 7. Read the per-file check: the friend change in db/table_cache.h, the comment at ValidateRemoteExactBlobMetadata() in db/compaction/compaction_job.h, then PrepareRemoteAllowedBlobGenerations() and ValidateRemoteExactBlobMetadata() in db/compaction/compaction_service_job.cc. The safety comment above ValidateBlobRelocationInputGenerations() in compaction_job.h (diff 10) explains the route retirement rule that the allowed generation numbers rely on. 8. Read CompactionJob::ProcessKeyValueCompactionWithCompactionService() in this order: the feature_aware_compaction flag and the coarse-output local path, the TryAgain handling, the call to step 6, the garbage filter, then the per-file check in the import loop and fall_back_after_validation. Last, read the tests in db/compaction/compaction_service_test.cc. Each test name says which path it covers. ## Terms used in this diff Stack terms: | Term | Meaning | |---|---| | origin | Blob file where the value was first written | | original offset | Value offset in the origin file | | stable identity | (origin, original offset). Never changes, even when the value moves | | generation number | The file number that a reference uses to find its value. A new value has generation number == origin. When indirect blob GC moves the value, the reference keeps its generation number, and a route sends the read to the relocation file | | unflattened reference | Index with generation number == origin (value not relocated, or not flattened yet) | | flattened reference | Index with generation number != origin: compaction rewrote the generation number to the relocation file that holds the value | | flatten | Compaction rewrites the generation number to the relocation file that holds the value, so old routes can go. Only flatten makes generation number != origin | | route | Old file number -> the relocation file that now holds its live values. One hop: a target is never itself routed. Stored in the MANIFEST | | route target | The relocation file that a route points to | | name | In code and comments, the generation number of a reference (its key in the Version file table) | | Version file table | The table in a Version that maps a generation number to a live file or to a route | | physical file | A file that holds value bytes: a normal blob file or a relocation file. A generation number can name a physical file, or it can be only the key of a route | | indirect origin | Normal blob file written with the option on; it is the origin of unflattened references | | relocation file | Block-based table keyed by (origin, original offset) that holds copied values. Each value is the stored bytes plus a 5-byte trailer (kSimpleGen2BlobTrailerSize) | | stored CRC | CRC32C of the stored bytes plus the compression byte | | logical CRC | CRC32C of the uncompressed value (BlobReference::stable_checksum) | | exact reference metadata | Per-SST sorted list of every blob reference the SST keeps, written when the SST is built. References with the same encoding class, generation number and origin form a group. The list is stored as data blocks, with an index of the groups. An SST that has it is an exact SST | | coarse SST | An SST without exact reference metadata. It records one frontier file number and pins every indirect-origin blob file at or above it. Frontier 0 means no coarse frontier | | census | Background count, per generation number, of the SSTs and references that use it, plus live bytes per physical file | | blob file retirement | MANIFEST record (kBlobFileRetirement) that removes a collected blob file from the Version | | route deletion | MANIFEST record that removes a route that no SST uses. Route retirement is the background job that writes these records | | running-compaction route rule | Route retirement skips route G -> X while G is a live blob file or relocation file in the input Version of any running compaction (diff 10, DBImpl::RetireBlobRelocationRoutes()) | | install check | At install, ValidateBlobRelocationOutputDependencies() (diff 10) re-checks each generation number of the output whose physical file has a changed file metadata object since the input Version | | format fence | A MANIFEST record that marks a column family (per-CF fence) or the whole DB (DB-global fence) as possibly holding blob indirection data. It never clears. See "MANIFEST records" in db/blob/blob_relocation_manifest.h | | SST-only output policy | A compaction that may not create blob files ends with TryAgain when a value needs blob-file output (diff 10, CompactionIterator::DeclineSstOnlyBlobOutput()) | "The option" is enable_blob_indirection. Terms of this diff: | Term | Meaning | |---|---| | feature-aware job | A compaction whose column family has the per-CF format fence (VersionStorageInfo::HasBlobIndirectionFormat()) or has the option on. Its remote run uses the contract of this diff | | legacy job | Any other compaction. It uses the existing remote BlobDB protocol | | contract values | The three values that a feature-aware request carries: the protocol version, the options fingerprint and the relocation state fingerprint | | options fingerprint | A 64-bit hash (ComputeBlobGCCompactionServiceOptionsFingerprint()) of the options and format versions that the worker must share with the primary. The comment at the CompactionServiceInput fields lists them | | relocation state fingerprint | A 128-bit hash of the relocation files, routes and format version of a Version (VersionStorageInfo::ComputeBlobRelocationStateFingerprint()). The descriptor fields route_snapshot_fingerprint_high and _low hold the same value | | decline | The worker or the primary refuses a remote attempt, and the primary runs the job locally (kUseLocal). The worker declines with an empty result (protocol version 0), or with TryAgain inside an OK result | | feature metadata | An exact descriptor or a coarse classification property on an output SST, or an indirect-origin blob file | | per-file check | The checks that the primary runs on each output SST after it imports the SST (ValidateRemoteExactBlobMetadata()) | | allowed generation numbers | The generation numbers of the indirect reference groups of the job's exact inputs, plus the route target of each of them in the input Version (CompactionJob::remote_allowed_blob_generations_). See "Allowed generation numbers" below | | kFailure | The CompactionServiceJobStatus that marks the compaction as failed. The primary does not run the job locally, and the error goes to the normal compaction error path | "blob_gc" in the new field and function names means the blob indirection feature of this stack, not legacy blob garbage collection. ## Previous behavior CompactionService had no versioned contract for blob indirection. A worker could return output without the exact reference metadata that the primary needs, or with generation numbers that the primary never checked. Legacy remote compaction already supports BlobDB: the primary imports the worker's blob file output, blob garbage and oldest_blob_file_number, and it removes imported files again when the install fails. The primary already declines a result that reports garbage for a blob file that is not in the input Version (for example, a blob file that GC already collected). ## New behavior ### Flow ``` primary picks a compaction | +-- legacy job ----------------> existing remote BlobDB protocol; the | primary declines a result with feature | metadata -> run locally feature-aware job | local-only case (see Which jobs) or coarse output? ----------------- yes --> run locally | no request with the contract values | worker: contract checks pass? ------ no ---> empty result -> run locally | yes worker: SST-only run; new blob file output needed? --------------- yes --> TryAgain in OK result | no -> run locally primary: check before the import --- fail -> run locally | primary: drop garbage of files that have no garbage counters | primary: import each SST, then the per-file check ---------------- fail -> delete the imported files, | run locally install (MANIFEST write) ``` Every failed contract or output check runs the job locally. Other import errors (for example a failed rename) keep their existing kFailure handling. If the cleanup after a failed per-file check fails, the job also fails with kFailure. ### Which jobs - A job is feature-aware when its column family has the per-CF format fence or has the option on. Legacy jobs use the existing remote BlobDB protocol unchanged, except for the feature metadata check below. - A feature-aware job whose output would be coarse (output level 0, the compaction is not bottommost, and no input has exact metadata; for example an intra-L0 compaction above older overlapping data, or a universal compaction that writes to level 0 and is not bottommost) runs locally from the start. The primary accepts only exact feature output, so a worker would do the work for nothing. - CompactionJob::ShouldUseLocalCompaction() (diff 10) also keeps these feature-aware jobs local: - The first activation compaction. Its input Version has no format fence yet, and the worker requires one. - A job with a manual override of enable_blob_garbage_collection or blob_garbage_collection_age_cutoff. The worker rebuilds these options from the OPTIONS file. - A job with several subcompactions. Feature-aware output must be accepted or rejected as a whole, and CompactionService runs each remote subcompaction as a separate job. - A job with the option off and blob GC on, in a CF with the format fence. It rewrites indirect values into legacy blob files or inline values. It runs locally as a conservative choice, because remote workers have not been validated for this rewrite. ### Request - A feature-aware request carries the three contract values: the protocol version (kBlobGCCompactionServiceProtocolVersion, value 1), the options fingerprint, and the relocation state fingerprint of the input Version. - The options fingerprint covers the options that change what the worker writes (enable_blob_indirection, which is also the flattening permission, enable_blob_garbage_collection and blob_garbage_collection_age_cutoff), the column family ID, the options file number, and the format versions of every on-disk part this feature writes. The comment at the CompactionServiceInput fields in compaction_job.h has the full list. The age cutoff is hashed after the same text round trip that the OPTIONS file uses, so a value such as 1/3 gives the same hash on both sides. - The worker declines the job when the protocol version differs, when its Version has no per-CF format fence, when its options fingerprint differs, or when its relocation state fingerprint differs. It returns an empty result, whose protocol version is 0, so the primary runs the job locally before it imports anything. ### Worker - A feature-aware worker never creates blob files (CompactionServiceCompactionJob::AllowBlobFileCreation() is false), so the primary reserves no blob file numbers for it. This turns on the SST-only output policy from diff 10. When a value, a merge result or legacy blob GC needs new blob output, the worker declines the whole attempt. It returns OK with TryAgain in its result, so a CompactionService that maps a failed worker call to kFailure still forwards the decline, and no background error is set. SST-only output is accepted when no new blob output is needed. - The worker flattens when its options allow it (CompactionServiceCompactionJob::AllowBlobIndexFlattening()). The result returns the worker's protocol version and whether it flattened references (has_flattened_blob_index). - Resumable compaction progress is off for feature-aware jobs, because the progress format does not store the contract. ### Check before the import ValidateBlobGCCompactionServiceResult() runs before any output is imported. - A legacy job whose worker returns feature metadata is declined and runs locally. This happens in two cases: 1. The worker's OPTIONS file has the option on, and the CF still has no format fence. Installing the worker's feature output would fail with Corruption, a background error that stops writes, because a Version without the fence rejects exact metadata. The local run writes legacy output, which installs. 2. The format fence is added after the job is built but before the worker opens the DB. The legacy install path does not check feature metadata (the fingerprint, the origins and the allowed generation numbers), so the primary declines the result instead of installing it unchecked. At this diff, the local run also fails to install (see "Validation and known limitations"). - A feature-aware result is declined when it has another protocol version, flattening that the primary's options do not allow, any blob file addition, or another output level. - In a feature-aware result, each output must have a valid SST file name and report an exact metadata descriptor that has repair metadata (the CF ID and the build SST file number that RepairDB needs; BlobReferenceMetadataDescriptor::HasRepairMetadata()), the primary's CF ID, its own SST file number, and the request's relocation state fingerprint. ### Garbage filter In a feature-aware result, the primary drops the garbage that the worker reports for a file that has no garbage counters in the Version: a routed generation number (a blob file that indirect blob GC collected, or a deleted relocation file), or a live relocation file. Local compaction drops the same garbage. The comment at the filter in compaction_service_job.cc has an example. ### Per-file check Inside the existing remote import loop, ValidateRemoteExactBlobMetadata() checks each imported SST in these steps: 1. The descriptor stored in the imported file must equal the reported one. 2. The file's real CF ID (from its table properties) must be the primary's, and the descriptor must name the same CF ID and the file's original file number. 3. TableReader::VerifyBlobReferenceMetadata() reads and checks all of the file's exact reference metadata (see the next section). 4. The primary reads the reference summary (the list of groups) of the file. The same short-lived reader serves steps 3 and 4. 5. For every reference group that is not kDirect (kDirect groups hold legacy references), Version::ValidateBlobReferenceGroup() checks the generation number and the origin against the captured input Version (its comment has the origin rules). The generation number must also be an allowed generation number, or a number that a coarse input SST of this job pins (see rule 3 in "Allowed generation numbers"). A failed check deletes the outputs that were already imported (existing cleanup helpers), releases the table readers that the check opened, and runs the job locally. Each imported SST is recorded as an output path, so the cleanup of an aborted compaction (CompactionJob::CleanupAbortedSubcompactions()) deletes it. ### Full metadata check (VerifyBlobReferenceMetadata) Where it runs: on the primary, in step 3 of the per-file check, after the SST is imported and before it is added to the compaction outputs. It runs without the DB mutex. TableReader has a default that returns NotSupported; BlockBasedTable implements it. What it checks: it reads the index block of the reference metadata and decodes it against the expected descriptor, which checks the index checksum and the descriptor totals. Then it reads every data block in file order, including the blocks of legacy kDirect references, and checks each block checksum, the key order across runs and blocks, and every group total. It does not read the table's key/value entries. The reads use readahead that starts at 256 KB and grows to 1 MB (estimates, not tuned), do not fill the block cache, and use the compaction's rate limiter priority. Why: the reference summary read in step 4 and the census read only the index block. A worker bug or a damaged file could leave bad data blocks that no install-time read sees. Indirect blob GC reads these data blocks later to find the live values. If it found the damage only then, the compaction inputs would already be gone. Checking at import lets the primary run the job locally from its intact inputs instead. A local compaction does not run this check, because the primary built that metadata itself. ### Allowed generation numbers Background: the worker is a separate process, and the primary does not trust it to write only safe generation numbers. A generation number in an output is safe only if its file or route cannot go away between the pick and the install. The install check does not cover all cases: it re-checks a number only when the file metadata of its physical file changed, so it does not see a route that was deleted while its target file stayed the same. Rule: the primary accepts an indirect group of a remote output only when its generation number is one of these: 1. A generation number that an indirect reference group of the job's exact inputs holds. The input SSTs stay in the current Version until the install, so the census still counts them, and route retirement keeps the route of such a number. 2. The route target of such a number in the input Version. The target is a live relocation file in the input Version, so the running-compaction route rule keeps any route from it that GC adds during the job. The worker flattens only to these targets: it runs only when its relocation state fingerprint equals the one of the primary's input Version, so it sees the same routes. 3. A number that a coarse input SST of this job pins (checked directly, not stored in the list). A coarse SST has no reference groups, so the primary cannot list its numbers, and it accepts any number that such an SST could hold. A coarse reference has generation number == origin and names an indirect-origin blob file at or above the frontier of its SST. So the primary accepts such a number only when all of these hold: it is at or above the lowest frontier of the job's own coarse inputs (remote_coarse_input_frontier_), it equals the group's origin, it has no route, it is not a relocation file, and it is a live blob file in the input Version with BlobFileMetaData::IsIndirectOrigin(). The frontier comes from the job's inputs only, not from the whole input Version, because an SST outside the job can be removed while the job runs. The coarse input stays in the current Version until the install, so the frontier of the current Version stays at or below the number. Route retirement and GC use that frontier, so the blob file is not collected while the job runs. A number outside these cases, for example a routed number that no input holds, could lose its route while the job runs, and the install check would not notice. The installed SST would then hold a reference that no read can resolve. So the primary rejects such an output and runs the job locally. A correct worker never writes such a number, because compaction only copies the generation numbers of its inputs or flattens them to their route targets. How the list is built: on the first per-file check of a job, PrepareRemoteAllowedBlobGenerations() visits the indirect groups of the exact inputs (ForEachIndirectInputReferenceGroup(), which reuses the input summaries that were read for the census, and reads a missing one). The list then stays fixed for the rest of the job. It is a sorted vector without duplicates, searched with binary search. While it is filled, each time the vector is full it drops duplicates, and it doubles its capacity (to at least 16 numbers, a small start size) only when it is still at least half full after that. This way it does not sort the whole list on almost every push. Most groups of one job share their generation numbers, so the list stays near the number of distinct numbers. Each growth is charged to the block cache (CacheEntryRole::kBlobReferenceMetadataBuildingBuffer, 8 bytes per number) through ReserveBlobRelocationDependencyMemory(), together with the output dependencies of the install check. If the charge fails, the job runs locally. ### Other - The checks open the imported SST with the compaction's file options (file_options_). They read the output's table properties through the shared table cache, and release those readers when the job falls back. The exact metadata reads use short-lived readers that do not insert table metadata into the shared block cache. ### Monitoring - REMOTE_COMPACT_COARSE_FALLBACK_COUNT (ticker) counts feature-aware compactions that run locally from the start because their output would be coarse (see "Which jobs"). Each one also has an INFO line. - REMOTE_COMPACT_OUTPUT_BLOB_REFERENCES (histogram) records the reference count of each remote output SST that the primary checks before the install. - The base ticker REMOTE_COMPACT_BLOB_RESULT_DECLINED_COUNT also counts two new declines, each with a WARN line: the blob GC contract check of the result (ValidateBlobGCCompactionServiceResult()) fails, or the check of an output file fails. After the second one, the primary removes the output files it already copied and runs the job locally. - Feature off: only feature-aware jobs reach these paths. ## What changed in the implementation "Parts of this diff" lists the files. These changes are not named there: - db/compaction/compaction_service_job.cc: the four new request fields and the two new result fields are in the OptionTypeInfo tables of CompactionServiceInput and CompactionServiceResult, next to the existing remote BlobDB fields. ProcessKeyValueCompactionWithCompactionService() returns kFailure when the compaction has no input Version, column family or table cache. - db/db_impl/db_impl_secondary.cc: CompactWithoutInstallation() returns InvalidArgument for a null result, and Corruption when the column family has no current Version. - table/block_based/block_based_table_reader.cc: the metadata block read takes an optional prefetch buffer, which only the full metadata check uses. - db/compaction/compaction_job_test.cc: the serialization test covers the new fields. - db/compaction/compaction_outputs.cc: a test hook (sync point CompactionOutputs::AddToOutput:Value) can replace the output value, so that tests can build malformed remote compaction output. When the hook replaced the value, AddToOutput() decodes the new value instead of using the compaction iterator's decode. ## Safety boundaries and unsupported cases - Public API: new ticker REMOTE_COMPACT_COARSE_FALLBACK_COUNT and new histogram REMOTE_COMPACT_OUTPUT_BLOB_REFERENCES, appended before TICKER_ENUM_MAX and HISTOGRAM_ENUM_MAX. The Java API maps the histogram (HistogramType and portal.h). The new ticker has no Java mapping, as for recent base tickers: all 256 byte values of org.rocksdb.TickerType are used. - The worker is trusted to list the references it writes correctly. The primary checks that the list is well formed and that every group is allowed, but it does not scan every key again. - A relocation file's MANIFEST record holds no list of its origins, so the group check cannot prove that a relocation file holds a given origin. A read checks the stored key (origin, original offset) instead. - Feature-aware jobs never use remote blob file output: a job that needs new blob output runs locally. Remote indirect blob GC is not supported. So in a column family with the option on that still has legacy blob data, with compaction-time blob GC on (or after lowering min_blob_size), the remote compactions that need blob output all run locally. - An interrupted feature-aware job restarts from the beginning. - Serialized fields: older peers ignore unknown fields. A worker that does not return the protocol version makes the primary run the job locally. A 64-bit options fingerprint collision is the only case that the worker check would miss; each output's descriptor is still checked. - The TryAgain decline reaches the primary only if the CompactionService returns the serialized result. If Wait() reports kFailure without a readable result, the normal remote failure path applies instead of the local run. - The worker declines a job when the relocation state fingerprint of its Version differs from the request. So any GC install or route retirement between the pick and the worker's open makes the job run locally. With continuous GC, many remote jobs may run locally. - The full metadata check reads all reference data blocks of each remote output once. This adds I/O on the primary in proportion to the size of the metadata, not to the size of the SST data. ## Validation and known limitations The Test Plan lists the tests. Known limitation at this diff: if the format fence is added after a legacy job is built, the primary declines the worker's feature output, and the local run writes an SST without exact reference metadata. The install of that SST fails with Corruption, because a Version with the format fence rejects such an SST. This sets a background error. Diff 12 closes this case: the MANIFEST writer marks such local outputs as legacy SSTs, and the fenced Version accepts them. ## Option gate At this diff, ColumnFamilyData::ValidateOptions rejects enable_blob_indirection outside tests (the option gate: diff 2 adds it, diff 14 removes it). The behavior described here with the option on is reachable only through the internal test hook ColumnFamilyData::TEST_SetAllowBlobIndirection, not in production. ## Stack context (same in diffs 2-14, except that diffs 10-14 add the diff 10 route rule to the install line; "<== this diff" marks this diff's part): Goal: remove garbage from blob files without rewriting any SST. A value gets a stable identity when it is first written. Indirect blob GC copies the live values of whole files into a relocation file and installs routes in the MANIFEST. Readers follow the routes. SSTs are not changed. How a read finds a value (P is the relocation file that holds the value now): ``` SST entry indirect BlobIndex { generation number, origin, original offset, size, CRCs } | v Version file table: generation number -> live file, or route old file -> P | | live file route found | | v v blob file: read at original offset relocation file P (always one hop): relocation file: look up key look up key (origin, original offset) (origin, original offset) ``` Lifecycle of one value (in this example, F is the blob file where the value is first written, and P is the relocation file that GC writes): ``` write new value goes to blob file F; index: origin = F, generation number = F | census per generation number: the SSTs and references that use it; | per physical file: live bytes, ranked by garbage ratio GC job copy the live values of whole input files (stored bytes, checked | with the stored CRC) into relocation file P, keyed by | (origin, original offset) install one MANIFEST edit: add P, route each input generation number still | in use to P (a relocation input that gets an output keeps a route | for all its generation numbers), re-point the routes into the | inputs to P, and remove each input (blob file retirement for a | blob file, relocation file deletion for a relocation file) flatten compaction sets generation number = P in the SSTs it rewrites | route route retirement: route deletion records remove the routes that delete no SST uses ``` Stack map: | Diff | Adds | |---|---| | 1 | Finalize blob metadata outside the DB mutex (prep) | | 2 | Indirect BlobIndex (kIndirectBlob) and exact SST reference metadata | | 3 | Relocation file format and stable read path | | 4 | MANIFEST routes and relocation state in the Version | | 5 | Exact reverse reference census | | 6 | MANIFEST apply outside the DB mutex, with a precondition re-check | | 7 | Coarse flush metadata, per-SST reference cap, epoch runs | | 8 | Indirect blob GC building blocks | | 9 | Indirect blob GC job | | 10 | Flatten generation numbers; route retirement | | 11 | Remote compaction validation <== this diff | | 12 | Dynamic options and operations | | 13 | Lost relocation files | | 14 | RepairDB rebuilds relocation state; option gate removed | | 15 | db_stress and crash test support | ## Stack context: terms for later diffs | Term | Meaning | |---|---| | legacy SST | An SST in a fenced CF that has no exact metadata and no coarse frontier, because it was created before the CF used the option. The MANIFEST marks it, so that the fenced Version accepts it (diff 12) | Tracking issue: meta#15301 Differential Revision: D124930422
Summary: Pull Request resolved: meta#15549 ## Why the change is required enable_blob_indirection can be changed with SetOptions(). A user can turn it on, turn it off and turn it on again while flushes, compactions and ingestions run. Data that was written while the option was on must stay readable after it is turned off. The files and metadata of such data must also stay correct in the maintenance APIs: metadata listing, ldb, export, import and ingestion. This diff makes the option a runtime policy. The option decides how new output is written. It is not a read gate. The diff tracks flushes and compactions that may still write output with the option on, checks every writer against that state, and makes the maintenance APIs handle relocation files and fenced column families. ## Parts of this diff 1. Pending activation tracking (db/column_family.h, db/memtable_list.{h,cc}, db/flush_job.cc, db/compaction/compaction.cc, db/db_impl/db_impl_compaction_flush.cc). Most other parts read this state. 2. Outputs made before activation (db/version_edit.{h,cc}, db/version_set.cc, db/compaction/compaction_job.cc, db/compaction/subcompaction_state.h): an output SST that may have been written before the column family used the option is marked. If it has no exact reference metadata and the MANIFEST writer sees or sets the fence, the writer classifies it as legacy. 3. Format checks (db/column_family.cc, db/db_impl/db_impl_open.cc, db/version_set.cc): SetOptions(), DB open and the MANIFEST writer reject a table format or compaction style that a fenced or pending column family cannot use. DB open also rejects blob direct write on a fenced column family. 4. Ingestion and import checks (db/external_sst_file_ingestion_job.{h,cc}, RejectBlobIndirectionTableProperties() in db/builder.h, db/import_column_family_job.cc). 5. Maintenance APIs (db/db_impl/db_impl.cc, utilities/checkpoint/checkpoint_impl.cc, include/rocksdb/utilities/checkpoint.h, tools/ldb_cmd.cc). 6. Census work on activation (db/version_set.cc, db/db_impl/db_impl.cc): the batch that activates a column family reserves a census install for it, and SetOptions() and CreateColumnFamily() schedule census work. 7. Removal of files with no live value while indirect blob GC cannot run normally (db/column_family.cc, db/db_impl/db_impl_compaction_flush.cc, include/rocksdb/advanced_options.h). 8. Ingestion can join an earlier MANIFEST batch (LogAndApplyWithJoinablePrecondition() in db/version_set.{h,cc}). A joined ingestion shares the MANIFEST write and sync of that batch instead of starting its own batch. It is only for throughput, and it is safe to skip on a first read. ## How to read this diff 1. Read "Terms used in this diff" near the top of this summary. Then read the format fence entries in the "MANIFEST records" section of db/blob/blob_relocation_manifest.h. The per-CF fence is read with VersionStorageInfo::HasBlobIndirectionFormat(). 2. Pending activation tracking. It depends on no other part of this diff, and most later checks read it. Read the "Pending activation" comment and HasPendingBlobIndirectionOutput() in db/column_family.h, then MemTableList::BeginBlobIndirectionActivation() and EndBlobIndirectionActivation() in db/memtable_list.{h,cc}. Then read the callers: db/flush_job.cc, db/compaction/compaction.cc, and FlushMemTablesToOutputFiles() and AtomicFlushMemTablesToOutputFiles() in db/db_impl/db_impl_compaction_flush.cc. 3. Outputs made before activation. Read FileMetaData::blob_reference_output_predates_indirection in db/version_edit.h and VersionEdit::MarkBlobReferenceOutputsPredatingIndirectionLegacy() in db/version_edit.cc. Then read where the flag is set: db/flush_job.cc, db/compaction/compaction_job.cc with SubcompactionState::AddOutputsEdit(), CopyTrivialCompactionFiles(), and ingestion. Then read how LogAndApplyImpl() in db/version_set.cc sets has_predating_blob_reference_output and activates_blob_indirection on each ManifestWriter. Last, read the classification block in ProcessManifestWrites() (MarkPredatingBlobReferenceOutputsUnlocked). 4. Format checks. Read ValidateBlobIndirectionOptionsAfterRecovery() (called from db/db_impl/db_impl_open.cc) and the new checks in PrepareMutableCFOptions() in db/column_family.cc. Then read the per-edit check in ProcessManifestWrites() ("SetOptions may check the options before"). 5. Ingestion and import checks. Read the "Blob indirection checks of an ingestion" comment in db/external_sst_file_ingestion_job.h, the Prepare() scan in the .cc file, and the SetBlobIndirectionPendingAtPrepare() calls in db/db_impl/db_impl.cc. Then read RejectBlobIndirectionTableProperties() in db/builder.h, which import also uses (db/import_column_family_job.cc). 6. Leaf parts, in any order. Maintenance APIs: GetColumnFamilyMetaData(), GetAllColumnFamilyMetaData() and GetColumnFamilyMetaDataForExport() in db/db_impl/db_impl.cc, ExportColumnFamily() in utilities/checkpoint/checkpoint_impl.cc, and DumpBlobFile() in tools/ldb_cmd.cc. Removal of files with no live value: NeedsCompaction() in db/column_family.cc and empty_files_only in BackgroundCompaction(). Census: ReserveBlobReferenceCensusInstalls() in db/version_set.cc, and the ScheduleBlobReferenceCensus() calls after SetOptions() and CreateColumnFamily() in db/db_impl/db_impl.cc. 7. Batch joining (throughput only). Read LogAndApplyWithJoinablePrecondition() in db/version_set.h and its caller in db/db_impl/db_impl.cc. Then read the ManifestWriter fields, ManifestBatchState and the numbered rules in VersionSet::ManifestBatchEndsBefore() in db/version_set.cc. Then read "Details: MANIFEST batching" near the end of this summary. 8. Tests: db/blob/db_blob_basic_test.cc (toggles, pending activation, import, census, removal of files with no live value), db/external_sst_file_test.cc (ingestion and batching), db/version_set_test.cc (batch barriers), tools/ldb_cmd_test.cc (relocation file dump), db/compaction/compaction_service_test.cc (a legacy remote job after the fence), db/memtable_list_test.cc (a pending blob indirection activation in the memtable list). ## Terms used in this diff Stack terms: | Term | Meaning | |---|---| | origin | Blob file where the value was first written | | original offset | Value offset in the origin file | | stable identity | (origin, original offset). Never changes, even when the value moves | | generation number | The file number that a reference uses to find its value. A new value has generation number == origin. When indirect blob GC moves the value, the reference keeps its generation number, and a route sends the read to the relocation file | | unflattened reference | Index with generation number == origin (value not relocated, or not flattened yet) | | flattened reference | Index with generation number != origin: compaction rewrote the generation number to the relocation file that holds the value | | flatten | Compaction rewrites the generation number to the relocation file that holds the value, so old routes can go. Only flatten makes generation number != origin | | route | Old file number -> the relocation file that now holds its live values. One hop: a target is never itself routed. Stored in the MANIFEST | | Version file table | The table in a Version that maps a generation number to a live file or to a route | | physical file | A file that holds value bytes: a normal blob file or a relocation file. A generation number can name a physical file, or it can be only the key of a route | | indirect origin | Normal blob file written with the option on; it is the origin of unflattened references | | relocation file | Block-based table keyed by (origin, original offset) that holds copied values. Each value is the stored bytes plus a 5-byte trailer (kSimpleGen2BlobTrailerSize) | | stored CRC | CRC32C of the stored bytes plus the compression byte | | logical CRC | CRC32C of the uncompressed value (BlobReference::stable_checksum) | | exact reference metadata | Per-SST sorted list of every blob reference the SST keeps, written when the SST is built. References with the same encoding class, generation number and origin form a group. The list is stored as data blocks, with an index of the groups. An SST that has it is an exact SST | | coarse SST | An SST without exact reference metadata. It records one frontier file number and pins every indirect-origin blob file at or above it. Frontier 0 means no coarse frontier | | census | Background count, per generation number, of the SSTs and references that use it, plus live bytes per physical file | | blob file retirement | MANIFEST record (kBlobFileRetirement) that removes a collected blob file from the Version | | route deletion | MANIFEST record that removes a route that no SST uses. Route retirement is the background job that writes these records | "The option" is enable_blob_indirection. Terms of this diff: | Term | Meaning | |---|---| | format fence | A MANIFEST record that marks a column family (per-CF fence, kBlobIndirectionFormatVersion) or the whole DB (DB-global fence) as possibly holding blob indirection data. Diff 4 adds it. It never clears, also after the option is turned off. See "MANIFEST records" in db/blob/blob_relocation_manifest.h | | fenced CF | A column family whose current Version has the per-CF fence | | activation | The MANIFEST edit that sets the per-CF fence of a column family for the first time. (The term list at the top of db/blob/blob_relocation_manifest.h uses the word for the first such edit in the whole DB; this diff uses it per column family.) The first flush or compaction output written with the option on carries it. The MANIFEST writer of that edit is the activating writer | | pending activation | A flush or compaction that copied or picked its options with the option on and has not finished. It can still activate the column family, also after the option is turned off. ColumnFamilyData tracks it with two counters and a per-memtable mark (HasPendingBlobIndirectionOutput()) | | predating output | An output SST that may have been written before the column family used the option. It is marked in memory with FileMetaData::blob_reference_output_predates_indirection | | legacy SST | An SST in a fenced CF that has no exact metadata and no coarse frontier, because it was created before activation. The MANIFEST stores this as FileMetaData::blob_reference_legacy_metadata, so that recovery accepts the SST | | candidate Version | The Version that a MANIFEST batch builds from its edits before the MANIFEST write. It is installed only if the write succeeds | | census install | The census update that a MANIFEST batch prepares for a new Version and installs together with it, so that the census follows the Version (BlobReferenceCensusInstallBatch in db/version_set.h, diff 5) | | same-file / separate-file reference | A same-file blob reference is a kBlob or kBlobTTL BlobIndex whose blob record is in the same file as the table entry (BlobIndex::IsSameFile()). The ingestion scan rejects every other BlobIndex: a reference to another blob file (legacy or indirect, flattened or not) and an inlined TTL value. The error says that ingestion supports only same-file blob references | | batch joining | An earlier MANIFEST queue leader runs the precondition of a later ingestion and takes the ingestion into its batch | ## Previous behavior The option could already change through SetOptions(), and diff 4 already set the per-CF fence on the first output written with the option on. But nothing tracked a flush or compaction that started with the option on and had not installed its output yet. Such a job could still set the fence after SetOptions() turned the option off. SetOptions(), DB open, ingestion, import and export did not check the fence or such pending jobs. Base import and export reject an SST whose metadata names a blob file (oldest_blob_file_number is not 0). But an SST whose blob references are all flattened, or all use a generation number that has a route, can have oldest_blob_file_number 0, so this base check does not catch every SST with indirect references. For example, SetOptions() allowed a switch to PlainTable on a fenced CF. GetColumnFamilyMetaData() and ldb did not know about relocation files. With automatic compactions disabled, files with no live value stayed on disk. File ingestion had no precondition. ## New behavior ### Policy - enable_blob_indirection decides how new output is written and how indirect blob GC picks files. It is not a read gate. With the option off, new output uses legacy blobs, existing indirect data stays readable, and ordinary compaction can rewrite it into legacy blobs. - The per-CF fence is set by the first flush or compaction output written with the option on. No MANIFEST edit clears it. ### Pending activation - The pending state has three parts: two counters and a per-memtable mark. A flush with the option on marks the first memtable it picks (ReadOnlyMemTable::pending_blob_indirection_activation_), and MemTableList counts the marked memtables. HasPendingBlobIndirectionOutput() is true while any count or mark is set. The option checks and the MANIFEST writer use it to reject incompatible writers, also after the option is turned off. - A flush whose options copy has the option on is pending from that copy on (BeginBlobIndirectionFlush()). This includes the time when it syncs closed WALs without the DB mutex, before it picks its memtables. After the pick, the mark on its first memtable keeps it pending until the memtable is installed or rolled back. - A compaction picked with the option on is pending until it releases its input Version. ### Outputs made before activation - Three producers mark an output SST as a predating output: a flush or compaction with the option off on an input Version without the fence; a file copy of an input SST that has no reference metadata and no coarse frontier; and every ingested file. A file copy also keeps the input's coarse frontier. - If the MANIFEST writer sees the fence, or the same writer sets it, it marks each predating output without exact reference metadata as a legacy SST. - This closes the remote compaction case from diff 11 where a legacy job is followed by the format fence. The primary declines the worker's result and runs the compaction locally. The local outputs are predating outputs (the compaction was picked with the option off on a Version without the fence), so the MANIFEST writer marks them as legacy SSTs, and the fenced Version accepts them. - Parallel flushes can complete in reverse order. When one MANIFEST caller holds both an older flush output made before activation and a newer activating flush, the older output is classified as legacy. Recovery accepts a legacy SST only after the fence of its column family was replayed, or in the edit that sets the fence. So the first edit of the activating writer also sets the fence. The edits are not reordered or split, so if the MANIFEST ends inside this writer's edits, recovery still sees the fence before any legacy SST. The comment on FileMetaData::blob_reference_output_predates_indirection in db/version_edit.h explains the whole path. ### Format checks - SetOptions() (PrepareMutableCFOptions()) treats a column family with a pending activation like a fenced CF: the new options need a Block-Based Table writer and a supported compaction style. - DB open (ValidateBlobIndirectionOptionsAfterRecovery()) checks the compaction style when the option is on, an activation is pending or the column family is fenced. It checks the Block-Based Table writer and blob direct write (which must be off) only for a fenced CF. - The MANIFEST writer checks the compaction style again for each edit, and for a fenced or fence-setting edit also checks for a Block-Based Table writer. SetOptions() can check the options before another queued writer sets the fence, and a pending activation can set the fence after SetOptions() turned the option off. ### Ingestion and import - While an activation is pending, an ingestion scans its files for blob references in Prepare(), even if the column family has no fence and the option is off. The pending state is read under the DB mutex before Prepare(). - At the MANIFEST install, ValidateBlobIndirectionPublication() returns TryAgain if the column family is fenced, has the option on or is pending, and an ingested file was not scanned (files that the ingestion generated itself are exempt). The TryAgain status goes back to the user. When the user calls the ingestion again, it reads the state again, and if the column family still needs the scan, Prepare() now scans, so the check passes. The comment "Blob indirection checks of an ingestion" in db/external_sst_file_ingestion_job.h lists the steps. - Base import rejects an SST whose import metadata names a blob file (oldest_blob_file_number is not 0). This is not enough for blob indirection: the caller writes the metadata, and an SST whose references are all flattened or routed can have oldest_blob_file_number 0. So import also checks the table properties of each SST. Ingestion into a legacy column family (no fence, option off) does the same check; base ingestion has no blob check. Both read only the table properties, not the entries. They reject an SST that has exact metadata or a coarse frontier other than 0. Such an SST can hold indirect references, which need the routes of the DB that wrote it. The target DB does not have those routes. The property check is enough, because every SST with indirect references carries one of these two properties. SstFileWriter cannot write an indirect BlobIndex. - These rejects are also the guard that indirect blob GC relies on: a GC install assumes that no ingested or imported SST holds an indirect reference to a generation number of a GC input. The base import check alone is not this guard, for the reason above. ### Maintenance APIs - GetColumnFamilyMetaData() (the overload without GetColumnFamilyMetaDataOptions) and GetAllColumnFamilyMetaData() list relocation files. They pin a Version and list the relocation files without the DB mutex, but only for a column family whose Version has relocation files. Other column families keep the base path with one lock. - ExportColumnFamily() returns NotSupported for a column family that has the option on, a pending activation or the fence, because the export does not carry relocation files and routes. This check runs before the export directory is created and the column family is flushed, and GetColumnFamilyMetaDataForExport() checks again under the lock that reads the SST list. The base export check, which rejects an SST whose metadata names a blob file, runs later and does not catch an SST whose references are all flattened or routed. The checkpoint.h comments describe this limit. - ldb dump recognizes a relocation file and does not decode it as a blob log file. - VerifyFileChecksums() also checks the full-file checksum of each relocation file in the current Version, in the same way as for blob files (DBImpl::VerifyChecksumInternal() in db/db_impl/db_impl.cc). - Checkpoints, backups and the live-file APIs already handle relocation files from diff 4. ### Census - A MANIFEST batch that activates a column family reserves a census install for it (ReserveBlobReferenceCensusInstalls()), also when the new options have the option off. A pending flush or compaction can set the fence after the option is turned off. The CF then holds indirect references, and the census must count them so that indirect blob GC and route retirement stay correct. - SetOptions() and CreateColumnFamily() call ScheduleBlobReferenceCensus() for the column family. It queues census work only if the column family is fenced and its census needs a run. - An ingestion with a replace range already uses the census rebuild path of diff 5. This diff adds only a test for it. ### Removal of files with no live value - NeedsCompaction() checks for blob files and relocation files with no live value (HasEmptyBlobGcCandidate()) before it checks disable_auto_compactions. - When normal indirect blob GC cannot run (blob files, blob GC or blob indirection off, or automatic compactions disabled), the GC job still removes such files (GetEmptyBlobGcCandidates(), empty_files_only). It writes the blob file retirement of a normal blob file, or a relocation file deletion record (kBlobRelocationFileDeletion) of a relocation file. No SST uses a value in such a file, so the job copies no value and rewrites no SST. - So with automatic compactions disabled, the DB still writes these MANIFEST records and deletes the files. The enable_blob_indirection comment in advanced_options.h says so. ### Batch joining File ingestion can join an earlier MANIFEST batch, so it shares that batch's MANIFEST write and sync instead of starting its own batch. Correctness does not need it. "Details: MANIFEST batching" near the end of this summary has the rules and the proof. A queue leader whose precondition cannot join ends the batch after its own writers (rule 1 of VersionSet::ManifestBatchEndsBefore(); test VersionSetTest.PreconditionedWriterIsManifestBatchBarrier). The rules of ManifestBatchEndsBefore() replace the two batch barrier checks that diff 6 added to the grouping loop of ProcessManifestWrites(). Rule 1 keeps the barrier after a leader whose precondition cannot join. Rule 6 keeps a later precondition writer out of the batch, unless its caller opted in to joining, no edit in the batch sets a format fence, and its precondition passes at that moment. ## What changed in the implementation "Parts of this diff" and "How to read this diff" list the files. These changes are not named there: - db/memtable.h: the per-memtable mark of a pending activation. - db/builder.{h,cc}: RejectBlobIndirectionTableProperties(), shared by import and legacy ingestion. - include/rocksdb/db.h: the CreateColumnFamilyWithImport() comment says that import returns NotSupported when the table properties of a file show that it can hold blob indirection references. - db/compaction/compaction_service_job.cc: the comment for the local rerun after the primary declines a remote result, and the sync point BlobGCResultDeclined for tests. - db/db_impl/db_impl.h: GetColumnFamilyMetaDataForExport(), and the empty_files_only argument of RunIndirectBlobGCBatch(). - db/db_impl/db_impl_debug.cc: the test hook passes empty_files_only=false. - db/compaction/compaction_job.h: the comment of the compaction safety protocol says that ingestion and import reject indirect references. ## Safety boundaries and unsupported cases - Activation does not run a one-time full-key scan. The binary cannot be downgraded after activation. - These are rejected: activation that conflicts with blob direct write, PlainTable transitions once an activation is pending or the column family is fenced, and ingestion of an external SST with a blob reference that is not a same-file reference when the column family is fenced, has the option on or has a pending activation. - An ingestion checks blob references before it copies files if it sees the activation during Prepare(). If the activation starts after Prepare(), the ingestion returns TryAgain before the MANIFEST install, and its copied files are rolled back. - ExportColumnFamily() refuses a fenced CF. Turning the option off does not lift this limit. Full checkpoints are supported. - A DB that never activated, with the option off, keeps its base behavior, including its MANIFEST group commit. The only exception: ingestion or import of an SST that carries blob indirection properties returns NotSupported (base import already rejects such an SST when its metadata names a blob file). ## Validation and known limitations Unit tests cover option toggles, pending activation for normal flush, atomic flush and compaction, legacy classification of outputs made before activation, ingestion scans and retries, batch joining and its barriers, import and legacy ingestion property checks, the census after a replace-range ingestion, the flush window before the memtable pick, ldb dump of a relocation file, and removal of files with no live value with automatic compactions off. ## Option gate At this diff, ColumnFamilyData::ValidateOptions rejects enable_blob_indirection outside tests (the option gate: diff 2 adds it, diff 14 removes it). The behavior described here with the option on is reachable only through the internal test hook ColumnFamilyData::TEST_SetAllowBlobIndirection, not in production. ## Details: MANIFEST batching - File ingestion checks its publication precondition (ValidateBlobIndirectionPublication()) at the MANIFEST queue head. An earlier queue leader may also run this check, with the DB mutex held, so that the ingestion can join the leader's MANIFEST batch. It joins only if the batch is a normal MANIFEST batch, no edit already in the batch sets a format fence, and the check passes. A failed join does not fail the batch. The ingestion then runs the check again at the queue head and returns TryAgain if needed. SetOptions(), indirect blob GC, route retirement and the compaction publication precondition do not join, and they end the batch they lead. - After an ingestion joins a batch, or leads it, the batch also ends before a later caller that activates a column family or publishes new CF options. Other callers follow the normal batching rules. Proof: the ingestion check reads four kinds of state. Inside one batch, only two kinds of MANIFEST callers can change the fence or the option value: a caller that activates a column family, and a caller that publishes new CF options. The pending state can change outside the MANIFEST queue, so the table explains it separately. - Diff 6 applies the edit of a queue leader that has a precondition without the DB mutex, and runs the precondition again afterwards. This diff keeps that only for preconditions that cannot join: indirect blob GC, route retirement, and a compaction that publishes blob relocation output dependencies. A file ingestion applies its edit with the DB mutex held, as the leader or as a joined follower. Its precondition does not run again, so it cannot fail after other callers joined the batch. An ingestion edit is small, so the lock hold is short. - A joinable precondition is checked against the current Versions, not against the candidate Version of the batch. This is why no ingestion joins after an edit that sets a fence or publishes CF options. | State the check reads | Who can change it inside one batch | Covered by | |---|---|---| | Format fence of the current Version | an edit that sets the fence (an activating flush or compaction) | the join is refused if an earlier edit sets the fence; the batch ends before a later activating caller | | Latest enable_blob_indirection | SetOptions(), which publishes options through the queue | the batch ends before a caller that publishes options | | Pending activation (two counters and the per-memtable marks) | a flush or compaction that copied or picked its options with the option on | A new pending activation can start only while the option is on, and the option cannot change while the batch holds the queue head. If the option is on, the check already requires every file to be scanned. | | Per-file validated and generated flags | only the ingestion itself, before it is queued | fixed | - Rule 3 keeps outputs made before activation and an activation from different callers out of one batch. Within one caller they stay together (see "Outputs made before activation"). - Lock cost of a join: the leader runs the ingestion check under the DB mutex. The check reads only the fence bit, the option value, the pending state (two counters and the count of marked memtables) and two flags per ingested file. It does no I/O. It allocates only when it fails (the TryAgain message). ## Stack context (same in diffs 2-14, except that diffs 10-14 add the diff 10 route rule to the install line; "<== this diff" marks this diff's part): Goal: remove garbage from blob files without rewriting any SST. A value gets a stable identity when it is first written. Indirect blob GC copies the live values of whole files into a relocation file and installs routes in the MANIFEST. Readers follow the routes. SSTs are not changed. How a read finds a value (P is the relocation file that holds the value now): ``` SST entry indirect BlobIndex { generation number, origin, original offset, size, CRCs } | v Version file table: generation number -> live file, or route old file -> P | | live file route found | | v v blob file: read at original offset relocation file P (always one hop): relocation file: look up key look up key (origin, original offset) (origin, original offset) ``` Lifecycle of one value (in this example, F is the blob file where the value is first written, and P is the relocation file that GC writes): ``` write new value goes to blob file F; index: origin = F, generation number = F | census per generation number: the SSTs and references that use it; | per physical file: live bytes, ranked by garbage ratio GC job copy the live values of whole input files (stored bytes, checked | with the stored CRC) into relocation file P, keyed by | (origin, original offset) install one MANIFEST edit: add P, route each input generation number still | in use to P (a relocation input that gets an output keeps a route | for all its generation numbers), re-point the routes into the | inputs to P, and remove each input (blob file retirement for a | blob file, relocation file deletion for a relocation file) flatten compaction sets generation number = P in the SSTs it rewrites | route route retirement: route deletion records remove the routes that delete no SST uses ``` Stack map: | Diff | Adds | |---|---| | 1 | Finalize blob metadata outside the DB mutex (prep) | | 2 | Indirect BlobIndex (kIndirectBlob) and exact SST reference metadata | | 3 | Relocation file format and stable read path | | 4 | MANIFEST routes and relocation state in the Version | | 5 | Exact reverse reference census | | 6 | MANIFEST apply outside the DB mutex, with a precondition re-check | | 7 | Coarse flush metadata, per-SST reference cap, epoch runs | | 8 | Indirect blob GC building blocks | | 9 | Indirect blob GC job | | 10 | Flatten generation numbers; route retirement | | 11 | Remote compaction validation | | 12 | Dynamic options and operations <== this diff | | 13 | Lost relocation files | | 14 | RepairDB rebuilds relocation state; option gate removed | | 15 | db_stress and crash test support | ## Stack context: terms for later diffs None. This diff uses no term that only a later diff adds. Tracking issue: meta#15301 Differential Revision: D124975457
Summary: Pull Request resolved: meta#15720 ## Why the change is required Base RocksDB has a repair open: with best_efforts_blob_db_repair, an open records a missing or corrupt normal blob file as dropped (kDroppedBlobFile), and reads and compactions then treat its values as deleted. A column family with blob indirection has two more kinds of physical file that can go missing: indirect-origin blob files and relocation files. Earlier diffs of this stack already apply the base dropped-file record to a missing indirect-origin blob file, and reads treat its values as deleted. Without this diff, three problems are left: - A compaction of an SST that holds an indirect reference to a dropped origin fails with Corruption (CompactionJob::ValidateBlobRelocationInputGenerations()). - A missing relocation file makes the repair open fall back in time. A fallback past an indirect blob GC edit can lose values that a relocation file on disk still holds. - skip_missing_blob_files drops live values that a route still serves. This diff makes a missing file leave the DB readable. The references to it go away without a read of the file, and no value whose data is still on disk is lost. It also adds an open check for missing relocation files (see "Open check"). Only a relocation file can be marked lost. A normal or indirect-origin blob file that is missing uses the base dropped-file record instead. A relocation file cannot use the base record, for two reasons: - The base dropped list keeps a file only while some SST has an oldest_blob_file_number at or below it. Flattened and routed references do not set oldest_blob_file_number, so the dropped list would forget a relocation file while SSTs still use it. - A route target must be a relocation file in the Version. So a lost relocation file stays in the Version, marked lost, until indirect blob GC removes it. ## Parts of this diff 1. Repair open (db/version_builder.cc, db/blob/blob_relocation_manifest.h): best_efforts_blob_db_repair marks a missing or corrupt relocation file lost. 2. Read path (db/blob/blob_fetcher.{h,cc}, db/version_set.{h,cc}, table/get_context.cc, db/db_iter.{h,cc}, db/merge_helper.cc): a lost value reads as a point deletion, and a wide-column entity loses only its lost blob columns. 3. Compaction (db/compaction/compaction.{h,cc}, db/compaction/compaction_job.cc, db/blob/missing_blob_repair_iterator.h): a compaction turns references to lost and dropped files into deletions or partial entities. 4. Fallback checks (db/version_builder.h, db/version_edit_handler.{h,cc}, db/version_set.cc): a repair open fails instead of falling back to a point that would drop values that a relocation file on disk still holds. 5. Open check (db/db_impl/db_impl_open.cc, db/db_impl/db_impl_files.cc): every normal open checks that each relocation file that is not marked lost exists. 6. skip_missing_blob_files guard (db/db_impl/db_impl_compaction_flush.cc): CompactRange with skip_missing_blob_files returns NotSupported on a column family that may hold blob indirection references. A first-time reader can skip this part; the rest of the diff does not depend on it. ## How to read this diff 1. Overview: read "Terms used in this diff" below. Then read "Lost relocation files" in db/blob/blob_relocation_manifest.h. It gives the life of a lost file and says why a relocation file is marked lost and not dropped. The "Accepted loss (kBlobPhysicalFileLoss)" bullet above it describes the record itself. The bullet title and a few comments say "accepted loss"; it means the same thing as a loss record. 2. How a file becomes lost: the relocation file loop in db/version_builder.cc (the branch under best_efforts_blob_db_repair). The open check in db/db_impl/db_impl_open.cc (DBImpl::CheckRelocationFilesExist() and its two calls in DBImpl::Recover()) is the case of a normal open (not a best-efforts recovery open). 3. The loss check: db/version_set.h and db/version_set.cc (VersionStorageInfo::HasLostBlobRelocationFiles(), VersionStorageInfo::IsBlobReferenceLost(), Version::IsBlobValueLost()), then db/blob/blob_fetcher.h (BlobFetcher::IsBlobValueLost(), HasBlobValueLoss(), has_lost_blob_relocation_files_). 4. Point reads and merges: table/get_context.cc (GetContext::SaveValue()) and db/merge_helper.cc. Both turn a lost value into kTypeDeletion and use the repaired entity. 5. Iterator: db/db_iter.h (CurrentValue(), SaveCurrentValue(), the loss fields), then db/db_iter.cc (ParseKey(), CheckDeletedByDroppedBlobFiles(), the forward and reverse range tombstone changes). 6. Compaction: db/compaction/compaction.h (RepairsLostBlobReferences(), CollectExactBlobReferences()), db/blob/missing_blob_repair_iterator.h (Mode), then db/compaction/compaction_job.cc (CreateInputIterator(), ShouldUseLocalCompaction(), ValidateBlobRelocationInputGenerations()). 7. Fallback checks: db/version_builder.h (RecordsMissingBlobFilesAsDropped(), IsBlobFileRemovedByGc()), db/version_edit_handler.cc (VersionEditHandlerPointInTime::CheckRepairFallbackAgainstGc()), then db/version_set.cc (VersionSet::TryRecover()). 8. Side changes: db/db_filesnapshot.cc (live-file lists skip lost files) and CheckSkipMissingBlobFilesSupported() in db/db_impl/db_impl_compaction_flush.cc. ## Terms used in this diff Stack terms: | Term | Meaning | |---|---| | origin | Blob file where the value was first written | | original offset | Value offset in the origin file | | generation number | The file number that a reference uses to find its value. A new value has generation number == origin. When indirect blob GC moves the value, the reference keeps its generation number, and a route sends the read to the relocation file | | unflattened reference | Index with generation number == origin (value not relocated, or not flattened yet) | | flattened reference | Index with generation number != origin: compaction rewrote the generation number to the relocation file that holds the value | | route | Old file number -> the relocation file that now holds its live values. One hop: a target is never itself routed. Stored in the MANIFEST | | physical file | A file that holds value bytes: a normal blob file or a relocation file. A generation number can name a physical file, or it can be only the key of a route | | indirect origin | Normal blob file written with enable_blob_indirection on; it is the origin of unflattened references | | relocation file | Block-based table keyed by (origin, original offset) that holds copied values | | exact reference metadata | Per-SST sorted list of every blob reference the SST keeps, written when the SST is built. An SST that has it is an exact SST | | coarse SST | An SST without exact reference metadata. It records one frontier file number and pins every indirect-origin blob file at or above it. Frontier 0 means no coarse frontier | | census | Background count, per generation number, of the SSTs and references that use it, plus live bytes per physical file. An SST that the census counts for a file is a holder of that file | | blob file retirement | MANIFEST record (kBlobFileRetirement) that removes a collected blob file from the Version | | format fence | A MANIFEST record that marks a column family (per-CF fence) or the whole DB (DB-global fence) as possibly holding blob indirection data. It never clears. See "MANIFEST records" in db/blob/blob_relocation_manifest.h | | loss record | MANIFEST record (kBlobPhysicalFileLoss) that marks a relocation file lost. A lost file stays in the Version, but its values read as deleted. Code comments also call it an accepted loss | Terms of this diff: | Term | Meaning | |---|---| | repair open | An open with best_efforts_blob_db_repair (base option) | | dropped blob file | A normal or indirect-origin blob file that a repair open recorded as dropped (kDroppedBlobFile, base). It leaves the Version, and reads treat its values as deleted | | lost reference | An indirect reference whose generation number is a lost relocation file, or is routed to one (VersionStorageInfo::IsBlobReferenceLost()) | | base gate | Compaction::RepairsMissingBlobReferences() (base). It holds when the input Version has dropped blob files and an input SST has an oldest_blob_file_number at or below the newest dropped file; the compaction then repairs references to dropped files. It also holds when skip_missing_blob_files is set; the compaction then repairs references to every missing blob file | | loss gate | Compaction::RepairsLostBlobReferences(): the input Version has a lost relocation file | | fallback in time | Best-efforts recovery opens an older point of the MANIFEST when the newest state needs a file that is missing, for example a non-L0 SST | ## Previous behavior Only tests could create a loss. A repair open already put a missing indirect-origin blob file on the base dropped list, even when all its values count as garbage, and reads treated its values as deleted. But a compaction of an exact SST that holds an indirect reference to a dropped origin failed with Corruption, because CompactionJob::ValidateBlobRelocationInputGenerations() found the generation number unreadable. VersionBuilder already applied a loss record and kept the file in the Version, marked lost. A fetch from a lost relocation file returned NotFound. Compaction failed on a reference to a lost file, because Version::ValidateBlobReferenceGeneration() returned NotFound. A wide-column entity with a lost blob column could not be read. A repair open treated a missing relocation file like a missing SST and fell back in time. DB::GetLiveFiles(), GetLiveFilesStorageInfo() and GetLiveFilesChecksumInfo() listed lost relocation files. A normal open did not check that relocation files exist, so a missing one failed only later reads. CompactRangeOptions::skip_missing_blob_files treated a routed (relocated) generation number as missing and dropped live values. ## New behavior ### Repair open - A repair open marks a missing or corrupt relocation file lost and writes the loss record into its new MANIFEST. A missing relocation file does not cause a fallback in time. Without the flag, a normal open fails with NotFound (see "Open check"), and a plain best_efforts_recovery open falls back to an earlier consistent point, or fails (see "Fallback checks"). - A dropped indirect origin needs nothing more: an origin that is still in the Version has no route (indirect blob GC adds the route and retires the origin in one edit), GC selects only files in the Version, and the census counts of a dropped origin drain to 0 as compactions remove its references. - Loss marks survive MANIFEST replay and rotation, checkpoints and backup. DB::GetLiveFiles(), GetLiveFilesStorageInfo() and GetLiveFilesChecksumInfo() skip lost files, because they have no bytes to copy. - VersionBuilder rejects a loss record that names no live relocation file of the edit's column family ("Blob relocation loss names no live target"). ### Reads - A lost reference acts as a same-sequence point deletion, before any cache lookup. An iterator uses kTypeDeletionWithTimestamp when user-defined timestamps are on; Get and merges use kTypeDeletion. A reference to a dropped origin reads as deleted through the base dropped check. - Every read path checks for loss before it fetches. A fetch that reaches a lost relocation file anyway returns Corruption, as base does for a dropped file. - A wide-column entity loses only its lost blob columns, as with skip_missing_blob_files. It becomes a deletion only when every column is lost or dropped. An iterator runs the base dropped check on the entity after it removes the lost columns (DBIter::CheckDeletedByDroppedBlobFiles() reads the repaired entity from DBIter::CurrentValue()). So an entity whose blob columns are partly lost and partly dropped, with no other column, reads as deleted. Get on an entity whose default column is lost or dropped returns an empty value. - Get, MultiGet, iterators (forward, reverse and with unprepared values) and merges use the base helper WideColumnSerialization::RemoveBlobColumns(), which MissingBlobRepairIterator also uses. - The blob fetcher records, when it is made, whether its Version has lost files. With no lost file, the loss check for each key never reads the Version. A tailing iterator keeps its fetcher after it drops its reference to that Version, so a flush can free the Version during a scan. - Read-path range tombstone conversion (min_tombstones_for_range_conversion) never counts lost keys. A lost key ends a tombstone run like a live key, in both iteration directions, so a loss never becomes a real range tombstone. ### Compaction - A compaction wraps its input in MissingBlobRepairIterator when the base gate or the loss gate holds. The loss gate is needed because flattened and routed references do not set oldest_blob_file_number, so an SST that holds only such references to a lost relocation file never matches the base gate. Without skip_missing_blob_files, the iterator uses MissingBlobRepairIterator::Mode::kDroppedOrLostFiles: it repairs references to dropped files (base) and lost references. skip_missing_blob_files is allowed only on a plain column family (see "skip_missing_blob_files guard") and keeps the base Mode::kMissingFiles. - With either gate, the compaction runs locally. With the loss gate, it also writes exact reference metadata at every output level, including L0. Without the loss gate, an L0 output is exact only when the compaction is bottommost or an L0 input is already exact (diff 7). The reason for the loss rule: a coarse SST records only its frontier, and it pins every generation number at or above the frontier. A lost relocation file stays pinned while an old number routed to it is at or above that frontier. So a coarse output could keep the lost file pinned, and the lost file would not be removed. - A lost reference becomes a same-sequence deletion (kTypeDeletionWithTimestamp when user-defined timestamps are on), also with snapshots or older keys below. A SingleDelete in the same compaction pairs with it as with skip_missing_blob_files. - A lost generation number is legal (Version::ValidateBlobReferenceGeneration() returns OK), so the compaction does not fail. The compaction's input check (CompactionJob::ValidateBlobRelocationInputGenerations()) also accepts a dropped origin without a route, because the repair iterator turns its references into deletions and the output never writes it. A generation number that the input Version cannot resolve in any other way still fails the job with Corruption. - CompactionIterator has its own loss check (diff 10). It is for callers that give it a blob fetcher without the repair iterator (BuildTable() for flush and recovery). In a CompactionJob run this check finds no lost reference, because the repair iterator has already turned lost references into deletions. - Holders are repaired only when a compaction rewrites them. Nothing schedules a compaction for them. - When the census finds no holder of a lost relocation file, indirect blob GC deletes it as an empty input, without reading it (diff 9). The loss record leaves with the file record. ### Fallback checks - Fallback in time past a GC edit: with best_efforts_blob_db_repair, a missing non-L0 SST can still make the open fall back to an older MANIFEST point. That point can be older than an indirect blob GC edit, so it names origins that GC already moved into a relocation file and deleted. Under the repair flag, base drops a blob file only when its data is gone from disk. A fallback past a GC edit would drop old origin files whose values are still on disk in a relocation file, and the purge at open would then delete that relocation file. So the open fails with Corruption instead, and it never loses a value whose data is still on disk. The check is VersionEditHandlerPointInTime::CheckRepairFallbackAgainstGc(). The message holds the column family ID and the file numbers. When this check fails, VersionSet::TryRecover() stops and does not try an older MANIFEST file: that file never saw the GC edit, so it would drop the same origins. - Older MANIFEST file: when recovery from the newest MANIFEST file fails for another reason (for example, the file is empty), base best-efforts recovery tries an older MANIFEST file. That file does not know the indirect blob GC edits of the newer ones, so the check above cannot run. So if a repair open succeeds from an older MANIFEST file, and a column family with the format fence has dropped blob files or lost relocation files, the open fails with Corruption. - Plain best_efforts_recovery (without the repair flag) never marks a file lost. With a missing relocation file, VersionBuilder refuses the Version, and recovery goes back to an earlier consistent point or fails, as for a missing SST. A read-only open with plain best_efforts_recovery can still fall back to an older MANIFEST file, as in base RocksDB. ### Open check Every read-write or read-only open checks that each relocation file that is not marked lost exists in its canonical path (the first cf_path of its column family), before any column family is visible. An open with best_efforts_recovery or best_efforts_blob_db_repair skips this check. If one is missing, the open fails with NotFound. The message names the file path, its number and the column family, and says to restore the file or to open with best_efforts_blob_db_repair. The check compares with a directory listing instead of calling FileExists per file. A read-write open reuses the listing that MaybeUpdateNextFileNumber() already makes of every DB path, so the check runs after it, still before WAL replay. A read-only open lists each needed path once. ### skip_missing_blob_files guard skip_missing_blob_files returns NotSupported on a column family that may hold blob indirection references: the option is on, an enabled flush or compaction may still publish the first format fence, or the per-CF fence is set. CompactRangeInternal() checks before any work. RunManualCompaction() checks again with the picked compaction's options and input Version, and it releases its inputs if rejected. Plain column families keep the base path. ### Monitoring - BLOB_DB_LOST_BLOB_READ_COUNT (ticker) counts entries that reads treat as deleted, or as wide-column entities with missing columns, because their values are in a lost relocation file. Point reads and iterators count. An iterator can count an entry again when it moves back to it. - rocksdb.blob-relocation-lost-files (DB::Properties::kBlobRelocationLostFiles, an int property) and the new map key lost-relocation-files of rocksdb.indirect-blob-gc-stats return the number of relocation files of the column family that are marked lost. - An open with best_efforts_blob_db_repair logs one ERROR line per column family that has lost relocation files (VersionSet::LogDroppedBlobFiles()), also for files that an earlier open marked lost: "Lost blob relocation files after best-efforts BlobDB repair (reads of their values return NotFound): count N, files #a #b ... and K more". The line names at most kMaxLoggedLostFileNumbers (16) files, so it stays short. The property has the count, and ldb manifest_dump lists every lost file. - Feature off: only a column family with blob indirection can have lost relocation files, so the ticker does not change and the count is 0. ## Option gate At this diff, ColumnFamilyData::ValidateOptions rejects enable_blob_indirection outside tests (the option gate: diff 2 adds it, diff 14 removes it). The behavior described here with the option on is reachable only through the internal test hook ColumnFamilyData::TEST_SetAllowBlobIndirection, not in production. ## Safety boundaries and unsupported cases - Public API: new ticker BLOB_DB_LOST_BLOB_READ_COUNT, appended before TICKER_ENUM_MAX, and new property rocksdb.blob-relocation-lost-files. The new ticker has no Java mapping, as for recent base tickers: all 256 byte values of org.rocksdb.TickerType are used. - No live repair API: only a repair open marks files lost or dropped. It marks a missing or corrupt relocation file lost, and, as in base, it records a missing or corrupt normal or origin blob file as dropped. - Legacy kBlobIndex references keep base behavior: a repair open drops a missing legacy blob file, and skip_missing_blob_files repairs it on a plain column family. - A lost Put repaired into a Delete makes a later SingleDelete of that key fail with Corruption when they meet in compaction (enforce_single_del_contracts). Base skip_missing_blob_files behaves the same. - Loss repair compaction runs locally, not on a remote compaction service. This is a conservative choice: a worker reads the loss records from the MANIFEST and could repair the references itself, but the primary does not check a worker's result for that. - When a Version has a loss record, a tailing iterator reads that Version again for each blob key. A flush can free that Version during the scan (see "Reads"), so the read can touch freed memory. Base RocksDB has the same problem when it fetches a blob value. This diff does not fix it. - A lost file is removed only after compactions have rewritten all its holders. With disable_auto_compactions, that needs a manual compaction. - The two fallback checks can fail a repair open in rare cases where nothing new would be lost: - The older MANIFEST file check fails when that file holds drops that an earlier repair open already recorded. - The check against indirect blob GC edits fails for an origin that GC retired with no route. - Both checks can fail for a relocation file that an earlier loss record marked lost and that GC later removed. Its values were already lost. A plain best_efforts_recovery open (without best_efforts_blob_db_repair) is the workaround. ## Validation and known limitations Unit tests cover the repair open, the read paths, loss repair compaction, the fallback checks, the open check and the skip_missing_blob_files guard (see the Test Plan). The tailing iterator limitation above is known and not fixed here. ## Stack context (same in diffs 2-14, except that diffs 10-14 add the diff 10 route rule to the install line; "<== this diff" marks this diff's part): Goal: remove garbage from blob files without rewriting any SST. A value gets a stable identity when it is first written. Indirect blob GC copies the live values of whole files into a relocation file and installs routes in the MANIFEST. Readers follow the routes. SSTs are not changed. How a read finds a value (P is the relocation file that holds the value now): ``` SST entry indirect BlobIndex { generation number, origin, original offset, size, CRCs } | v Version file table: generation number -> live file, or route old file -> P | | live file route found | | v v blob file: read at original offset relocation file P (always one hop): relocation file: look up key look up key (origin, original offset) (origin, original offset) ``` Lifecycle of one value (in this example, F is the blob file where the value is first written, and P is the relocation file that GC writes): ``` write new value goes to blob file F; index: origin = F, generation number = F | census per generation number: the SSTs and references that use it; | per physical file: live bytes, ranked by garbage ratio GC job copy the live values of whole input files (stored bytes, checked | with the stored CRC) into relocation file P, keyed by | (origin, original offset) install one MANIFEST edit: add P, route each input generation number still | in use to P (a relocation input that gets an output keeps a route | for all its generation numbers), re-point the routes into the | inputs to P, and remove each input (blob file retirement for a | blob file, relocation file deletion for a relocation file) flatten compaction sets generation number = P in the SSTs it rewrites | route route retirement: route deletion records remove the routes that delete no SST uses ``` Stack map: | Diff | Adds | |---|---| | 1 | Finalize blob metadata outside the DB mutex (prep) | | 2 | Indirect BlobIndex (kIndirectBlob) and exact SST reference metadata | | 3 | Relocation file format and stable read path | | 4 | MANIFEST routes and relocation state in the Version | | 5 | Exact reverse reference census | | 6 | MANIFEST apply outside the DB mutex, with a precondition re-check | | 7 | Coarse flush metadata, per-SST reference cap, epoch runs | | 8 | Indirect blob GC building blocks | | 9 | Indirect blob GC job | | 10 | Flatten generation numbers; route retirement | | 11 | Remote compaction validation | | 12 | Dynamic options and operations | | 13 | Lost relocation files <== this diff | | 14 | RepairDB rebuilds relocation state; option gate removed | | 15 | db_stress and crash test support | Tracking issue: meta#15301 Differential Revision: D132305446
meta#15550) Summary: Pull Request resolved: meta#15550 ## Why the change is required RepairDB must recover indirect references when the MANIFEST routes cannot be trusted, and it must never publish an incomplete or empty replacement image. Without this diff, RepairDB on a DB that uses blob indirection can only refuse to run: a legacy repair writes a new MANIFEST from the files it finds, so it would lose the routes and relocation files that the old MANIFEST records. RepairDB must also keep the losses that are already accepted: the loss records of lost relocation files and the dropped-file records of missing blob files (an open with best_efforts_blob_db_repair writes both: diff 13 adds the loss records; the dropped-file records are the base mechanism, which earlier diffs of this stack also apply to indirect origins). This diff also completes the feature: it removes the option gate added in diff 2, so enable_blob_indirection can be used in production from here. ## Parts of this diff 1. Core: feature RepairDB (db/repair.cc; db/blob/blob_relocation_manifest.h has the comment that RepairDB writes the loss record again). When the DB has persisted blob indirection evidence, RepairDB runs a crash-safe transaction and rebuilds the routes and relocation files. The "Feature repair" overview at the top of db/repair.cc lists its steps. 2. Repair catalog and evidence rules (file/filename.{h,cc}). The catalog lets a crash or a retry finish or roll back a repair. Opens (db/db_impl/db_impl_open.cc, db/db_impl/db_impl.h, db/db_impl/db_impl_secondary.cc), best-efforts recovery (VersionSet::TryRecover() in db/version_set.cc) and DestroyDB (db/db_impl/db_impl.cc) check the catalog. 3. Staged MANIFEST (db/version_set.{h,cc}): VersionSet::RecoverForRepair() and SyncAndCloseRepairManifest(). Repair writes its new MANIFEST under a temporary name, and VersionSet never starts another MANIFEST while it stages one. 4. Readers that repair uses. BlobReferenceIterator, a cursor over the exact references of one SST (db/blob/blob_reference_metadata.h, table/table_reader.h, table/block_based/block_based_table_reader.{h,cc}). BlobFileReader::CreateFromPathForRepair() and BlobRelocationFileReader::OpenForRepair() open a file at a given path without MANIFEST data (db/blob/blob_file_reader.{h,cc}). BlobLogSequentialReader reads a record with a size limit and in two steps, header then payload (db/blob/blob_log_sequential_reader.{h,cc}). TableReaderOptions::isolate_repair_caches keeps repair readers out of shared caches (table/table_builder.h, db/table_cache.{h,cc}, table/block_based/block_based_table_factory.cc). Safe to skim on a first read. 5. Option gate removal, the public RepairDB comment and three db_bench flags (db/column_family.{h,cc}, db/db_test_util.cc, include/rocksdb/advanced_options.h, include/rocksdb/db.h, tools/db_bench_tool.cc). Safe to skip on a first read. ## How to read this diff 1. Read "Terms used in this diff" in this summary ("Stack context" at the end has the read path and the lifecycle of a value). Then read the "Feature repair" overview at the top of db/repair.cc and skim Repairer::Run() for the order of the steps. Look up "format fence", "Accepted loss (kBlobPhysicalFileLoss)" (which this summary calls a loss record) and "Lost relocation files" in db/blob/blob_relocation_manifest.h. 2. The repair catalog: file/filename.h (RepairCatalogRecordType, the field table at RepairCatalogRecord, ReadRepairCatalog, ResolveRepairCatalog, CheckRepairCatalogForOpen), then file/filename.cc (ReadRepairCatalogRecords, and the step list in ResolveRepairCatalog). It depends on nothing new. The code of reading steps 5 to 8 (StartRepairCatalog() and later) writes records to it. The readers of step 3 and the replay of step 4 do not. 3. The readers: BlobReferenceIterator in db/blob/blob_reference_metadata.h and BlockBasedBlobReferenceIterator in table/block_based/block_based_table_reader.cc; OpenForRepair and CreateFromPathForRepair in db/blob/blob_file_reader.h; isolate_repair_caches in table/table_builder.h. They depend on nothing new in this diff. 4. Feature evidence and the original MANIFEST replay: file/filename.h (IsBlobIndirection*Evidence), then db/repair.cc (DetectBlobIndirection, RepairOriginalManifestHandler, ReadOriginalManifest). These choose feature repair and keep the losses, the routes that point to a lost relocation file, the dropped-file records (kDroppedBlobFile), and the log floors and SST lists. Repair rebuilds all other routes in RepairBlobState() (reading step 7). The "Lost files" and "Dropped blob files" paragraphs in the db/repair.cc overview give the full rules. 5. Reference runs in db/repair.cc: RepairScratchFile, RepairReferenceRun, AppendScannedReference, PrepareExactTableRuns, MergeRepairRuns and VisitRepairReferences. The SST scan writes the runs, so repair reads the data of each SST only once. PrepareExactTableRuns uses BlobReferenceIterator from step 3. 6. WAL conversion in db/repair.cc: the file overview parts (b) and (d), PrepareOriginalReplayCoverage, RepairBatchValidator, ConvertLogFilesToTables and ConvertLogToTable. PrepareOriginalReplayCoverage runs after the SST scan. It compares the SSTs that repair found with the SST lists from step 4 and writes policy records. WAL conversion writes coverage records. 7. Route rebuild in db/repair.cc: the comment above RepairBlobState, then its body, DiscoverBlobFiles and RepairRelocationPack. This step uses steps 4 and 5. 8. Commit and open: db/repair.cc (AddTables, PublishManifest), db/version_set.h (RecoverForRepair, SyncAndCloseRepairManifest), VersionSet::TryRecover in db/version_set.cc, db/db_impl/db_impl_open.cc (the CheckRepairCatalogForOpen calls and DBImpl::CheckUnresolvedRepairCatalog) and DestroyDB in db/db_impl/db_impl.cc. ## Terms used in this diff Stack terms: | Term | Meaning | |---|---| | origin | Blob file where the value was first written | | original offset | Value offset in the origin file | | stable identity | (origin, original offset). Never changes, even when the value moves | | generation number | The file number that a reference uses to find its value. A new value has generation number == origin. When indirect blob GC moves the value, the reference keeps its generation number, and a route sends the read to the relocation file | | unflattened reference | Index with generation number == origin (value not relocated, or not flattened yet) | | flattened reference | Index with generation number != origin: compaction rewrote the generation number to the relocation file that holds the value | | flatten | Compaction rewrites the generation number to the relocation file that holds the value, so old routes can go. Only flatten makes generation number != origin | | Version file table | The table in a Version that maps a generation number to a live file or to a route | | route | Old file number -> the relocation file that now holds its live values. One hop: a target is never itself routed. Stored in the MANIFEST | | physical file | A file that holds value bytes: a normal blob file or a relocation file. A generation number can name a physical file, or it can be only the key of a route | | indirect origin | Normal blob file written with enable_blob_indirection on; it is the origin of unflattened references | | relocation file | Block-based table keyed by (origin, original offset) that holds copied values. Each value is the stored bytes plus a 5-byte trailer (kSimpleGen2BlobTrailerSize) | | served files | The old file numbers that a relocation file lists as the ones it may serve (see db/blob/blob_relocation_file.h). Repair rebuilds routes from them | | stored CRC | CRC32C of the stored bytes plus the compression byte | | logical CRC | CRC32C of the uncompressed value (BlobReference::stable_checksum) | | exact reference metadata | Per-SST sorted list of every blob reference the SST keeps, written when the SST is built. References with the same encoding class, generation number and origin form a group. The list is stored as data blocks, with an index of the groups. An SST that has it is an exact SST | | coarse SST | An SST without exact reference metadata. It records one frontier file number and pins every indirect-origin blob file at or above it. Frontier 0 means no coarse frontier | | census | Background count, per generation number, of the SSTs and references that use it, plus live bytes per physical file. An SST that the census counts for a file is a holder of that file | | blob file retirement | MANIFEST record (kBlobFileRetirement) that removes a collected blob file from the Version | | route deletion | MANIFEST record that removes a route that no SST uses. Route retirement is the background job that writes these records | | format fence | A MANIFEST record that marks a column family (per-CF fence) or the whole DB (DB-global fence) as possibly holding blob indirection data. It never clears. See "MANIFEST records" in db/blob/blob_relocation_manifest.h | | loss record | MANIFEST record (kBlobPhysicalFileLoss) that marks a relocation file lost. A lost file stays in the Version, but its values read as deleted. | Terms of this diff. The repair terms image, repair catalog, staged MANIFEST, source (with its encoding class), reference run, repair group, visit, donor and lost file are defined in the "Feature repair" overview at the top of db/repair.cc. The WAL terms are defined in "WAL coverage" above PrepareOriginalReplayCoverage() in db/repair.cc. The rows below are the other terms: | Term | Meaning | |---|---| | legacy | Written the way base RocksDB writes it, without blob indirection | | legacy reference | A kBlob or kBlobTTL index. It names its blob file directly and cannot be routed | | classification with frontier 0 | A classification property (kBlobReferenceClassificationPropertyName) whose frontier is 0: the SST holds no indirect reference. It is still feature evidence | | feature repair | The RepairDB mode for a DB with persisted blob indirection evidence. Legacy repair is the base mode | | canonical path | The first cf_path of a column family. Repair writes its rebuilt and copied blob files there, and only relocation files there can be route targets | | legacy run | A reference run that holds legacy references, each with its SST user key | | legacy group | The repair group of the legacy references of one blob file. Legacy references cannot be routed, so repair keeps that blob file | | fallback input | A legacy run of an exact SST that is not in offset order. That input reads the SST's exact metadata instead of the run | ## Previous behavior Up to diff 13, RepairDB recovered legacy SST and blob files only. A temporary guard returned NotSupported, with inputs untouched, whenever it found positive feature evidence. ColumnFamilyData::ValidateOptions rejected enable_blob_indirection unless a test called ColumnFamilyData::TEST_SetAllowBlobIndirection(true). No open, recovery or DestroyDB step knew about a repair catalog. ## New behavior ### RepairDB transaction - A feature repair writes a repair catalog. Its record types are header, input, intent, seal, archive, coverage, candidate, policy and sync marker; file/filename.h has the field use of each type. Repair starts the catalog by writing the header and the input records of the files found so far to a temporary file. It syncs that file once, renames it to REPAIR-IN-PROGRESS and fsyncs the directory, so a crash cannot leave an empty or torn catalog. A later search after the SST scan adds input records for the blob files it finds. - A catalog record must be durable before the action it allows. So repair syncs the catalog only right before such an action: the rename above, the creation of an output file (its temporary and final intents share one sync), and the candidate. Seals (also a seal written again to mark an output unused), coverage, policy and archive records allow no action before the next sync. Losing them in a crash only makes the retry redo work. - Repair writes a sync marker record just before each catalog sync. A crash can damage records written after the last completed sync. When the reader finds damage after at least one sync marker, and at most one more marker follows the damage (the last marker may itself be unsynced), it treats the damage as a torn unsynced tail and reads the catalog up to the last marker before the damage. Damage before the first marker is Corruption. Damage followed by two or more markers is in synced records and is Corruption. A bad checksum makes the log reader drop the rest of its 32 KB block, so damage to synced records in the last block also reads as a torn tail (at worst it leaves an output of that attempt untracked). - Repair writes its new MANIFEST as a staged MANIFEST (VersionSet::RecoverForRepair()). VersionSet appends every repair edit to that file and never starts another MANIFEST while it stages one. The file gets its MANIFEST name just before CURRENT changes. - Repair converts the WALs one whole WAL at a time and writes the certified floor to the catalog after each WAL (a coverage record). It writes the staged MANIFEST. In it, each CF's log number is the larger of the certified floor and the original floor, so the next open does not replay converted WAL records again. The staged MANIFEST records as its last sequence the larger of the original MANIFEST's last sequence (raised by each converted WAL) and the largest SST sequence. This number can be larger than every SST sequence, for example after a bottommost compaction. CURRENT is written last. - If the original MANIFEST marks a physical file of a column family lost, and repair cannot prove which WAL records that MANIFEST covers (an SST is missing), repair fails with Corruption when WAL files exist: replaying them could bring lost values back. - A retry rolls back the unfinished outputs of a failed attempt. A committed attempt cleans up its inputs. - Each WAL batch is fully decoded before any part of it is inserted. For wide-column entities this uses WideColumnSerialization::Deserialize plus a strict column name order check and BlobIndex::Validate for each blob column, because ForEachBlobIndex does not check the framing of entities without blob columns. A batch that fails these checks is reported as corrupt and skipped. - DestroyDB removes a recognized catalog that has an ownership header (an attempt number and an owner identity), and the repair's REPAIR-<number>-*.dbtmp files. A read-write open that finds no catalog, or finishes a committed one, deletes leftover REPAIR-<number>-*.dbtmp files. Read-write, read-only, best-efforts and secondary opens read the catalog first. A read-write open finishes the cleanup of a committed attempt. Read-only and secondary opens check its seals and change no file. An uncommitted, damaged or ambiguous attempt fails the open. While a recognized catalog exists, best-efforts recovery (VersionSet::TryRecover()) resolves it and then recovers only from the MANIFEST that CURRENT names, so it never picks a MANIFEST that repair has not published. - A REPAIR-IN-PROGRESS file that cannot be read or is not a catalog fails the open only if the replayed MANIFEST shows the DB-global fence, a per-CF fence or relocation files (DBImpl::CheckUnresolvedRepairCatalog()). The check runs right after MANIFEST replay, before WAL replay and before any file is written or deleted. Only when there is no CURRENT and create_if_missing would create a new DB does the open probe the MANIFESTs, SSTs and blob files for feature evidence (HasBlobIndirectionFileEvidence(); the probe has no MANIFEST file checksums, so it opens files with kNoFileChecksumFuncName). Open and RepairDB use the same evidence rules. ### RepairDB route rebuild - After repair, each generation number must be readable in one of two ways. A live generation number is a normal blob file or a relocation file that holds every value that the references with this generation number point to. A routed generation number has a route to a live relocation file. That file serves the generation number (its served files list it) and holds all these values. - Repair groups the references of all SSTs into repair groups. A legacy group keeps its blob file. In a column family with the format fence, there are two exceptions: a blob file with a dropped-file record, and a legacy blob file that is not on disk. Repair records both as dropped (see the "Dropped blob files" bullet in Safety boundaries). - Repair decides the generation numbers of a column family from the largest to the smallest. A relocation file serves only smaller file numbers, so every possible target of a generation number is decided before it. A file that is a routed generation number, or that the original MANIFEST marks lost, is never a target. - For one generation number, repair takes the first candidate that holds all its values, checked with the stored CRC: the normal blob file of that number (only when every reference is unflattened), the relocation file of that number, then each relocation file in the canonical path that serves it, newest first. A newer file that serves a generation number replaced the older ones, which hold more garbage. The first two keep the generation number live; the last one routes it. If no file holds all the values, the generation number routes to a rebuilt relocation file. For an unflattened generation number, repair keeps the normal blob file when a copy passes the checks (ScanNormalBlobFile() checks the header and footer, and ValidateNormalBlobReference() checks every reference of the group): the copy in the canonical path, or a copy in another path of the column family, which repair then copies to the canonical path. If the canonical path holds a file that fails the checks, repair never replaces it before CURRENT commits, and the generation number routes to a relocation file: an existing one that serves it, or a rebuilt one, which can copy the values from the good copy (a donor). - The values of the generation numbers that no file can serve are copied into one rebuilt relocation file per column family, in key order (origin, original offset), and these generation numbers are routed to it. A value that several generation numbers reference is written once. The rebuilt file serves exactly these generation numbers. It is not split by blob_file_size: the values of one generation number can come from several origins and interleave in key order with the values of others, and one route must point to one file. Its number must be at most kMaxBlobRelocationFileNumber (2^62 - 1); a larger number fails the repair with Corruption. - BlobRelocationFileReader::OpenForRepair returns the checked directory of the file, so repair can read its origins and served files. It checks the directory region with the tail size of the file, because repair has no tail size from a MANIFEST. TableReaderOptions::isolate_repair_caches makes a repair reader use no block cache, no persistent cache and no table reader cache reservation. ### RepairDB at scale - The SST scan writes the references of each (SST, source) as a reference run in a repair scratch file (REPAIR-<number>-<session>-refs.dbtmp, an output intent of the catalog). It writes all references of an SST without exact metadata, and the legacy references of an exact SST, each legacy one with its SST user key. While the scan still has an exact SST open, it copies the SST's metadata groups into runs through BlobReferenceIterator and checks that each legacy run equals the SST's metadata group. So the scan reads the data of each SST once. Detection and the scan also read the SST's properties, and a fallback input opens the SST again. A scratch error fails the repair; it never sets an SST aside. - A visit reads runs, not SSTs. So each SST is read once by the scan instead of once per source and visit, a legacy key is checked against the blob record key without an SST lookup, and a visit that reads only runs holds no table reader. At most 64 runs (kMaxRepairCursorFanIn) are merged at once, which keeps one merge at about 256 KB of chunk buffers (64 runs x 4096-byte chunks, an estimate). A larger group is first merged in passes into intermediate runs, all in one scratch file per group that is deleted when the group is done (one catalog intent per such group, plus one directory sync at commit). Equal references, also within one run and in the merge passes, must have the same metadata. Equal legacy references must also have the same user key, and a reference repeated inside one run must have the same user key. A run is freed when its group is done. - A fallback input reads the exact metadata through BlobReferenceIterator and looks up its keys in the SST. Fallback inputs join the visit's merge on top of the 64-run limit. They are not merged in passes, and each holds a table reader during the visit. ### Lost and dropped files in RepairDB Diff 13 adds the lost-file handling that repair builds on: an open with best_efforts_blob_db_repair marks a missing relocation file lost with a loss record and records a missing normal or origin blob file as dropped (kDroppedBlobFile, the base mechanism; earlier diffs apply it to indirect origins too), and reads and compactions treat their values as deleted. This diff makes RepairDB keep these losses, so a restored or leftover copy of such a file never brings its values back: - RepairDB keeps the file record and the loss record of each lost relocation file that a kept SST still references, and the routes that point to it. A lost file that no kept SST uses is dropped. A lost file is never a live generation number or a route target. - In a column family with the format fence, RepairDB keeps the dropped-file records and adds one for a missing legacy blob file (see "Dropped blob files" at the top of db/repair.cc). ### Base RepairDB salvage rules - Only positive persisted evidence selects feature repair: a MANIFEST fence or feature record, a decodable SST descriptor or classification (including a classification with frontier 0), or a validated indirect-origin or relocation blob file. Runtime options and failed probes are not evidence. - Legacy repair output never collects exact metadata, so it never becomes unfenced feature evidence. A legacy repair keeps no per-SST blob reference state. - In legacy repair, a feature SST that detection could not read but the scan can is moved to lost/, and the scan continues with the other SSTs. - A persisted classification with frontier 0 marks its CF as a feature CF, so the repaired MANIFEST fences that CF. ### Other changes - This diff removes the option gate added in diff 2 and its test hook ColumnFamilyData::TEST_SetAllowBlobIndirection. enable_blob_indirection is usable in production from this diff. - Repair attaches the reference summaries of the exact SSTs it writes to its MANIFEST edit, as flush and compaction do. The repaired DB gets its census from the normal open-time build. - BlobReferenceIterator (TableReader::NewBlobReferenceIterator()) walks the exact references of one group of one SST. It holds only the SST's reference index and one run at a time, so a reader that merges one group over many SSTs never holds all their references at once. When the caller gives it a cache reservation manager, it charges its memory there. If the cache refuses the charge (a block cache with strict_capacity_limit), the error, MemoryLimit, is returned. In RepairDB, only fallback inputs charge the column family's block cache. The scan's copy of the metadata is not charged. This charge is a memory reservation, not cached blocks, so it does not conflict with isolate_repair_caches. RepairDB is its only user. - RepairDB can add back an obsolete coarse SST that is still on disk and rebuild the route of the missing old file that the SST references. So a coarse SST can reference a routed generation number, and indirect blob GC keeps the coarse pin of a relocation input. Route retirement also keeps any route whose old file number is at or above the coarse frontier (the lowest frontier of all coarse SSTs), so the route that such an SST reads through stays. The comments in db/blob/blob_reference_metadata.h and DBImpl::IndirectBlobGCJob::SelectInputs() give this reason. - The public db.h comment for RepairDB describes feature repair: the repair catalog, the retry rule, the Corruption when RepairDB cannot tell which MANIFEST is valid, and that it does not make the DB safe to open with an older RocksDB release. - db_bench gets --enable_blob_indirection, --blob_indirection_max_references_per_sst and --indirect_blob_gc_read_coalesce_bytes flags. ### Implementation notes - One set of used output paths and blob file numbers is built once. Relocation candidates are indexed by (CF, origin) and by (CF, served file). A relocation file's owner CF is read from its table properties before it is opened once. - One read of a normal blob record checks both the record CRC and the stored CRC. BlobLogSequentialReader checks record sizes against a read limit before it allocates buffers, and it can skip a payload that repair does not need. - The hash maps and sets that this diff adds to the repair code use UnorderedMap and UnorderedSet (util/hash_containers.h). - The commit step skips the seal check of the kept outputs, the candidate and the archives that this repair sealed itself (a retry still verifies them). It still checks the seal of an unused output before it deletes it, and it reads the candidate MANIFEST to check the log floors and the last sequence. - When repair replays the original MANIFEST, an atomic group keeps an undo log of the entries it changes (a CF drop saves that CF's maps). Original SSTs are kept as {size, unique id}. ### Monitoring - After the route rebuild, RepairDB logs one WARN line per column family with the format fence: the blob files it kept, the relocation files (and how many it rebuilt), the routes, the lost files and the dropped blob files. - Release notes (unreleased_history) describe the feature, its monitoring, and the new enum values BlobFileCreationReason::kIndirectBlobGC and TableReaderCaller::kBlobRelocationFileRead. ## Safety boundaries and unsupported cases - A follower open (DB::OpenAsFollower) does not check the repair catalog. Do not open a follower on a DB while RepairDB runs or after a failed RepairDB. - Dropped blob files (the full rules are in the "Dropped blob files" paragraph at the top of db/repair.cc): - A dropped-file record wins over any copy of the file on disk, valid or corrupt, so a restored or leftover copy never brings its values back. - In a column family with the format fence, RepairDB also records a missing legacy blob file as dropped, so a read of such a value returns NotFound. Base RepairDB keeps the reference, and the read gets Corruption. The reason: a column family with the format fence rejects skip_missing_blob_files, so without the record a later compaction of such a value fails and stops writes. - Any other missing value makes repair fail with Corruption. When the MANIFEST is lost, there are no dropped-file records, so a missing origin always makes repair fail. - In a feature repair, a column family without the format fence keeps base behavior for a missing blob file: the SSTs keep their references, repair writes no record, and a read of such a value gets Corruption. - WAL conversion and dropped files: the WAL guard (the Corruption rule for a column family with a lost file, see "RepairDB transaction") covers only lost relocation files, not dropped blob files. A WAL that is converted again can bring back a value whose blob file has a dropped-file record. Base RepairDB does the same, and the WAL record is the original write of the value. The WAL coverage comment above PrepareOriginalReplayCoverage() in db/repair.cc says this. - RepairDB writes one rebuilt relocation file per column family, also when it is larger than blob_file_size. Rebuilding is needed only when no existing file holds all values of a generation number, for example after a relocation file was damaged. - A legacy reference cannot be routed. If the copy of its blob file in the canonical path is not valid and a good copy is in another path, repair fails with Corruption and asks to move the canonical file away and run RepairDB again. - Feature RepairDB needs temporary space for the reference runs: about the total size of the scanned SSTs (more when the SSTs are compressed; an estimate), plus up to about twice the largest merged group for merge passes. Repair logs the estimate before the scan and the main scratch bytes used after the route rebuild. A scratch write error, such as no space, fails the repair with that error. The scratch files are deleted, and a crash leaves them as catalog intents that the retry removes. - If a catalog is damaged inside its synced records (for example by media corruption), RepairDB and every open return Corruption. Manual recovery: keep a copy of the DB directory; delete REPAIR-IN-PROGRESS and the REPAIR-<number>-*.dbtmp files; then run RepairDB again. Outputs of the damaged attempt that remain (salvage SSTs, rebuilt blob files) are then treated as inputs; this can repeat WAL salvage but loses no data. The same manual steps are needed if a crash loses the candidate record after CURRENT already names the repaired MANIFEST (possible only through the 32 KB block limit above): the catalog then names neither image, and RepairDB and open return "Repair authority is ambiguous" without deleting anything. - An open with a damaged REPAIR-IN-PROGRESS file is not refused when the replayed MANIFEST has no fence and no relocation record; RepairDB is needed for such a DB anyway. - RepairDB does not recover to a consistent point in time, as in base. Feature-format binary downgrade is not supported, and RepairDB does not make the DB safe to open with an older RocksDB release. ## Validation and known limitations - The Test Plan lists the new and updated tests. They cover the catalog, crash and retry points, the route rebuild, dropped and lost files, scale limits and the reference cursor. - This diff does not benchmark repair with a block cache with strict_capacity_limit, where a fallback input can fail with MemoryLimit. - The limits in "Safety boundaries" (one rebuilt file per column family, temporary space, manual recovery from a damaged catalog) are known and documented. ## Stack context (same in diffs 2-14, except that diffs 10-14 add the diff 10 route rule to the install line; "<== this diff" marks this diff's part): Goal: remove garbage from blob files without rewriting any SST. A value gets a stable identity when it is first written. Indirect blob GC copies the live values of whole files into a relocation file and installs routes in the MANIFEST. Readers follow the routes. SSTs are not changed. How a read finds a value (P is the relocation file that holds the value now): ``` SST entry indirect BlobIndex { generation number, origin, original offset, size, CRCs } | v Version file table: generation number -> live file, or route old file -> P | | live file route found | | v v blob file: read at original offset relocation file P (always one hop): relocation file: look up key look up key (origin, original offset) (origin, original offset) ``` Lifecycle of one value (in this example, F is the blob file where the value is first written, and P is the relocation file that GC writes): ``` write new value goes to blob file F; index: origin = F, generation number = F | census per generation number: the SSTs and references that use it; | per physical file: live bytes, ranked by garbage ratio GC job copy the live values of whole input files (stored bytes, checked | with the stored CRC) into relocation file P, keyed by | (origin, original offset) install one MANIFEST edit: add P, route each input generation number still | in use to P (a relocation input that gets an output keeps a route | for all its generation numbers), re-point the routes into the | inputs to P, and remove each input (blob file retirement for a | blob file, relocation file deletion for a relocation file) flatten compaction sets generation number = P in the SSTs it rewrites | route route retirement: route deletion records remove the routes that delete no SST uses ``` Stack map: | Diff | Adds | |---|---| | 1 | Finalize blob metadata outside the DB mutex (prep) | | 2 | Indirect BlobIndex (kIndirectBlob) and exact SST reference metadata | | 3 | Relocation file format and stable read path | | 4 | MANIFEST routes and relocation state in the Version | | 5 | Exact reverse reference census | | 6 | MANIFEST apply outside the DB mutex, with a precondition re-check | | 7 | Coarse flush metadata, per-SST reference cap, epoch runs | | 8 | Indirect blob GC building blocks | | 9 | Indirect blob GC job | | 10 | Flatten generation numbers; route retirement | | 11 | Remote compaction validation | | 12 | Dynamic options and operations | | 13 | Lost relocation files | | 14 | RepairDB rebuilds relocation state; option gate removed <== this diff | | 15 | db_stress and crash test support | Tracking issue: meta#15301 Differential Revision: D125050780
Summary: Pull Request resolved: meta#15683 ## Background This is diff 15 of 15. It tests the feature that diffs 1-14 add, and it uses these terms: - Indirect blob GC: with enable_blob_indirection on, a background job moves the live values out of blob files or relocation files that are at least half garbage, without rewriting any SST. Compaction-time blob GC (base BlobDB GC) moves a value only when compaction rewrites the SST that references it. - Relocation file: the new file that indirect blob GC writes with the moved values. - Route: a MANIFEST record that sends reads of the old blob file number to the relocation file that holds its values. So SSTs keep their old references, and reads still find the values. - Route retirement: a background job that removes the routes that no SST and no running job needs any more, with one MANIFEST edit. - Census: the per-column-family index of which SSTs reference which blob files (BlobReferenceCensus). Route retirement uses it to pick routes that no SST needs. - Coarse SST: an SST that records only its oldest referenced blob file, not an exact list (flush output and some L0 compaction output). - Format fence: a MANIFEST record that marks a column family that may hold blob indirection data. It stays set after the option is turned off (see "MANIFEST records" in db/blob/blob_relocation_manifest.h). For the option, read the enable_blob_indirection comment in include/rocksdb/advanced_options.h. For a worked example of relocation files and routes, read the top comment of db/blob/blob_relocation_manifest.h. ## Why the change is required Without this diff, the Blob GC stack (diffs 1-14) has no stress coverage. Indirect blob GC, route retirement, turning enable_blob_indirection on and off, and crash recovery during a GC install are tested only by unit tests. ## Parts of this diff 1. RocksDB library (small): 4 TEST_KILL_RANDOM points in indirect blob GC and route retirement. The crash test needs them. 2. db_stress: new flags, startup checks, SetOptions choices, two reports, and two errors that db_stress accepts. 3. db_crashtest.py: two parameter groups for blob indirection and a fourth kill mode. It uses the db_stress flags of part 2. ## How to read this diff 1. Read "Background" above. Then read the enable_blob_indirection comment in include/rocksdb/advanced_options.h and the top comment of db/blob/blob_relocation_manifest.h. 2. Library changes (they depend on nothing else here). Read the 4 TEST_KILL_RANDOM points in db/db_impl/db_impl_compaction_flush.cc. 3. db_stress flags. Read the new flags in db_stress_tool/db_stress_gflags.cc (declared in db_stress_common.h). Then read where InitializeOptionsFromFlags() in db_stress_test_base.cc copies them into the options. Every later step uses these flags. 4. Startup rules and the debug hook. In db_stress_tool/db_stress_tool.cc, read the validation block and the sync point callbacks for blob_indirection_retire_routes_at_once. 5. Runtime behavior. Read the SetOptions table in StressTest::BuildOptionsTable() (db_stress_test_base.cc). Then read the two reports, ReportBlobIndirectionState() and ReportBlobIndirectionGc() (db_stress_test_base.h and .cc), and where db_stress_driver.cc calls them. 6. Accepted errors. Read IsRetryableBlobIndirectionIngestionError() in no_batched_ops_stress.cc and the local fallback in db_stress_compaction_service.cc. 7. Crash test last. In tools/db_crashtest.py, read _blob_indirection_gc_choice(), blob_indirection_params and blob_indirection_toggle_params, the blob indirection block in finalize_and_sanitize(), the blob_profiles choice in gen_cmd_params(), and kill mode 3 in whitebox_crash_main(). ## Previous behavior db_stress and db_crashtest.py could not turn on blob indirection, could not change it with SetOptions, and had no way to show whether indirect blob GC ran. When the remote worker used up its reserved blob file numbers, the db_stress compaction service returned a failure, and with local fallback off this became a background error. ## New behavior ### RocksDB library - New TEST_KILL_RANDOM points for whitebox crash tests: before GC renames its outputs, before and after the GC MANIFEST write, and before the route retirement MANIFEST write. ### db_stress - New flags: - enable_blob_indirection: the option, fixed for the run. - allow_setting_blob_indirection_dynamically: SetOptions turns the option on and off, mostly on. - blob_indirection_require_gc: fail the run if indirect blob GC collected no input file. It needs --statistics. - blob_indirection_max_references_per_sst: the per-SST blob reference cap of compaction outputs. - pin_blob_relocation_location_blocks: keep the location blocks of relocation files in memory. - blob_indirection_retire_routes_at_once (debug builds only): route retirement removes the routes it picked at once. By default it waits for a batch of 64 routes or for 10 minutes (kBlobRouteRetirementBatchSize and kBlobRouteRetirementMaxPendingMicros). - When enable_blob_indirection or allow_setting_blob_indirection_dynamically is set, db_stress rejects these combinations at startup: no blob files, blob direct write, FIFO, and a table format that is not block-based (the rules of ColumnFamilyData::ValidateOptions), and stacked BlobDB (a db_stress-only rule). It rejects FIFO also when only allow_setting_blob_indirection_dynamically is set and the option starts off, because a column family that has used blob indirection keeps rejecting FIFO. - When blob options are dynamic (allow_setting_blob_options_dynamically) and the option is on or may be turned on, SetOptions keeps blob files on, picks small blob file sizes, keeps enable_blob_garbage_collection on most of the time, picks the reference cap from 4096, 64K and 1M, and may change pin_blob_relocation_location_blocks. - At each start, db_stress prints the relocation files and routes that the DB holds. Whitebox crash tests kill most runs before the end-of-run report, so this line shows the GC work that earlier runs made durable. - At the end of a run, db_stress prints how many input files indirect blob GC collected, how many relocation files it wrote, and how many routes route retirement removed. It reads the tickers BLOB_DB_GC_NUM_FILES, BLOB_DB_GC_NUM_NEW_FILES and BLOB_RELOCATION_ROUTES_RETIRED (diff 10). For example, the line reads "Blob indirection GC: 104 input files retired, 44 relocation files written, 19 routes retired" (example counts; "retired" is the current log text). - While the option may be on, db_stress accepts TryAgain from file ingestion instead of failing the run. Ingestion returns TryAgain when one of these became true after the external files were prepared: the option is on, the column family has the format fence, or a flush or compaction that was picked with the option on is still running. An ingestion that returns TryAgain ingests nothing. The check accepts any TryAgain, because that case has no status subcode of its own. - The db_stress compaction service runs a remote job locally when the worker uses up the range of blob file numbers that the primary reserved for it (Incomplete "Blob file number range exhausted"). Nothing is installed in that case. This gap exists without this stack too; small blob files only make it common. ### db_crashtest.py - Two parameter groups: fixed (the option stays the same across restarts) and toggle (SetOptions and each restart turn it on and off, on 3 of 4 times). When a run picks a blob profile, the fixed group and the toggle group are added to the choice with weights 4 and 1. --test_blob_indirection and --test_blob_indirection_toggle force them. - Both groups make indirect blob GC run within a short test: - a small key space: 10000 keys 3 in 4 times, else 100000; - a write-heavy operation mix: 55% writes and 14% deletes; - small memtables, small SSTs and small blob files; - a reference cap of 4096, 64K or 1M, so that small caps often start a new compaction output. - Half of the runs run route retirement at once (blob_indirection_retire_routes_at_once). The fourth kill mode (see below) always does. - Each db_stress start in the first 300 seconds after the first db_stress command turns enable_blob_garbage_collection on (300 seconds is a fixed choice, not measured). After that, each restart may start with it off. SetOptions may also turn it off for a while when blob options are dynamic. - A step in finalize_and_sanitize() turns off both enable_blob_indirection and allow_setting_blob_indirection_dynamically when the run has no blob files, blob direct write, stacked BlobDB, FIFO or a full external table (the db_stress startup rules). It runs after the other sanitizing steps, because they can change these options. - Whitebox runs with blob indirection add a fourth kill mode. It sets kill_random_test to 20, so each kill point that is not excluded crashes the process about once in 20 calls. It excludes the frequent kill points (file writes, MANIFEST sync, CURRENT and LogAndApply), and runs route retirement at once. So the indirect blob GC and route retirement kill points get a chance to fire. The kill mode 3 comment in db_crashtest.py explains the choices. ## Implementation Stress options are set in InitializeOptionsFromFlags() and in the SetOptions table (StressTest::BuildOptionsTable()) in db_stress_test_base.cc. The indirect blob GC report runs from the driver before statistics are printed. Data integrity is checked by the existing expected-state verification, the reopen checks and VerifyChecksum, which also reads relocation files. ## Safety boundaries and limitations - Test-only code, except the TEST_KILL_RANDOM points (compiled out in release builds). No public API change. - No end-of-run check that every obsolete blob file or relocation file was deleted: injected MANIFEST faults keep files on purpose, and deletion can be asynchronous. - db_crashtest.py does not use blob_indirection_require_gc, because a short crash run may see no indirect blob GC run. - Blackbox runs are killed, so they do not print the end-of-run GC report. The start-of-run line (relocation files and routes in the DB) shows the GC work of earlier runs. Tracking issue: meta#15301 ## Stack context | Diff | Adds | |---|---| | 1 | Finalize blob metadata outside the DB mutex (prep) | | 2 | Indirect BlobIndex (kIndirectBlob) and exact SST reference metadata | | 3 | Relocation file format and stable read path | | 4 | MANIFEST routes and relocation state in the Version | | 5 | Exact reverse reference census | | 6 | MANIFEST apply outside the DB mutex, with a precondition re-check | | 7 | Coarse flush metadata, per-SST reference cap, epoch runs | | 8 | Indirect blob GC building blocks | | 9 | Indirect blob GC job | | 10 | Flatten generation numbers; route retirement | | 11 | Remote compaction validation | | 12 | Dynamic options and operations | | 13 | Lost relocation files | | 14 | RepairDB rebuilds relocation state; option gate removed | | 15 | db_stress and crash test support <== this diff | Differential Revision: D129346853
xingbowang
force-pushed
the
export-D129346853
branch
from
November 18, 2026 19:01
f9a3440 to
4e6942d
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary:
Background
This is diff 15 of 15. It tests the feature that diffs 1-14 add, and it uses these terms:
For the option, read the enable_blob_indirection comment in include/rocksdb/advanced_options.h. For a worked example of relocation files and routes, read the top comment of db/blob/blob_relocation_manifest.h.
Why the change is required
Without this diff, the Blob GC stack (diffs 1-14) has no stress coverage. Indirect blob GC, route retirement, turning enable_blob_indirection on and off, and crash recovery during a GC install are tested only by unit tests.
Parts of this diff
How to read this diff
Previous behavior
db_stress and db_crashtest.py could not turn on blob indirection, could not change it with SetOptions, and had no way to show whether indirect blob GC ran. When the remote worker used up its reserved blob file numbers, the db_stress compaction service returned a failure, and with local fallback off this became a background error.
New behavior
RocksDB library
db_stress
db_crashtest.py
Implementation
Stress options are set in InitializeOptionsFromFlags() and in the SetOptions table (StressTest::BuildOptionsTable()) in db_stress_test_base.cc. The indirect blob GC report runs from the driver before statistics are printed. Data integrity is checked by the existing expected-state verification, the reopen checks and VerifyChecksum, which also reads relocation files.
Safety boundaries and limitations
Tracking issue: #15301
Stack context
Differential Revision: D129346853