Repository navigation
2/15: Add indirect BlobIndex and exact SST metadata (#15540) - #15540
xingbowang wants to merge 2 commits into
Conversation
|
@xingbowang has exported this pull request. If you are a Meta employee, you can view the originating Diff in D124652097. |
|
| Check | Count |
|---|---|
bugprone-argument-comment |
4 |
cppcoreguidelines-avoid-non-const-global-variables |
2 |
cppcoreguidelines-missing-std-forward |
1 |
performance-inefficient-vector-operation |
1 |
performance-no-automatic-move |
1 |
| Total | 9 |
Details
db/blob/blob_garbage_meter_test.cc (2 warning(s))
db/blob/blob_garbage_meter_test.cc:160:45: warning: argument name 'generation' in comment does not match parameter name 'generation_file_number' [bugprone-argument-comment]
db/blob/blob_garbage_meter_test.cc:168:37: warning: argument name 'seq' in comment does not match parameter name 's' [bugprone-argument-comment]
db/blob/blob_reference_metadata.cc (2 warning(s))
db/blob/blob_reference_metadata.cc:34:23: warning: variable 'unsorted_group_fallbacks' is non-const and globally accessible, consider making it const [cppcoreguidelines-avoid-non-const-global-variables]
db/blob/blob_reference_metadata.cc:884:10: warning: constness of 'release_status' prevents automatic move [performance-no-automatic-move]
db/column_family.cc (1 warning(s))
db/column_family.cc:1554:19: warning: variable 'allow_blob_indirection_for_test' is non-const and globally accessible, consider making it const [cppcoreguidelines-avoid-non-const-global-variables]
db/db_options_test.cc (1 warning(s))
db/db_options_test.cc:2233:5: warning: 'push_back' is called inside a loop; consider pre-allocating the container capacity before the loop [performance-inefficient-vector-operation]
db/memtable_list.cc (1 warning(s))
db/memtable_list.cc:1147:25: warning: argument name 'column_family_options' in comment does not match parameter name 'new_cf_options' [bugprone-argument-comment]
db/version_edit_test.cc (1 warning(s))
db/version_edit_test.cc:898:45: warning: argument name 'generation' in comment does not match parameter name 'generation_file_number' [bugprone-argument-comment]
db/wide/wide_column_serialization.cc (1 warning(s))
db/wide/wide_column_serialization.cc:352:55: warning: forwarding reference parameter 'visit' is never forwarded inside the function body [cppcoreguidelines-missing-std-forward]
Claude Code Review - OBSOLETESuperseded by a newer AI review. Expand to see the original review.✅ Claude Code ReviewAuto-triggered after CI passed — reviewing commit b0bad4a SummaryLarge, well-structured PR that adds the foundation for stable blob indirection: a new BlobIndex type, SST-level blob reference metadata (collector, block codecs, descriptor), a new CF option gated behind test-only validation, and supporting infrastructure (CacheEntryRole, CacheReservationHandle::Resize). The design is defense-in-depth with checksums at every layer and extensive bounds checking against crafted inputs. The diff is truncated so some changes (src.mk, Makefile, WideColumnSerialization::ForEachBlobIndex, additional tests, table builder integration) could not be verified. High-severity findings (2):
Full review (click to expand)Findings🔴 HIGHH1. Build system: src.mk and Makefile missing
|
| Context | Does code execute? | Assumptions hold? | Action needed? |
|---|---|---|---|
| WritePreparedTxnDB | YES (flush/compaction) | YES - gated by option | Safe |
| ReadOnly DB | NO (no writes) | N/A | Safe |
| CompactionService | YES | Verify MutableCFOptions serialization | Check |
| User-defined timestamps | YES | YES - metadata is key-independent | Safe |
| MemPurge | YES | YES - same BlobFileBuilder path | Safe |
| SstFileWriter | Partially - skips collection | YES | Safe |
| FIFO/Universal compaction | YES | YES | Safe |
Positive Observations
- Defense in depth: Every serialized block has magic, version, checksums. Canonical varint validation rejects non-canonical encodings. Extensive bounds checking with explicit size limits (
kMaxReferenceCount,kMaxGroupCount,kMaxDataBlockSize,kMaxTopLevelMetadataBlockSize). - Memory accounting: The collector tracks reservation memory including vector capacity, hash table nodes, and bucket arrays.
ReserveCapacityensures reservation is updated before allocation. - Fail-closed design: All encoding errors, count overflows, and checksum mismatches return Status errors.
- Test coverage: Visible tests cover resize semantics, shared reservation quantum, failed resize ownership, and identity blob round-trip.
- Clear documentation: PR description thoroughly documents safety, compatibility, limitations, and the prefix gate strategy.
ℹ️ About this response
Generated by Claude Code.
Review methodology: claude_md/ci_review_prompt.md
Limitations:
- Claude may miss context from files not in the diff
- Large PRs may be truncated
- Always apply human judgment to AI suggestions
Commands:
/claude-review [context]— Request a code review/claude-query <question>— Ask about the PR or codebase
Summary: Pull Request resolved: meta#15540 Why the change is required: The goal of this stack is a blob GC that does not depend on SSTs: it removes garbage from blob files without rewriting any SST. Such a GC needs two things that RocksDB does not have: - A value identity that does not change when the value moves to another file. - A checked list of the blob references that each SST keeps. The list must include older internal versions of a key that the SST still keeps (for example for snapshots), and the blob columns of wide-column entities. This diff adds both. Later diffs build the census (an in-memory reverse index from generation number to the SSTs that use it), GC and repair on top of this list. This diff in one paragraph: With the new column family option enable_blob_indirection on, the blob file writer gives each new value an indirect BlobIndex (type 3, kIndirectBlob). It stores the value's stable identity (origin file, original offset) and its CRCs (an uncompressed value stores only the CRC of the uncompressed value; a compressed value also stores the CRC of its stored bytes). Also with the option on, the block-based table builder collects every blob reference that the SST keeps. It writes them as "exact reference metadata": data blocks, one reference index block, and a 72-byte descriptor table property. In this diff only the table builder writes this metadata, and only tests read it. A temporary option gate (see "option gate" under Terms) keeps the option off outside tests. Separately, wide-column entities with blob columns are now fully checked before their blob indexes are used, also with the option off. Parts of this diff: 1. Indirect BlobIndex (core): types 3 and 4 in db/blob/blob_index.h, and the blob file writer that writes type 3. 2. Exact reference metadata format and decoders (core): db/blob/blob_reference_metadata.{h,cc}. 3. Collector and table builder integration (core): BlobReferenceMetadataCollector, its block cache charge, and BlockBasedTableBuilder. 4. Option enable_blob_indirection and the option gate. Tests need it to turn the feature on. You can skip this part on a first read. 5. Wide-column validation (WideColumnSerialization::ForEachBlobIndex). The table builder uses it to give the blob columns of each entity to the collector. It also changes behavior with the option off (see New behavior). You can skip this part on a first read. 6. Small helpers: canonical varints (util/coding.h) and CacheReservationHandle::Resize(). You can skip these on a first read. How to read this diff: 1. Overview: read "Why" and "This diff in one paragraph" above, and "Terms used in this diff" right below. Then read the top comment of db/blob/blob_reference_metadata.h (terms, layout diagrams, run and entry layout, index layout). 2. Indirect BlobIndex (db/blob/blob_index.h) and canonical varints (util/coding.h GetCanonicalVarint32/64): read the overview comment and the kIndirectBlob / kIndirectFlattenedBlob code. Every later part uses these fields: origin, original offset, generation number, logical CRC and stored CRC. The new formats read varints only in the canonical form. 3. Blob file writer (db/blob/blob_file_builder.cc, blob_log_format.cc, blob_log_writer.cc): see how a new value gets a type-3 index and how its CRC is computed once. 4. Format types and decoders: the structs in db/blob/blob_reference_metadata.h, then in the .cc the constants, GapBase, DecodeEntryBody, DecodeRunEntriesAt, DecodeBlobReferenceIndex and BlobReferenceGroupDecoder. They use only BlobIndex and the canonical varints. Learn the on-disk format here, before you read the writer. 5. Collector and its cache charge: first read the new CacheEntryRole kBlobReferenceMetadataBuildingBuffer (include/rocksdb/cache.h) and CacheReservationHandle::Resize (cache/cache_reservation_manager.h). Then read BlobReferenceMetadataCollector in this order: the class comment, Add, AppendSorted, BuildAndWriteDataBlocks, EncodeIndex, FinishOnce. Then read the unsorted fallback (AppendUnsorted, SortUnsortedGroup, SortAndEncodeGroup). 6. Wide-column change (db/wide/wide_column_serialization.cc, ForEachBlobIndex and WalkV2Columns): it gives every blob index of an entity to a callback after it checks the whole entity. The table builder uses it to feed the collector. UpdateBoundaries, BlobGarbageMeter and the compaction iterator also use it, with the option on or off. 7. Option: enable_blob_indirection (include/rocksdb/advanced_options.h, options/cf_options.cc), and the option checks and gate in db/column_family.cc (ValidateOptions, TEST_SetAllowBlobIndirection), db/db_impl/db_impl_open.cc and db_impl_secondary.cc (the FIFO check for opens that skip ValidateOptions). 8. Table integration: table/block_based/block_based_table_factory.cc (the shared reservation manager), then block_based_table_builder.cc (Rep constructor, Add, WriteBlobReferenceMetadata, EstimatedTailSize). See where the collector is created, fed and finished. block_based_table_reader.cc maps the new block name to a block type. Terms used in this diff: Stack terms: | Term | Meaning | |---|---| | origin | Blob file where the value was first written | | original offset | Value offset in the origin file | | stable identity | (origin, original offset). Never changes, even when the value moves | | generation number | The file number that a reference uses to find its value. A new value has generation number == origin. When GC moves the value, the reference keeps its generation number, and a route sends the read to the relocation file | | indirect BlobIndex | A kIndirectBlob (type 3) or kIndirectFlattenedBlob (type 4) index. It stores the stable identity, so the value can move without an SST rewrite | | unflattened reference | Index with generation number == origin: the value was not moved, or it was moved but no compaction has rewritten its reference yet (see "flatten" in "Stack context: terms for later diffs") | | flattened reference | Index with generation number != origin (type 4). Only flatten (diff 8) writes it; this diff defines the encoding | | route | Maps an old file number (a file that GC has removed from the Version) to the relocation file that now holds its live values. One hop: a target is never itself routed. Stored in the MANIFEST (diff 4) | | relocation file | Block-based table keyed by (origin, original offset) that holds copied values. Each value is the stored bytes plus a 5-byte trailer (kSimpleGen2BlobTrailerSize) | | logical CRC | CRC32C of the uncompressed value (BlobIndex::stable_checksum()) | | stored CRC | CRC32C of the stored bytes plus the compression byte | | physical file | A file that holds value bytes: a normal blob file or a relocation file. A generation number can name a physical file, or it can be only the key of a route | | exact reference metadata | Per-SST sorted list of every blob reference the SST keeps, written when the SST is built. References with the same encoding class, generation number and origin form a group. The list is stored as data blocks, with an index of the groups. An SST that has it is an exact SST | | option gate | Until diff 11, ColumnFamilyData::ValidateOptions returns NotSupported for enable_blob_indirection on a read-write open, CreateColumnFamily and SetOptions, unless a test calls ColumnFamilyData::TEST_SetAllowBlobIndirection(true). It keeps the option off in production while the stack is not complete | Terms of this diff (all defined at the top of db/blob/blob_reference_metadata.h): | Term | Meaning | |---|---| | encoding class (class) | kDirect (a legacy kBlob or kBlobTTL index), unflattened, or flattened | | reference group (group) | All references of one SST with the same (encoding class, generation number, origin) | | run | The part of one group inside one data block | | record bytes | The bytes the value's record takes in the file named by its generation number | | gap | How far a record is from where it would be if the records were back to back. A gap of 0 means back to back | | descriptor | The 72-byte table property that points to the reference index block and holds the totals, the CF ID, the SST file number and the relocation-state fingerprint | | relocation-state fingerprint | 128-bit hash of the routes and relocation files of the Version that an SST was built from (fields route_snapshot_fingerprint_*). Written as 0 in this diff | | legacy | Written the way RocksDB writes it without blob indirection | Previous behavior: - A BlobIndex named only a physical blob file and an offset. A value could not move to another file without rewriting every SST that referenced it. - An SST had no metadata that listed its blob references. The only way to find them was to scan its user keys. - These readers get the blob indexes of a wide-column entity through ForEachBlobFileNumber: UpdateBoundaries, BlobGarbageMeter, the compaction iterator, WideColumnSerialization::RemoveBlobColumns, and the referenced_blob_bytes table property count in BlockBasedTableBuilder::Add. ForEachBlobFileNumber decoded each blob index, but it did not check the rest of the entity framing (name order, section sizes, exact input size), and it could call the callback before it found a later error. New behavior: Separate change (option on or off): - The same readers (UpdateBoundaries, BlobGarbageMeter, the compaction iterator, RemoveBlobColumns, and the referenced_blob_bytes count in BlockBasedTableBuilder::Add) use WideColumnSerialization::ForEachBlobIndex, with the option on or off. With the option on, the table builder also uses it to give the blob columns of each entity to the collector. For an entity with blob columns, it checks the full framing, then decodes and validates every blob index before any callback runs. So an entity with a malformed framing now fails the flush or compaction, even with the option off. The referenced_blob_bytes count itself does not fail the build on a malformed entity. It removes that property from the SST, as in base. Option and option gate: - This diff adds a new column family option, `enable_blob_indirection`. It is off by default, and SetOptions can change it. - The option gate (see Terms) keeps the option off outside tests until the stack is complete. ColumnFamilyData::ValidateOptions rejects the option with NotSupported on a read-write open, CreateColumnFamily and SetOptions. Read-only opens (DBImplReadOnly and CompactedDBImpl) and secondary opens (including DB::OpenAndCompact) do not run ValidateOptions; they check only the FIFO rule (see Implementation). Only tests can turn the option on, through the internal ColumnFamilyData::TEST_SetAllowBlobIndirection(). DBTestBase sets it. - When the option is on, ValidateOptions also requires enable_blob_files, the block-based table format, no blob direct write, and a compaction style other than FIFO. Blob file writer (option on): - The blob file writer encodes each new value with a stable identity (origin file, original offset) and two checksums: - the logical CRC: the CRC32C of the uncompressed value. - the stored CRC: the CRC32C of the exact stored (maybe compressed) bytes, extended with the compression byte. - For an uncompressed value the stored CRC is not written, because the reader can compute it from the logical CRC. - The checksums have no version field of their own. The BlobIndex type byte gives the format version: the two new types, 3 (kIndirectBlob) and 4 (kIndirectFlattenedBlob, see Implementation), are BlobIndex::kIndirectBlobFormatVersion (1). A later format uses new type values. - The read path does not check these checksums yet. The check starts in diff 3. Exact reference metadata (option on): 1. What: a sorted list of every blob reference that the SST keeps, in plain blob values and in the blob columns of wide-column entities. 2. Where: data blocks (target 16 KB each before compression; small blocks waste less I/O, because GC reads a whole block to get a few groups) and one reference index block, written after the SST data blocks and before the SST tail, plus a 72-byte descriptor table property. 3. Who reads it: only tests in this diff. The census, GC and RepairDB read it in later diffs. 4. Details: the top comment of db/blob/blob_reference_metadata.h has the full format with diagrams, the record-bytes rules and the gap rule. - Not collected: SstFileWriter output (it has no DB file number) and any output whose CF is unknown (it has no CF ID). The descriptor needs both values, so these outputs stay in the legacy form. Checks on the metadata: - A repeated or overlapping reference in one group fails the SST build with Corruption. Readers check every count, order, size and total against the index and the descriptor, so corrupt metadata returns Corruption. The writer never makes a data block larger than kTargetDataBlockSize (16 KB), and readers accept up to kMaxDataBlockSize (64 KB). Details: "Readers" and "Integrity" at the top of db/blob/blob_reference_metadata.h, and BlobReferenceGroupDecoder. Collection and memory: - Each reference is encoded when it is added, in the on-disk entry encoding, into its group's chain of 512-byte chunks. Only a group that takes the unsorted fallback is encoded a second time, in Finish(). Inside a group, references normally arrive in rising offset order, so no reference is sorted; at the end only the groups are sorted. A group that gets a lower offset takes the unsorted fallback. The BlobReferenceMetadataCollector class comment explains the chunks, the chunk size and the fallback. - The collector keeps every reference of the SST in memory until the SST is finished, and charges this memory to the block cache under a new CacheEntryRole, kBlobReferenceMetadataBuildingBuffer. The measured cost is about 8.3 to 9 bytes per reference for uncompressed values and 12.8 to 13.5 bytes for compressed values. "Memory usage" and "Memory charge" in the BlobReferenceMetadataCollector class comment give the measurement setup and the charge rule. - There is no fixed limit on references. The collector fails with MemoryLimit above 2^20 groups or 2^20 data blocks in one SST; these are persisted format limits. - There is no charge when the table has no block cache or when charging for this role is disabled. Each block-based table factory holds one cache reservation manager for this role, shared by its builders and the SST summaries. If the charge fails, the SST build fails, so the flush or compaction fails with a background error. The buffers and their charge are released when the SST is finished or abandoned. Implementation: - BlobIndex adds two indirect encodings. The legacy encodings do not change. - Type 3, kIndirectBlob (unflattened, generation number == origin), stores the origin, the original offset, the stored size, the compression type, the logical CRC and, for compressed values, the stored CRC. It does not store the generation number. This saves the size of one varint file number per reference (about 3 to 4 bytes for typical file numbers; an estimate, not measured). - Type 4, kIndirectFlattenedBlob (flattened), stores the generation number first, then the same fields. Its generation number must be above the origin. The decoder rejects type 4 with generation number <= origin, so each reference has one encoding. - The encoders take the fields as one BlobIndex::IndirectBlobFields, set by field name, so values of the same type (file number, offset and size; the two CRCs) cannot be swapped. EncodeIndirectBlob() takes the generation number as a separate argument; EncodeUnflattenedIndirectBlob() uses the origin, so the common path cannot leave the generation number unset. - The new files blob_reference_metadata.{h,cc} define the exact reference metadata. The files also contain: - BlobReferenceMetadataCollector, the collector described above. It takes the SST identity (CF ID, SST file number and the relocation-state fingerprint) as one BlobReferenceSstIdentity, set by field name. - BlobReferenceGroupKey, the key of a group: (encoding class, generation number, origin). It has the order, the hash and the validity check that the collector map, the sort, the decoder and FindBlobReferenceGroup() use. BlobReference and BlobReferenceGroup keep their flat fields and return the key through group_key(). - The index codec and the descriptor table property. - BlobReferenceGroupDecoder, the only decoder of reference runs. It reads a group one data block at a time and holds every corruption check of a group walk. Readers that visit one group (found by a binary search of the index) use it, and so do readers that visit every block in order. - Callers in this diff: only the table builder runs the collector. The decoders (DecodeBlobReferenceIndex, FindBlobReferenceGroup, BlobReferenceGroupDecoder, ForEachBlobReference*) are used only by tests. kMaxDecodedBlobReferenceBytes is used only in compile-time checks of the count limits. The census (diff 5) and GC (diffs 6 and 7) are their first production users. The census is also the first user of BlobReferenceSstSummary (see the last Implementation bullet). - Descriptor: the CF ID and the SST file number are the repair fields. They let RepairDB tell which column family and SST the list belongs to without a MANIFEST. The decoder rejects a descriptor whose repair fields are not set (unknown CF ID or SST file number 0). The descriptor also has a 128-bit relocation-state fingerprint (the route_snapshot_fingerprint fields). Remote compaction uses it in diff 9 to check that an SST built elsewhere used the same routes as the job that asked for it. In this diff the table builder always writes 0, and DecodeFrom() does not check it. The field is in the format now, so the descriptor keeps one format version (1) through the stack. While the option gate is on, no data in this format exists outside tests. - The collector validates each BlobIndex once, when it is added. Its Finish() writes the data blocks and then the reference index block through one block writer, and fills the descriptor, including the index handle. Destroying the collector frees its memory and then its charge, also when the build is abandoned. - BlockBasedTableBuilder creates the collector with the CF ID and SST file number from TableBuilderOptions. It passes the size of each entry's user key to the collector, so the collector can compute the exact size of each blob record. It adds the collector's EstimatedEncodedSize() to its tail size estimate: the blocks are written before the tail, but output file size planning uses this estimate. The reference blocks are written with BlockType::kUserDefinedIndex, which keeps their contents as opaque bytes. Readers read them without the block cache, so prepopulate_block_cache does not warm them into the block cache (WriteMaybeCompressedBlock() with warm_cache false). - With the option on, the blob file builder computes the CRC32C of each stored value once. The blob log record CRC combines it with the key CRC (crc32c::Crc32cCombine). The stored CRC extends it with the compression byte. For an uncompressed value it is also the logical CRC. The bytes written do not change. - WideColumnSerialization::ForEachBlobIndex replaces ForEachBlobFileNumber. It reads the entity header and column types once. For a V1 entity (no column types and no blob columns), it returns right after the header. For a V2 entity without blob columns, it returns right after the type scan. It shares one V2 column walker (WalkV2Columns) with Deserialize(), so the entity is parsed in one pass. - Read-only opens (DBImplReadOnly, CompactedDBImpl) and secondary opens skip ValidateOptions, so they check the compaction style against the option once per column family: in DBImpl::Recover() for read-only opens, and at the start of OpenAsSecondaryImpl() for secondary opens. DB::OpenAndCompact opens through OpenAsSecondaryImpl(), so it gets the same check. - CacheReservationHandle gets Resize(), so each collector or summary can change its own part of the shared charge. - util/coding.h adds GetCanonicalVarint32/64, which reject overlong varint encodings. The new formats use them, so each value has one encoding. - BlobReferenceSstSummary is the in-memory list of an SST's groups and totals that Finish() can return. It has no caller yet: the table builder passes a null summary output. It is checked and does not change after Finish(), so the finished builder, the SST metadata and the census (its first user, in diff 5) can share it. It is in this diff because Finish() hands the group buffer and its cache charge over to the summary. Adding the summary later would mean rewriting the charge code of Finish() in diff 5, so the charge code is written and tested once, here. Safety boundaries and unsupported cases: - Public API: this diff adds the option `enable_blob_indirection` with the C API functions `rocksdb_options_set_enable_blob_indirection` and `rocksdb_options_get_enable_blob_indirection`. It also inserts a new CacheEntryRole value before kMisc, which changes the numeric value of kMisc and kNumCacheEntryRoles. - With the option off, new blob values and SSTs are written in the legacy form. Legacy blob indexes stay readable. - One behavior change applies with the option off too: UpdateBoundaries, BlobGarbageMeter and the compaction iterator fully validate wide-column entities with blob columns and their blob indexes (see "Separate change" above). - Exact reference metadata has one format (version 1) with a fixed 72-byte descriptor. The decoders reject any other version as Corruption. - The metadata blocks are protected by the table's block checksum only. With checksum = kNoChecksum, corrupt metadata is detected only when it breaks the format or order checks. - The index decoder checks each decoded count against the remaining input size before it reserves memory. So a small corrupt index cannot make it allocate a large vector. - Inlined-TTL and same-file blob indexes are not collected and cannot be relocated. Encoding, count and checksum errors return an error; a bad reference is never skipped. - Option gate (see Terms): outside tests no DB writes data in the new format. The gate checks only the option value, never persisted state. Diff 11 removes it, once data written with the option stays readable through the end of the stack and the safety checks for turning the option on are in place. - Downgrading the binary after data was written with the option on is not supported. Validation and known limitations: - Unit tests cover the BlobIndex encodings and their rejection of bad input, the metadata layout and every corruption check, the collector (sorted path, unsorted fallback, memory charge), the flush integration, the option checks and option gate, wide-column validation, and canonical varints. See the Test Plan. - The collection buffer is not spilled to disk. One SST with very many blob references needs a matching amount of charged memory: the per-reference amount given under "Collection and memory", plus up to one partly used 512-byte chunk per group. While a fallback group is sorted, it also needs 16 bytes of sort key per reference and a second copy of its entries. For a test group of 200K references with every other one out of order, the test comment estimates about 44 bytes per reference, and the test checks that the peak charge is at most 56 bytes per reference. - Flush still writes a single output file. Until diff 7, one compaction output collects every reference it gets, charged to the block cache. Diff 7 cuts compaction outputs (including L0) at a reference cap and keeps each user key's history in one output. With the option gate, a large output is reachable only through the test hook. Stack context (same in diffs 2-11, except that diffs 8-11 add the diff 8 route rule to the install line; "<== this diff" marks this diff's part): Goal: remove garbage from blob files without rewriting any SST. A value gets a stable identity when it is first written. GC copies the live values of whole files into a relocation file and installs routes in the MANIFEST. Readers follow the routes. SSTs are not changed. How a read finds a value (P is the relocation file that holds the value now): ``` SST entry indirect BlobIndex { generation number, origin, original offset, size, CRCs } | v Version file table: generation number -> live file, or route old file -> P | | live file route found | | v v blob file: read at original offset relocation file P (always one hop): relocation file: look up key look up key (origin, original offset) (origin, original offset) ``` Lifecycle of one value (in this example, F is the blob file where the value is first written, and P is the relocation file that GC writes): ``` write new value goes to blob file F; index: origin = F, generation number = F | census per generation number: the SSTs and references that use it; | per physical file: live bytes, ranked by garbage ratio GC job copy the live values of whole input files (stored bytes, checked | with the stored CRC) into relocation file P, keyed by | (origin, original offset) install one MANIFEST edit: add P, route each input generation number still | in use to P, re-point the routes into the inputs to P, and remove | each input (blob file retirement for a blob file, relocation | file deletion for a relocation file) flatten compaction sets generation number = P in the SSTs it rewrites | route route retirement: route deletion records remove the routes that delete no SST uses ``` Stack map: | Diff | Adds | |---|---| | 1 | Finalize blob metadata outside the DB mutex (prep) | | 2 | Indirect BlobIndex (kIndirectBlob) and exact SST reference metadata <== this diff | | 3 | Relocation file format and stable read path | | 4 | MANIFEST routes and Version ownership | | 5 | Exact reverse reference census | | 6 | Standalone relocation GC (one input file) | | 7 | Pack whole input files; reclaim relocation files | | 8 | Flatten generation numbers; route retirement | | 9 | Remote compaction validation | | 10 | Dynamic options and operations | | 11 | RepairDB rebuilds relocation state; option gate removed | | 12 | db_stress and crash test support | Stack context: terms for later diffs: | Term | Meaning | |---|---| | census | Background count, per generation number, of the SSTs and references that use it, plus live bytes per physical file (diff 5) | | indirect origin | Normal blob file written with the option on; it is the origin of unflattened references | | flatten | Compaction rewrites the generation number to the relocation file that holds the value, so old routes can go. Only flatten makes generation number != origin (diff 8) | | blob file retirement | MANIFEST record (kBlobFileRetirement) that removes a collected blob file from the Version (diff 4) | | route deletion | MANIFEST record that removes a route that no SST uses. Route retirement is the background job that writes these records (diff 8) | Tracking issue: meta#15301 Differential Revision: D124652097
Summary: Pull Request resolved: meta#15540 Why the change is required: The goal of this stack is a blob GC that does not depend on SSTs: it removes garbage from blob files without rewriting any SST. Such a GC needs two things that RocksDB does not have: - A value identity that does not change when the value moves to another file. - A checked list of the blob references that each SST keeps. The list must include older internal versions of a key that the SST still keeps (for example for snapshots), and the blob columns of wide-column entities. This diff adds both. Later diffs build the census (an in-memory reverse index from generation number to the SSTs that use it), GC and repair on top of this list. This diff in one paragraph: With the new column family option enable_blob_indirection on, the blob file writer gives each new value an indirect BlobIndex (type 3, kIndirectBlob). It stores the value's stable identity (origin file, original offset) and its CRCs (an uncompressed value stores only the CRC of the uncompressed value; a compressed value also stores the CRC of its stored bytes). Also with the option on, the block-based table builder collects every blob reference that the SST keeps. It writes them as "exact reference metadata": data blocks, one reference index block, and a 72-byte descriptor table property. In this diff only the table builder writes this metadata, and only tests read it. A temporary option gate (see "option gate" under Terms) keeps the option off outside tests. Separately, wide-column entities with blob columns are now fully checked before their blob indexes are used, also with the option off. Parts of this diff: 1. Indirect BlobIndex (core): types 3 and 4 in db/blob/blob_index.h, and the blob file writer that writes type 3. 2. Exact reference metadata format and decoders (core): db/blob/blob_reference_metadata.{h,cc}. 3. Collector and table builder integration (core): BlobReferenceMetadataCollector, its block cache charge, and BlockBasedTableBuilder. 4. Option enable_blob_indirection and the option gate. Tests need it to turn the feature on. You can skip this part on a first read. 5. Wide-column validation (WideColumnSerialization::ForEachBlobIndex). The table builder uses it to give the blob columns of each entity to the collector. It also changes behavior with the option off (see New behavior). You can skip this part on a first read. 6. Small helpers: canonical varints (util/coding.h) and CacheReservationHandle::Resize(). You can skip these on a first read. How to read this diff: 1. Overview: read "Why" and "This diff in one paragraph" above, and "Terms used in this diff" right below. Then read the top comment of db/blob/blob_reference_metadata.h (terms, layout diagrams, run and entry layout, index layout). 2. Indirect BlobIndex (db/blob/blob_index.h) and canonical varints (util/coding.h GetCanonicalVarint32/64): read the overview comment and the kIndirectBlob / kIndirectFlattenedBlob code. Every later part uses these fields: origin, original offset, generation number, logical CRC and stored CRC. The new formats read varints only in the canonical form. 3. Blob file writer (db/blob/blob_file_builder.cc, blob_log_format.cc, blob_log_writer.cc): see how a new value gets a type-3 index and how its CRC is computed once. 4. Format types and decoders: the structs in db/blob/blob_reference_metadata.h, then in the .cc the constants, GapBase, DecodeEntryBody, DecodeRunEntriesAt, DecodeBlobReferenceIndex and BlobReferenceGroupDecoder. They use only BlobIndex and the canonical varints. Learn the on-disk format here, before you read the writer. 5. Collector and its cache charge: first read the new CacheEntryRole kBlobReferenceMetadataBuildingBuffer (include/rocksdb/cache.h) and CacheReservationHandle::Resize (cache/cache_reservation_manager.h). Then read BlobReferenceMetadataCollector in this order: the class comment, Add, AppendSorted, BuildAndWriteDataBlocks, EncodeIndex, FinishOnce. Then read the unsorted fallback (AppendUnsorted, SortUnsortedGroup, SortAndEncodeGroup). 6. Wide-column change (db/wide/wide_column_serialization.cc, ForEachBlobIndex and WalkV2Columns): it gives every blob index of an entity to a callback after it checks the whole entity. The table builder uses it to feed the collector. UpdateBoundaries, BlobGarbageMeter and the compaction iterator also use it, with the option on or off. 7. Option: enable_blob_indirection (include/rocksdb/advanced_options.h, options/cf_options.cc), and the option checks and gate in db/column_family.cc (ValidateOptions, TEST_SetAllowBlobIndirection), db/db_impl/db_impl_open.cc and db_impl_secondary.cc (the FIFO check for opens that skip ValidateOptions). 8. Table integration: table/block_based/block_based_table_factory.cc (the shared reservation manager), then block_based_table_builder.cc (Rep constructor, Add, WriteBlobReferenceMetadata, EstimatedTailSize). See where the collector is created, fed and finished. block_based_table_reader.cc maps the new block name to a block type. Terms used in this diff: Stack terms: | Term | Meaning | |---|---| | origin | Blob file where the value was first written | | original offset | Value offset in the origin file | | stable identity | (origin, original offset). Never changes, even when the value moves | | generation number | The file number that a reference uses to find its value. A new value has generation number == origin. When GC moves the value, the reference keeps its generation number, and a route sends the read to the relocation file | | indirect BlobIndex | A kIndirectBlob (type 3) or kIndirectFlattenedBlob (type 4) index. It stores the stable identity, so the value can move without an SST rewrite | | unflattened reference | Index with generation number == origin: the value was not moved, or it was moved but no compaction has rewritten its reference yet (see "flatten" in "Stack context: terms for later diffs") | | flattened reference | Index with generation number != origin (type 4). Only flatten (diff 8) writes it; this diff defines the encoding | | route | Maps an old file number (a file that GC has removed from the Version) to the relocation file that now holds its live values. One hop: a target is never itself routed. Stored in the MANIFEST (diff 4) | | relocation file | Block-based table keyed by (origin, original offset) that holds copied values. Each value is the stored bytes plus a 5-byte trailer (kSimpleGen2BlobTrailerSize) | | logical CRC | CRC32C of the uncompressed value (BlobIndex::stable_checksum()) | | stored CRC | CRC32C of the stored bytes plus the compression byte | | physical file | A file that holds value bytes: a normal blob file or a relocation file. A generation number can name a physical file, or it can be only the key of a route | | exact reference metadata | Per-SST sorted list of every blob reference the SST keeps, written when the SST is built. References with the same encoding class, generation number and origin form a group. The list is stored as data blocks, with an index of the groups. An SST that has it is an exact SST | | option gate | Until diff 11, ColumnFamilyData::ValidateOptions returns NotSupported for enable_blob_indirection on a read-write open, CreateColumnFamily and SetOptions, unless a test calls ColumnFamilyData::TEST_SetAllowBlobIndirection(true). It keeps the option off in production while the stack is not complete | Terms of this diff (all defined at the top of db/blob/blob_reference_metadata.h): | Term | Meaning | |---|---| | encoding class (class) | kDirect (a legacy kBlob or kBlobTTL index), unflattened, or flattened | | reference group (group) | All references of one SST with the same (encoding class, generation number, origin) | | run | The part of one group inside one data block | | record bytes | The bytes the value's record takes in the file named by its generation number | | gap | How far a record is from where it would be if the records were back to back. A gap of 0 means back to back | | descriptor | The 72-byte table property that points to the reference index block and holds the totals, the CF ID, the SST file number and the relocation-state fingerprint | | relocation-state fingerprint | 128-bit hash of the routes and relocation files of the Version that an SST was built from (fields route_snapshot_fingerprint_*). Written as 0 in this diff | | legacy | Written the way RocksDB writes it without blob indirection | Previous behavior: - A BlobIndex named only a physical blob file and an offset. A value could not move to another file without rewriting every SST that referenced it. - An SST had no metadata that listed its blob references. The only way to find them was to scan its user keys. - These readers get the blob indexes of a wide-column entity through ForEachBlobFileNumber: UpdateBoundaries, BlobGarbageMeter, the compaction iterator, WideColumnSerialization::RemoveBlobColumns, and the referenced_blob_bytes table property count in BlockBasedTableBuilder::Add. ForEachBlobFileNumber decoded each blob index, but it did not check the rest of the entity framing (name order, section sizes, exact input size), and it could call the callback before it found a later error. New behavior: Separate change (option on or off): - The same readers (UpdateBoundaries, BlobGarbageMeter, the compaction iterator, RemoveBlobColumns, and the referenced_blob_bytes count in BlockBasedTableBuilder::Add) use WideColumnSerialization::ForEachBlobIndex, with the option on or off. With the option on, the table builder also uses it to give the blob columns of each entity to the collector. For an entity with blob columns, it checks the full framing, then decodes and validates every blob index before any callback runs. So an entity with a malformed framing now fails the flush or compaction, even with the option off. The referenced_blob_bytes count itself does not fail the build on a malformed entity. It removes that property from the SST, as in base. Option and option gate: - This diff adds a new column family option, `enable_blob_indirection`. It is off by default, and SetOptions can change it. - The option gate (see Terms) keeps the option off outside tests until the stack is complete. ColumnFamilyData::ValidateOptions rejects the option with NotSupported on a read-write open, CreateColumnFamily and SetOptions. Read-only opens (DBImplReadOnly and CompactedDBImpl) and secondary opens (including DB::OpenAndCompact) do not run ValidateOptions; they check only the FIFO rule (see Implementation). Only tests can turn the option on, through the internal ColumnFamilyData::TEST_SetAllowBlobIndirection(). DBTestBase sets it. - When the option is on, ValidateOptions also requires enable_blob_files, the block-based table format, no blob direct write, and a compaction style other than FIFO. Blob file writer (option on): - The blob file writer encodes each new value with a stable identity (origin file, original offset) and two checksums: - the logical CRC: the CRC32C of the uncompressed value. - the stored CRC: the CRC32C of the exact stored (maybe compressed) bytes, extended with the compression byte. - For an uncompressed value the stored CRC is not written, because the reader can compute it from the logical CRC. - The checksums have no version field of their own. The BlobIndex type byte gives the format version: the two new types, 3 (kIndirectBlob) and 4 (kIndirectFlattenedBlob, see Implementation), are BlobIndex::kIndirectBlobFormatVersion (1). A later format uses new type values. - The read path does not check these checksums yet. The check starts in diff 3. Exact reference metadata (option on): 1. What: a sorted list of every blob reference that the SST keeps, in plain blob values and in the blob columns of wide-column entities. 2. Where: data blocks (target 16 KB each before compression; small blocks waste less I/O, because GC reads a whole block to get a few groups) and one reference index block, written after the SST data blocks and before the SST tail, plus a 72-byte descriptor table property. 3. Who reads it: only tests in this diff. The census, GC and RepairDB read it in later diffs. 4. Details: the top comment of db/blob/blob_reference_metadata.h has the full format with diagrams, the record-bytes rules and the gap rule. - Not collected: SstFileWriter output (it has no DB file number) and any output whose CF is unknown (it has no CF ID). The descriptor needs both values, so these outputs stay in the legacy form. Checks on the metadata: - A repeated or overlapping reference in one group fails the SST build with Corruption. Readers check every count, order, size and total against the index and the descriptor, so corrupt metadata returns Corruption. The writer never makes a data block larger than kTargetDataBlockSize (16 KB), and readers accept up to kMaxDataBlockSize (64 KB). Details: "Readers" and "Integrity" at the top of db/blob/blob_reference_metadata.h, and BlobReferenceGroupDecoder. Collection and memory: - Each reference is encoded when it is added, in the on-disk entry encoding, into its group's chain of 512-byte chunks. Only a group that takes the unsorted fallback is encoded a second time, in Finish(). Inside a group, references normally arrive in rising offset order, so no reference is sorted; at the end only the groups are sorted. A group that gets a lower offset takes the unsorted fallback. The BlobReferenceMetadataCollector class comment explains the chunks, the chunk size and the fallback. - The collector keeps every reference of the SST in memory until the SST is finished, and charges this memory to the block cache under a new CacheEntryRole, kBlobReferenceMetadataBuildingBuffer. The measured cost is about 8.3 to 9 bytes per reference for uncompressed values and 12.8 to 13.5 bytes for compressed values. "Memory usage" and "Memory charge" in the BlobReferenceMetadataCollector class comment give the measurement setup and the charge rule. - There is no fixed limit on references. The collector fails with MemoryLimit above 2^20 groups or 2^20 data blocks in one SST; these are persisted format limits. - There is no charge when the table has no block cache or when charging for this role is disabled. Each block-based table factory holds one cache reservation manager for this role, shared by its builders and the SST summaries. If the charge fails, the SST build fails, so the flush or compaction fails with a background error. The buffers and their charge are released when the SST is finished or abandoned. Implementation: - BlobIndex adds two indirect encodings. The legacy encodings do not change. - Type 3, kIndirectBlob (unflattened, generation number == origin), stores the origin, the original offset, the stored size, the compression type, the logical CRC and, for compressed values, the stored CRC. It does not store the generation number. This saves the size of one varint file number per reference (about 3 to 4 bytes for typical file numbers; an estimate, not measured). - Type 4, kIndirectFlattenedBlob (flattened), stores the generation number first, then the same fields. Its generation number must be above the origin. The decoder rejects type 4 with generation number <= origin, so each reference has one encoding. - The encoders take the fields as one BlobIndex::IndirectBlobFields, set by field name, so values of the same type (file number, offset and size; the two CRCs) cannot be swapped. EncodeIndirectBlob() takes the generation number as a separate argument; EncodeUnflattenedIndirectBlob() uses the origin, so the common path cannot leave the generation number unset. - The new files blob_reference_metadata.{h,cc} define the exact reference metadata. The files also contain: - BlobReferenceMetadataCollector, the collector described above. It takes the SST identity (CF ID, SST file number and the relocation-state fingerprint) as one BlobReferenceSstIdentity, set by field name. - BlobReferenceGroupKey, the key of a group: (encoding class, generation number, origin). It has the order, the hash and the validity check that the collector map, the sort, the decoder and FindBlobReferenceGroup() use. BlobReference and BlobReferenceGroup keep their flat fields and return the key through group_key(). - The index codec and the descriptor table property. - BlobReferenceGroupDecoder, the only decoder of reference runs. It reads a group one data block at a time and holds every corruption check of a group walk. Readers that visit one group (found by a binary search of the index) use it, and so do readers that visit every block in order. - Callers in this diff: only the table builder runs the collector. The decoders (DecodeBlobReferenceIndex, FindBlobReferenceGroup, BlobReferenceGroupDecoder, ForEachBlobReference*) are used only by tests. kMaxDecodedBlobReferenceBytes is used only in compile-time checks of the count limits. The census (diff 5) and GC (diffs 6 and 7) are their first production users. The census is also the first user of BlobReferenceSstSummary (see the last Implementation bullet). - Descriptor: the CF ID and the SST file number are the repair fields. They let RepairDB tell which column family and SST the list belongs to without a MANIFEST. The decoder rejects a descriptor whose repair fields are not set (unknown CF ID or SST file number 0). The descriptor also has a 128-bit relocation-state fingerprint (the route_snapshot_fingerprint fields). Remote compaction uses it in diff 9 to check that an SST built elsewhere used the same routes as the job that asked for it. In this diff the table builder always writes 0, and DecodeFrom() does not check it. The field is in the format now, so the descriptor keeps one format version (1) through the stack. While the option gate is on, no data in this format exists outside tests. - The collector validates each BlobIndex once, when it is added. Its Finish() writes the data blocks and then the reference index block through one block writer, and fills the descriptor, including the index handle. Destroying the collector frees its memory and then its charge, also when the build is abandoned. - BlockBasedTableBuilder creates the collector with the CF ID and SST file number from TableBuilderOptions. It passes the size of each entry's user key to the collector, so the collector can compute the exact size of each blob record. It adds the collector's EstimatedEncodedSize() to its tail size estimate: the blocks are written before the tail, but output file size planning uses this estimate. The reference blocks are written with BlockType::kUserDefinedIndex, which keeps their contents as opaque bytes. Readers read them without the block cache, so prepopulate_block_cache does not warm them into the block cache (WriteMaybeCompressedBlock() with warm_cache false). - With the option on, the blob file builder computes the CRC32C of each stored value once. The blob log record CRC combines it with the key CRC (crc32c::Crc32cCombine). The stored CRC extends it with the compression byte. For an uncompressed value it is also the logical CRC. The bytes written do not change. - WideColumnSerialization::ForEachBlobIndex replaces ForEachBlobFileNumber. It reads the entity header and column types once. For a V1 entity (no column types and no blob columns), it returns right after the header. For a V2 entity without blob columns, it returns right after the type scan. It shares one V2 column walker (WalkV2Columns) with Deserialize(), so the entity is parsed in one pass. - Read-only opens (DBImplReadOnly, CompactedDBImpl) and secondary opens skip ValidateOptions, so they check the compaction style against the option once per column family: in DBImpl::Recover() for read-only opens, and at the start of OpenAsSecondaryImpl() for secondary opens. DB::OpenAndCompact opens through OpenAsSecondaryImpl(), so it gets the same check. - CacheReservationHandle gets Resize(), so each collector or summary can change its own part of the shared charge. - util/coding.h adds GetCanonicalVarint32/64, which reject overlong varint encodings. The new formats use them, so each value has one encoding. - BlobReferenceSstSummary is the in-memory list of an SST's groups and totals that Finish() can return. It has no caller yet: the table builder passes a null summary output. It is checked and does not change after Finish(), so the finished builder, the SST metadata and the census (its first user, in diff 5) can share it. It is in this diff because Finish() hands the group buffer and its cache charge over to the summary. Adding the summary later would mean rewriting the charge code of Finish() in diff 5, so the charge code is written and tested once, here. Safety boundaries and unsupported cases: - Public API: this diff adds the option `enable_blob_indirection` with the C API functions `rocksdb_options_set_enable_blob_indirection` and `rocksdb_options_get_enable_blob_indirection`. It also inserts a new CacheEntryRole value before kMisc, which changes the numeric value of kMisc and kNumCacheEntryRoles. - With the option off, new blob values and SSTs are written in the legacy form. Legacy blob indexes stay readable. - One behavior change applies with the option off too: UpdateBoundaries, BlobGarbageMeter and the compaction iterator fully validate wide-column entities with blob columns and their blob indexes (see "Separate change" above). - Exact reference metadata has one format (version 1) with a fixed 72-byte descriptor. The decoders reject any other version as Corruption. - The metadata blocks are protected by the table's block checksum only. With checksum = kNoChecksum, corrupt metadata is detected only when it breaks the format or order checks. - The index decoder checks each decoded count against the remaining input size before it reserves memory. So a small corrupt index cannot make it allocate a large vector. - Inlined-TTL and same-file blob indexes are not collected and cannot be relocated. Encoding, count and checksum errors return an error; a bad reference is never skipped. - Option gate (see Terms): outside tests no DB writes data in the new format. The gate checks only the option value, never persisted state. Diff 11 removes it, once data written with the option stays readable through the end of the stack and the safety checks for turning the option on are in place. - Downgrading the binary after data was written with the option on is not supported. Validation and known limitations: - Unit tests cover the BlobIndex encodings and their rejection of bad input, the metadata layout and every corruption check, the collector (sorted path, unsorted fallback, memory charge), the flush integration, the option checks and option gate, wide-column validation, and canonical varints. See the Test Plan. - The collection buffer is not spilled to disk. One SST with very many blob references needs a matching amount of charged memory: the per-reference amount given under "Collection and memory", plus up to one partly used 512-byte chunk per group. While a fallback group is sorted, it also needs 16 bytes of sort key per reference and a second copy of its entries. For a test group of 200K references with every other one out of order, the test comment estimates about 44 bytes per reference, and the test checks that the peak charge is at most 56 bytes per reference. - Flush still writes a single output file. Until diff 7, one compaction output collects every reference it gets, charged to the block cache. Diff 7 cuts compaction outputs (including L0) at a reference cap and keeps each user key's history in one output. With the option gate, a large output is reachable only through the test hook. Stack context (same in diffs 2-11, except that diffs 8-11 add the diff 8 route rule to the install line; "<== this diff" marks this diff's part): Goal: remove garbage from blob files without rewriting any SST. A value gets a stable identity when it is first written. GC copies the live values of whole files into a relocation file and installs routes in the MANIFEST. Readers follow the routes. SSTs are not changed. How a read finds a value (P is the relocation file that holds the value now): ``` SST entry indirect BlobIndex { generation number, origin, original offset, size, CRCs } | v Version file table: generation number -> live file, or route old file -> P | | live file route found | | v v blob file: read at original offset relocation file P (always one hop): relocation file: look up key look up key (origin, original offset) (origin, original offset) ``` Lifecycle of one value (in this example, F is the blob file where the value is first written, and P is the relocation file that GC writes): ``` write new value goes to blob file F; index: origin = F, generation number = F | census per generation number: the SSTs and references that use it; | per physical file: live bytes, ranked by garbage ratio GC job copy the live values of whole input files (stored bytes, checked | with the stored CRC) into relocation file P, keyed by | (origin, original offset) install one MANIFEST edit: add P, route each input generation number still | in use to P, re-point the routes into the inputs to P, and remove | each input (blob file retirement for a blob file, relocation | file deletion for a relocation file) flatten compaction sets generation number = P in the SSTs it rewrites | route route retirement: route deletion records remove the routes that delete no SST uses ``` Stack map: | Diff | Adds | |---|---| | 1 | Finalize blob metadata outside the DB mutex (prep) | | 2 | Indirect BlobIndex (kIndirectBlob) and exact SST reference metadata <== this diff | | 3 | Relocation file format and stable read path | | 4 | MANIFEST routes and Version ownership | | 5 | Exact reverse reference census | | 6 | Standalone relocation GC (one input file) | | 7 | Pack whole input files; reclaim relocation files | | 8 | Flatten generation numbers; route retirement | | 9 | Remote compaction validation | | 10 | Dynamic options and operations | | 11 | RepairDB rebuilds relocation state; option gate removed | | 12 | db_stress and crash test support | Stack context: terms for later diffs: | Term | Meaning | |---|---| | census | Background count, per generation number, of the SSTs and references that use it, plus live bytes per physical file (diff 5) | | indirect origin | Normal blob file written with the option on; it is the origin of unflattened references | | flatten | Compaction rewrites the generation number to the relocation file that holds the value, so old routes can go. Only flatten makes generation number != origin (diff 8) | | blob file retirement | MANIFEST record (kBlobFileRetirement) that removes a collected blob file from the Version (diff 4) | | route deletion | MANIFEST record that removes a route that no SST uses. Route retirement is the background job that writes these records (diff 8) | Tracking issue: meta#15301 Differential Revision: D124652097
b0bad4a to
80c866c
Compare
Summary: Pull Request resolved: meta#15540 Why the change is required: The goal of this stack is a blob GC that does not depend on SSTs: it removes garbage from blob files without rewriting any SST. Such a GC needs two things that RocksDB does not have: - A value identity that does not change when the value moves to another file. - A checked list of the blob references that each SST keeps. The list must include older internal versions of a key that the SST still keeps (for example for snapshots), and the blob columns of wide-column entities. This diff adds both. Later diffs build the census (an in-memory reverse index from generation number to the SSTs that use it), GC and repair on top of this list. This diff in one paragraph: With the new column family option enable_blob_indirection on, the blob file writer gives each new value an indirect BlobIndex (type 3, kIndirectBlob). It stores the value's stable identity (origin file, original offset) and its CRCs (an uncompressed value stores only the CRC of the uncompressed value; a compressed value also stores the CRC of its stored bytes). Also with the option on, the block-based table builder collects every blob reference that the SST keeps. It writes them as "exact reference metadata": data blocks, one reference index block, and a 72-byte descriptor table property. In this diff only the table builder writes this metadata, and only tests read it. A temporary option gate (see "option gate" under Terms) keeps the option off outside tests. Separately, wide-column entities with blob columns are now fully checked before their blob indexes are used, also with the option off. Parts of this diff: 1. Indirect BlobIndex (core): types 3 and 4 in db/blob/blob_index.h, and the blob file writer that writes type 3. 2. Exact reference metadata format and decoders (core): db/blob/blob_reference_metadata.{h,cc}. 3. Collector and table builder integration (core): BlobReferenceMetadataCollector, its block cache charge, and BlockBasedTableBuilder. 4. Option enable_blob_indirection and the option gate. Tests need it to turn the feature on. You can skip this part on a first read. 5. Wide-column validation (WideColumnSerialization::ForEachBlobIndex). The table builder uses it to give the blob columns of each entity to the collector. It also changes behavior with the option off (see New behavior). You can skip this part on a first read. 6. Small helpers: canonical varints (util/coding.h) and CacheReservationHandle::Resize(). You can skip these on a first read. How to read this diff: 1. Overview: read "Why" and "This diff in one paragraph" above, and "Terms used in this diff" right below. Then read the top comment of db/blob/blob_reference_metadata.h (terms, layout diagrams, run and entry layout, index layout). 2. Indirect BlobIndex (db/blob/blob_index.h) and canonical varints (util/coding.h GetCanonicalVarint32/64): read the overview comment and the kIndirectBlob / kIndirectFlattenedBlob code. Every later part uses these fields: origin, original offset, generation number, logical CRC and stored CRC. The new formats read varints only in the canonical form. 3. Blob file writer (db/blob/blob_file_builder.cc, blob_log_format.cc, blob_log_writer.cc): see how a new value gets a type-3 index and how its CRC is computed once. 4. Format types and decoders: the structs in db/blob/blob_reference_metadata.h, then in the .cc the constants, GapBase, DecodeEntryBody, DecodeRunEntriesAt, DecodeBlobReferenceIndex and BlobReferenceGroupDecoder. They use only BlobIndex and the canonical varints. Learn the on-disk format here, before you read the writer. 5. Collector and its cache charge: first read the new CacheEntryRole kBlobReferenceMetadataBuildingBuffer (include/rocksdb/cache.h) and CacheReservationHandle::Resize (cache/cache_reservation_manager.h). Then read BlobReferenceMetadataCollector in this order: the class comment, Add, AppendSorted, BuildAndWriteDataBlocks, EncodeIndex, FinishOnce. Then read the unsorted fallback (AppendUnsorted, SortUnsortedGroup, SortAndEncodeGroup). 6. Wide-column change (db/wide/wide_column_serialization.cc, ForEachBlobIndex and WalkV2Columns): it gives every blob index of an entity to a callback after it checks the whole entity. The table builder uses it to feed the collector. UpdateBoundaries, BlobGarbageMeter and the compaction iterator also use it, with the option on or off. 7. Option: enable_blob_indirection (include/rocksdb/advanced_options.h, options/cf_options.cc), and the option checks and gate in db/column_family.cc (ValidateOptions, TEST_SetAllowBlobIndirection), db/db_impl/db_impl_open.cc and db_impl_secondary.cc (the FIFO check for opens that skip ValidateOptions). 8. Table integration: table/block_based/block_based_table_factory.cc (the shared reservation manager), then block_based_table_builder.cc (Rep constructor, Add, WriteBlobReferenceMetadata, EstimatedTailSize). See where the collector is created, fed and finished. block_based_table_reader.cc maps the new block name to a block type. Terms used in this diff: Stack terms: | Term | Meaning | |---|---| | origin | Blob file where the value was first written | | original offset | Value offset in the origin file | | stable identity | (origin, original offset). Never changes, even when the value moves | | generation number | The file number that a reference uses to find its value. A new value has generation number == origin. When GC moves the value, the reference keeps its generation number, and a route sends the read to the relocation file | | indirect BlobIndex | A kIndirectBlob (type 3) or kIndirectFlattenedBlob (type 4) index. It stores the stable identity, so the value can move without an SST rewrite | | unflattened reference | Index with generation number == origin: the value was not moved, or it was moved but no compaction has rewritten its reference yet (see "flatten" in "Stack context: terms for later diffs") | | flattened reference | Index with generation number != origin (type 4). Only flatten (diff 8) writes it; this diff defines the encoding | | route | Maps an old file number (a file that GC has removed from the Version) to the relocation file that now holds its live values. One hop: a target is never itself routed. Stored in the MANIFEST (diff 4) | | relocation file | Block-based table keyed by (origin, original offset) that holds copied values. Each value is the stored bytes plus a 5-byte trailer (kSimpleGen2BlobTrailerSize) | | logical CRC | CRC32C of the uncompressed value (BlobIndex::stable_checksum()) | | stored CRC | CRC32C of the stored bytes plus the compression byte | | physical file | A file that holds value bytes: a normal blob file or a relocation file. A generation number can name a physical file, or it can be only the key of a route | | exact reference metadata | Per-SST sorted list of every blob reference the SST keeps, written when the SST is built. References with the same encoding class, generation number and origin form a group. The list is stored as data blocks, with an index of the groups. An SST that has it is an exact SST | | option gate | Until diff 11, ColumnFamilyData::ValidateOptions returns NotSupported for enable_blob_indirection on a read-write open, CreateColumnFamily and SetOptions, unless a test calls ColumnFamilyData::TEST_SetAllowBlobIndirection(true). It keeps the option off in production while the stack is not complete | Terms of this diff (all defined at the top of db/blob/blob_reference_metadata.h): | Term | Meaning | |---|---| | encoding class (class) | kDirect (a legacy kBlob or kBlobTTL index), unflattened, or flattened | | reference group (group) | All references of one SST with the same (encoding class, generation number, origin) | | run | The part of one group inside one data block | | record bytes | The bytes the value's record takes in the file named by its generation number | | gap | How far a record is from where it would be if the records were back to back. A gap of 0 means back to back | | descriptor | The 72-byte table property that points to the reference index block and holds the totals, the CF ID, the SST file number and the relocation-state fingerprint | | relocation-state fingerprint | 128-bit hash of the routes and relocation files of the Version that an SST was built from (fields route_snapshot_fingerprint_*). Written as 0 in this diff | | legacy | Written the way RocksDB writes it without blob indirection | Previous behavior: - A BlobIndex named only a physical blob file and an offset. A value could not move to another file without rewriting every SST that referenced it. - An SST had no metadata that listed its blob references. The only way to find them was to scan its user keys. - These readers get the blob indexes of a wide-column entity through ForEachBlobFileNumber: UpdateBoundaries, BlobGarbageMeter, the compaction iterator, WideColumnSerialization::RemoveBlobColumns, and the referenced_blob_bytes table property count in BlockBasedTableBuilder::Add. ForEachBlobFileNumber decoded each blob index, but it did not check the rest of the entity framing (name order, section sizes, exact input size), and it could call the callback before it found a later error. New behavior: Separate change (option on or off): - The same readers (UpdateBoundaries, BlobGarbageMeter, the compaction iterator, RemoveBlobColumns, and the referenced_blob_bytes count in BlockBasedTableBuilder::Add) use WideColumnSerialization::ForEachBlobIndex, with the option on or off. With the option on, the table builder also uses it to give the blob columns of each entity to the collector. For an entity with blob columns, it checks the full framing, then decodes and validates every blob index before any callback runs. So an entity with a malformed framing now fails the flush or compaction, even with the option off. The referenced_blob_bytes count itself does not fail the build on a malformed entity. It removes that property from the SST, as in base. Option and option gate: - This diff adds a new column family option, `enable_blob_indirection`. It is off by default, and SetOptions can change it. - The option gate (see Terms) keeps the option off outside tests until the stack is complete. ColumnFamilyData::ValidateOptions rejects the option with NotSupported on a read-write open, CreateColumnFamily and SetOptions. Read-only opens (DBImplReadOnly and CompactedDBImpl) and secondary opens (including DB::OpenAndCompact) do not run ValidateOptions; they check only the FIFO rule (see Implementation). Only tests can turn the option on, through the internal ColumnFamilyData::TEST_SetAllowBlobIndirection(). DBTestBase sets it. - When the option is on, ValidateOptions also requires enable_blob_files, the block-based table format, no blob direct write, and a compaction style other than FIFO. Blob file writer (option on): - The blob file writer encodes each new value with a stable identity (origin file, original offset) and two checksums: - the logical CRC: the CRC32C of the uncompressed value. - the stored CRC: the CRC32C of the exact stored (maybe compressed) bytes, extended with the compression byte. - For an uncompressed value the stored CRC is not written, because the reader can compute it from the logical CRC. - The checksums have no version field of their own. The BlobIndex type byte gives the format version: the two new types, 3 (kIndirectBlob) and 4 (kIndirectFlattenedBlob, see Implementation), are BlobIndex::kIndirectBlobFormatVersion (1). A later format uses new type values. - The read path does not check these checksums yet. The check starts in diff 3. Exact reference metadata (option on): 1. What: a sorted list of every blob reference that the SST keeps, in plain blob values and in the blob columns of wide-column entities. 2. Where: data blocks (target 16 KB each before compression; small blocks waste less I/O, because GC reads a whole block to get a few groups) and one reference index block, written after the SST data blocks and before the SST tail, plus a 72-byte descriptor table property. 3. Who reads it: only tests in this diff. The census, GC and RepairDB read it in later diffs. 4. Details: the top comment of db/blob/blob_reference_metadata.h has the full format with diagrams, the record-bytes rules and the gap rule. - Not collected: SstFileWriter output (it has no DB file number) and any output whose CF is unknown (it has no CF ID). The descriptor needs both values, so these outputs stay in the legacy form. Checks on the metadata: - A repeated or overlapping reference in one group fails the SST build with Corruption. Readers check every count, order, size and total against the index and the descriptor, so corrupt metadata returns Corruption. The writer never makes a data block larger than kTargetDataBlockSize (16 KB), and readers accept up to kMaxDataBlockSize (64 KB). Details: "Readers" and "Integrity" at the top of db/blob/blob_reference_metadata.h, and BlobReferenceGroupDecoder. Collection and memory: - Each reference is encoded when it is added, in the on-disk entry encoding, into its group's chain of 512-byte chunks. Only a group that takes the unsorted fallback is encoded a second time, in Finish(). Inside a group, references normally arrive in rising offset order, so no reference is sorted; at the end only the groups are sorted. A group that gets a lower offset takes the unsorted fallback. The BlobReferenceMetadataCollector class comment explains the chunks, the chunk size and the fallback. - The collector keeps every reference of the SST in memory until the SST is finished, and charges this memory to the block cache under a new CacheEntryRole, kBlobReferenceMetadataBuildingBuffer. The measured cost is about 8.3 to 9 bytes per reference for uncompressed values and 12.8 to 13.5 bytes for compressed values. "Memory usage" and "Memory charge" in the BlobReferenceMetadataCollector class comment give the measurement setup and the charge rule. - There is no fixed limit on references. The collector fails with MemoryLimit above 2^20 groups or 2^20 data blocks in one SST; these are persisted format limits. - There is no charge when the table has no block cache or when charging for this role is disabled. Each block-based table factory holds one cache reservation manager for this role, shared by its builders and the SST summaries. If the charge fails, the SST build fails, so the flush or compaction fails with a background error. The buffers and their charge are released when the SST is finished or abandoned. Implementation: - BlobIndex adds two indirect encodings. The legacy encodings do not change. - Type 3, kIndirectBlob (unflattened, generation number == origin), stores the origin, the original offset, the stored size, the compression type, the logical CRC and, for compressed values, the stored CRC. It does not store the generation number. This saves the size of one varint file number per reference (about 3 to 4 bytes for typical file numbers; an estimate, not measured). - Type 4, kIndirectFlattenedBlob (flattened), stores the generation number first, then the same fields. Its generation number must be above the origin. The decoder rejects type 4 with generation number <= origin, so each reference has one encoding. - The encoders take the fields as one BlobIndex::IndirectBlobFields, set by field name, so values of the same type (file number, offset and size; the two CRCs) cannot be swapped. EncodeIndirectBlob() takes the generation number as a separate argument; EncodeUnflattenedIndirectBlob() uses the origin, so the common path cannot leave the generation number unset. - The new files blob_reference_metadata.{h,cc} define the exact reference metadata. The files also contain: - BlobReferenceMetadataCollector, the collector described above. It takes the SST identity (CF ID, SST file number and the relocation-state fingerprint) as one BlobReferenceSstIdentity, set by field name. - BlobReferenceGroupKey, the key of a group: (encoding class, generation number, origin). It has the order, the hash and the validity check that the collector map, the sort, the decoder and FindBlobReferenceGroup() use. BlobReference and BlobReferenceGroup keep their flat fields and return the key through group_key(). - The index codec and the descriptor table property. - BlobReferenceGroupDecoder, the only decoder of reference runs. It reads a group one data block at a time and holds every corruption check of a group walk. Readers that visit one group (found by a binary search of the index) use it, and so do readers that visit every block in order. - Callers in this diff: only the table builder runs the collector. The decoders (DecodeBlobReferenceIndex, FindBlobReferenceGroup, BlobReferenceGroupDecoder, ForEachBlobReference*) are used only by tests. kMaxDecodedBlobReferenceBytes is used only in compile-time checks of the count limits. The census (diff 5) and GC (diffs 6 and 7) are their first production users. The census is also the first user of BlobReferenceSstSummary (see the last Implementation bullet). - Descriptor: the CF ID and the SST file number are the repair fields. They let RepairDB tell which column family and SST the list belongs to without a MANIFEST. The decoder rejects a descriptor whose repair fields are not set (unknown CF ID or SST file number 0). The descriptor also has a 128-bit relocation-state fingerprint (the route_snapshot_fingerprint fields). Remote compaction uses it in diff 9 to check that an SST built elsewhere used the same routes as the job that asked for it. In this diff the table builder always writes 0, and DecodeFrom() does not check it. The field is in the format now, so the descriptor keeps one format version (1) through the stack. While the option gate is on, no data in this format exists outside tests. - The collector validates each BlobIndex once, when it is added. Its Finish() writes the data blocks and then the reference index block through one block writer, and fills the descriptor, including the index handle. Destroying the collector frees its memory and then its charge, also when the build is abandoned. - BlockBasedTableBuilder creates the collector with the CF ID and SST file number from TableBuilderOptions. It passes the size of each entry's user key to the collector, so the collector can compute the exact size of each blob record. It adds the collector's EstimatedEncodedSize() to its tail size estimate: the blocks are written before the tail, but output file size planning uses this estimate. The reference blocks are written with BlockType::kUserDefinedIndex, which keeps their contents as opaque bytes. Readers read them without the block cache, so prepopulate_block_cache does not warm them into the block cache (WriteMaybeCompressedBlock() with warm_cache false). - With the option on, the blob file builder computes the CRC32C of each stored value once. The blob log record CRC combines it with the key CRC (crc32c::Crc32cCombine). The stored CRC extends it with the compression byte. For an uncompressed value it is also the logical CRC. The bytes written do not change. - WideColumnSerialization::ForEachBlobIndex replaces ForEachBlobFileNumber. It reads the entity header and column types once. For a V1 entity (no column types and no blob columns), it returns right after the header. For a V2 entity without blob columns, it returns right after the type scan. It shares one V2 column walker (WalkV2Columns) with Deserialize(), so the entity is parsed in one pass. - Read-only opens (DBImplReadOnly, CompactedDBImpl) and secondary opens skip ValidateOptions, so they check the compaction style against the option once per column family: in DBImpl::Recover() for read-only opens, and at the start of OpenAsSecondaryImpl() for secondary opens. DB::OpenAndCompact opens through OpenAsSecondaryImpl(), so it gets the same check. - CacheReservationHandle gets Resize(), so each collector or summary can change its own part of the shared charge. - util/coding.h adds GetCanonicalVarint32/64, which reject overlong varint encodings. The new formats use them, so each value has one encoding. - BlobReferenceSstSummary is the in-memory list of an SST's groups and totals that Finish() can return. It has no caller yet: the table builder passes a null summary output. It is checked and does not change after Finish(), so the finished builder, the SST metadata and the census (its first user, in diff 5) can share it. It is in this diff because Finish() hands the group buffer and its cache charge over to the summary. Adding the summary later would mean rewriting the charge code of Finish() in diff 5, so the charge code is written and tested once, here. Safety boundaries and unsupported cases: - Public API: this diff adds the option `enable_blob_indirection` with the C API functions `rocksdb_options_set_enable_blob_indirection` and `rocksdb_options_get_enable_blob_indirection`. It also inserts a new CacheEntryRole value before kMisc, which changes the numeric value of kMisc and kNumCacheEntryRoles. - With the option off, new blob values and SSTs are written in the legacy form. Legacy blob indexes stay readable. - One behavior change applies with the option off too: UpdateBoundaries, BlobGarbageMeter and the compaction iterator fully validate wide-column entities with blob columns and their blob indexes (see "Separate change" above). - Exact reference metadata has one format (version 1) with a fixed 72-byte descriptor. The decoders reject any other version as Corruption. - The metadata blocks are protected by the table's block checksum only. With checksum = kNoChecksum, corrupt metadata is detected only when it breaks the format or order checks. - The index decoder checks each decoded count against the remaining input size before it reserves memory. So a small corrupt index cannot make it allocate a large vector. - Inlined-TTL and same-file blob indexes are not collected and cannot be relocated. Encoding, count and checksum errors return an error; a bad reference is never skipped. - Option gate (see Terms): outside tests no DB writes data in the new format. The gate checks only the option value, never persisted state. Diff 11 removes it, once data written with the option stays readable through the end of the stack and the safety checks for turning the option on are in place. - Downgrading the binary after data was written with the option on is not supported. Validation and known limitations: - Unit tests cover the BlobIndex encodings and their rejection of bad input, the metadata layout and every corruption check, the collector (sorted path, unsorted fallback, memory charge), the flush integration, the option checks and option gate, wide-column validation, and canonical varints. See the Test Plan. - The collection buffer is not spilled to disk. One SST with very many blob references needs a matching amount of charged memory: the per-reference amount given under "Collection and memory", plus up to one partly used 512-byte chunk per group. While a fallback group is sorted, it also needs 16 bytes of sort key per reference and a second copy of its entries. For a test group of 200K references with every other one out of order, the test comment estimates about 44 bytes per reference, and the test checks that the peak charge is at most 56 bytes per reference. - Flush still writes a single output file. Until diff 7, one compaction output collects every reference it gets, charged to the block cache. Diff 7 cuts compaction outputs (including L0) at a reference cap and keeps each user key's history in one output. With the option gate, a large output is reachable only through the test hook. Stack context (same in diffs 2-11, except that diffs 8-11 add the diff 8 route rule to the install line; "<== this diff" marks this diff's part): Goal: remove garbage from blob files without rewriting any SST. A value gets a stable identity when it is first written. GC copies the live values of whole files into a relocation file and installs routes in the MANIFEST. Readers follow the routes. SSTs are not changed. How a read finds a value (P is the relocation file that holds the value now): ``` SST entry indirect BlobIndex { generation number, origin, original offset, size, CRCs } | v Version file table: generation number -> live file, or route old file -> P | | live file route found | | v v blob file: read at original offset relocation file P (always one hop): relocation file: look up key look up key (origin, original offset) (origin, original offset) ``` Lifecycle of one value (in this example, F is the blob file where the value is first written, and P is the relocation file that GC writes): ``` write new value goes to blob file F; index: origin = F, generation number = F | census per generation number: the SSTs and references that use it; | per physical file: live bytes, ranked by garbage ratio GC job copy the live values of whole input files (stored bytes, checked | with the stored CRC) into relocation file P, keyed by | (origin, original offset) install one MANIFEST edit: add P, route each input generation number still | in use to P, re-point the routes into the inputs to P, and remove | each input (blob file retirement for a blob file, relocation | file deletion for a relocation file) flatten compaction sets generation number = P in the SSTs it rewrites | route route retirement: route deletion records remove the routes that delete no SST uses ``` Stack map: | Diff | Adds | |---|---| | 1 | Finalize blob metadata outside the DB mutex (prep) | | 2 | Indirect BlobIndex (kIndirectBlob) and exact SST reference metadata <== this diff | | 3 | Relocation file format and stable read path | | 4 | MANIFEST routes and Version ownership | | 5 | Exact reverse reference census | | 6 | Standalone relocation GC (one input file) | | 7 | Pack whole input files; reclaim relocation files | | 8 | Flatten generation numbers; route retirement | | 9 | Remote compaction validation | | 10 | Dynamic options and operations | | 11 | RepairDB rebuilds relocation state; option gate removed | | 12 | db_stress and crash test support | Stack context: terms for later diffs: | Term | Meaning | |---|---| | census | Background count, per generation number, of the SSTs and references that use it, plus live bytes per physical file (diff 5) | | indirect origin | Normal blob file written with the option on; it is the origin of unflattened references | | flatten | Compaction rewrites the generation number to the relocation file that holds the value, so old routes can go. Only flatten makes generation number != origin (diff 8) | | blob file retirement | MANIFEST record (kBlobFileRetirement) that removes a collected blob file from the Version (diff 4) | | route deletion | MANIFEST record that removes a route that no SST uses. Route retirement is the background job that writes these records (diff 8) | Tracking issue: meta#15301 Differential Revision: D124652097
Summary: Pull Request resolved: meta#15540 Why the change is required: The goal of this stack is a blob GC that does not depend on SSTs: it removes garbage from blob files without rewriting any SST. Such a GC needs two things that RocksDB does not have: - A value identity that does not change when the value moves to another file. - A checked list of the blob references that each SST keeps. The list must include older internal versions of a key that the SST still keeps (for example for snapshots), and the blob columns of wide-column entities. This diff adds both. Later diffs build the census (an in-memory reverse index from generation number to the SSTs that use it), GC and repair on top of this list. This diff in one paragraph: With the new column family option enable_blob_indirection on, the blob file writer gives each new value an indirect BlobIndex (type 3, kIndirectBlob). It stores the value's stable identity (origin file, original offset) and its CRCs (an uncompressed value stores only the CRC of the uncompressed value; a compressed value also stores the CRC of its stored bytes). Also with the option on, the block-based table builder collects every blob reference that the SST keeps. It writes them as "exact reference metadata": data blocks, one reference index block, and a 72-byte descriptor table property. In this diff only the table builder writes this metadata, and only tests read it. A temporary option gate (see "option gate" under Terms) keeps the option off outside tests. Separately, wide-column entities with blob columns are now fully checked before their blob indexes are used, also with the option off. Parts of this diff: 1. Indirect BlobIndex (core): types 3 and 4 in db/blob/blob_index.h, and the blob file writer that writes type 3. 2. Exact reference metadata format and decoders (core): db/blob/blob_reference_metadata.{h,cc}. 3. Collector and table builder integration (core): BlobReferenceMetadataCollector, its block cache charge, and BlockBasedTableBuilder. 4. Option enable_blob_indirection and the option gate. Tests need it to turn the feature on. You can skip this part on a first read. 5. Wide-column validation (WideColumnSerialization::ForEachBlobIndex). The table builder uses it to give the blob columns of each entity to the collector. It also changes behavior with the option off (see New behavior). You can skip this part on a first read. 6. Small helpers: canonical varints (util/coding.h) and CacheReservationHandle::Resize(). You can skip these on a first read. How to read this diff: 1. Overview: read "Why" and "This diff in one paragraph" above, and "Terms used in this diff" right below. Then read the top comment of db/blob/blob_reference_metadata.h (terms, layout diagrams, run and entry layout, index layout). 2. Indirect BlobIndex (db/blob/blob_index.h) and canonical varints (util/coding.h GetCanonicalVarint32/64): read the overview comment and the kIndirectBlob / kIndirectFlattenedBlob code. Every later part uses these fields: origin, original offset, generation number, logical CRC and stored CRC. The new formats read varints only in the canonical form. 3. Blob file writer (db/blob/blob_file_builder.cc, blob_log_format.cc, blob_log_writer.cc): see how a new value gets a type-3 index and how its CRC is computed once. 4. Format types and decoders: the structs in db/blob/blob_reference_metadata.h, then in the .cc the constants, GapBase, DecodeEntryBody, DecodeRunEntriesAt, DecodeBlobReferenceIndex and BlobReferenceGroupDecoder. They use only BlobIndex and the canonical varints. Learn the on-disk format here, before you read the writer. 5. Collector and its cache charge: first read the new CacheEntryRole kBlobReferenceMetadataBuildingBuffer (include/rocksdb/cache.h) and CacheReservationHandle::Resize (cache/cache_reservation_manager.h). Then read BlobReferenceMetadataCollector in this order: the class comment, Add, AppendSorted, BuildAndWriteDataBlocks, EncodeIndex, FinishOnce. Then read the unsorted fallback (AppendUnsorted, SortUnsortedGroup, SortAndEncodeGroup). 6. Wide-column change (db/wide/wide_column_serialization.cc, ForEachBlobIndex and WalkV2Columns): it gives every blob index of an entity to a callback after it checks the whole entity. The table builder uses it to feed the collector. UpdateBoundaries, BlobGarbageMeter and the compaction iterator also use it, with the option on or off. 7. Option: enable_blob_indirection (include/rocksdb/advanced_options.h, options/cf_options.cc), and the option checks and gate in db/column_family.cc (ValidateOptions, TEST_SetAllowBlobIndirection), db/db_impl/db_impl_open.cc and db_impl_secondary.cc (the FIFO check for opens that skip ValidateOptions). 8. Table integration: table/block_based/block_based_table_factory.cc (the shared reservation manager), then block_based_table_builder.cc (Rep constructor, Add, WriteBlobReferenceMetadata, EstimatedTailSize). See where the collector is created, fed and finished. block_based_table_reader.cc maps the new block name to a block type. Terms used in this diff: Stack terms: | Term | Meaning | |---|---| | origin | Blob file where the value was first written | | original offset | Value offset in the origin file | | stable identity | (origin, original offset). Never changes, even when the value moves | | generation number | The file number that a reference uses to find its value. A new value has generation number == origin. When GC moves the value, the reference keeps its generation number, and a route sends the read to the relocation file | | indirect BlobIndex | A kIndirectBlob (type 3) or kIndirectFlattenedBlob (type 4) index. It stores the stable identity, so the value can move without an SST rewrite | | unflattened reference | Index with generation number == origin: the value was not moved, or it was moved but no compaction has rewritten its reference yet (see "flatten" in "Stack context: terms for later diffs") | | flattened reference | Index with generation number != origin (type 4). Only flatten (diff 8) writes it; this diff defines the encoding | | route | Maps an old file number (a file that GC has removed from the Version) to the relocation file that now holds its live values. One hop: a target is never itself routed. Stored in the MANIFEST (diff 4) | | relocation file | Block-based table keyed by (origin, original offset) that holds copied values. Each value is the stored bytes plus a 5-byte trailer (kSimpleGen2BlobTrailerSize) | | logical CRC | CRC32C of the uncompressed value (BlobIndex::stable_checksum()) | | stored CRC | CRC32C of the stored bytes plus the compression byte | | physical file | A file that holds value bytes: a normal blob file or a relocation file. A generation number can name a physical file, or it can be only the key of a route | | exact reference metadata | Per-SST sorted list of every blob reference the SST keeps, written when the SST is built. References with the same encoding class, generation number and origin form a group. The list is stored as data blocks, with an index of the groups. An SST that has it is an exact SST | | option gate | Until diff 11, ColumnFamilyData::ValidateOptions returns NotSupported for enable_blob_indirection on a read-write open, CreateColumnFamily and SetOptions, unless a test calls ColumnFamilyData::TEST_SetAllowBlobIndirection(true). It keeps the option off in production while the stack is not complete | Terms of this diff (all defined at the top of db/blob/blob_reference_metadata.h): | Term | Meaning | |---|---| | encoding class (class) | kDirect (a legacy kBlob or kBlobTTL index), unflattened, or flattened | | reference group (group) | All references of one SST with the same (encoding class, generation number, origin) | | run | The part of one group inside one data block | | record bytes | The bytes the value's record takes in the file named by its generation number | | gap | How far a record is from where it would be if the records were back to back. A gap of 0 means back to back | | descriptor | The 72-byte table property that points to the reference index block and holds the totals, the CF ID, the SST file number and the relocation-state fingerprint | | relocation-state fingerprint | 128-bit hash of the routes and relocation files of the Version that an SST was built from (fields route_snapshot_fingerprint_*). Written as 0 in this diff | | legacy | Written the way RocksDB writes it without blob indirection | Previous behavior: - A BlobIndex named only a physical blob file and an offset. A value could not move to another file without rewriting every SST that referenced it. - An SST had no metadata that listed its blob references. The only way to find them was to scan its user keys. - These readers get the blob indexes of a wide-column entity through ForEachBlobFileNumber: UpdateBoundaries, BlobGarbageMeter, the compaction iterator, WideColumnSerialization::RemoveBlobColumns, and the referenced_blob_bytes table property count in BlockBasedTableBuilder::Add. ForEachBlobFileNumber decoded each blob index, but it did not check the rest of the entity framing (name order, section sizes, exact input size), and it could call the callback before it found a later error. New behavior: Separate change (option on or off): - The same readers (UpdateBoundaries, BlobGarbageMeter, the compaction iterator, RemoveBlobColumns, and the referenced_blob_bytes count in BlockBasedTableBuilder::Add) use WideColumnSerialization::ForEachBlobIndex, with the option on or off. With the option on, the table builder also uses it to give the blob columns of each entity to the collector. For an entity with blob columns, it checks the full framing, then decodes and validates every blob index before any callback runs. So an entity with a malformed framing now fails the flush or compaction, even with the option off. The referenced_blob_bytes count itself does not fail the build on a malformed entity. It removes that property from the SST, as in base. Option and option gate: - This diff adds a new column family option, `enable_blob_indirection`. It is off by default, and SetOptions can change it. - The option gate (see Terms) keeps the option off outside tests until the stack is complete. ColumnFamilyData::ValidateOptions rejects the option with NotSupported on a read-write open, CreateColumnFamily and SetOptions. Read-only opens (DBImplReadOnly and CompactedDBImpl) and secondary opens (including DB::OpenAndCompact) do not run ValidateOptions; they check only the FIFO rule (see Implementation). Only tests can turn the option on, through the internal ColumnFamilyData::TEST_SetAllowBlobIndirection(). DBTestBase sets it. - When the option is on, ValidateOptions also requires enable_blob_files, the block-based table format, no blob direct write, and a compaction style other than FIFO. Blob file writer (option on): - The blob file writer encodes each new value with a stable identity (origin file, original offset) and two checksums: - the logical CRC: the CRC32C of the uncompressed value. - the stored CRC: the CRC32C of the exact stored (maybe compressed) bytes, extended with the compression byte. - For an uncompressed value the stored CRC is not written, because the reader can compute it from the logical CRC. - The checksums have no version field of their own. The BlobIndex type byte gives the format version: the two new types, 3 (kIndirectBlob) and 4 (kIndirectFlattenedBlob, see Implementation), are BlobIndex::kIndirectBlobFormatVersion (1). A later format uses new type values. - The read path does not check these checksums yet. The check starts in diff 3. Exact reference metadata (option on): 1. What: a sorted list of every blob reference that the SST keeps, in plain blob values and in the blob columns of wide-column entities. 2. Where: data blocks (target 16 KB each before compression; small blocks waste less I/O, because GC reads a whole block to get a few groups) and one reference index block, written after the SST data blocks and before the SST tail, plus a 72-byte descriptor table property. 3. Who reads it: only tests in this diff. The census, GC and RepairDB read it in later diffs. 4. Details: the top comment of db/blob/blob_reference_metadata.h has the full format with diagrams, the record-bytes rules and the gap rule. - Not collected: SstFileWriter output (it has no DB file number) and any output whose CF is unknown (it has no CF ID). The descriptor needs both values, so these outputs stay in the legacy form. Checks on the metadata: - A repeated or overlapping reference in one group fails the SST build with Corruption. Readers check every count, order, size and total against the index and the descriptor, so corrupt metadata returns Corruption. The writer never makes a data block larger than kTargetDataBlockSize (16 KB), and readers accept up to kMaxDataBlockSize (64 KB). Details: "Readers" and "Integrity" at the top of db/blob/blob_reference_metadata.h, and BlobReferenceGroupDecoder. Collection and memory: - Each reference is encoded when it is added, in the on-disk entry encoding, into its group's chain of 512-byte chunks. Only a group that takes the unsorted fallback is encoded a second time, in Finish(). Inside a group, references normally arrive in rising offset order, so no reference is sorted; at the end only the groups are sorted. A group that gets a lower offset takes the unsorted fallback. The BlobReferenceMetadataCollector class comment explains the chunks, the chunk size and the fallback. - The collector keeps every reference of the SST in memory until the SST is finished, and charges this memory to the block cache under a new CacheEntryRole, kBlobReferenceMetadataBuildingBuffer. The measured cost is about 8.3 to 9 bytes per reference for uncompressed values and 12.8 to 13.5 bytes for compressed values. "Memory usage" and "Memory charge" in the BlobReferenceMetadataCollector class comment give the measurement setup and the charge rule. - There is no fixed limit on references. The collector fails with MemoryLimit above 2^20 groups or 2^20 data blocks in one SST; these are persisted format limits. - There is no charge when the table has no block cache or when charging for this role is disabled. Each block-based table factory holds one cache reservation manager for this role, shared by its builders and the SST summaries. If the charge fails, the SST build fails, so the flush or compaction fails with a background error. The buffers and their charge are released when the SST is finished or abandoned. Implementation: - BlobIndex adds two indirect encodings. The legacy encodings do not change. - Type 3, kIndirectBlob (unflattened, generation number == origin), stores the origin, the original offset, the stored size, the compression type, the logical CRC and, for compressed values, the stored CRC. It does not store the generation number. This saves the size of one varint file number per reference (about 3 to 4 bytes for typical file numbers; an estimate, not measured). - Type 4, kIndirectFlattenedBlob (flattened), stores the generation number first, then the same fields. Its generation number must be above the origin. The decoder rejects type 4 with generation number <= origin, so each reference has one encoding. - The encoders take the fields as one BlobIndex::IndirectBlobFields, set by field name, so values of the same type (file number, offset and size; the two CRCs) cannot be swapped. EncodeIndirectBlob() takes the generation number as a separate argument; EncodeUnflattenedIndirectBlob() uses the origin, so the common path cannot leave the generation number unset. - The new files blob_reference_metadata.{h,cc} define the exact reference metadata. The files also contain: - BlobReferenceMetadataCollector, the collector described above. It takes the SST identity (CF ID, SST file number and the relocation-state fingerprint) as one BlobReferenceSstIdentity, set by field name. - BlobReferenceGroupKey, the key of a group: (encoding class, generation number, origin). It has the order, the hash and the validity check that the collector map, the sort, the decoder and FindBlobReferenceGroup() use. BlobReference and BlobReferenceGroup keep their flat fields and return the key through group_key(). - The index codec and the descriptor table property. - BlobReferenceGroupDecoder, the only decoder of reference runs. It reads a group one data block at a time and holds every corruption check of a group walk. Readers that visit one group (found by a binary search of the index) use it, and so do readers that visit every block in order. - Callers in this diff: only the table builder runs the collector. The decoders (DecodeBlobReferenceIndex, FindBlobReferenceGroup, BlobReferenceGroupDecoder, ForEachBlobReference*) are used only by tests. kMaxDecodedBlobReferenceBytes is used only in compile-time checks of the count limits. The census (diff 5) and GC (diffs 6 and 7) are their first production users. The census is also the first user of BlobReferenceSstSummary (see the last Implementation bullet). - Descriptor: the CF ID and the SST file number are the repair fields. They let RepairDB tell which column family and SST the list belongs to without a MANIFEST. The decoder rejects a descriptor whose repair fields are not set (unknown CF ID or SST file number 0). The descriptor also has a 128-bit relocation-state fingerprint (the route_snapshot_fingerprint fields). Remote compaction uses it in diff 9 to check that an SST built elsewhere used the same routes as the job that asked for it. In this diff the table builder always writes 0, and DecodeFrom() does not check it. The field is in the format now, so the descriptor keeps one format version (1) through the stack. While the option gate is on, no data in this format exists outside tests. - The collector validates each BlobIndex once, when it is added. Its Finish() writes the data blocks and then the reference index block through one block writer, and fills the descriptor, including the index handle. Destroying the collector frees its memory and then its charge, also when the build is abandoned. - BlockBasedTableBuilder creates the collector with the CF ID and SST file number from TableBuilderOptions. It passes the size of each entry's user key to the collector, so the collector can compute the exact size of each blob record. It adds the collector's EstimatedEncodedSize() to its tail size estimate: the blocks are written before the tail, but output file size planning uses this estimate. The reference blocks are written with BlockType::kUserDefinedIndex, which keeps their contents as opaque bytes. Readers read them without the block cache, so prepopulate_block_cache does not warm them into the block cache (WriteMaybeCompressedBlock() with warm_cache false). - With the option on, the blob file builder computes the CRC32C of each stored value once. The blob log record CRC combines it with the key CRC (crc32c::Crc32cCombine). The stored CRC extends it with the compression byte. For an uncompressed value it is also the logical CRC. The bytes written do not change. - WideColumnSerialization::ForEachBlobIndex replaces ForEachBlobFileNumber. It reads the entity header and column types once. For a V1 entity (no column types and no blob columns), it returns right after the header. For a V2 entity without blob columns, it returns right after the type scan. It shares one V2 column walker (WalkV2Columns) with Deserialize(), so the entity is parsed in one pass. - Read-only opens (DBImplReadOnly, CompactedDBImpl) and secondary opens skip ValidateOptions, so they check the compaction style against the option once per column family: in DBImpl::Recover() for read-only opens, and at the start of OpenAsSecondaryImpl() for secondary opens. DB::OpenAndCompact opens through OpenAsSecondaryImpl(), so it gets the same check. - CacheReservationHandle gets Resize(), so each collector or summary can change its own part of the shared charge. - util/coding.h adds GetCanonicalVarint32/64, which reject overlong varint encodings. The new formats use them, so each value has one encoding. - BlobReferenceSstSummary is the in-memory list of an SST's groups and totals that Finish() can return. It has no caller yet: the table builder passes a null summary output. It is checked and does not change after Finish(), so the finished builder, the SST metadata and the census (its first user, in diff 5) can share it. It is in this diff because Finish() hands the group buffer and its cache charge over to the summary. Adding the summary later would mean rewriting the charge code of Finish() in diff 5, so the charge code is written and tested once, here. Safety boundaries and unsupported cases: - Public API: this diff adds the option `enable_blob_indirection` with the C API functions `rocksdb_options_set_enable_blob_indirection` and `rocksdb_options_get_enable_blob_indirection`. It also inserts a new CacheEntryRole value before kMisc, which changes the numeric value of kMisc and kNumCacheEntryRoles. - With the option off, new blob values and SSTs are written in the legacy form. Legacy blob indexes stay readable. - One behavior change applies with the option off too: UpdateBoundaries, BlobGarbageMeter and the compaction iterator fully validate wide-column entities with blob columns and their blob indexes (see "Separate change" above). - Exact reference metadata has one format (version 1) with a fixed 72-byte descriptor. The decoders reject any other version as Corruption. - The metadata blocks are protected by the table's block checksum only. With checksum = kNoChecksum, corrupt metadata is detected only when it breaks the format or order checks. - The index decoder checks each decoded count against the remaining input size before it reserves memory. So a small corrupt index cannot make it allocate a large vector. - Inlined-TTL and same-file blob indexes are not collected and cannot be relocated. Encoding, count and checksum errors return an error; a bad reference is never skipped. - Option gate (see Terms): outside tests no DB writes data in the new format. The gate checks only the option value, never persisted state. Diff 11 removes it, once data written with the option stays readable through the end of the stack and the safety checks for turning the option on are in place. - Downgrading the binary after data was written with the option on is not supported. Validation and known limitations: - Unit tests cover the BlobIndex encodings and their rejection of bad input, the metadata layout and every corruption check, the collector (sorted path, unsorted fallback, memory charge), the flush integration, the option checks and option gate, wide-column validation, and canonical varints. See the Test Plan. - The collection buffer is not spilled to disk. One SST with very many blob references needs a matching amount of charged memory: the per-reference amount given under "Collection and memory", plus up to one partly used 512-byte chunk per group. While a fallback group is sorted, it also needs 16 bytes of sort key per reference and a second copy of its entries. For a test group of 200K references with every other one out of order, the test comment estimates about 44 bytes per reference, and the test checks that the peak charge is at most 56 bytes per reference. - Flush still writes a single output file. Until diff 7, one compaction output collects every reference it gets, charged to the block cache. Diff 7 cuts compaction outputs (including L0) at a reference cap and keeps each user key's history in one output. With the option gate, a large output is reachable only through the test hook. Stack context (same in diffs 2-11, except that diffs 8-11 add the diff 8 route rule to the install line; "<== this diff" marks this diff's part): Goal: remove garbage from blob files without rewriting any SST. A value gets a stable identity when it is first written. GC copies the live values of whole files into a relocation file and installs routes in the MANIFEST. Readers follow the routes. SSTs are not changed. How a read finds a value (P is the relocation file that holds the value now): ``` SST entry indirect BlobIndex { generation number, origin, original offset, size, CRCs } | v Version file table: generation number -> live file, or route old file -> P | | live file route found | | v v blob file: read at original offset relocation file P (always one hop): relocation file: look up key look up key (origin, original offset) (origin, original offset) ``` Lifecycle of one value (in this example, F is the blob file where the value is first written, and P is the relocation file that GC writes): ``` write new value goes to blob file F; index: origin = F, generation number = F | census per generation number: the SSTs and references that use it; | per physical file: live bytes, ranked by garbage ratio GC job copy the live values of whole input files (stored bytes, checked | with the stored CRC) into relocation file P, keyed by | (origin, original offset) install one MANIFEST edit: add P, route each input generation number still | in use to P, re-point the routes into the inputs to P, and remove | each input (blob file retirement for a blob file, relocation | file deletion for a relocation file) flatten compaction sets generation number = P in the SSTs it rewrites | route route retirement: route deletion records remove the routes that delete no SST uses ``` Stack map: | Diff | Adds | |---|---| | 1 | Finalize blob metadata outside the DB mutex (prep) | | 2 | Indirect BlobIndex (kIndirectBlob) and exact SST reference metadata <== this diff | | 3 | Relocation file format and stable read path | | 4 | MANIFEST routes and Version ownership | | 5 | Exact reverse reference census | | 6 | Standalone relocation GC (one input file) | | 7 | Pack whole input files; reclaim relocation files | | 8 | Flatten generation numbers; route retirement | | 9 | Remote compaction validation | | 10 | Dynamic options and operations | | 11 | RepairDB rebuilds relocation state; option gate removed | | 12 | db_stress and crash test support | Stack context: terms for later diffs: | Term | Meaning | |---|---| | census | Background count, per generation number, of the SSTs and references that use it, plus live bytes per physical file (diff 5) | | indirect origin | Normal blob file written with the option on; it is the origin of unflattened references | | flatten | Compaction rewrites the generation number to the relocation file that holds the value, so old routes can go. Only flatten makes generation number != origin (diff 8) | | blob file retirement | MANIFEST record (kBlobFileRetirement) that removes a collected blob file from the Version (diff 4) | | route deletion | MANIFEST record that removes a route that no SST uses. Route retirement is the background job that writes these records (diff 8) | Tracking issue: meta#15301 Differential Revision: D124652097
Summary: Pull Request resolved: meta#15540 Why the change is required: The goal of this stack is a blob GC that does not depend on SSTs: it removes garbage from blob files without rewriting any SST. Such a GC needs two things that RocksDB does not have: - A value identity that does not change when the value moves to another file. - A checked list of the blob references that each SST keeps. The list must include older internal versions of a key that the SST still keeps (for example for snapshots), and the blob columns of wide-column entities. This diff adds both. Later diffs build the census (an in-memory reverse index from generation number to the SSTs that use it), GC and repair on top of this list. This diff in one paragraph: With the new column family option enable_blob_indirection on, the blob file writer gives each new value an indirect BlobIndex (type 3, kIndirectBlob). It stores the value's stable identity (origin file, original offset) and its CRCs (an uncompressed value stores only the CRC of the uncompressed value; a compressed value also stores the CRC of its stored bytes). Also with the option on, the block-based table builder collects every blob reference that the SST keeps. It writes them as "exact reference metadata": data blocks, one reference index block, and a 72-byte descriptor table property. In this diff only the table builder writes this metadata, and only tests read it. A temporary option gate (see "option gate" under Terms) keeps the option off outside tests. Separately, wide-column entities with blob columns are now fully checked before their blob indexes are used, also with the option off. Parts of this diff: 1. Indirect BlobIndex (core): types 3 and 4 in db/blob/blob_index.h, and the blob file writer that writes type 3. 2. Exact reference metadata format and decoders (core): db/blob/blob_reference_metadata.{h,cc}. 3. Collector and table builder integration (core): BlobReferenceMetadataCollector, its block cache charge, and BlockBasedTableBuilder. 4. Option enable_blob_indirection and the option gate. Tests need it to turn the feature on. You can skip this part on a first read. 5. Wide-column validation (WideColumnSerialization::ForEachBlobIndex). The table builder uses it to give the blob columns of each entity to the collector. It also changes behavior with the option off (see New behavior). You can skip this part on a first read. 6. Small helpers: canonical varints (util/coding.h) and CacheReservationHandle::Resize(). You can skip these on a first read. How to read this diff: 1. Overview: read "Why" and "This diff in one paragraph" above, and "Terms used in this diff" right below. Then read the top comment of db/blob/blob_reference_metadata.h (terms, layout diagrams, run and entry layout, index layout). 2. Indirect BlobIndex (db/blob/blob_index.h) and canonical varints (util/coding.h GetCanonicalVarint32/64): read the overview comment and the kIndirectBlob / kIndirectFlattenedBlob code. Every later part uses these fields: origin, original offset, generation number, logical CRC and stored CRC. The new formats read varints only in the canonical form. 3. Blob file writer (db/blob/blob_file_builder.cc, blob_log_format.cc, blob_log_writer.cc): see how a new value gets a type-3 index and how its CRC is computed once. 4. Format types and decoders: the structs in db/blob/blob_reference_metadata.h, then in the .cc the constants, GapBase, DecodeEntryBody, DecodeRunEntriesAt, DecodeBlobReferenceIndex and BlobReferenceGroupDecoder. They use only BlobIndex and the canonical varints. Learn the on-disk format here, before you read the writer. 5. Collector and its cache charge: first read the new CacheEntryRole kBlobReferenceMetadataBuildingBuffer (include/rocksdb/cache.h) and CacheReservationHandle::Resize (cache/cache_reservation_manager.h). Then read BlobReferenceMetadataCollector in this order: the class comment, Add, AppendSorted, BuildAndWriteDataBlocks, EncodeIndex, FinishOnce. Then read the unsorted fallback (AppendUnsorted, SortUnsortedGroup, SortAndEncodeGroup). 6. Wide-column change (db/wide/wide_column_serialization.cc, ForEachBlobIndex and WalkV2Columns): it gives every blob index of an entity to a callback after it checks the whole entity. The table builder uses it to feed the collector. UpdateBoundaries, BlobGarbageMeter and the compaction iterator also use it, with the option on or off. 7. Option: enable_blob_indirection (include/rocksdb/advanced_options.h, options/cf_options.cc), and the option checks and gate in db/column_family.cc (ValidateOptions, TEST_SetAllowBlobIndirection), db/db_impl/db_impl_open.cc and db_impl_secondary.cc (the FIFO check for opens that skip ValidateOptions). 8. Table integration: table/block_based/block_based_table_factory.cc (the shared reservation manager), then block_based_table_builder.cc (Rep constructor, Add, WriteBlobReferenceMetadata, EstimatedTailSize). See where the collector is created, fed and finished. block_based_table_reader.cc maps the new block name to a block type. Terms used in this diff: Stack terms: | Term | Meaning | |---|---| | origin | Blob file where the value was first written | | original offset | Value offset in the origin file | | stable identity | (origin, original offset). Never changes, even when the value moves | | generation number | The file number that a reference uses to find its value. A new value has generation number == origin. When GC moves the value, the reference keeps its generation number, and a route sends the read to the relocation file | | indirect BlobIndex | A kIndirectBlob (type 3) or kIndirectFlattenedBlob (type 4) index. It stores the stable identity, so the value can move without an SST rewrite | | unflattened reference | Index with generation number == origin: the value was not moved, or it was moved but no compaction has rewritten its reference yet (see "flatten" in "Stack context: terms for later diffs") | | flattened reference | Index with generation number != origin (type 4). Only flatten (diff 8) writes it; this diff defines the encoding | | route | Maps an old file number (a file that GC has removed from the Version) to the relocation file that now holds its live values. One hop: a target is never itself routed. Stored in the MANIFEST (diff 4) | | relocation file | Block-based table keyed by (origin, original offset) that holds copied values. Each value is the stored bytes plus a 5-byte trailer (kSimpleGen2BlobTrailerSize) | | logical CRC | CRC32C of the uncompressed value (BlobIndex::stable_checksum()) | | stored CRC | CRC32C of the stored bytes plus the compression byte | | physical file | A file that holds value bytes: a normal blob file or a relocation file. A generation number can name a physical file, or it can be only the key of a route | | exact reference metadata | Per-SST sorted list of every blob reference the SST keeps, written when the SST is built. References with the same encoding class, generation number and origin form a group. The list is stored as data blocks, with an index of the groups. An SST that has it is an exact SST | | option gate | Until diff 11, ColumnFamilyData::ValidateOptions returns NotSupported for enable_blob_indirection on a read-write open, CreateColumnFamily and SetOptions, unless a test calls ColumnFamilyData::TEST_SetAllowBlobIndirection(true). It keeps the option off in production while the stack is not complete | Terms of this diff (all defined at the top of db/blob/blob_reference_metadata.h): | Term | Meaning | |---|---| | encoding class (class) | kDirect (a legacy kBlob or kBlobTTL index), unflattened, or flattened | | reference group (group) | All references of one SST with the same (encoding class, generation number, origin) | | run | The part of one group inside one data block | | record bytes | The bytes the value's record takes in the file named by its generation number | | gap | How far a record is from where it would be if the records were back to back. A gap of 0 means back to back | | descriptor | The 72-byte table property that points to the reference index block and holds the totals, the CF ID, the SST file number and the relocation-state fingerprint | | relocation-state fingerprint | 128-bit hash of the routes and relocation files of the Version that an SST was built from (fields route_snapshot_fingerprint_*). Written as 0 in this diff | | legacy | Written the way RocksDB writes it without blob indirection | Previous behavior: - A BlobIndex named only a physical blob file and an offset. A value could not move to another file without rewriting every SST that referenced it. - An SST had no metadata that listed its blob references. The only way to find them was to scan its user keys. - These readers get the blob indexes of a wide-column entity through ForEachBlobFileNumber: UpdateBoundaries, BlobGarbageMeter, the compaction iterator, WideColumnSerialization::RemoveBlobColumns, and the referenced_blob_bytes table property count in BlockBasedTableBuilder::Add. ForEachBlobFileNumber decoded each blob index, but it did not check the rest of the entity framing (name order, section sizes, exact input size), and it could call the callback before it found a later error. New behavior: Separate change (option on or off): - The same readers (UpdateBoundaries, BlobGarbageMeter, the compaction iterator, RemoveBlobColumns, and the referenced_blob_bytes count in BlockBasedTableBuilder::Add) use WideColumnSerialization::ForEachBlobIndex, with the option on or off. With the option on, the table builder also uses it to give the blob columns of each entity to the collector. For an entity with blob columns, it checks the full framing, then decodes and validates every blob index before any callback runs. So an entity with a malformed framing now fails the flush or compaction, even with the option off. The referenced_blob_bytes count itself does not fail the build on a malformed entity. It removes that property from the SST, as in base. Option and option gate: - This diff adds a new column family option, `enable_blob_indirection`. It is off by default, and SetOptions can change it. - The option gate (see Terms) keeps the option off outside tests until the stack is complete. ColumnFamilyData::ValidateOptions rejects the option with NotSupported on a read-write open, CreateColumnFamily and SetOptions. Read-only opens (DBImplReadOnly and CompactedDBImpl) and secondary opens (including DB::OpenAndCompact) do not run ValidateOptions; they check only the FIFO rule (see Implementation). Only tests can turn the option on, through the internal ColumnFamilyData::TEST_SetAllowBlobIndirection(). DBTestBase sets it. - When the option is on, ValidateOptions also requires enable_blob_files, the block-based table format, no blob direct write, and a compaction style other than FIFO. Blob file writer (option on): - The blob file writer encodes each new value with a stable identity (origin file, original offset) and two checksums: - the logical CRC: the CRC32C of the uncompressed value. - the stored CRC: the CRC32C of the exact stored (maybe compressed) bytes, extended with the compression byte. - For an uncompressed value the stored CRC is not written, because the reader can compute it from the logical CRC. - The checksums have no version field of their own. The BlobIndex type byte gives the format version: the two new types, 3 (kIndirectBlob) and 4 (kIndirectFlattenedBlob, see Implementation), are BlobIndex::kIndirectBlobFormatVersion (1). A later format uses new type values. - The read path does not check these checksums yet. The check starts in diff 3. Exact reference metadata (option on): 1. What: a sorted list of every blob reference that the SST keeps, in plain blob values and in the blob columns of wide-column entities. 2. Where: data blocks (target 16 KB each before compression; small blocks waste less I/O, because GC reads a whole block to get a few groups) and one reference index block, written after the SST data blocks and before the SST tail, plus a 72-byte descriptor table property. 3. Who reads it: only tests in this diff. The census, GC and RepairDB read it in later diffs. 4. Details: the top comment of db/blob/blob_reference_metadata.h has the full format with diagrams, the record-bytes rules and the gap rule. - Not collected: SstFileWriter output (it has no DB file number) and any output whose CF is unknown (it has no CF ID). The descriptor needs both values, so these outputs stay in the legacy form. Checks on the metadata: - A repeated or overlapping reference in one group fails the SST build with Corruption. Readers check every count, order, size and total against the index and the descriptor, so corrupt metadata returns Corruption. The writer never makes a data block larger than kTargetDataBlockSize (16 KB), and readers accept up to kMaxDataBlockSize (64 KB). Details: "Readers" and "Integrity" at the top of db/blob/blob_reference_metadata.h, and BlobReferenceGroupDecoder. Collection and memory: - Each reference is encoded when it is added, in the on-disk entry encoding, into its group's chain of 512-byte chunks. Only a group that takes the unsorted fallback is encoded a second time, in Finish(). Inside a group, references normally arrive in rising offset order, so no reference is sorted; at the end only the groups are sorted. A group that gets a lower offset takes the unsorted fallback. The BlobReferenceMetadataCollector class comment explains the chunks, the chunk size and the fallback. - The collector keeps every reference of the SST in memory until the SST is finished, and charges this memory to the block cache under a new CacheEntryRole, kBlobReferenceMetadataBuildingBuffer. The measured cost is about 8.3 to 9 bytes per reference for uncompressed values and 12.8 to 13.5 bytes for compressed values. "Memory usage" and "Memory charge" in the BlobReferenceMetadataCollector class comment give the measurement setup and the charge rule. - There is no fixed limit on references. The collector fails with MemoryLimit above 2^20 groups or 2^20 data blocks in one SST; these are persisted format limits. - There is no charge when the table has no block cache or when charging for this role is disabled. Each block-based table factory holds one cache reservation manager for this role, shared by its builders and the SST summaries. If the charge fails, the SST build fails, so the flush or compaction fails with a background error. The buffers and their charge are released when the SST is finished or abandoned. Implementation: - BlobIndex adds two indirect encodings. The legacy encodings do not change. - Type 3, kIndirectBlob (unflattened, generation number == origin), stores the origin, the original offset, the stored size, the compression type, the logical CRC and, for compressed values, the stored CRC. It does not store the generation number. This saves the size of one varint file number per reference (about 3 to 4 bytes for typical file numbers; an estimate, not measured). - Type 4, kIndirectFlattenedBlob (flattened), stores the generation number first, then the same fields. Its generation number must be above the origin. The decoder rejects type 4 with generation number <= origin, so each reference has one encoding. - The encoders take the fields as one BlobIndex::IndirectBlobFields, set by field name, so values of the same type (file number, offset and size; the two CRCs) cannot be swapped. EncodeIndirectBlob() takes the generation number as a separate argument; EncodeUnflattenedIndirectBlob() uses the origin, so the common path cannot leave the generation number unset. - The new files blob_reference_metadata.{h,cc} define the exact reference metadata. The files also contain: - BlobReferenceMetadataCollector, the collector described above. It takes the SST identity (CF ID, SST file number and the relocation-state fingerprint) as one BlobReferenceSstIdentity, set by field name. - BlobReferenceGroupKey, the key of a group: (encoding class, generation number, origin). It has the order, the hash and the validity check that the collector map, the sort, the decoder and FindBlobReferenceGroup() use. BlobReference and BlobReferenceGroup keep their flat fields and return the key through group_key(). - The index codec and the descriptor table property. - BlobReferenceGroupDecoder, the only decoder of reference runs. It reads a group one data block at a time and holds every corruption check of a group walk. Readers that visit one group (found by a binary search of the index) use it, and so do readers that visit every block in order. - Callers in this diff: only the table builder runs the collector. The decoders (DecodeBlobReferenceIndex, FindBlobReferenceGroup, BlobReferenceGroupDecoder, ForEachBlobReference*) are used only by tests. kMaxDecodedBlobReferenceBytes is used only in compile-time checks of the count limits. The census (diff 5) and GC (diffs 6 and 7) are their first production users. The census is also the first user of BlobReferenceSstSummary (see the last Implementation bullet). - Descriptor: the CF ID and the SST file number are the repair fields. They let RepairDB tell which column family and SST the list belongs to without a MANIFEST. The decoder rejects a descriptor whose repair fields are not set (unknown CF ID or SST file number 0). The descriptor also has a 128-bit relocation-state fingerprint (the route_snapshot_fingerprint fields). Remote compaction uses it in diff 9 to check that an SST built elsewhere used the same routes as the job that asked for it. In this diff the table builder always writes 0, and DecodeFrom() does not check it. The field is in the format now, so the descriptor keeps one format version (1) through the stack. While the option gate is on, no data in this format exists outside tests. - The collector validates each BlobIndex once, when it is added. Its Finish() writes the data blocks and then the reference index block through one block writer, and fills the descriptor, including the index handle. Destroying the collector frees its memory and then its charge, also when the build is abandoned. - BlockBasedTableBuilder creates the collector with the CF ID and SST file number from TableBuilderOptions. It passes the size of each entry's user key to the collector, so the collector can compute the exact size of each blob record. It adds the collector's EstimatedEncodedSize() to its tail size estimate: the blocks are written before the tail, but output file size planning uses this estimate. The reference blocks are written with BlockType::kUserDefinedIndex, which keeps their contents as opaque bytes. Readers read them without the block cache, so prepopulate_block_cache does not warm them into the block cache (WriteMaybeCompressedBlock() with warm_cache false). - With the option on, the blob file builder computes the CRC32C of each stored value once. The blob log record CRC combines it with the key CRC (crc32c::Crc32cCombine). The stored CRC extends it with the compression byte. For an uncompressed value it is also the logical CRC. The bytes written do not change. - WideColumnSerialization::ForEachBlobIndex replaces ForEachBlobFileNumber. It reads the entity header and column types once. For a V1 entity (no column types and no blob columns), it returns right after the header. For a V2 entity without blob columns, it returns right after the type scan. It shares one V2 column walker (WalkV2Columns) with Deserialize(), so the entity is parsed in one pass. - Read-only opens (DBImplReadOnly, CompactedDBImpl) and secondary opens skip ValidateOptions, so they check the compaction style against the option once per column family: in DBImpl::Recover() for read-only opens, and at the start of OpenAsSecondaryImpl() for secondary opens. DB::OpenAndCompact opens through OpenAsSecondaryImpl(), so it gets the same check. - CacheReservationHandle gets Resize(), so each collector or summary can change its own part of the shared charge. - util/coding.h adds GetCanonicalVarint32/64, which reject overlong varint encodings. The new formats use them, so each value has one encoding. - BlobReferenceSstSummary is the in-memory list of an SST's groups and totals that Finish() can return. It has no caller yet: the table builder passes a null summary output. It is checked and does not change after Finish(), so the finished builder, the SST metadata and the census (its first user, in diff 5) can share it. It is in this diff because Finish() hands the group buffer and its cache charge over to the summary. Adding the summary later would mean rewriting the charge code of Finish() in diff 5, so the charge code is written and tested once, here. Safety boundaries and unsupported cases: - Public API: this diff adds the option `enable_blob_indirection` with the C API functions `rocksdb_options_set_enable_blob_indirection` and `rocksdb_options_get_enable_blob_indirection`. It also inserts a new CacheEntryRole value before kMisc, which changes the numeric value of kMisc and kNumCacheEntryRoles. - With the option off, new blob values and SSTs are written in the legacy form. Legacy blob indexes stay readable. - One behavior change applies with the option off too: UpdateBoundaries, BlobGarbageMeter and the compaction iterator fully validate wide-column entities with blob columns and their blob indexes (see "Separate change" above). - Exact reference metadata has one format (version 1) with a fixed 72-byte descriptor. The decoders reject any other version as Corruption. - The metadata blocks are protected by the table's block checksum only. With checksum = kNoChecksum, corrupt metadata is detected only when it breaks the format or order checks. - The index decoder checks each decoded count against the remaining input size before it reserves memory. So a small corrupt index cannot make it allocate a large vector. - Inlined-TTL and same-file blob indexes are not collected and cannot be relocated. Encoding, count and checksum errors return an error; a bad reference is never skipped. - Option gate (see Terms): outside tests no DB writes data in the new format. The gate checks only the option value, never persisted state. Diff 11 removes it, once data written with the option stays readable through the end of the stack and the safety checks for turning the option on are in place. - Downgrading the binary after data was written with the option on is not supported. Validation and known limitations: - Unit tests cover the BlobIndex encodings and their rejection of bad input, the metadata layout and every corruption check, the collector (sorted path, unsorted fallback, memory charge), the flush integration, the option checks and option gate, wide-column validation, and canonical varints. See the Test Plan. - The collection buffer is not spilled to disk. One SST with very many blob references needs a matching amount of charged memory: the per-reference amount given under "Collection and memory", plus up to one partly used 512-byte chunk per group. While a fallback group is sorted, it also needs 16 bytes of sort key per reference and a second copy of its entries. For a test group of 200K references with every other one out of order, the test comment estimates about 44 bytes per reference, and the test checks that the peak charge is at most 56 bytes per reference. - Flush still writes a single output file. Until diff 7, one compaction output collects every reference it gets, charged to the block cache. Diff 7 cuts compaction outputs (including L0) at a reference cap and keeps each user key's history in one output. With the option gate, a large output is reachable only through the test hook. Stack context (same in diffs 2-11, except that diffs 8-11 add the diff 8 route rule to the install line; "<== this diff" marks this diff's part): Goal: remove garbage from blob files without rewriting any SST. A value gets a stable identity when it is first written. GC copies the live values of whole files into a relocation file and installs routes in the MANIFEST. Readers follow the routes. SSTs are not changed. How a read finds a value (P is the relocation file that holds the value now): ``` SST entry indirect BlobIndex { generation number, origin, original offset, size, CRCs } | v Version file table: generation number -> live file, or route old file -> P | | live file route found | | v v blob file: read at original offset relocation file P (always one hop): relocation file: look up key look up key (origin, original offset) (origin, original offset) ``` Lifecycle of one value (in this example, F is the blob file where the value is first written, and P is the relocation file that GC writes): ``` write new value goes to blob file F; index: origin = F, generation number = F | census per generation number: the SSTs and references that use it; | per physical file: live bytes, ranked by garbage ratio GC job copy the live values of whole input files (stored bytes, checked | with the stored CRC) into relocation file P, keyed by | (origin, original offset) install one MANIFEST edit: add P, route each input generation number still | in use to P, re-point the routes into the inputs to P, and remove | each input (blob file retirement for a blob file, relocation | file deletion for a relocation file) flatten compaction sets generation number = P in the SSTs it rewrites | route route retirement: route deletion records remove the routes that delete no SST uses ``` Stack map: | Diff | Adds | |---|---| | 1 | Finalize blob metadata outside the DB mutex (prep) | | 2 | Indirect BlobIndex (kIndirectBlob) and exact SST reference metadata <== this diff | | 3 | Relocation file format and stable read path | | 4 | MANIFEST routes and Version ownership | | 5 | Exact reverse reference census | | 6 | Standalone relocation GC (one input file) | | 7 | Pack whole input files; reclaim relocation files | | 8 | Flatten generation numbers; route retirement | | 9 | Remote compaction validation | | 10 | Dynamic options and operations | | 11 | RepairDB rebuilds relocation state; option gate removed | | 12 | db_stress and crash test support | Stack context: terms for later diffs: | Term | Meaning | |---|---| | census | Background count, per generation number, of the SSTs and references that use it, plus live bytes per physical file (diff 5) | | indirect origin | Normal blob file written with the option on; it is the origin of unflattened references | | flatten | Compaction rewrites the generation number to the relocation file that holds the value, so old routes can go. Only flatten makes generation number != origin (diff 8) | | blob file retirement | MANIFEST record (kBlobFileRetirement) that removes a collected blob file from the Version (diff 4) | | route deletion | MANIFEST record that removes a route that no SST uses. Route retirement is the background job that writes these records (diff 8) | Tracking issue: meta#15301 Differential Revision: D124652097
Summary: Pull Request resolved: meta#15540 Why the change is required: The goal of this stack is a blob GC that does not depend on SSTs: it removes garbage from blob files without rewriting any SST. Such a GC needs two things that RocksDB does not have: - A value identity that does not change when the value moves to another file. - A checked list of the blob references that each SST keeps. The list must include older internal versions of a key that the SST still keeps (for example for snapshots), and the blob columns of wide-column entities. This diff adds both. Later diffs build the census (an in-memory reverse index from generation number to the SSTs that use it), GC and repair on top of this list. This diff in one paragraph: With the new column family option enable_blob_indirection on, the blob file writer gives each new value an indirect BlobIndex (type 3, kIndirectBlob). It stores the value's stable identity (origin file, original offset) and its CRCs (an uncompressed value stores only the CRC of the uncompressed value; a compressed value also stores the CRC of its stored bytes). Also with the option on, the block-based table builder collects every blob reference that the SST keeps. It writes them as "exact reference metadata": data blocks, one reference index block, and a 72-byte descriptor table property. In this diff only the table builder writes this metadata, and only tests read it. A temporary option gate (see "option gate" under Terms) keeps the option off outside tests. Separately, wide-column entities with blob columns are now fully checked before their blob indexes are used, also with the option off. Parts of this diff: 1. Indirect BlobIndex (core): types 3 and 4 in db/blob/blob_index.h, and the blob file writer that writes type 3. 2. Exact reference metadata format and decoders (core): db/blob/blob_reference_metadata.{h,cc}. 3. Collector and table builder integration (core): BlobReferenceMetadataCollector, its block cache charge, and BlockBasedTableBuilder. 4. Option enable_blob_indirection and the option gate. Tests need it to turn the feature on. You can skip this part on a first read. 5. Wide-column validation (WideColumnSerialization::ForEachBlobIndex). The table builder uses it to give the blob columns of each entity to the collector. It also changes behavior with the option off (see New behavior). You can skip this part on a first read. 6. Small helpers: canonical varints (util/coding.h) and CacheReservationHandle::Resize(). You can skip these on a first read. How to read this diff: 1. Overview: read "Why" and "This diff in one paragraph" above, and "Terms used in this diff" right below. Then read the top comment of db/blob/blob_reference_metadata.h (terms, layout diagrams, run and entry layout, index layout). 2. Indirect BlobIndex (db/blob/blob_index.h) and canonical varints (util/coding.h GetCanonicalVarint32/64): read the overview comment and the kIndirectBlob / kIndirectFlattenedBlob code. Every later part uses these fields: origin, original offset, generation number, logical CRC and stored CRC. The new formats read varints only in the canonical form. 3. Blob file writer (db/blob/blob_file_builder.cc, blob_log_format.cc, blob_log_writer.cc): see how a new value gets a type-3 index and how its CRC is computed once. 4. Format types and decoders: the structs in db/blob/blob_reference_metadata.h, then in the .cc the constants, GapBase, DecodeEntryBody, DecodeRunEntriesAt, DecodeBlobReferenceIndex and BlobReferenceGroupDecoder. They use only BlobIndex and the canonical varints. Learn the on-disk format here, before you read the writer. 5. Collector and its cache charge: first read the new CacheEntryRole kBlobReferenceMetadataBuildingBuffer (include/rocksdb/cache.h) and CacheReservationHandle::Resize (cache/cache_reservation_manager.h). Then read BlobReferenceMetadataCollector in this order: the class comment, Add, AppendSorted, BuildAndWriteDataBlocks, EncodeIndex, FinishOnce. Then read the unsorted fallback (AppendUnsorted, SortUnsortedGroup, SortAndEncodeGroup). 6. Wide-column change (db/wide/wide_column_serialization.cc, ForEachBlobIndex and WalkV2Columns): it gives every blob index of an entity to a callback after it checks the whole entity. The table builder uses it to feed the collector. UpdateBoundaries, BlobGarbageMeter and the compaction iterator also use it, with the option on or off. 7. Option: enable_blob_indirection (include/rocksdb/advanced_options.h, options/cf_options.cc), and the option checks and gate in db/column_family.cc (ValidateOptions, TEST_SetAllowBlobIndirection), db/db_impl/db_impl_open.cc and db_impl_secondary.cc (the FIFO check for opens that skip ValidateOptions). 8. Table integration: table/block_based/block_based_table_factory.cc (the shared reservation manager), then block_based_table_builder.cc (Rep constructor, Add, WriteBlobReferenceMetadata, EstimatedTailSize). See where the collector is created, fed and finished. block_based_table_reader.cc maps the new block name to a block type. Terms used in this diff: Stack terms: | Term | Meaning | |---|---| | origin | Blob file where the value was first written | | original offset | Value offset in the origin file | | stable identity | (origin, original offset). Never changes, even when the value moves | | generation number | The file number that a reference uses to find its value. A new value has generation number == origin. When GC moves the value, the reference keeps its generation number, and a route sends the read to the relocation file | | indirect BlobIndex | A kIndirectBlob (type 3) or kIndirectFlattenedBlob (type 4) index. It stores the stable identity, so the value can move without an SST rewrite | | unflattened reference | Index with generation number == origin: the value was not moved, or it was moved but no compaction has rewritten its reference yet (see "flatten" in "Stack context: terms for later diffs") | | flattened reference | Index with generation number != origin (type 4). Only flatten (diff 8) writes it; this diff defines the encoding | | route | Maps an old file number (a file that GC has removed from the Version) to the relocation file that now holds its live values. One hop: a target is never itself routed. Stored in the MANIFEST (diff 4) | | relocation file | Block-based table keyed by (origin, original offset) that holds copied values. Each value is the stored bytes plus a 5-byte trailer (kSimpleGen2BlobTrailerSize) | | logical CRC | CRC32C of the uncompressed value (BlobIndex::stable_checksum()) | | stored CRC | CRC32C of the stored bytes plus the compression byte | | physical file | A file that holds value bytes: a normal blob file or a relocation file. A generation number can name a physical file, or it can be only the key of a route | | exact reference metadata | Per-SST sorted list of every blob reference the SST keeps, written when the SST is built. References with the same encoding class, generation number and origin form a group. The list is stored as data blocks, with an index of the groups. An SST that has it is an exact SST | | option gate | Until diff 11, ColumnFamilyData::ValidateOptions returns NotSupported for enable_blob_indirection on a read-write open, CreateColumnFamily and SetOptions, unless a test calls ColumnFamilyData::TEST_SetAllowBlobIndirection(true). It keeps the option off in production while the stack is not complete | Terms of this diff (all defined at the top of db/blob/blob_reference_metadata.h): | Term | Meaning | |---|---| | encoding class (class) | kDirect (a legacy kBlob or kBlobTTL index), unflattened, or flattened | | reference group (group) | All references of one SST with the same (encoding class, generation number, origin) | | run | The part of one group inside one data block | | record bytes | The bytes the value's record takes in the file named by its generation number | | gap | How far a record is from where it would be if the records were back to back. A gap of 0 means back to back | | descriptor | The 72-byte table property that points to the reference index block and holds the totals, the CF ID, the SST file number and the relocation-state fingerprint | | relocation-state fingerprint | 128-bit hash of the routes and relocation files of the Version that an SST was built from (fields route_snapshot_fingerprint_*). Written as 0 in this diff | | legacy | Written the way RocksDB writes it without blob indirection | Previous behavior: - A BlobIndex named only a physical blob file and an offset. A value could not move to another file without rewriting every SST that referenced it. - An SST had no metadata that listed its blob references. The only way to find them was to scan its user keys. - These readers get the blob indexes of a wide-column entity through ForEachBlobFileNumber: UpdateBoundaries, BlobGarbageMeter, the compaction iterator, WideColumnSerialization::RemoveBlobColumns, and the referenced_blob_bytes table property count in BlockBasedTableBuilder::Add. ForEachBlobFileNumber decoded each blob index, but it did not check the rest of the entity framing (name order, section sizes, exact input size), and it could call the callback before it found a later error. New behavior: Separate change (option on or off): - The same readers (UpdateBoundaries, BlobGarbageMeter, the compaction iterator, RemoveBlobColumns, and the referenced_blob_bytes count in BlockBasedTableBuilder::Add) use WideColumnSerialization::ForEachBlobIndex, with the option on or off. With the option on, the table builder also uses it to give the blob columns of each entity to the collector. For an entity with blob columns, it checks the full framing, then decodes and validates every blob index before any callback runs. So an entity with a malformed framing now fails the flush or compaction, even with the option off. The referenced_blob_bytes count itself does not fail the build on a malformed entity. It removes that property from the SST, as in base. Option and option gate: - This diff adds a new column family option, `enable_blob_indirection`. It is off by default, and SetOptions can change it. - The option gate (see Terms) keeps the option off outside tests until the stack is complete. ColumnFamilyData::ValidateOptions rejects the option with NotSupported on a read-write open, CreateColumnFamily and SetOptions. Read-only opens (DBImplReadOnly and CompactedDBImpl) and secondary opens (including DB::OpenAndCompact) do not run ValidateOptions; they check only the FIFO rule (see Implementation). Only tests can turn the option on, through the internal ColumnFamilyData::TEST_SetAllowBlobIndirection(). DBTestBase sets it. - When the option is on, ValidateOptions also requires enable_blob_files, the block-based table format, no blob direct write, and a compaction style other than FIFO. Blob file writer (option on): - The blob file writer encodes each new value with a stable identity (origin file, original offset) and two checksums: - the logical CRC: the CRC32C of the uncompressed value. - the stored CRC: the CRC32C of the exact stored (maybe compressed) bytes, extended with the compression byte. - For an uncompressed value the stored CRC is not written, because the reader can compute it from the logical CRC. - The checksums have no version field of their own. The BlobIndex type byte gives the format version: the two new types, 3 (kIndirectBlob) and 4 (kIndirectFlattenedBlob, see Implementation), are BlobIndex::kIndirectBlobFormatVersion (1). A later format uses new type values. - The read path does not check these checksums yet. The check starts in diff 3. Exact reference metadata (option on): 1. What: a sorted list of every blob reference that the SST keeps, in plain blob values and in the blob columns of wide-column entities. 2. Where: data blocks (target 16 KB each before compression; small blocks waste less I/O, because GC reads a whole block to get a few groups) and one reference index block, written after the SST data blocks and before the SST tail, plus a 72-byte descriptor table property. 3. Who reads it: only tests in this diff. The census, GC and RepairDB read it in later diffs. 4. Details: the top comment of db/blob/blob_reference_metadata.h has the full format with diagrams, the record-bytes rules and the gap rule. - Not collected: SstFileWriter output (it has no DB file number) and any output whose CF is unknown (it has no CF ID). The descriptor needs both values, so these outputs stay in the legacy form. Checks on the metadata: - A repeated or overlapping reference in one group fails the SST build with Corruption. Readers check every count, order, size and total against the index and the descriptor, so corrupt metadata returns Corruption. The writer never makes a data block larger than kTargetDataBlockSize (16 KB), and readers accept up to kMaxDataBlockSize (64 KB). Details: "Readers" and "Integrity" at the top of db/blob/blob_reference_metadata.h, and BlobReferenceGroupDecoder. Collection and memory: - Each reference is encoded when it is added, in the on-disk entry encoding, into its group's chain of 512-byte chunks. Only a group that takes the unsorted fallback is encoded a second time, in Finish(). Inside a group, references normally arrive in rising offset order, so no reference is sorted; at the end only the groups are sorted. A group that gets a lower offset takes the unsorted fallback. The BlobReferenceMetadataCollector class comment explains the chunks, the chunk size and the fallback. - The collector keeps every reference of the SST in memory until the SST is finished, and charges this memory to the block cache under a new CacheEntryRole, kBlobReferenceMetadataBuildingBuffer. The measured cost is about 8.3 to 9 bytes per reference for uncompressed values and 12.8 to 13.5 bytes for compressed values. "Memory usage" and "Memory charge" in the BlobReferenceMetadataCollector class comment give the measurement setup and the charge rule. - There is no fixed limit on references. The collector fails with MemoryLimit above 2^20 groups or 2^20 data blocks in one SST; these are persisted format limits. - There is no charge when the table has no block cache or when charging for this role is disabled. Each block-based table factory holds one cache reservation manager for this role, shared by its builders and the SST summaries. If the charge fails, the SST build fails, so the flush or compaction fails with a background error. The buffers and their charge are released when the SST is finished or abandoned. Implementation: - BlobIndex adds two indirect encodings. The legacy encodings do not change. - Type 3, kIndirectBlob (unflattened, generation number == origin), stores the origin, the original offset, the stored size, the compression type, the logical CRC and, for compressed values, the stored CRC. It does not store the generation number. This saves the size of one varint file number per reference (about 3 to 4 bytes for typical file numbers; an estimate, not measured). - Type 4, kIndirectFlattenedBlob (flattened), stores the generation number first, then the same fields. Its generation number must be above the origin. The decoder rejects type 4 with generation number <= origin, so each reference has one encoding. - The encoders take the fields as one BlobIndex::IndirectBlobFields, set by field name, so values of the same type (file number, offset and size; the two CRCs) cannot be swapped. EncodeIndirectBlob() takes the generation number as a separate argument; EncodeUnflattenedIndirectBlob() uses the origin, so the common path cannot leave the generation number unset. - The new files blob_reference_metadata.{h,cc} define the exact reference metadata. The files also contain: - BlobReferenceMetadataCollector, the collector described above. It takes the SST identity (CF ID, SST file number and the relocation-state fingerprint) as one BlobReferenceSstIdentity, set by field name. - BlobReferenceGroupKey, the key of a group: (encoding class, generation number, origin). It has the order, the hash and the validity check that the collector map, the sort, the decoder and FindBlobReferenceGroup() use. BlobReference and BlobReferenceGroup keep their flat fields and return the key through group_key(). - The index codec and the descriptor table property. - BlobReferenceGroupDecoder, the only decoder of reference runs. It reads a group one data block at a time and holds every corruption check of a group walk. Readers that visit one group (found by a binary search of the index) use it, and so do readers that visit every block in order. - Callers in this diff: only the table builder runs the collector. The decoders (DecodeBlobReferenceIndex, FindBlobReferenceGroup, BlobReferenceGroupDecoder, ForEachBlobReference*) are used only by tests. kMaxDecodedBlobReferenceBytes is used only in compile-time checks of the count limits. The census (diff 5) and GC (diffs 6 and 7) are their first production users. The census is also the first user of BlobReferenceSstSummary (see the last Implementation bullet). - Descriptor: the CF ID and the SST file number are the repair fields. They let RepairDB tell which column family and SST the list belongs to without a MANIFEST. The decoder rejects a descriptor whose repair fields are not set (unknown CF ID or SST file number 0). The descriptor also has a 128-bit relocation-state fingerprint (the route_snapshot_fingerprint fields). Remote compaction uses it in diff 9 to check that an SST built elsewhere used the same routes as the job that asked for it. In this diff the table builder always writes 0, and DecodeFrom() does not check it. The field is in the format now, so the descriptor keeps one format version (1) through the stack. While the option gate is on, no data in this format exists outside tests. - The collector validates each BlobIndex once, when it is added. Its Finish() writes the data blocks and then the reference index block through one block writer, and fills the descriptor, including the index handle. Destroying the collector frees its memory and then its charge, also when the build is abandoned. - BlockBasedTableBuilder creates the collector with the CF ID and SST file number from TableBuilderOptions. It passes the size of each entry's user key to the collector, so the collector can compute the exact size of each blob record. It adds the collector's EstimatedEncodedSize() to its tail size estimate: the blocks are written before the tail, but output file size planning uses this estimate. The reference blocks are written with BlockType::kUserDefinedIndex, which keeps their contents as opaque bytes. Readers read them without the block cache, so prepopulate_block_cache does not warm them into the block cache (WriteMaybeCompressedBlock() with warm_cache false). - With the option on, the blob file builder computes the CRC32C of each stored value once. The blob log record CRC combines it with the key CRC (crc32c::Crc32cCombine). The stored CRC extends it with the compression byte. For an uncompressed value it is also the logical CRC. The bytes written do not change. - WideColumnSerialization::ForEachBlobIndex replaces ForEachBlobFileNumber. It reads the entity header and column types once. For a V1 entity (no column types and no blob columns), it returns right after the header. For a V2 entity without blob columns, it returns right after the type scan. It shares one V2 column walker (WalkV2Columns) with Deserialize(), so the entity is parsed in one pass. - Read-only opens (DBImplReadOnly, CompactedDBImpl) and secondary opens skip ValidateOptions, so they check the compaction style against the option once per column family: in DBImpl::Recover() for read-only opens, and at the start of OpenAsSecondaryImpl() for secondary opens. DB::OpenAndCompact opens through OpenAsSecondaryImpl(), so it gets the same check. - CacheReservationHandle gets Resize(), so each collector or summary can change its own part of the shared charge. - util/coding.h adds GetCanonicalVarint32/64, which reject overlong varint encodings. The new formats use them, so each value has one encoding. - BlobReferenceSstSummary is the in-memory list of an SST's groups and totals that Finish() can return. It has no caller yet: the table builder passes a null summary output. It is checked and does not change after Finish(), so the finished builder, the SST metadata and the census (its first user, in diff 5) can share it. It is in this diff because Finish() hands the group buffer and its cache charge over to the summary. Adding the summary later would mean rewriting the charge code of Finish() in diff 5, so the charge code is written and tested once, here. Safety boundaries and unsupported cases: - Public API: this diff adds the option `enable_blob_indirection` with the C API functions `rocksdb_options_set_enable_blob_indirection` and `rocksdb_options_get_enable_blob_indirection`. It also inserts a new CacheEntryRole value before kMisc, which changes the numeric value of kMisc and kNumCacheEntryRoles. - With the option off, new blob values and SSTs are written in the legacy form. Legacy blob indexes stay readable. - One behavior change applies with the option off too: UpdateBoundaries, BlobGarbageMeter and the compaction iterator fully validate wide-column entities with blob columns and their blob indexes (see "Separate change" above). - Exact reference metadata has one format (version 1) with a fixed 72-byte descriptor. The decoders reject any other version as Corruption. - The metadata blocks are protected by the table's block checksum only. With checksum = kNoChecksum, corrupt metadata is detected only when it breaks the format or order checks. - The index decoder checks each decoded count against the remaining input size before it reserves memory. So a small corrupt index cannot make it allocate a large vector. - Inlined-TTL and same-file blob indexes are not collected and cannot be relocated. Encoding, count and checksum errors return an error; a bad reference is never skipped. - Option gate (see Terms): outside tests no DB writes data in the new format. The gate checks only the option value, never persisted state. Diff 11 removes it, once data written with the option stays readable through the end of the stack and the safety checks for turning the option on are in place. - Downgrading the binary after data was written with the option on is not supported. Validation and known limitations: - Unit tests cover the BlobIndex encodings and their rejection of bad input, the metadata layout and every corruption check, the collector (sorted path, unsorted fallback, memory charge), the flush integration, the option checks and option gate, wide-column validation, and canonical varints. See the Test Plan. - The collection buffer is not spilled to disk. One SST with very many blob references needs a matching amount of charged memory: the per-reference amount given under "Collection and memory", plus up to one partly used 512-byte chunk per group. While a fallback group is sorted, it also needs 16 bytes of sort key per reference and a second copy of its entries. For a test group of 200K references with every other one out of order, the test comment estimates about 44 bytes per reference, and the test checks that the peak charge is at most 56 bytes per reference. - Flush still writes a single output file. Until diff 7, one compaction output collects every reference it gets, charged to the block cache. Diff 7 cuts compaction outputs (including L0) at a reference cap and keeps each user key's history in one output. With the option gate, a large output is reachable only through the test hook. Stack context (same in diffs 2-11, except that diffs 8-11 add the diff 8 route rule to the install line; "<== this diff" marks this diff's part): Goal: remove garbage from blob files without rewriting any SST. A value gets a stable identity when it is first written. GC copies the live values of whole files into a relocation file and installs routes in the MANIFEST. Readers follow the routes. SSTs are not changed. How a read finds a value (P is the relocation file that holds the value now): ``` SST entry indirect BlobIndex { generation number, origin, original offset, size, CRCs } | v Version file table: generation number -> live file, or route old file -> P | | live file route found | | v v blob file: read at original offset relocation file P (always one hop): relocation file: look up key look up key (origin, original offset) (origin, original offset) ``` Lifecycle of one value (in this example, F is the blob file where the value is first written, and P is the relocation file that GC writes): ``` write new value goes to blob file F; index: origin = F, generation number = F | census per generation number: the SSTs and references that use it; | per physical file: live bytes, ranked by garbage ratio GC job copy the live values of whole input files (stored bytes, checked | with the stored CRC) into relocation file P, keyed by | (origin, original offset) install one MANIFEST edit: add P, route each input generation number still | in use to P, re-point the routes into the inputs to P, and remove | each input (blob file retirement for a blob file, relocation | file deletion for a relocation file) flatten compaction sets generation number = P in the SSTs it rewrites | route route retirement: route deletion records remove the routes that delete no SST uses ``` Stack map: | Diff | Adds | |---|---| | 1 | Finalize blob metadata outside the DB mutex (prep) | | 2 | Indirect BlobIndex (kIndirectBlob) and exact SST reference metadata <== this diff | | 3 | Relocation file format and stable read path | | 4 | MANIFEST routes and Version ownership | | 5 | Exact reverse reference census | | 6 | Standalone relocation GC (one input file) | | 7 | Pack whole input files; reclaim relocation files | | 8 | Flatten generation numbers; route retirement | | 9 | Remote compaction validation | | 10 | Dynamic options and operations | | 11 | RepairDB rebuilds relocation state; option gate removed | | 12 | db_stress and crash test support | Stack context: terms for later diffs: | Term | Meaning | |---|---| | census | Background count, per generation number, of the SSTs and references that use it, plus live bytes per physical file (diff 5) | | indirect origin | Normal blob file written with the option on; it is the origin of unflattened references | | flatten | Compaction rewrites the generation number to the relocation file that holds the value, so old routes can go. Only flatten makes generation number != origin (diff 8) | | blob file retirement | MANIFEST record (kBlobFileRetirement) that removes a collected blob file from the Version (diff 4) | | route deletion | MANIFEST record that removes a route that no SST uses. Route retirement is the background job that writes these records (diff 8) | Tracking issue: meta#15301 Differential Revision: D124652097
Summary: Pull Request resolved: meta#15540 Why the change is required: The goal of this stack is a blob GC that does not depend on SSTs: it removes garbage from blob files without rewriting any SST. Such a GC needs two things that RocksDB does not have: - A value identity that does not change when the value moves to another file. - A checked list of the blob references that each SST keeps. The list must include older internal versions of a key that the SST still keeps (for example for snapshots), and the blob columns of wide-column entities. This diff adds both. Later diffs build the census (an in-memory reverse index from generation number to the SSTs that use it), GC and repair on top of this list. This diff in one paragraph: With the new column family option enable_blob_indirection on, the blob file writer gives each new value an indirect BlobIndex (type 3, kIndirectBlob). It stores the value's stable identity (origin file, original offset) and its CRCs (an uncompressed value stores only the CRC of the uncompressed value; a compressed value also stores the CRC of its stored bytes). Also with the option on, the block-based table builder collects every blob reference that the SST keeps. It writes them as "exact reference metadata": data blocks, one reference index block, and a 72-byte descriptor table property. In this diff only the table builder writes this metadata, and only tests read it. A temporary option gate (see "option gate" under Terms) keeps the option off outside tests. Separately, wide-column entities with blob columns are now fully checked before their blob indexes are used, also with the option off. Parts of this diff: 1. Indirect BlobIndex (core): types 3 and 4 in db/blob/blob_index.h, and the blob file writer that writes type 3. 2. Exact reference metadata format and decoders (core): db/blob/blob_reference_metadata.{h,cc}. 3. Collector and table builder integration (core): BlobReferenceMetadataCollector, its block cache charge, and BlockBasedTableBuilder. 4. Option enable_blob_indirection and the option gate. Tests need it to turn the feature on. You can skip this part on a first read. 5. Wide-column validation (WideColumnSerialization::ForEachBlobIndex). The table builder uses it to give the blob columns of each entity to the collector. It also changes behavior with the option off (see New behavior). You can skip this part on a first read. 6. Small helpers: canonical varints (util/coding.h) and CacheReservationHandle::Resize(). You can skip these on a first read. How to read this diff: 1. Overview: read "Why" and "This diff in one paragraph" above, and "Terms used in this diff" right below. Then read the top comment of db/blob/blob_reference_metadata.h (terms, layout diagrams, run and entry layout, index layout). 2. Indirect BlobIndex (db/blob/blob_index.h) and canonical varints (util/coding.h GetCanonicalVarint32/64): read the overview comment and the kIndirectBlob / kIndirectFlattenedBlob code. Every later part uses these fields: origin, original offset, generation number, logical CRC and stored CRC. The new formats read varints only in the canonical form. 3. Blob file writer (db/blob/blob_file_builder.cc, blob_log_format.cc, blob_log_writer.cc): see how a new value gets a type-3 index and how its CRC is computed once. 4. Format types and decoders: the structs in db/blob/blob_reference_metadata.h, then in the .cc the constants, GapBase, DecodeEntryBody, DecodeRunEntriesAt, DecodeBlobReferenceIndex and BlobReferenceGroupDecoder. They use only BlobIndex and the canonical varints. Learn the on-disk format here, before you read the writer. 5. Collector and its cache charge: first read the new CacheEntryRole kBlobReferenceMetadataBuildingBuffer (include/rocksdb/cache.h) and CacheReservationHandle::Resize (cache/cache_reservation_manager.h). Then read BlobReferenceMetadataCollector in this order: the class comment, Add, AppendSorted, BuildAndWriteDataBlocks, EncodeIndex, FinishOnce. Then read the unsorted fallback (AppendUnsorted, SortUnsortedGroup, SortAndEncodeGroup). 6. Wide-column change (db/wide/wide_column_serialization.cc, ForEachBlobIndex and WalkV2Columns): it gives every blob index of an entity to a callback after it checks the whole entity. The table builder uses it to feed the collector. UpdateBoundaries, BlobGarbageMeter and the compaction iterator also use it, with the option on or off. 7. Option: enable_blob_indirection (include/rocksdb/advanced_options.h, options/cf_options.cc), and the option checks and gate in db/column_family.cc (ValidateOptions, TEST_SetAllowBlobIndirection), db/db_impl/db_impl_open.cc and db_impl_secondary.cc (the FIFO check for opens that skip ValidateOptions). 8. Table integration: table/block_based/block_based_table_factory.cc (the shared reservation manager), then block_based_table_builder.cc (Rep constructor, Add, WriteBlobReferenceMetadata, EstimatedTailSize). See where the collector is created, fed and finished. block_based_table_reader.cc maps the new block name to a block type. Terms used in this diff: Stack terms: | Term | Meaning | |---|---| | origin | Blob file where the value was first written | | original offset | Value offset in the origin file | | stable identity | (origin, original offset). Never changes, even when the value moves | | generation number | The file number that a reference uses to find its value. A new value has generation number == origin. When GC moves the value, the reference keeps its generation number, and a route sends the read to the relocation file | | indirect BlobIndex | A kIndirectBlob (type 3) or kIndirectFlattenedBlob (type 4) index. It stores the stable identity, so the value can move without an SST rewrite | | unflattened reference | Index with generation number == origin: the value was not moved, or it was moved but no compaction has rewritten its reference yet (see "flatten" in "Stack context: terms for later diffs") | | flattened reference | Index with generation number != origin (type 4). Only flatten (diff 8) writes it; this diff defines the encoding | | route | Maps an old file number (a file that GC has removed from the Version) to the relocation file that now holds its live values. One hop: a target is never itself routed. Stored in the MANIFEST (diff 4) | | relocation file | Block-based table keyed by (origin, original offset) that holds copied values. Each value is the stored bytes plus a 5-byte trailer (kSimpleGen2BlobTrailerSize) | | logical CRC | CRC32C of the uncompressed value (BlobIndex::stable_checksum()) | | stored CRC | CRC32C of the stored bytes plus the compression byte | | physical file | A file that holds value bytes: a normal blob file or a relocation file. A generation number can name a physical file, or it can be only the key of a route | | exact reference metadata | Per-SST sorted list of every blob reference the SST keeps, written when the SST is built. References with the same encoding class, generation number and origin form a group. The list is stored as data blocks, with an index of the groups. An SST that has it is an exact SST | | option gate | Until diff 11, ColumnFamilyData::ValidateOptions returns NotSupported for enable_blob_indirection on a read-write open, CreateColumnFamily and SetOptions, unless a test calls ColumnFamilyData::TEST_SetAllowBlobIndirection(true). It keeps the option off in production while the stack is not complete | Terms of this diff (all defined at the top of db/blob/blob_reference_metadata.h): | Term | Meaning | |---|---| | encoding class (class) | kDirect (a legacy kBlob or kBlobTTL index), unflattened, or flattened | | reference group (group) | All references of one SST with the same (encoding class, generation number, origin) | | run | The part of one group inside one data block | | record bytes | The bytes the value's record takes in the file named by its generation number | | gap | How far a record is from where it would be if the records were back to back. A gap of 0 means back to back | | descriptor | The 72-byte table property that points to the reference index block and holds the totals, the CF ID, the SST file number and the relocation-state fingerprint | | relocation-state fingerprint | 128-bit hash of the routes and relocation files of the Version that an SST was built from (fields route_snapshot_fingerprint_*). Written as 0 in this diff | | legacy | Written the way RocksDB writes it without blob indirection | Previous behavior: - A BlobIndex named only a physical blob file and an offset. A value could not move to another file without rewriting every SST that referenced it. - An SST had no metadata that listed its blob references. The only way to find them was to scan its user keys. - These readers get the blob indexes of a wide-column entity through ForEachBlobFileNumber: UpdateBoundaries, BlobGarbageMeter, the compaction iterator, WideColumnSerialization::RemoveBlobColumns, and the referenced_blob_bytes table property count in BlockBasedTableBuilder::Add. ForEachBlobFileNumber decoded each blob index, but it did not check the rest of the entity framing (name order, section sizes, exact input size), and it could call the callback before it found a later error. New behavior: Separate change (option on or off): - The same readers (UpdateBoundaries, BlobGarbageMeter, the compaction iterator, RemoveBlobColumns, and the referenced_blob_bytes count in BlockBasedTableBuilder::Add) use WideColumnSerialization::ForEachBlobIndex, with the option on or off. With the option on, the table builder also uses it to give the blob columns of each entity to the collector. For an entity with blob columns, it checks the full framing, then decodes and validates every blob index before any callback runs. So an entity with a malformed framing now fails the flush or compaction, even with the option off. The referenced_blob_bytes count itself does not fail the build on a malformed entity. It removes that property from the SST, as in base. Option and option gate: - This diff adds a new column family option, `enable_blob_indirection`. It is off by default, and SetOptions can change it. - The option gate (see Terms) keeps the option off outside tests until the stack is complete. ColumnFamilyData::ValidateOptions rejects the option with NotSupported on a read-write open, CreateColumnFamily and SetOptions. Read-only opens (DBImplReadOnly and CompactedDBImpl) and secondary opens (including DB::OpenAndCompact) do not run ValidateOptions; they check only the FIFO rule (see Implementation). Only tests can turn the option on, through the internal ColumnFamilyData::TEST_SetAllowBlobIndirection(). DBTestBase sets it. - When the option is on, ValidateOptions also requires enable_blob_files, the block-based table format, no blob direct write, and a compaction style other than FIFO. Blob file writer (option on): - The blob file writer encodes each new value with a stable identity (origin file, original offset) and two checksums: - the logical CRC: the CRC32C of the uncompressed value. - the stored CRC: the CRC32C of the exact stored (maybe compressed) bytes, extended with the compression byte. - For an uncompressed value the stored CRC is not written, because the reader can compute it from the logical CRC. - The checksums have no version field of their own. The BlobIndex type byte gives the format version: the two new types, 3 (kIndirectBlob) and 4 (kIndirectFlattenedBlob, see Implementation), are BlobIndex::kIndirectBlobFormatVersion (1). A later format uses new type values. - The read path does not check these checksums yet. The check starts in diff 3. Exact reference metadata (option on): 1. What: a sorted list of every blob reference that the SST keeps, in plain blob values and in the blob columns of wide-column entities. 2. Where: data blocks (target 16 KB each before compression; small blocks waste less I/O, because GC reads a whole block to get a few groups) and one reference index block, written after the SST data blocks and before the SST tail, plus a 72-byte descriptor table property. 3. Who reads it: only tests in this diff. The census, GC and RepairDB read it in later diffs. 4. Details: the top comment of db/blob/blob_reference_metadata.h has the full format with diagrams, the record-bytes rules and the gap rule. - Not collected: SstFileWriter output (it has no DB file number) and any output whose CF is unknown (it has no CF ID). The descriptor needs both values, so these outputs stay in the legacy form. Checks on the metadata: - A repeated or overlapping reference in one group fails the SST build with Corruption. Readers check every count, order, size and total against the index and the descriptor, so corrupt metadata returns Corruption. The writer never makes a data block larger than kTargetDataBlockSize (16 KB), and readers accept up to kMaxDataBlockSize (64 KB). Details: "Readers" and "Integrity" at the top of db/blob/blob_reference_metadata.h, and BlobReferenceGroupDecoder. Collection and memory: - Each reference is encoded when it is added, in the on-disk entry encoding, into its group's chain of 512-byte chunks. Only a group that takes the unsorted fallback is encoded a second time, in Finish(). Inside a group, references normally arrive in rising offset order, so no reference is sorted; at the end only the groups are sorted. A group that gets a lower offset takes the unsorted fallback. The BlobReferenceMetadataCollector class comment explains the chunks, the chunk size and the fallback. - The collector keeps every reference of the SST in memory until the SST is finished, and charges this memory to the block cache under a new CacheEntryRole, kBlobReferenceMetadataBuildingBuffer. The measured cost is about 8.3 to 9 bytes per reference for uncompressed values and 12.8 to 13.5 bytes for compressed values. "Memory usage" and "Memory charge" in the BlobReferenceMetadataCollector class comment give the measurement setup and the charge rule. - There is no fixed limit on references. The collector fails with MemoryLimit above 2^20 groups or 2^20 data blocks in one SST; these are persisted format limits. - There is no charge when the table has no block cache or when charging for this role is disabled. Each block-based table factory holds one cache reservation manager for this role, shared by its builders and the SST summaries. If the charge fails, the SST build fails, so the flush or compaction fails with a background error. The buffers and their charge are released when the SST is finished or abandoned. Implementation: - BlobIndex adds two indirect encodings. The legacy encodings do not change. - Type 3, kIndirectBlob (unflattened, generation number == origin), stores the origin, the original offset, the stored size, the compression type, the logical CRC and, for compressed values, the stored CRC. It does not store the generation number. This saves the size of one varint file number per reference (about 3 to 4 bytes for typical file numbers; an estimate, not measured). - Type 4, kIndirectFlattenedBlob (flattened), stores the generation number first, then the same fields. Its generation number must be above the origin. The decoder rejects type 4 with generation number <= origin, so each reference has one encoding. - The encoders take the fields as one BlobIndex::IndirectBlobFields, set by field name, so values of the same type (file number, offset and size; the two CRCs) cannot be swapped. EncodeIndirectBlob() takes the generation number as a separate argument; EncodeUnflattenedIndirectBlob() uses the origin, so the common path cannot leave the generation number unset. - The new files blob_reference_metadata.{h,cc} define the exact reference metadata. The files also contain: - BlobReferenceMetadataCollector, the collector described above. It takes the SST identity (CF ID, SST file number and the relocation-state fingerprint) as one BlobReferenceSstIdentity, set by field name. - BlobReferenceGroupKey, the key of a group: (encoding class, generation number, origin). It has the order, the hash and the validity check that the collector map, the sort, the decoder and FindBlobReferenceGroup() use. BlobReference and BlobReferenceGroup keep their flat fields and return the key through group_key(). - The index codec and the descriptor table property. - BlobReferenceGroupDecoder, the only decoder of reference runs. It reads a group one data block at a time and holds every corruption check of a group walk. Readers that visit one group (found by a binary search of the index) use it, and so do readers that visit every block in order. - Callers in this diff: only the table builder runs the collector. The decoders (DecodeBlobReferenceIndex, FindBlobReferenceGroup, BlobReferenceGroupDecoder, ForEachBlobReference*) are used only by tests. kMaxDecodedBlobReferenceBytes is used only in compile-time checks of the count limits. The census (diff 5) and GC (diffs 6 and 7) are their first production users. The census is also the first user of BlobReferenceSstSummary (see the last Implementation bullet). - Descriptor: the CF ID and the SST file number are the repair fields. They let RepairDB tell which column family and SST the list belongs to without a MANIFEST. The decoder rejects a descriptor whose repair fields are not set (unknown CF ID or SST file number 0). The descriptor also has a 128-bit relocation-state fingerprint (the route_snapshot_fingerprint fields). Remote compaction uses it in diff 9 to check that an SST built elsewhere used the same routes as the job that asked for it. In this diff the table builder always writes 0, and DecodeFrom() does not check it. The field is in the format now, so the descriptor keeps one format version (1) through the stack. While the option gate is on, no data in this format exists outside tests. - The collector validates each BlobIndex once, when it is added. Its Finish() writes the data blocks and then the reference index block through one block writer, and fills the descriptor, including the index handle. Destroying the collector frees its memory and then its charge, also when the build is abandoned. - BlockBasedTableBuilder creates the collector with the CF ID and SST file number from TableBuilderOptions. It passes the size of each entry's user key to the collector, so the collector can compute the exact size of each blob record. It adds the collector's EstimatedEncodedSize() to its tail size estimate: the blocks are written before the tail, but output file size planning uses this estimate. The reference blocks are written with BlockType::kUserDefinedIndex, which keeps their contents as opaque bytes. Readers read them without the block cache, so prepopulate_block_cache does not warm them into the block cache (WriteMaybeCompressedBlock() with warm_cache false). - With the option on, the blob file builder computes the CRC32C of each stored value once. The blob log record CRC combines it with the key CRC (crc32c::Crc32cCombine). The stored CRC extends it with the compression byte. For an uncompressed value it is also the logical CRC. The bytes written do not change. - WideColumnSerialization::ForEachBlobIndex replaces ForEachBlobFileNumber. It reads the entity header and column types once. For a V1 entity (no column types and no blob columns), it returns right after the header. For a V2 entity without blob columns, it returns right after the type scan. It shares one V2 column walker (WalkV2Columns) with Deserialize(), so the entity is parsed in one pass. - Read-only opens (DBImplReadOnly, CompactedDBImpl) and secondary opens skip ValidateOptions, so they check the compaction style against the option once per column family: in DBImpl::Recover() for read-only opens, and at the start of OpenAsSecondaryImpl() for secondary opens. DB::OpenAndCompact opens through OpenAsSecondaryImpl(), so it gets the same check. - CacheReservationHandle gets Resize(), so each collector or summary can change its own part of the shared charge. - util/coding.h adds GetCanonicalVarint32/64, which reject overlong varint encodings. The new formats use them, so each value has one encoding. - BlobReferenceSstSummary is the in-memory list of an SST's groups and totals that Finish() can return. It has no caller yet: the table builder passes a null summary output. It is checked and does not change after Finish(), so the finished builder, the SST metadata and the census (its first user, in diff 5) can share it. It is in this diff because Finish() hands the group buffer and its cache charge over to the summary. Adding the summary later would mean rewriting the charge code of Finish() in diff 5, so the charge code is written and tested once, here. Safety boundaries and unsupported cases: - Public API: this diff adds the option `enable_blob_indirection` with the C API functions `rocksdb_options_set_enable_blob_indirection` and `rocksdb_options_get_enable_blob_indirection`. It also inserts a new CacheEntryRole value before kMisc, which changes the numeric value of kMisc and kNumCacheEntryRoles. - With the option off, new blob values and SSTs are written in the legacy form. Legacy blob indexes stay readable. - One behavior change applies with the option off too: UpdateBoundaries, BlobGarbageMeter and the compaction iterator fully validate wide-column entities with blob columns and their blob indexes (see "Separate change" above). - Exact reference metadata has one format (version 1) with a fixed 72-byte descriptor. The decoders reject any other version as Corruption. - The metadata blocks are protected by the table's block checksum only. With checksum = kNoChecksum, corrupt metadata is detected only when it breaks the format or order checks. - The index decoder checks each decoded count against the remaining input size before it reserves memory. So a small corrupt index cannot make it allocate a large vector. - Inlined-TTL and same-file blob indexes are not collected and cannot be relocated. Encoding, count and checksum errors return an error; a bad reference is never skipped. - Option gate (see Terms): outside tests no DB writes data in the new format. The gate checks only the option value, never persisted state. Diff 11 removes it, once data written with the option stays readable through the end of the stack and the safety checks for turning the option on are in place. - Downgrading the binary after data was written with the option on is not supported. Validation and known limitations: - Unit tests cover the BlobIndex encodings and their rejection of bad input, the metadata layout and every corruption check, the collector (sorted path, unsorted fallback, memory charge), the flush integration, the option checks and option gate, wide-column validation, and canonical varints. See the Test Plan. - The collection buffer is not spilled to disk. One SST with very many blob references needs a matching amount of charged memory: the per-reference amount given under "Collection and memory", plus up to one partly used 512-byte chunk per group. While a fallback group is sorted, it also needs 16 bytes of sort key per reference and a second copy of its entries. For a test group of 200K references with every other one out of order, the test comment estimates about 44 bytes per reference, and the test checks that the peak charge is at most 56 bytes per reference. - Flush still writes a single output file. Until diff 7, one compaction output collects every reference it gets, charged to the block cache. Diff 7 cuts compaction outputs (including L0) at a reference cap and keeps each user key's history in one output. With the option gate, a large output is reachable only through the test hook. Stack context (same in diffs 2-11, except that diffs 8-11 add the diff 8 route rule to the install line; "<== this diff" marks this diff's part): Goal: remove garbage from blob files without rewriting any SST. A value gets a stable identity when it is first written. GC copies the live values of whole files into a relocation file and installs routes in the MANIFEST. Readers follow the routes. SSTs are not changed. How a read finds a value (P is the relocation file that holds the value now): ``` SST entry indirect BlobIndex { generation number, origin, original offset, size, CRCs } | v Version file table: generation number -> live file, or route old file -> P | | live file route found | | v v blob file: read at original offset relocation file P (always one hop): relocation file: look up key look up key (origin, original offset) (origin, original offset) ``` Lifecycle of one value (in this example, F is the blob file where the value is first written, and P is the relocation file that GC writes): ``` write new value goes to blob file F; index: origin = F, generation number = F | census per generation number: the SSTs and references that use it; | per physical file: live bytes, ranked by garbage ratio GC job copy the live values of whole input files (stored bytes, checked | with the stored CRC) into relocation file P, keyed by | (origin, original offset) install one MANIFEST edit: add P, route each input generation number still | in use to P, re-point the routes into the inputs to P, and remove | each input (blob file retirement for a blob file, relocation | file deletion for a relocation file) flatten compaction sets generation number = P in the SSTs it rewrites | route route retirement: route deletion records remove the routes that delete no SST uses ``` Stack map: | Diff | Adds | |---|---| | 1 | Finalize blob metadata outside the DB mutex (prep) | | 2 | Indirect BlobIndex (kIndirectBlob) and exact SST reference metadata <== this diff | | 3 | Relocation file format and stable read path | | 4 | MANIFEST routes and Version ownership | | 5 | Exact reverse reference census | | 6 | Standalone relocation GC (one input file) | | 7 | Pack whole input files; reclaim relocation files | | 8 | Flatten generation numbers; route retirement | | 9 | Remote compaction validation | | 10 | Dynamic options and operations | | 11 | RepairDB rebuilds relocation state; option gate removed | | 12 | db_stress and crash test support | Stack context: terms for later diffs: | Term | Meaning | |---|---| | census | Background count, per generation number, of the SSTs and references that use it, plus live bytes per physical file (diff 5) | | indirect origin | Normal blob file written with the option on; it is the origin of unflattened references | | flatten | Compaction rewrites the generation number to the relocation file that holds the value, so old routes can go. Only flatten makes generation number != origin (diff 8) | | blob file retirement | MANIFEST record (kBlobFileRetirement) that removes a collected blob file from the Version (diff 4) | | route deletion | MANIFEST record that removes a route that no SST uses. Route retirement is the background job that writes these records (diff 8) | Tracking issue: meta#15301 Differential Revision: D124652097
Summary: Pull Request resolved: meta#15540 Why the change is required: The goal of this stack is a blob GC that does not depend on SSTs: it removes garbage from blob files without rewriting any SST. Such a GC needs two things that RocksDB does not have: - A value identity that does not change when the value moves to another file. - A checked list of the blob references that each SST keeps. The list must include older internal versions of a key that the SST still keeps (for example for snapshots), and the blob columns of wide-column entities. This diff adds both. Later diffs build the census (an in-memory reverse index from generation number to the SSTs that use it), GC and repair on top of this list. This diff in one paragraph: With the new column family option enable_blob_indirection on, the blob file writer gives each new value an indirect BlobIndex (type 3, kIndirectBlob). It stores the value's stable identity (origin file, original offset) and its CRCs (an uncompressed value stores only the CRC of the uncompressed value; a compressed value also stores the CRC of its stored bytes). Also with the option on, the block-based table builder collects every blob reference that the SST keeps. It writes them as "exact reference metadata": data blocks, one reference index block, and a 72-byte descriptor table property. In this diff only the table builder writes this metadata, and only tests read it. A temporary option gate (see "option gate" under Terms) keeps the option off outside tests. Separately, wide-column entities with blob columns are now fully checked before their blob indexes are used, also with the option off. Parts of this diff: 1. Indirect BlobIndex (core): types 3 and 4 in db/blob/blob_index.h, and the blob file writer that writes type 3. 2. Exact reference metadata format and decoders (core): db/blob/blob_reference_metadata.{h,cc}. 3. Collector and table builder integration (core): BlobReferenceMetadataCollector, its block cache charge, and BlockBasedTableBuilder. 4. Option enable_blob_indirection and the option gate. Tests need it to turn the feature on. You can skip this part on a first read. 5. Wide-column validation (WideColumnSerialization::ForEachBlobIndex). The table builder uses it to give the blob columns of each entity to the collector. It also changes behavior with the option off (see New behavior). You can skip this part on a first read. 6. Small helpers: canonical varints (util/coding.h) and CacheReservationHandle::Resize(). You can skip these on a first read. How to read this diff: 1. Overview: read "Why" and "This diff in one paragraph" above, and "Terms used in this diff" right below. Then read the top comment of db/blob/blob_reference_metadata.h (terms, layout diagrams, run and entry layout, index layout). 2. Indirect BlobIndex (db/blob/blob_index.h) and canonical varints (util/coding.h GetCanonicalVarint32/64): read the overview comment and the kIndirectBlob / kIndirectFlattenedBlob code. Every later part uses these fields: origin, original offset, generation number, logical CRC and stored CRC. The new formats read varints only in the canonical form. 3. Blob file writer (db/blob/blob_file_builder.cc, blob_log_format.cc, blob_log_writer.cc): see how a new value gets a type-3 index and how its CRC is computed once. 4. Format types and decoders: the structs in db/blob/blob_reference_metadata.h, then in the .cc the constants, GapBase, DecodeEntryBody, DecodeRunEntriesAt, DecodeBlobReferenceIndex and BlobReferenceGroupDecoder. They use only BlobIndex and the canonical varints. Learn the on-disk format here, before you read the writer. 5. Collector and its cache charge: first read the new CacheEntryRole kBlobReferenceMetadataBuildingBuffer (include/rocksdb/cache.h) and CacheReservationHandle::Resize (cache/cache_reservation_manager.h). Then read BlobReferenceMetadataCollector in this order: the class comment, Add, AppendSorted, BuildAndWriteDataBlocks, EncodeIndex, FinishOnce. Then read the unsorted fallback (AppendUnsorted, SortUnsortedGroup, SortAndEncodeGroup). 6. Wide-column change (db/wide/wide_column_serialization.cc, ForEachBlobIndex and WalkV2Columns): it gives every blob index of an entity to a callback after it checks the whole entity. The table builder uses it to feed the collector. UpdateBoundaries, BlobGarbageMeter and the compaction iterator also use it, with the option on or off. 7. Option: enable_blob_indirection (include/rocksdb/advanced_options.h, options/cf_options.cc), and the option checks and gate in db/column_family.cc (ValidateOptions, TEST_SetAllowBlobIndirection), db/db_impl/db_impl_open.cc and db_impl_secondary.cc (the FIFO check for opens that skip ValidateOptions). 8. Table integration: table/block_based/block_based_table_factory.cc (the shared reservation manager), then block_based_table_builder.cc (Rep constructor, Add, WriteBlobReferenceMetadata, EstimatedTailSize). See where the collector is created, fed and finished. block_based_table_reader.cc maps the new block name to a block type. Terms used in this diff: Stack terms: | Term | Meaning | |---|---| | origin | Blob file where the value was first written | | original offset | Value offset in the origin file | | stable identity | (origin, original offset). Never changes, even when the value moves | | generation number | The file number that a reference uses to find its value. A new value has generation number == origin. When GC moves the value, the reference keeps its generation number, and a route sends the read to the relocation file | | indirect BlobIndex | A kIndirectBlob (type 3) or kIndirectFlattenedBlob (type 4) index. It stores the stable identity, so the value can move without an SST rewrite | | unflattened reference | Index with generation number == origin: the value was not moved, or it was moved but no compaction has rewritten its reference yet (see "flatten" in "Stack context: terms for later diffs") | | flattened reference | Index with generation number != origin (type 4). Only flatten (diff 8) writes it; this diff defines the encoding | | route | Maps an old file number (a file that GC has removed from the Version) to the relocation file that now holds its live values. One hop: a target is never itself routed. Stored in the MANIFEST (diff 4) | | relocation file | Block-based table keyed by (origin, original offset) that holds copied values. Each value is the stored bytes plus a 5-byte trailer (kSimpleGen2BlobTrailerSize) | | logical CRC | CRC32C of the uncompressed value (BlobIndex::stable_checksum()) | | stored CRC | CRC32C of the stored bytes plus the compression byte | | physical file | A file that holds value bytes: a normal blob file or a relocation file. A generation number can name a physical file, or it can be only the key of a route | | exact reference metadata | Per-SST sorted list of every blob reference the SST keeps, written when the SST is built. References with the same encoding class, generation number and origin form a group. The list is stored as data blocks, with an index of the groups. An SST that has it is an exact SST | | option gate | Until diff 11, ColumnFamilyData::ValidateOptions returns NotSupported for enable_blob_indirection on a read-write open, CreateColumnFamily and SetOptions, unless a test calls ColumnFamilyData::TEST_SetAllowBlobIndirection(true). It keeps the option off in production while the stack is not complete | Terms of this diff (all defined at the top of db/blob/blob_reference_metadata.h): | Term | Meaning | |---|---| | encoding class (class) | kDirect (a legacy kBlob or kBlobTTL index), unflattened, or flattened | | reference group (group) | All references of one SST with the same (encoding class, generation number, origin) | | run | The part of one group inside one data block | | record bytes | The bytes the value's record takes in the file named by its generation number | | gap | How far a record is from where it would be if the records were back to back. A gap of 0 means back to back | | descriptor | The 72-byte table property that points to the reference index block and holds the totals, the CF ID, the SST file number and the relocation-state fingerprint | | relocation-state fingerprint | 128-bit hash of the routes and relocation files of the Version that an SST was built from (fields route_snapshot_fingerprint_*). Written as 0 in this diff | | legacy | Written the way RocksDB writes it without blob indirection | Previous behavior: - A BlobIndex named only a physical blob file and an offset. A value could not move to another file without rewriting every SST that referenced it. - An SST had no metadata that listed its blob references. The only way to find them was to scan its user keys. - These readers get the blob indexes of a wide-column entity through ForEachBlobFileNumber: UpdateBoundaries, BlobGarbageMeter, the compaction iterator, WideColumnSerialization::RemoveBlobColumns, and the referenced_blob_bytes table property count in BlockBasedTableBuilder::Add. ForEachBlobFileNumber decoded each blob index, but it did not check the rest of the entity framing (name order, section sizes, exact input size), and it could call the callback before it found a later error. New behavior: Separate change (option on or off): - The same readers (UpdateBoundaries, BlobGarbageMeter, the compaction iterator, RemoveBlobColumns, and the referenced_blob_bytes count in BlockBasedTableBuilder::Add) use WideColumnSerialization::ForEachBlobIndex, with the option on or off. With the option on, the table builder also uses it to give the blob columns of each entity to the collector. For an entity with blob columns, it checks the full framing, then decodes and validates every blob index before any callback runs. So an entity with a malformed framing now fails the flush or compaction, even with the option off. The referenced_blob_bytes count itself does not fail the build on a malformed entity. It removes that property from the SST, as in base. Option and option gate: - This diff adds a new column family option, `enable_blob_indirection`. It is off by default, and SetOptions can change it. - The option gate (see Terms) keeps the option off outside tests until the stack is complete. ColumnFamilyData::ValidateOptions rejects the option with NotSupported on a read-write open, CreateColumnFamily and SetOptions. Read-only opens (DBImplReadOnly and CompactedDBImpl) and secondary opens (including DB::OpenAndCompact) do not run ValidateOptions; they check only the FIFO rule (see Implementation). Only tests can turn the option on, through the internal ColumnFamilyData::TEST_SetAllowBlobIndirection(). DBTestBase sets it. - When the option is on, ValidateOptions also requires enable_blob_files, the block-based table format, no blob direct write, and a compaction style other than FIFO. Blob file writer (option on): - The blob file writer encodes each new value with a stable identity (origin file, original offset) and two checksums: - the logical CRC: the CRC32C of the uncompressed value. - the stored CRC: the CRC32C of the exact stored (maybe compressed) bytes, extended with the compression byte. - For an uncompressed value the stored CRC is not written, because the reader can compute it from the logical CRC. - The checksums have no version field of their own. The BlobIndex type byte gives the format version: the two new types, 3 (kIndirectBlob) and 4 (kIndirectFlattenedBlob, see Implementation), are BlobIndex::kIndirectBlobFormatVersion (1). A later format uses new type values. - The read path does not check these checksums yet. The check starts in diff 3. Exact reference metadata (option on): 1. What: a sorted list of every blob reference that the SST keeps, in plain blob values and in the blob columns of wide-column entities. 2. Where: data blocks (target 16 KB each before compression; small blocks waste less I/O, because GC reads a whole block to get a few groups) and one reference index block, written after the SST data blocks and before the SST tail, plus a 72-byte descriptor table property. 3. Who reads it: only tests in this diff. The census, GC and RepairDB read it in later diffs. 4. Details: the top comment of db/blob/blob_reference_metadata.h has the full format with diagrams, the record-bytes rules and the gap rule. - Not collected: SstFileWriter output (it has no DB file number) and any output whose CF is unknown (it has no CF ID). The descriptor needs both values, so these outputs stay in the legacy form. Checks on the metadata: - A repeated or overlapping reference in one group fails the SST build with Corruption. Readers check every count, order, size and total against the index and the descriptor, so corrupt metadata returns Corruption. The writer never makes a data block larger than kTargetDataBlockSize (16 KB), and readers accept up to kMaxDataBlockSize (64 KB). Details: "Readers" and "Integrity" at the top of db/blob/blob_reference_metadata.h, and BlobReferenceGroupDecoder. Collection and memory: - Each reference is encoded when it is added, in the on-disk entry encoding, into its group's chain of 512-byte chunks. Only a group that takes the unsorted fallback is encoded a second time, in Finish(). Inside a group, references normally arrive in rising offset order, so no reference is sorted; at the end only the groups are sorted. A group that gets a lower offset takes the unsorted fallback. The BlobReferenceMetadataCollector class comment explains the chunks, the chunk size and the fallback. - The collector keeps every reference of the SST in memory until the SST is finished, and charges this memory to the block cache under a new CacheEntryRole, kBlobReferenceMetadataBuildingBuffer. The measured cost is about 8.3 to 9 bytes per reference for uncompressed values and 12.8 to 13.5 bytes for compressed values. "Memory usage" and "Memory charge" in the BlobReferenceMetadataCollector class comment give the measurement setup and the charge rule. - There is no fixed limit on references. The collector fails with MemoryLimit above 2^20 groups or 2^20 data blocks in one SST; these are persisted format limits. - There is no charge when the table has no block cache or when charging for this role is disabled. Each block-based table factory holds one cache reservation manager for this role, shared by its builders and the SST summaries. If the charge fails, the SST build fails, so the flush or compaction fails with a background error. The buffers and their charge are released when the SST is finished or abandoned. Implementation: - BlobIndex adds two indirect encodings. The legacy encodings do not change. - Type 3, kIndirectBlob (unflattened, generation number == origin), stores the origin, the original offset, the stored size, the compression type, the logical CRC and, for compressed values, the stored CRC. It does not store the generation number. This saves the size of one varint file number per reference (about 3 to 4 bytes for typical file numbers; an estimate, not measured). - Type 4, kIndirectFlattenedBlob (flattened), stores the generation number first, then the same fields. Its generation number must be above the origin. The decoder rejects type 4 with generation number <= origin, so each reference has one encoding. - The encoders take the fields as one BlobIndex::IndirectBlobFields, set by field name, so values of the same type (file number, offset and size; the two CRCs) cannot be swapped. EncodeIndirectBlob() takes the generation number as a separate argument; EncodeUnflattenedIndirectBlob() uses the origin, so the common path cannot leave the generation number unset. - The new files blob_reference_metadata.{h,cc} define the exact reference metadata. The files also contain: - BlobReferenceMetadataCollector, the collector described above. It takes the SST identity (CF ID, SST file number and the relocation-state fingerprint) as one BlobReferenceSstIdentity, set by field name. - BlobReferenceGroupKey, the key of a group: (encoding class, generation number, origin). It has the order, the hash and the validity check that the collector map, the sort, the decoder and FindBlobReferenceGroup() use. BlobReference and BlobReferenceGroup keep their flat fields and return the key through group_key(). - The index codec and the descriptor table property. - BlobReferenceGroupDecoder, the only decoder of reference runs. It reads a group one data block at a time and holds every corruption check of a group walk. Readers that visit one group (found by a binary search of the index) use it, and so do readers that visit every block in order. - Callers in this diff: only the table builder runs the collector. The decoders (DecodeBlobReferenceIndex, FindBlobReferenceGroup, BlobReferenceGroupDecoder, ForEachBlobReference*) are used only by tests. kMaxDecodedBlobReferenceBytes is used only in compile-time checks of the count limits. The census (diff 5) and GC (diffs 6 and 7) are their first production users. The census is also the first user of BlobReferenceSstSummary (see the last Implementation bullet). - Descriptor: the CF ID and the SST file number are the repair fields. They let RepairDB tell which column family and SST the list belongs to without a MANIFEST. The decoder rejects a descriptor whose repair fields are not set (unknown CF ID or SST file number 0). The descriptor also has a 128-bit relocation-state fingerprint (the route_snapshot_fingerprint fields). Remote compaction uses it in diff 9 to check that an SST built elsewhere used the same routes as the job that asked for it. In this diff the table builder always writes 0, and DecodeFrom() does not check it. The field is in the format now, so the descriptor keeps one format version (1) through the stack. While the option gate is on, no data in this format exists outside tests. - The collector validates each BlobIndex once, when it is added. Its Finish() writes the data blocks and then the reference index block through one block writer, and fills the descriptor, including the index handle. Destroying the collector frees its memory and then its charge, also when the build is abandoned. - BlockBasedTableBuilder creates the collector with the CF ID and SST file number from TableBuilderOptions. It passes the size of each entry's user key to the collector, so the collector can compute the exact size of each blob record. It adds the collector's EstimatedEncodedSize() to its tail size estimate: the blocks are written before the tail, but output file size planning uses this estimate. The reference blocks are written with BlockType::kUserDefinedIndex, which keeps their contents as opaque bytes. Readers read them without the block cache, so prepopulate_block_cache does not warm them into the block cache (WriteMaybeCompressedBlock() with warm_cache false). - With the option on, the blob file builder computes the CRC32C of each stored value once. The blob log record CRC combines it with the key CRC (crc32c::Crc32cCombine). The stored CRC extends it with the compression byte. For an uncompressed value it is also the logical CRC. The bytes written do not change. - WideColumnSerialization::ForEachBlobIndex replaces ForEachBlobFileNumber. It reads the entity header and column types once. For a V1 entity (no column types and no blob columns), it returns right after the header. For a V2 entity without blob columns, it returns right after the type scan. It shares one V2 column walker (WalkV2Columns) with Deserialize(), so the entity is parsed in one pass. - Read-only opens (DBImplReadOnly, CompactedDBImpl) and secondary opens skip ValidateOptions, so they check the compaction style against the option once per column family: in DBImpl::Recover() for read-only opens, and at the start of OpenAsSecondaryImpl() for secondary opens. DB::OpenAndCompact opens through OpenAsSecondaryImpl(), so it gets the same check. - CacheReservationHandle gets Resize(), so each collector or summary can change its own part of the shared charge. - util/coding.h adds GetCanonicalVarint32/64, which reject overlong varint encodings. The new formats use them, so each value has one encoding. - BlobReferenceSstSummary is the in-memory list of an SST's groups and totals that Finish() can return. It has no caller yet: the table builder passes a null summary output. It is checked and does not change after Finish(), so the finished builder, the SST metadata and the census (its first user, in diff 5) can share it. It is in this diff because Finish() hands the group buffer and its cache charge over to the summary. Adding the summary later would mean rewriting the charge code of Finish() in diff 5, so the charge code is written and tested once, here. Safety boundaries and unsupported cases: - Public API: this diff adds the option `enable_blob_indirection` with the C API functions `rocksdb_options_set_enable_blob_indirection` and `rocksdb_options_get_enable_blob_indirection`. It also inserts a new CacheEntryRole value before kMisc, which changes the numeric value of kMisc and kNumCacheEntryRoles. - With the option off, new blob values and SSTs are written in the legacy form. Legacy blob indexes stay readable. - One behavior change applies with the option off too: UpdateBoundaries, BlobGarbageMeter and the compaction iterator fully validate wide-column entities with blob columns and their blob indexes (see "Separate change" above). - Exact reference metadata has one format (version 1) with a fixed 72-byte descriptor. The decoders reject any other version as Corruption. - The metadata blocks are protected by the table's block checksum only. With checksum = kNoChecksum, corrupt metadata is detected only when it breaks the format or order checks. - The index decoder checks each decoded count against the remaining input size before it reserves memory. So a small corrupt index cannot make it allocate a large vector. - Inlined-TTL and same-file blob indexes are not collected and cannot be relocated. Encoding, count and checksum errors return an error; a bad reference is never skipped. - Option gate (see Terms): outside tests no DB writes data in the new format. The gate checks only the option value, never persisted state. Diff 11 removes it, once data written with the option stays readable through the end of the stack and the safety checks for turning the option on are in place. - Downgrading the binary after data was written with the option on is not supported. Validation and known limitations: - Unit tests cover the BlobIndex encodings and their rejection of bad input, the metadata layout and every corruption check, the collector (sorted path, unsorted fallback, memory charge), the flush integration, the option checks and option gate, wide-column validation, and canonical varints. See the Test Plan. - The collection buffer is not spilled to disk. One SST with very many blob references needs a matching amount of charged memory: the per-reference amount given under "Collection and memory", plus up to one partly used 512-byte chunk per group. While a fallback group is sorted, it also needs 16 bytes of sort key per reference and a second copy of its entries. For a test group of 200K references with every other one out of order, the test comment estimates about 44 bytes per reference, and the test checks that the peak charge is at most 56 bytes per reference. - Flush still writes a single output file. Until diff 7, one compaction output collects every reference it gets, charged to the block cache. Diff 7 cuts compaction outputs (including L0) at a reference cap and keeps each user key's history in one output. With the option gate, a large output is reachable only through the test hook. Stack context (same in diffs 2-11, except that diffs 8-11 add the diff 8 route rule to the install line; "<== this diff" marks this diff's part): Goal: remove garbage from blob files without rewriting any SST. A value gets a stable identity when it is first written. GC copies the live values of whole files into a relocation file and installs routes in the MANIFEST. Readers follow the routes. SSTs are not changed. How a read finds a value (P is the relocation file that holds the value now): ``` SST entry indirect BlobIndex { generation number, origin, original offset, size, CRCs } | v Version file table: generation number -> live file, or route old file -> P | | live file route found | | v v blob file: read at original offset relocation file P (always one hop): relocation file: look up key look up key (origin, original offset) (origin, original offset) ``` Lifecycle of one value (in this example, F is the blob file where the value is first written, and P is the relocation file that GC writes): ``` write new value goes to blob file F; index: origin = F, generation number = F | census per generation number: the SSTs and references that use it; | per physical file: live bytes, ranked by garbage ratio GC job copy the live values of whole input files (stored bytes, checked | with the stored CRC) into relocation file P, keyed by | (origin, original offset) install one MANIFEST edit: add P, route each input generation number still | in use to P, re-point the routes into the inputs to P, and remove | each input (blob file retirement for a blob file, relocation | file deletion for a relocation file) flatten compaction sets generation number = P in the SSTs it rewrites | route route retirement: route deletion records remove the routes that delete no SST uses ``` Stack map: | Diff | Adds | |---|---| | 1 | Finalize blob metadata outside the DB mutex (prep) | | 2 | Indirect BlobIndex (kIndirectBlob) and exact SST reference metadata <== this diff | | 3 | Relocation file format and stable read path | | 4 | MANIFEST routes and Version ownership | | 5 | Exact reverse reference census | | 6 | Standalone relocation GC (one input file) | | 7 | Pack whole input files; reclaim relocation files | | 8 | Flatten generation numbers; route retirement | | 9 | Remote compaction validation | | 10 | Dynamic options and operations | | 11 | RepairDB rebuilds relocation state; option gate removed | | 12 | db_stress and crash test support | Stack context: terms for later diffs: | Term | Meaning | |---|---| | census | Background count, per generation number, of the SSTs and references that use it, plus live bytes per physical file (diff 5) | | indirect origin | Normal blob file written with the option on; it is the origin of unflattened references | | flatten | Compaction rewrites the generation number to the relocation file that holds the value, so old routes can go. Only flatten makes generation number != origin (diff 8) | | blob file retirement | MANIFEST record (kBlobFileRetirement) that removes a collected blob file from the Version (diff 4) | | route deletion | MANIFEST record that removes a route that no SST uses. Route retirement is the background job that writes these records (diff 8) | Tracking issue: meta#15301 Differential Revision: D124652097
80c866c to
64afe7c
Compare
Summary: Pull Request resolved: meta#15540 Why the change is required: The goal of this stack is a blob GC that does not depend on SSTs: it removes garbage from blob files without rewriting any SST. Such a GC needs two things that RocksDB does not have: - A value identity that does not change when the value moves to another file. - A checked list of the blob references that each SST keeps. The list must include older internal versions of a key that the SST still keeps (for example for snapshots), and the blob columns of wide-column entities. This diff adds both. Later diffs build the census (an in-memory reverse index from generation number to the SSTs that use it), GC and repair on top of this list. This diff in one paragraph: With the new column family option enable_blob_indirection on, the blob file writer gives each new value an indirect BlobIndex (type 3, kIndirectBlob). It stores the value's stable identity (origin file, original offset) and its CRCs (an uncompressed value stores only the CRC of the uncompressed value; a compressed value also stores the CRC of its stored bytes). Also with the option on, the block-based table builder collects every blob reference that the SST keeps. It writes them as "exact reference metadata": data blocks, one reference index block, and a 72-byte descriptor table property. In this diff only the table builder writes this metadata, and only tests read it. A temporary option gate (see "option gate" under Terms) keeps the option off outside tests. Separately, wide-column entities with blob columns are now fully checked before their blob indexes are used, also with the option off. Parts of this diff: 1. Indirect BlobIndex (core): types 3 and 4 in db/blob/blob_index.h, and the blob file writer that writes type 3. 2. Exact reference metadata format and decoders (core): db/blob/blob_reference_metadata.{h,cc}. 3. Collector and table builder integration (core): BlobReferenceMetadataCollector, its block cache charge, and BlockBasedTableBuilder. 4. Option enable_blob_indirection and the option gate. Tests need it to turn the feature on. You can skip this part on a first read. 5. Wide-column validation (WideColumnSerialization::ForEachBlobIndex). The table builder uses it to give the blob columns of each entity to the collector. It also changes behavior with the option off (see New behavior). You can skip this part on a first read. 6. Small helpers: canonical varints (util/coding.h) and CacheReservationHandle::Resize(). You can skip these on a first read. How to read this diff: 1. Overview: read "Why" and "This diff in one paragraph" above, and "Terms used in this diff" right below. Then read the top comment of db/blob/blob_reference_metadata.h (terms, layout diagrams, run and entry layout, index layout). 2. Indirect BlobIndex (db/blob/blob_index.h) and canonical varints (util/coding.h GetCanonicalVarint32/64): read the overview comment and the kIndirectBlob / kIndirectFlattenedBlob code. Every later part uses these fields: origin, original offset, generation number, logical CRC and stored CRC. The new formats read varints only in the canonical form. 3. Blob file writer (db/blob/blob_file_builder.cc, blob_log_format.cc, blob_log_writer.cc): see how a new value gets a type-3 index and how its CRC is computed once. 4. Format types and decoders: the structs in db/blob/blob_reference_metadata.h, then in the .cc the constants, GapBase, DecodeEntryBody, DecodeRunEntriesAt, DecodeBlobReferenceIndex and BlobReferenceGroupDecoder. They use only BlobIndex and the canonical varints. Learn the on-disk format here, before you read the writer. 5. Collector and its cache charge: first read the new CacheEntryRole kBlobReferenceMetadataBuildingBuffer (include/rocksdb/cache.h) and CacheReservationHandle::Resize (cache/cache_reservation_manager.h). Then read BlobReferenceMetadataCollector in this order: the class comment, Add, AppendSorted, BuildAndWriteDataBlocks, EncodeIndex, FinishOnce. Then read the unsorted fallback (AppendUnsorted, SortUnsortedGroup, SortAndEncodeGroup). 6. Wide-column change (db/wide/wide_column_serialization.cc, ForEachBlobIndex and WalkV2Columns): it gives every blob index of an entity to a callback after it checks the whole entity. The table builder uses it to feed the collector. UpdateBoundaries, BlobGarbageMeter and the compaction iterator also use it, with the option on or off. 7. Option: enable_blob_indirection (include/rocksdb/advanced_options.h, options/cf_options.cc), and the option checks and gate in db/column_family.cc (ValidateOptions, TEST_SetAllowBlobIndirection), db/db_impl/db_impl_open.cc and db_impl_secondary.cc (the FIFO check for opens that skip ValidateOptions). 8. Table integration: table/block_based/block_based_table_factory.cc (the shared reservation manager), then block_based_table_builder.cc (Rep constructor, Add, WriteBlobReferenceMetadata, EstimatedTailSize). See where the collector is created, fed and finished. block_based_table_reader.cc maps the new block name to a block type. Terms used in this diff: Stack terms: | Term | Meaning | |---|---| | origin | Blob file where the value was first written | | original offset | Value offset in the origin file | | stable identity | (origin, original offset). Never changes, even when the value moves | | generation number | The file number that a reference uses to find its value. A new value has generation number == origin. When GC moves the value, the reference keeps its generation number, and a route sends the read to the relocation file | | indirect BlobIndex | A kIndirectBlob (type 3) or kIndirectFlattenedBlob (type 4) index. It stores the stable identity, so the value can move without an SST rewrite | | unflattened reference | Index with generation number == origin: the value was not moved, or it was moved but no compaction has rewritten its reference yet (see "flatten" in "Stack context: terms for later diffs") | | flattened reference | Index with generation number != origin (type 4). Only flatten (diff 8) writes it; this diff defines the encoding | | route | Maps an old file number (a file that GC has removed from the Version) to the relocation file that now holds its live values. One hop: a target is never itself routed. Stored in the MANIFEST (diff 4) | | relocation file | Block-based table keyed by (origin, original offset) that holds copied values. Each value is the stored bytes plus a 5-byte trailer (kSimpleGen2BlobTrailerSize) | | logical CRC | CRC32C of the uncompressed value (BlobIndex::stable_checksum()) | | stored CRC | CRC32C of the stored bytes plus the compression byte | | physical file | A file that holds value bytes: a normal blob file or a relocation file. A generation number can name a physical file, or it can be only the key of a route | | exact reference metadata | Per-SST sorted list of every blob reference the SST keeps, written when the SST is built. References with the same encoding class, generation number and origin form a group. The list is stored as data blocks, with an index of the groups. An SST that has it is an exact SST | | option gate | Until diff 11, ColumnFamilyData::ValidateOptions returns NotSupported for enable_blob_indirection on a read-write open, CreateColumnFamily and SetOptions, unless a test calls ColumnFamilyData::TEST_SetAllowBlobIndirection(true). It keeps the option off in production while the stack is not complete | Terms of this diff (all defined at the top of db/blob/blob_reference_metadata.h): | Term | Meaning | |---|---| | encoding class (class) | kDirect (a legacy kBlob or kBlobTTL index), unflattened, or flattened | | reference group (group) | All references of one SST with the same (encoding class, generation number, origin) | | run | The part of one group inside one data block | | record bytes | The bytes the value's record takes in the file named by its generation number | | gap | How far a record is from where it would be if the records were back to back. A gap of 0 means back to back | | descriptor | The 72-byte table property that points to the reference index block and holds the totals, the CF ID, the SST file number and the relocation-state fingerprint | | relocation-state fingerprint | 128-bit hash of the routes and relocation files of the Version that an SST was built from (fields route_snapshot_fingerprint_*). Written as 0 in this diff | | legacy | Written the way RocksDB writes it without blob indirection | Previous behavior: - A BlobIndex named only a physical blob file and an offset. A value could not move to another file without rewriting every SST that referenced it. - An SST had no metadata that listed its blob references. The only way to find them was to scan its user keys. - These readers get the blob indexes of a wide-column entity through ForEachBlobFileNumber: UpdateBoundaries, BlobGarbageMeter, the compaction iterator, WideColumnSerialization::RemoveBlobColumns, and the referenced_blob_bytes table property count in BlockBasedTableBuilder::Add. ForEachBlobFileNumber decoded each blob index, but it did not check the rest of the entity framing (name order, section sizes, exact input size), and it could call the callback before it found a later error. New behavior: Separate change (option on or off): - The same readers (UpdateBoundaries, BlobGarbageMeter, the compaction iterator, RemoveBlobColumns, and the referenced_blob_bytes count in BlockBasedTableBuilder::Add) use WideColumnSerialization::ForEachBlobIndex, with the option on or off. With the option on, the table builder also uses it to give the blob columns of each entity to the collector. For an entity with blob columns, it checks the full framing, then decodes and validates every blob index before any callback runs. So an entity with a malformed framing now fails the flush or compaction, even with the option off. The referenced_blob_bytes count itself does not fail the build on a malformed entity. It removes that property from the SST, as in base. Option and option gate: - This diff adds a new column family option, `enable_blob_indirection`. It is off by default, and SetOptions can change it. - The option gate (see Terms) keeps the option off outside tests until the stack is complete. ColumnFamilyData::ValidateOptions rejects the option with NotSupported on a read-write open, CreateColumnFamily and SetOptions. Read-only opens (DBImplReadOnly and CompactedDBImpl) and secondary opens (including DB::OpenAndCompact) do not run ValidateOptions; they check only the FIFO rule (see Implementation). Only tests can turn the option on, through the internal ColumnFamilyData::TEST_SetAllowBlobIndirection(). DBTestBase sets it. - When the option is on, ValidateOptions also requires enable_blob_files, the block-based table format, no blob direct write, and a compaction style other than FIFO. Blob file writer (option on): - The blob file writer encodes each new value with a stable identity (origin file, original offset) and two checksums: - the logical CRC: the CRC32C of the uncompressed value. - the stored CRC: the CRC32C of the exact stored (maybe compressed) bytes, extended with the compression byte. - For an uncompressed value the stored CRC is not written, because the reader can compute it from the logical CRC. - The checksums have no version field of their own. The BlobIndex type byte gives the format version: the two new types, 3 (kIndirectBlob) and 4 (kIndirectFlattenedBlob, see Implementation), are BlobIndex::kIndirectBlobFormatVersion (1). A later format uses new type values. - The read path does not check these checksums yet. The check starts in diff 3. Exact reference metadata (option on): 1. What: a sorted list of every blob reference that the SST keeps, in plain blob values and in the blob columns of wide-column entities. 2. Where: data blocks (target 16 KB each before compression; small blocks waste less I/O, because GC reads a whole block to get a few groups) and one reference index block, written after the SST data blocks and before the SST tail, plus a 72-byte descriptor table property. 3. Who reads it: only tests in this diff. The census, GC and RepairDB read it in later diffs. 4. Details: the top comment of db/blob/blob_reference_metadata.h has the full format with diagrams, the record-bytes rules and the gap rule. - Not collected: SstFileWriter output (it has no DB file number) and any output whose CF is unknown (it has no CF ID). The descriptor needs both values, so these outputs stay in the legacy form. Checks on the metadata: - A repeated or overlapping reference in one group fails the SST build with Corruption. Readers check every count, order, size and total against the index and the descriptor, so corrupt metadata returns Corruption. The writer never makes a data block larger than kTargetDataBlockSize (16 KB), and readers accept up to kMaxDataBlockSize (64 KB). Details: "Readers" and "Integrity" at the top of db/blob/blob_reference_metadata.h, and BlobReferenceGroupDecoder. Collection and memory: - Each reference is encoded when it is added, in the on-disk entry encoding, into its group's chain of 512-byte chunks. Only a group that takes the unsorted fallback is encoded a second time, in Finish(). Inside a group, references normally arrive in rising offset order, so no reference is sorted; at the end only the groups are sorted. A group that gets a lower offset takes the unsorted fallback. The BlobReferenceMetadataCollector class comment explains the chunks, the chunk size and the fallback. - The collector keeps every reference of the SST in memory until the SST is finished, and charges this memory to the block cache under a new CacheEntryRole, kBlobReferenceMetadataBuildingBuffer. The measured cost is about 8.3 to 9 bytes per reference for uncompressed values and 12.8 to 13.5 bytes for compressed values. "Memory usage" and "Memory charge" in the BlobReferenceMetadataCollector class comment give the measurement setup and the charge rule. - There is no fixed limit on references. The collector fails with MemoryLimit above 2^20 groups or 2^20 data blocks in one SST; these are persisted format limits. - There is no charge when the table has no block cache or when charging for this role is disabled. Each block-based table factory holds one cache reservation manager for this role, shared by its builders and the SST summaries. If the charge fails, the SST build fails, so the flush or compaction fails with a background error. The buffers and their charge are released when the SST is finished or abandoned. Implementation: - BlobIndex adds two indirect encodings. The legacy encodings do not change. - Type 3, kIndirectBlob (unflattened, generation number == origin), stores the origin, the original offset, the stored size, the compression type, the logical CRC and, for compressed values, the stored CRC. It does not store the generation number. This saves the size of one varint file number per reference (about 3 to 4 bytes for typical file numbers; an estimate, not measured). - Type 4, kIndirectFlattenedBlob (flattened), stores the generation number first, then the same fields. Its generation number must be above the origin. The decoder rejects type 4 with generation number <= origin, so each reference has one encoding. - The encoders take the fields as one BlobIndex::IndirectBlobFields, set by field name, so values of the same type (file number, offset and size; the two CRCs) cannot be swapped. EncodeIndirectBlob() takes the generation number as a separate argument; EncodeUnflattenedIndirectBlob() uses the origin, so the common path cannot leave the generation number unset. - The new files blob_reference_metadata.{h,cc} define the exact reference metadata. The files also contain: - BlobReferenceMetadataCollector, the collector described above. It takes the SST identity (CF ID, SST file number and the relocation-state fingerprint) as one BlobReferenceSstIdentity, set by field name. - BlobReferenceGroupKey, the key of a group: (encoding class, generation number, origin). It has the order, the hash and the validity check that the collector map, the sort, the decoder and FindBlobReferenceGroup() use. BlobReference and BlobReferenceGroup keep their flat fields and return the key through group_key(). - The index codec and the descriptor table property. - BlobReferenceGroupDecoder, the only decoder of reference runs. It reads a group one data block at a time and holds every corruption check of a group walk. Readers that visit one group (found by a binary search of the index) use it, and so do readers that visit every block in order. - Callers in this diff: only the table builder runs the collector. The decoders (DecodeBlobReferenceIndex, FindBlobReferenceGroup, BlobReferenceGroupDecoder, ForEachBlobReference*) are used only by tests. kMaxDecodedBlobReferenceBytes is used only in compile-time checks of the count limits. The census (diff 5) and GC (diffs 6 and 7) are their first production users. The census is also the first user of BlobReferenceSstSummary (see the last Implementation bullet). - Descriptor: the CF ID and the SST file number are the repair fields. They let RepairDB tell which column family and SST the list belongs to without a MANIFEST. The decoder rejects a descriptor whose repair fields are not set (unknown CF ID or SST file number 0). The descriptor also has a 128-bit relocation-state fingerprint (the route_snapshot_fingerprint fields). Remote compaction uses it in diff 9 to check that an SST built elsewhere used the same routes as the job that asked for it. In this diff the table builder always writes 0, and DecodeFrom() does not check it. The field is in the format now, so the descriptor keeps one format version (1) through the stack. While the option gate is on, no data in this format exists outside tests. - The collector validates each BlobIndex once, when it is added. Its Finish() writes the data blocks and then the reference index block through one block writer, and fills the descriptor, including the index handle. Destroying the collector frees its memory and then its charge, also when the build is abandoned. - BlockBasedTableBuilder creates the collector with the CF ID and SST file number from TableBuilderOptions. It passes the size of each entry's user key to the collector, so the collector can compute the exact size of each blob record. It adds the collector's EstimatedEncodedSize() to its tail size estimate: the blocks are written before the tail, but output file size planning uses this estimate. The reference blocks are written with BlockType::kUserDefinedIndex, which keeps their contents as opaque bytes. Readers read them without the block cache, so prepopulate_block_cache does not warm them into the block cache (WriteMaybeCompressedBlock() with warm_cache false). - With the option on, the blob file builder computes the CRC32C of each stored value once. The blob log record CRC combines it with the key CRC (crc32c::Crc32cCombine). The stored CRC extends it with the compression byte. For an uncompressed value it is also the logical CRC. The bytes written do not change. - WideColumnSerialization::ForEachBlobIndex replaces ForEachBlobFileNumber. It reads the entity header and column types once. For a V1 entity (no column types and no blob columns), it returns right after the header. For a V2 entity without blob columns, it returns right after the type scan. It shares one V2 column walker (WalkV2Columns) with Deserialize(), so the entity is parsed in one pass. - Read-only opens (DBImplReadOnly, CompactedDBImpl) and secondary opens skip ValidateOptions, so they check the compaction style against the option once per column family: in DBImpl::Recover() for read-only opens, and at the start of OpenAsSecondaryImpl() for secondary opens. DB::OpenAndCompact opens through OpenAsSecondaryImpl(), so it gets the same check. - CacheReservationHandle gets Resize(), so each collector or summary can change its own part of the shared charge. - util/coding.h adds GetCanonicalVarint32/64, which reject overlong varint encodings. The new formats use them, so each value has one encoding. - BlobReferenceSstSummary is the in-memory list of an SST's groups and totals that Finish() can return. It has no caller yet: the table builder passes a null summary output. It is checked and does not change after Finish(), so the finished builder, the SST metadata and the census (its first user, in diff 5) can share it. It is in this diff because Finish() hands the group buffer and its cache charge over to the summary. Adding the summary later would mean rewriting the charge code of Finish() in diff 5, so the charge code is written and tested once, here. Safety boundaries and unsupported cases: - Public API: this diff adds the option `enable_blob_indirection` with the C API functions `rocksdb_options_set_enable_blob_indirection` and `rocksdb_options_get_enable_blob_indirection`. It also inserts a new CacheEntryRole value before kMisc, which changes the numeric value of kMisc and kNumCacheEntryRoles. - With the option off, new blob values and SSTs are written in the legacy form. Legacy blob indexes stay readable. - One behavior change applies with the option off too: UpdateBoundaries, BlobGarbageMeter and the compaction iterator fully validate wide-column entities with blob columns and their blob indexes (see "Separate change" above). - Exact reference metadata has one format (version 1) with a fixed 72-byte descriptor. The decoders reject any other version as Corruption. - The metadata blocks are protected by the table's block checksum only. With checksum = kNoChecksum, corrupt metadata is detected only when it breaks the format or order checks. - The index decoder checks each decoded count against the remaining input size before it reserves memory. So a small corrupt index cannot make it allocate a large vector. - Inlined-TTL and same-file blob indexes are not collected and cannot be relocated. Encoding, count and checksum errors return an error; a bad reference is never skipped. - Option gate (see Terms): outside tests no DB writes data in the new format. The gate checks only the option value, never persisted state. Diff 11 removes it, once data written with the option stays readable through the end of the stack and the safety checks for turning the option on are in place. - Downgrading the binary after data was written with the option on is not supported. Validation and known limitations: - Unit tests cover the BlobIndex encodings and their rejection of bad input, the metadata layout and every corruption check, the collector (sorted path, unsorted fallback, memory charge), the flush integration, the option checks and option gate, wide-column validation, and canonical varints. See the Test Plan. - The collection buffer is not spilled to disk. One SST with very many blob references needs a matching amount of charged memory: the per-reference amount given under "Collection and memory", plus up to one partly used 512-byte chunk per group. While a fallback group is sorted, it also needs 16 bytes of sort key per reference and a second copy of its entries. For a test group of 200K references with every other one out of order, the test comment estimates about 44 bytes per reference, and the test checks that the peak charge is at most 56 bytes per reference. - Flush still writes a single output file. Until diff 7, one compaction output collects every reference it gets, charged to the block cache. Diff 7 cuts compaction outputs (including L0) at a reference cap and keeps each user key's history in one output. With the option gate, a large output is reachable only through the test hook. Stack context (same in diffs 2-11, except that diffs 8-11 add the diff 8 route rule to the install line; "<== this diff" marks this diff's part): Goal: remove garbage from blob files without rewriting any SST. A value gets a stable identity when it is first written. GC copies the live values of whole files into a relocation file and installs routes in the MANIFEST. Readers follow the routes. SSTs are not changed. How a read finds a value (P is the relocation file that holds the value now): ``` SST entry indirect BlobIndex { generation number, origin, original offset, size, CRCs } | v Version file table: generation number -> live file, or route old file -> P | | live file route found | | v v blob file: read at original offset relocation file P (always one hop): relocation file: look up key look up key (origin, original offset) (origin, original offset) ``` Lifecycle of one value (in this example, F is the blob file where the value is first written, and P is the relocation file that GC writes): ``` write new value goes to blob file F; index: origin = F, generation number = F | census per generation number: the SSTs and references that use it; | per physical file: live bytes, ranked by garbage ratio GC job copy the live values of whole input files (stored bytes, checked | with the stored CRC) into relocation file P, keyed by | (origin, original offset) install one MANIFEST edit: add P, route each input generation number still | in use to P, re-point the routes into the inputs to P, and remove | each input (blob file retirement for a blob file, relocation | file deletion for a relocation file) flatten compaction sets generation number = P in the SSTs it rewrites | route route retirement: route deletion records remove the routes that delete no SST uses ``` Stack map: | Diff | Adds | |---|---| | 1 | Finalize blob metadata outside the DB mutex (prep) | | 2 | Indirect BlobIndex (kIndirectBlob) and exact SST reference metadata <== this diff | | 3 | Relocation file format and stable read path | | 4 | MANIFEST routes and Version ownership | | 5 | Exact reverse reference census | | 6 | Standalone relocation GC (one input file) | | 7 | Pack whole input files; reclaim relocation files | | 8 | Flatten generation numbers; route retirement | | 9 | Remote compaction validation | | 10 | Dynamic options and operations | | 11 | RepairDB rebuilds relocation state; option gate removed | | 12 | db_stress and crash test support | Stack context: terms for later diffs: | Term | Meaning | |---|---| | census | Background count, per generation number, of the SSTs and references that use it, plus live bytes per physical file (diff 5) | | indirect origin | Normal blob file written with the option on; it is the origin of unflattened references | | flatten | Compaction rewrites the generation number to the relocation file that holds the value, so old routes can go. Only flatten makes generation number != origin (diff 8) | | blob file retirement | MANIFEST record (kBlobFileRetirement) that removes a collected blob file from the Version (diff 4) | | route deletion | MANIFEST record that removes a route that no SST uses. Route retirement is the background job that writes these records (diff 8) | Tracking issue: meta#15301 Differential Revision: D124652097
Summary: Pull Request resolved: meta#15540 Why the change is required: The goal of this stack is a blob GC that does not depend on SSTs: it removes garbage from blob files without rewriting any SST. Such a GC needs two things that RocksDB does not have: - A value identity that does not change when the value moves to another file. - A checked list of the blob references that each SST keeps. The list must include older internal versions of a key that the SST still keeps (for example for snapshots), and the blob columns of wide-column entities. This diff adds both. Later diffs build the census (an in-memory reverse index from generation number to the SSTs that use it), GC and repair on top of this list. This diff in one paragraph: With the new column family option enable_blob_indirection on, the blob file writer gives each new value an indirect BlobIndex (type 3, kIndirectBlob). It stores the value's stable identity (origin file, original offset) and its CRCs (an uncompressed value stores only the CRC of the uncompressed value; a compressed value also stores the CRC of its stored bytes). Also with the option on, the block-based table builder collects every blob reference that the SST keeps. It writes them as "exact reference metadata": data blocks, one reference index block, and a 72-byte descriptor table property. In this diff only the table builder writes this metadata, and only tests read it. A temporary option gate (see "option gate" under Terms) keeps the option off outside tests. Separately, wide-column entities with blob columns are now fully checked before their blob indexes are used, also with the option off. Parts of this diff: 1. Indirect BlobIndex (core): types 3 and 4 in db/blob/blob_index.h, and the blob file writer that writes type 3. 2. Exact reference metadata format and decoders (core): db/blob/blob_reference_metadata.{h,cc}. 3. Collector and table builder integration (core): BlobReferenceMetadataCollector, its block cache charge, and BlockBasedTableBuilder. 4. Option enable_blob_indirection and the option gate. Tests need it to turn the feature on. You can skip this part on a first read. 5. Wide-column validation (WideColumnSerialization::ForEachBlobIndex). The table builder uses it to give the blob columns of each entity to the collector. It also changes behavior with the option off (see New behavior). You can skip this part on a first read. 6. Small helpers: canonical varints (util/coding.h) and CacheReservationHandle::Resize(). You can skip these on a first read. How to read this diff: 1. Overview: read "Why" and "This diff in one paragraph" above, and "Terms used in this diff" right below. Then read the top comment of db/blob/blob_reference_metadata.h (terms, layout diagrams, run and entry layout, index layout). 2. Indirect BlobIndex (db/blob/blob_index.h) and canonical varints (util/coding.h GetCanonicalVarint32/64): read the overview comment and the kIndirectBlob / kIndirectFlattenedBlob code. Every later part uses these fields: origin, original offset, generation number, logical CRC and stored CRC. The new formats read varints only in the canonical form. 3. Blob file writer (db/blob/blob_file_builder.cc, blob_log_format.cc, blob_log_writer.cc): see how a new value gets a type-3 index and how its CRC is computed once. 4. Format types and decoders: the structs in db/blob/blob_reference_metadata.h, then in the .cc the constants, GapBase, DecodeEntryBody, DecodeRunEntriesAt, DecodeBlobReferenceIndex and BlobReferenceGroupDecoder. They use only BlobIndex and the canonical varints. Learn the on-disk format here, before you read the writer. 5. Collector and its cache charge: first read the new CacheEntryRole kBlobReferenceMetadataBuildingBuffer (include/rocksdb/cache.h) and CacheReservationHandle::Resize (cache/cache_reservation_manager.h). Then read BlobReferenceMetadataCollector in this order: the class comment, Add, AppendSorted, BuildAndWriteDataBlocks, EncodeIndex, FinishOnce. Then read the unsorted fallback (AppendUnsorted, SortUnsortedGroup, SortAndEncodeGroup). 6. Wide-column change (db/wide/wide_column_serialization.cc, ForEachBlobIndex and WalkV2Columns): it gives every blob index of an entity to a callback after it checks the whole entity. The table builder uses it to feed the collector. UpdateBoundaries, BlobGarbageMeter and the compaction iterator also use it, with the option on or off. 7. Option: enable_blob_indirection (include/rocksdb/advanced_options.h, options/cf_options.cc), and the option checks and gate in db/column_family.cc (ValidateOptions, TEST_SetAllowBlobIndirection), db/db_impl/db_impl_open.cc and db_impl_secondary.cc (the FIFO check for opens that skip ValidateOptions). 8. Table integration: table/block_based/block_based_table_factory.cc (the shared reservation manager), then block_based_table_builder.cc (Rep constructor, Add, WriteBlobReferenceMetadata, EstimatedTailSize). See where the collector is created, fed and finished. block_based_table_reader.cc maps the new block name to a block type. Terms used in this diff: Stack terms: | Term | Meaning | |---|---| | origin | Blob file where the value was first written | | original offset | Value offset in the origin file | | stable identity | (origin, original offset). Never changes, even when the value moves | | generation number | The file number that a reference uses to find its value. A new value has generation number == origin. When GC moves the value, the reference keeps its generation number, and a route sends the read to the relocation file | | indirect BlobIndex | A kIndirectBlob (type 3) or kIndirectFlattenedBlob (type 4) index. It stores the stable identity, so the value can move without an SST rewrite | | unflattened reference | Index with generation number == origin: the value was not moved, or it was moved but no compaction has rewritten its reference yet (see "flatten" in "Stack context: terms for later diffs") | | flattened reference | Index with generation number != origin (type 4). Only flatten (diff 8) writes it; this diff defines the encoding | | route | Maps an old file number (a file that GC has removed from the Version) to the relocation file that now holds its live values. One hop: a target is never itself routed. Stored in the MANIFEST (diff 4) | | relocation file | Block-based table keyed by (origin, original offset) that holds copied values. Each value is the stored bytes plus a 5-byte trailer (kSimpleGen2BlobTrailerSize) | | logical CRC | CRC32C of the uncompressed value (BlobIndex::stable_checksum()) | | stored CRC | CRC32C of the stored bytes plus the compression byte | | physical file | A file that holds value bytes: a normal blob file or a relocation file. A generation number can name a physical file, or it can be only the key of a route | | exact reference metadata | Per-SST sorted list of every blob reference the SST keeps, written when the SST is built. References with the same encoding class, generation number and origin form a group. The list is stored as data blocks, with an index of the groups. An SST that has it is an exact SST | | option gate | Until diff 11, ColumnFamilyData::ValidateOptions returns NotSupported for enable_blob_indirection on a read-write open, CreateColumnFamily and SetOptions, unless a test calls ColumnFamilyData::TEST_SetAllowBlobIndirection(true). It keeps the option off in production while the stack is not complete | Terms of this diff (all defined at the top of db/blob/blob_reference_metadata.h): | Term | Meaning | |---|---| | encoding class (class) | kDirect (a legacy kBlob or kBlobTTL index), unflattened, or flattened | | reference group (group) | All references of one SST with the same (encoding class, generation number, origin) | | run | The part of one group inside one data block | | record bytes | The bytes the value's record takes in the file named by its generation number | | gap | How far a record is from where it would be if the records were back to back. A gap of 0 means back to back | | descriptor | The 72-byte table property that points to the reference index block and holds the totals, the CF ID, the SST file number and the relocation-state fingerprint | | relocation-state fingerprint | 128-bit hash of the routes and relocation files of the Version that an SST was built from (fields route_snapshot_fingerprint_*). Written as 0 in this diff | | legacy | Written the way RocksDB writes it without blob indirection | Previous behavior: - A BlobIndex named only a physical blob file and an offset. A value could not move to another file without rewriting every SST that referenced it. - An SST had no metadata that listed its blob references. The only way to find them was to scan its user keys. - These readers get the blob indexes of a wide-column entity through ForEachBlobFileNumber: UpdateBoundaries, BlobGarbageMeter, the compaction iterator, WideColumnSerialization::RemoveBlobColumns, and the referenced_blob_bytes table property count in BlockBasedTableBuilder::Add. ForEachBlobFileNumber decoded each blob index, but it did not check the rest of the entity framing (name order, section sizes, exact input size), and it could call the callback before it found a later error. New behavior: Separate change (option on or off): - The same readers (UpdateBoundaries, BlobGarbageMeter, the compaction iterator, RemoveBlobColumns, and the referenced_blob_bytes count in BlockBasedTableBuilder::Add) use WideColumnSerialization::ForEachBlobIndex, with the option on or off. With the option on, the table builder also uses it to give the blob columns of each entity to the collector. For an entity with blob columns, it checks the full framing, then decodes and validates every blob index before any callback runs. So an entity with a malformed framing now fails the flush or compaction, even with the option off. The referenced_blob_bytes count itself does not fail the build on a malformed entity. It removes that property from the SST, as in base. Option and option gate: - This diff adds a new column family option, `enable_blob_indirection`. It is off by default, and SetOptions can change it. - The option gate (see Terms) keeps the option off outside tests until the stack is complete. ColumnFamilyData::ValidateOptions rejects the option with NotSupported on a read-write open, CreateColumnFamily and SetOptions. Read-only opens (DBImplReadOnly and CompactedDBImpl) and secondary opens (including DB::OpenAndCompact) do not run ValidateOptions; they check only the FIFO rule (see Implementation). Only tests can turn the option on, through the internal ColumnFamilyData::TEST_SetAllowBlobIndirection(). DBTestBase sets it. - When the option is on, ValidateOptions also requires enable_blob_files, the block-based table format, no blob direct write, and a compaction style other than FIFO. Blob file writer (option on): - The blob file writer encodes each new value with a stable identity (origin file, original offset) and two checksums: - the logical CRC: the CRC32C of the uncompressed value. - the stored CRC: the CRC32C of the exact stored (maybe compressed) bytes, extended with the compression byte. - For an uncompressed value the stored CRC is not written, because the reader can compute it from the logical CRC. - The checksums have no version field of their own. The BlobIndex type byte gives the format version: the two new types, 3 (kIndirectBlob) and 4 (kIndirectFlattenedBlob, see Implementation), are BlobIndex::kIndirectBlobFormatVersion (1). A later format uses new type values. - The read path does not check these checksums yet. The check starts in diff 3. Exact reference metadata (option on): 1. What: a sorted list of every blob reference that the SST keeps, in plain blob values and in the blob columns of wide-column entities. 2. Where: data blocks (target 16 KB each before compression; small blocks waste less I/O, because GC reads a whole block to get a few groups) and one reference index block, written after the SST data blocks and before the SST tail, plus a 72-byte descriptor table property. 3. Who reads it: only tests in this diff. The census, GC and RepairDB read it in later diffs. 4. Details: the top comment of db/blob/blob_reference_metadata.h has the full format with diagrams, the record-bytes rules and the gap rule. - Not collected: SstFileWriter output (it has no DB file number) and any output whose CF is unknown (it has no CF ID). The descriptor needs both values, so these outputs stay in the legacy form. Checks on the metadata: - A repeated or overlapping reference in one group fails the SST build with Corruption. Readers check every count, order, size and total against the index and the descriptor, so corrupt metadata returns Corruption. The writer never makes a data block larger than kTargetDataBlockSize (16 KB), and readers accept up to kMaxDataBlockSize (64 KB). Details: "Readers" and "Integrity" at the top of db/blob/blob_reference_metadata.h, and BlobReferenceGroupDecoder. Collection and memory: - Each reference is encoded when it is added, in the on-disk entry encoding, into its group's chain of 512-byte chunks. Only a group that takes the unsorted fallback is encoded a second time, in Finish(). Inside a group, references normally arrive in rising offset order, so no reference is sorted; at the end only the groups are sorted. A group that gets a lower offset takes the unsorted fallback. The BlobReferenceMetadataCollector class comment explains the chunks, the chunk size and the fallback. - The collector keeps every reference of the SST in memory until the SST is finished, and charges this memory to the block cache under a new CacheEntryRole, kBlobReferenceMetadataBuildingBuffer. The measured cost is about 8.3 to 9 bytes per reference for uncompressed values and 12.8 to 13.5 bytes for compressed values. "Memory usage" and "Memory charge" in the BlobReferenceMetadataCollector class comment give the measurement setup and the charge rule. - There is no fixed limit on references. The collector fails with MemoryLimit above 2^20 groups or 2^20 data blocks in one SST; these are persisted format limits. - There is no charge when the table has no block cache or when charging for this role is disabled. Each block-based table factory holds one cache reservation manager for this role, shared by its builders and the SST summaries. If the charge fails, the SST build fails, so the flush or compaction fails with a background error. The buffers and their charge are released when the SST is finished or abandoned. Implementation: - BlobIndex adds two indirect encodings. The legacy encodings do not change. - Type 3, kIndirectBlob (unflattened, generation number == origin), stores the origin, the original offset, the stored size, the compression type, the logical CRC and, for compressed values, the stored CRC. It does not store the generation number. This saves the size of one varint file number per reference (about 3 to 4 bytes for typical file numbers; an estimate, not measured). - Type 4, kIndirectFlattenedBlob (flattened), stores the generation number first, then the same fields. Its generation number must be above the origin. The decoder rejects type 4 with generation number <= origin, so each reference has one encoding. - The encoders take the fields as one BlobIndex::IndirectBlobFields, set by field name, so values of the same type (file number, offset and size; the two CRCs) cannot be swapped. EncodeIndirectBlob() takes the generation number as a separate argument; EncodeUnflattenedIndirectBlob() uses the origin, so the common path cannot leave the generation number unset. - The new files blob_reference_metadata.{h,cc} define the exact reference metadata. The files also contain: - BlobReferenceMetadataCollector, the collector described above. It takes the SST identity (CF ID, SST file number and the relocation-state fingerprint) as one BlobReferenceSstIdentity, set by field name. - BlobReferenceGroupKey, the key of a group: (encoding class, generation number, origin). It has the order, the hash and the validity check that the collector map, the sort, the decoder and FindBlobReferenceGroup() use. BlobReference and BlobReferenceGroup keep their flat fields and return the key through group_key(). - The index codec and the descriptor table property. - BlobReferenceGroupDecoder, the only decoder of reference runs. It reads a group one data block at a time and holds every corruption check of a group walk. Readers that visit one group (found by a binary search of the index) use it, and so do readers that visit every block in order. - Callers in this diff: only the table builder runs the collector. The decoders (DecodeBlobReferenceIndex, FindBlobReferenceGroup, BlobReferenceGroupDecoder, ForEachBlobReference*) are used only by tests. kMaxDecodedBlobReferenceBytes is used only in compile-time checks of the count limits. The census (diff 5) and GC (diffs 6 and 7) are their first production users. The census is also the first user of BlobReferenceSstSummary (see the last Implementation bullet). - Descriptor: the CF ID and the SST file number are the repair fields. They let RepairDB tell which column family and SST the list belongs to without a MANIFEST. The decoder rejects a descriptor whose repair fields are not set (unknown CF ID or SST file number 0). The descriptor also has a 128-bit relocation-state fingerprint (the route_snapshot_fingerprint fields). Remote compaction uses it in diff 9 to check that an SST built elsewhere used the same routes as the job that asked for it. In this diff the table builder always writes 0, and DecodeFrom() does not check it. The field is in the format now, so the descriptor keeps one format version (1) through the stack. While the option gate is on, no data in this format exists outside tests. - The collector validates each BlobIndex once, when it is added. Its Finish() writes the data blocks and then the reference index block through one block writer, and fills the descriptor, including the index handle. Destroying the collector frees its memory and then its charge, also when the build is abandoned. - BlockBasedTableBuilder creates the collector with the CF ID and SST file number from TableBuilderOptions. It passes the size of each entry's user key to the collector, so the collector can compute the exact size of each blob record. It adds the collector's EstimatedEncodedSize() to its tail size estimate: the blocks are written before the tail, but output file size planning uses this estimate. The reference blocks are written with BlockType::kUserDefinedIndex, which keeps their contents as opaque bytes. Readers read them without the block cache, so prepopulate_block_cache does not warm them into the block cache (WriteMaybeCompressedBlock() with warm_cache false). - With the option on, the blob file builder computes the CRC32C of each stored value once. The blob log record CRC combines it with the key CRC (crc32c::Crc32cCombine). The stored CRC extends it with the compression byte. For an uncompressed value it is also the logical CRC. The bytes written do not change. - WideColumnSerialization::ForEachBlobIndex replaces ForEachBlobFileNumber. It reads the entity header and column types once. For a V1 entity (no column types and no blob columns), it returns right after the header. For a V2 entity without blob columns, it returns right after the type scan. It shares one V2 column walker (WalkV2Columns) with Deserialize(), so the entity is parsed in one pass. - Read-only opens (DBImplReadOnly, CompactedDBImpl) and secondary opens skip ValidateOptions, so they check the compaction style against the option once per column family: in DBImpl::Recover() for read-only opens, and at the start of OpenAsSecondaryImpl() for secondary opens. DB::OpenAndCompact opens through OpenAsSecondaryImpl(), so it gets the same check. - CacheReservationHandle gets Resize(), so each collector or summary can change its own part of the shared charge. - util/coding.h adds GetCanonicalVarint32/64, which reject overlong varint encodings. The new formats use them, so each value has one encoding. - BlobReferenceSstSummary is the in-memory list of an SST's groups and totals that Finish() can return. It has no caller yet: the table builder passes a null summary output. It is checked and does not change after Finish(), so the finished builder, the SST metadata and the census (its first user, in diff 5) can share it. It is in this diff because Finish() hands the group buffer and its cache charge over to the summary. Adding the summary later would mean rewriting the charge code of Finish() in diff 5, so the charge code is written and tested once, here. Safety boundaries and unsupported cases: - Public API: this diff adds the option `enable_blob_indirection` with the C API functions `rocksdb_options_set_enable_blob_indirection` and `rocksdb_options_get_enable_blob_indirection`. It also inserts a new CacheEntryRole value before kMisc, which changes the numeric value of kMisc and kNumCacheEntryRoles. - With the option off, new blob values and SSTs are written in the legacy form. Legacy blob indexes stay readable. - One behavior change applies with the option off too: UpdateBoundaries, BlobGarbageMeter and the compaction iterator fully validate wide-column entities with blob columns and their blob indexes (see "Separate change" above). - Exact reference metadata has one format (version 1) with a fixed 72-byte descriptor. The decoders reject any other version as Corruption. - The metadata blocks are protected by the table's block checksum only. With checksum = kNoChecksum, corrupt metadata is detected only when it breaks the format or order checks. - The index decoder checks each decoded count against the remaining input size before it reserves memory. So a small corrupt index cannot make it allocate a large vector. - Inlined-TTL and same-file blob indexes are not collected and cannot be relocated. Encoding, count and checksum errors return an error; a bad reference is never skipped. - Option gate (see Terms): outside tests no DB writes data in the new format. The gate checks only the option value, never persisted state. Diff 11 removes it, once data written with the option stays readable through the end of the stack and the safety checks for turning the option on are in place. - Downgrading the binary after data was written with the option on is not supported. Validation and known limitations: - Unit tests cover the BlobIndex encodings and their rejection of bad input, the metadata layout and every corruption check, the collector (sorted path, unsorted fallback, memory charge), the flush integration, the option checks and option gate, wide-column validation, and canonical varints. See the Test Plan. - The collection buffer is not spilled to disk. One SST with very many blob references needs a matching amount of charged memory: the per-reference amount given under "Collection and memory", plus up to one partly used 512-byte chunk per group. While a fallback group is sorted, it also needs 16 bytes of sort key per reference and a second copy of its entries. For a test group of 200K references with every other one out of order, the test comment estimates about 44 bytes per reference, and the test checks that the peak charge is at most 56 bytes per reference. - Flush still writes a single output file. Until diff 7, one compaction output collects every reference it gets, charged to the block cache. Diff 7 cuts compaction outputs (including L0) at a reference cap and keeps each user key's history in one output. With the option gate, a large output is reachable only through the test hook. Stack context (same in diffs 2-11, except that diffs 8-11 add the diff 8 route rule to the install line; "<== this diff" marks this diff's part): Goal: remove garbage from blob files without rewriting any SST. A value gets a stable identity when it is first written. GC copies the live values of whole files into a relocation file and installs routes in the MANIFEST. Readers follow the routes. SSTs are not changed. How a read finds a value (P is the relocation file that holds the value now): ``` SST entry indirect BlobIndex { generation number, origin, original offset, size, CRCs } | v Version file table: generation number -> live file, or route old file -> P | | live file route found | | v v blob file: read at original offset relocation file P (always one hop): relocation file: look up key look up key (origin, original offset) (origin, original offset) ``` Lifecycle of one value (in this example, F is the blob file where the value is first written, and P is the relocation file that GC writes): ``` write new value goes to blob file F; index: origin = F, generation number = F | census per generation number: the SSTs and references that use it; | per physical file: live bytes, ranked by garbage ratio GC job copy the live values of whole input files (stored bytes, checked | with the stored CRC) into relocation file P, keyed by | (origin, original offset) install one MANIFEST edit: add P, route each input generation number still | in use to P, re-point the routes into the inputs to P, and remove | each input (blob file retirement for a blob file, relocation | file deletion for a relocation file) flatten compaction sets generation number = P in the SSTs it rewrites | route route retirement: route deletion records remove the routes that delete no SST uses ``` Stack map: | Diff | Adds | |---|---| | 1 | Finalize blob metadata outside the DB mutex (prep) | | 2 | Indirect BlobIndex (kIndirectBlob) and exact SST reference metadata <== this diff | | 3 | Relocation file format and stable read path | | 4 | MANIFEST routes and Version ownership | | 5 | Exact reverse reference census | | 6 | Standalone relocation GC (one input file) | | 7 | Pack whole input files; reclaim relocation files | | 8 | Flatten generation numbers; route retirement | | 9 | Remote compaction validation | | 10 | Dynamic options and operations | | 11 | RepairDB rebuilds relocation state; option gate removed | | 12 | db_stress and crash test support | Stack context: terms for later diffs: | Term | Meaning | |---|---| | census | Background count, per generation number, of the SSTs and references that use it, plus live bytes per physical file (diff 5) | | indirect origin | Normal blob file written with the option on; it is the origin of unflattened references | | flatten | Compaction rewrites the generation number to the relocation file that holds the value, so old routes can go. Only flatten makes generation number != origin (diff 8) | | blob file retirement | MANIFEST record (kBlobFileRetirement) that removes a collected blob file from the Version (diff 4) | | route deletion | MANIFEST record that removes a route that no SST uses. Route retirement is the background job that writes these records (diff 8) | Tracking issue: meta#15301 Differential Revision: D124652097
Summary: Pull Request resolved: meta#15540 Why the change is required: The goal of this stack is a blob GC that does not depend on SSTs: it removes garbage from blob files without rewriting any SST. Such a GC needs two things that RocksDB does not have: - A value identity that does not change when the value moves to another file. - A checked list of the blob references that each SST keeps. The list must include older internal versions of a key that the SST still keeps (for example for snapshots), and the blob columns of wide-column entities. This diff adds both. Later diffs build the census (an in-memory reverse index from generation number to the SSTs that use it), GC and repair on top of this list. This diff in one paragraph: With the new column family option enable_blob_indirection on, the blob file writer gives each new value an indirect BlobIndex (type 3, kIndirectBlob). It stores the value's stable identity (origin file, original offset) and its CRCs (an uncompressed value stores only the CRC of the uncompressed value; a compressed value also stores the CRC of its stored bytes). Also with the option on, the block-based table builder collects every blob reference that the SST keeps. It writes them as "exact reference metadata": data blocks, one reference index block, and a 72-byte descriptor table property. In this diff only the table builder writes this metadata, and only tests read it. A temporary option gate (see "option gate" under Terms) keeps the option off outside tests. Separately, wide-column entities with blob columns are now fully checked before their blob indexes are used, also with the option off. Parts of this diff: 1. Indirect BlobIndex (core): types 3 and 4 in db/blob/blob_index.h, and the blob file writer that writes type 3. 2. Exact reference metadata format and decoders (core): db/blob/blob_reference_metadata.{h,cc}. 3. Collector and table builder integration (core): BlobReferenceMetadataCollector, its block cache charge, and BlockBasedTableBuilder. 4. Option enable_blob_indirection and the option gate. Tests need it to turn the feature on. You can skip this part on a first read. 5. Wide-column validation (WideColumnSerialization::ForEachBlobIndex). The table builder uses it to give the blob columns of each entity to the collector. It also changes behavior with the option off (see New behavior). You can skip this part on a first read. 6. Small helpers: canonical varints (util/coding.h) and CacheReservationHandle::Resize(). You can skip these on a first read. How to read this diff: 1. Overview: read "Why" and "This diff in one paragraph" above, and "Terms used in this diff" right below. Then read the top comment of db/blob/blob_reference_metadata.h (terms, layout diagrams, run and entry layout, index layout). 2. Indirect BlobIndex (db/blob/blob_index.h) and canonical varints (util/coding.h GetCanonicalVarint32/64): read the overview comment and the kIndirectBlob / kIndirectFlattenedBlob code. Every later part uses these fields: origin, original offset, generation number, logical CRC and stored CRC. The new formats read varints only in the canonical form. 3. Blob file writer (db/blob/blob_file_builder.cc, blob_log_format.cc, blob_log_writer.cc): see how a new value gets a type-3 index and how its CRC is computed once. 4. Format types and decoders: the structs in db/blob/blob_reference_metadata.h, then in the .cc the constants, GapBase, DecodeEntryBody, DecodeRunEntriesAt, DecodeBlobReferenceIndex and BlobReferenceGroupDecoder. They use only BlobIndex and the canonical varints. Learn the on-disk format here, before you read the writer. 5. Collector and its cache charge: first read the new CacheEntryRole kBlobReferenceMetadataBuildingBuffer (include/rocksdb/cache.h) and CacheReservationHandle::Resize (cache/cache_reservation_manager.h). Then read BlobReferenceMetadataCollector in this order: the class comment, Add, AppendSorted, BuildAndWriteDataBlocks, EncodeIndex, FinishOnce. Then read the unsorted fallback (AppendUnsorted, SortUnsortedGroup, SortAndEncodeGroup). 6. Wide-column change (db/wide/wide_column_serialization.cc, ForEachBlobIndex and WalkV2Columns): it gives every blob index of an entity to a callback after it checks the whole entity. The table builder uses it to feed the collector. UpdateBoundaries, BlobGarbageMeter and the compaction iterator also use it, with the option on or off. 7. Option: enable_blob_indirection (include/rocksdb/advanced_options.h, options/cf_options.cc), and the option checks and gate in db/column_family.cc (ValidateOptions, TEST_SetAllowBlobIndirection), db/db_impl/db_impl_open.cc and db_impl_secondary.cc (the FIFO check for opens that skip ValidateOptions). 8. Table integration: table/block_based/block_based_table_factory.cc (the shared reservation manager), then block_based_table_builder.cc (Rep constructor, Add, WriteBlobReferenceMetadata, EstimatedTailSize). See where the collector is created, fed and finished. block_based_table_reader.cc maps the new block name to a block type. Terms used in this diff: Stack terms: | Term | Meaning | |---|---| | origin | Blob file where the value was first written | | original offset | Value offset in the origin file | | stable identity | (origin, original offset). Never changes, even when the value moves | | generation number | The file number that a reference uses to find its value. A new value has generation number == origin. When GC moves the value, the reference keeps its generation number, and a route sends the read to the relocation file | | indirect BlobIndex | A kIndirectBlob (type 3) or kIndirectFlattenedBlob (type 4) index. It stores the stable identity, so the value can move without an SST rewrite | | unflattened reference | Index with generation number == origin: the value was not moved, or it was moved but no compaction has rewritten its reference yet (see "flatten" in "Stack context: terms for later diffs") | | flattened reference | Index with generation number != origin (type 4). Only flatten (diff 8) writes it; this diff defines the encoding | | route | Maps an old file number (a file that GC has removed from the Version) to the relocation file that now holds its live values. One hop: a target is never itself routed. Stored in the MANIFEST (diff 4) | | relocation file | Block-based table keyed by (origin, original offset) that holds copied values. Each value is the stored bytes plus a 5-byte trailer (kSimpleGen2BlobTrailerSize) | | logical CRC | CRC32C of the uncompressed value (BlobIndex::stable_checksum()) | | stored CRC | CRC32C of the stored bytes plus the compression byte | | physical file | A file that holds value bytes: a normal blob file or a relocation file. A generation number can name a physical file, or it can be only the key of a route | | exact reference metadata | Per-SST sorted list of every blob reference the SST keeps, written when the SST is built. References with the same encoding class, generation number and origin form a group. The list is stored as data blocks, with an index of the groups. An SST that has it is an exact SST | | option gate | Until diff 11, ColumnFamilyData::ValidateOptions returns NotSupported for enable_blob_indirection on a read-write open, CreateColumnFamily and SetOptions, unless a test calls ColumnFamilyData::TEST_SetAllowBlobIndirection(true). It keeps the option off in production while the stack is not complete | Terms of this diff (all defined at the top of db/blob/blob_reference_metadata.h): | Term | Meaning | |---|---| | encoding class (class) | kDirect (a legacy kBlob or kBlobTTL index), unflattened, or flattened | | reference group (group) | All references of one SST with the same (encoding class, generation number, origin) | | run | The part of one group inside one data block | | record bytes | The bytes the value's record takes in the file named by its generation number | | gap | How far a record is from where it would be if the records were back to back. A gap of 0 means back to back | | descriptor | The 72-byte table property that points to the reference index block and holds the totals, the CF ID, the SST file number and the relocation-state fingerprint | | relocation-state fingerprint | 128-bit hash of the routes and relocation files of the Version that an SST was built from (fields route_snapshot_fingerprint_*). Written as 0 in this diff | | legacy | Written the way RocksDB writes it without blob indirection | Previous behavior: - A BlobIndex named only a physical blob file and an offset. A value could not move to another file without rewriting every SST that referenced it. - An SST had no metadata that listed its blob references. The only way to find them was to scan its user keys. - These readers get the blob indexes of a wide-column entity through ForEachBlobFileNumber: UpdateBoundaries, BlobGarbageMeter, the compaction iterator, WideColumnSerialization::RemoveBlobColumns, and the referenced_blob_bytes table property count in BlockBasedTableBuilder::Add. ForEachBlobFileNumber decoded each blob index, but it did not check the rest of the entity framing (name order, section sizes, exact input size), and it could call the callback before it found a later error. New behavior: Separate change (option on or off): - The same readers (UpdateBoundaries, BlobGarbageMeter, the compaction iterator, RemoveBlobColumns, and the referenced_blob_bytes count in BlockBasedTableBuilder::Add) use WideColumnSerialization::ForEachBlobIndex, with the option on or off. With the option on, the table builder also uses it to give the blob columns of each entity to the collector. For an entity with blob columns, it checks the full framing, then decodes and validates every blob index before any callback runs. So an entity with a malformed framing now fails the flush or compaction, even with the option off. The referenced_blob_bytes count itself does not fail the build on a malformed entity. It removes that property from the SST, as in base. Option and option gate: - This diff adds a new column family option, `enable_blob_indirection`. It is off by default, and SetOptions can change it. - The option gate (see Terms) keeps the option off outside tests until the stack is complete. ColumnFamilyData::ValidateOptions rejects the option with NotSupported on a read-write open, CreateColumnFamily and SetOptions. Read-only opens (DBImplReadOnly and CompactedDBImpl) and secondary opens (including DB::OpenAndCompact) do not run ValidateOptions; they check only the FIFO rule (see Implementation). Only tests can turn the option on, through the internal ColumnFamilyData::TEST_SetAllowBlobIndirection(). DBTestBase sets it. - When the option is on, ValidateOptions also requires enable_blob_files, the block-based table format, no blob direct write, and a compaction style other than FIFO. Blob file writer (option on): - The blob file writer encodes each new value with a stable identity (origin file, original offset) and two checksums: - the logical CRC: the CRC32C of the uncompressed value. - the stored CRC: the CRC32C of the exact stored (maybe compressed) bytes, extended with the compression byte. - For an uncompressed value the stored CRC is not written, because the reader can compute it from the logical CRC. - The checksums have no version field of their own. The BlobIndex type byte gives the format version: the two new types, 3 (kIndirectBlob) and 4 (kIndirectFlattenedBlob, see Implementation), are BlobIndex::kIndirectBlobFormatVersion (1). A later format uses new type values. - The read path does not check these checksums yet. The check starts in diff 3. Exact reference metadata (option on): 1. What: a sorted list of every blob reference that the SST keeps, in plain blob values and in the blob columns of wide-column entities. 2. Where: data blocks (target 16 KB each before compression; small blocks waste less I/O, because GC reads a whole block to get a few groups) and one reference index block, written after the SST data blocks and before the SST tail, plus a 72-byte descriptor table property. 3. Who reads it: only tests in this diff. The census, GC and RepairDB read it in later diffs. 4. Details: the top comment of db/blob/blob_reference_metadata.h has the full format with diagrams, the record-bytes rules and the gap rule. - Not collected: SstFileWriter output (it has no DB file number) and any output whose CF is unknown (it has no CF ID). The descriptor needs both values, so these outputs stay in the legacy form. Checks on the metadata: - A repeated or overlapping reference in one group fails the SST build with Corruption. Readers check every count, order, size and total against the index and the descriptor, so corrupt metadata returns Corruption. The writer never makes a data block larger than kTargetDataBlockSize (16 KB), and readers accept up to kMaxDataBlockSize (64 KB). Details: "Readers" and "Integrity" at the top of db/blob/blob_reference_metadata.h, and BlobReferenceGroupDecoder. Collection and memory: - Each reference is encoded when it is added, in the on-disk entry encoding, into its group's chain of 512-byte chunks. Only a group that takes the unsorted fallback is encoded a second time, in Finish(). Inside a group, references normally arrive in rising offset order, so no reference is sorted; at the end only the groups are sorted. A group that gets a lower offset takes the unsorted fallback. The BlobReferenceMetadataCollector class comment explains the chunks, the chunk size and the fallback. - The collector keeps every reference of the SST in memory until the SST is finished, and charges this memory to the block cache under a new CacheEntryRole, kBlobReferenceMetadataBuildingBuffer. The measured cost is about 8.3 to 9 bytes per reference for uncompressed values and 12.8 to 13.5 bytes for compressed values. "Memory usage" and "Memory charge" in the BlobReferenceMetadataCollector class comment give the measurement setup and the charge rule. - There is no fixed limit on references. The collector fails with MemoryLimit above 2^20 groups or 2^20 data blocks in one SST; these are persisted format limits. - There is no charge when the table has no block cache or when charging for this role is disabled. Each block-based table factory holds one cache reservation manager for this role, shared by its builders and the SST summaries. If the charge fails, the SST build fails, so the flush or compaction fails with a background error. The buffers and their charge are released when the SST is finished or abandoned. Implementation: - BlobIndex adds two indirect encodings. The legacy encodings do not change. - Type 3, kIndirectBlob (unflattened, generation number == origin), stores the origin, the original offset, the stored size, the compression type, the logical CRC and, for compressed values, the stored CRC. It does not store the generation number. This saves the size of one varint file number per reference (about 3 to 4 bytes for typical file numbers; an estimate, not measured). - Type 4, kIndirectFlattenedBlob (flattened), stores the generation number first, then the same fields. Its generation number must be above the origin. The decoder rejects type 4 with generation number <= origin, so each reference has one encoding. - The encoders take the fields as one BlobIndex::IndirectBlobFields, set by field name, so values of the same type (file number, offset and size; the two CRCs) cannot be swapped. EncodeIndirectBlob() takes the generation number as a separate argument; EncodeUnflattenedIndirectBlob() uses the origin, so the common path cannot leave the generation number unset. - The new files blob_reference_metadata.{h,cc} define the exact reference metadata. The files also contain: - BlobReferenceMetadataCollector, the collector described above. It takes the SST identity (CF ID, SST file number and the relocation-state fingerprint) as one BlobReferenceSstIdentity, set by field name. - BlobReferenceGroupKey, the key of a group: (encoding class, generation number, origin). It has the order, the hash and the validity check that the collector map, the sort, the decoder and FindBlobReferenceGroup() use. BlobReference and BlobReferenceGroup keep their flat fields and return the key through group_key(). - The index codec and the descriptor table property. - BlobReferenceGroupDecoder, the only decoder of reference runs. It reads a group one data block at a time and holds every corruption check of a group walk. Readers that visit one group (found by a binary search of the index) use it, and so do readers that visit every block in order. - Callers in this diff: only the table builder runs the collector. The decoders (DecodeBlobReferenceIndex, FindBlobReferenceGroup, BlobReferenceGroupDecoder, ForEachBlobReference*) are used only by tests. kMaxDecodedBlobReferenceBytes is used only in compile-time checks of the count limits. The census (diff 5) and GC (diffs 6 and 7) are their first production users. The census is also the first user of BlobReferenceSstSummary (see the last Implementation bullet). - Descriptor: the CF ID and the SST file number are the repair fields. They let RepairDB tell which column family and SST the list belongs to without a MANIFEST. The decoder rejects a descriptor whose repair fields are not set (unknown CF ID or SST file number 0). The descriptor also has a 128-bit relocation-state fingerprint (the route_snapshot_fingerprint fields). Remote compaction uses it in diff 9 to check that an SST built elsewhere used the same routes as the job that asked for it. In this diff the table builder always writes 0, and DecodeFrom() does not check it. The field is in the format now, so the descriptor keeps one format version (1) through the stack. While the option gate is on, no data in this format exists outside tests. - The collector validates each BlobIndex once, when it is added. Its Finish() writes the data blocks and then the reference index block through one block writer, and fills the descriptor, including the index handle. Destroying the collector frees its memory and then its charge, also when the build is abandoned. - BlockBasedTableBuilder creates the collector with the CF ID and SST file number from TableBuilderOptions. It passes the size of each entry's user key to the collector, so the collector can compute the exact size of each blob record. It adds the collector's EstimatedEncodedSize() to its tail size estimate: the blocks are written before the tail, but output file size planning uses this estimate. The reference blocks are written with BlockType::kUserDefinedIndex, which keeps their contents as opaque bytes. Readers read them without the block cache, so prepopulate_block_cache does not warm them into the block cache (WriteMaybeCompressedBlock() with warm_cache false). - With the option on, the blob file builder computes the CRC32C of each stored value once. The blob log record CRC combines it with the key CRC (crc32c::Crc32cCombine). The stored CRC extends it with the compression byte. For an uncompressed value it is also the logical CRC. The bytes written do not change. - WideColumnSerialization::ForEachBlobIndex replaces ForEachBlobFileNumber. It reads the entity header and column types once. For a V1 entity (no column types and no blob columns), it returns right after the header. For a V2 entity without blob columns, it returns right after the type scan. It shares one V2 column walker (WalkV2Columns) with Deserialize(), so the entity is parsed in one pass. - Read-only opens (DBImplReadOnly, CompactedDBImpl) and secondary opens skip ValidateOptions, so they check the compaction style against the option once per column family: in DBImpl::Recover() for read-only opens, and at the start of OpenAsSecondaryImpl() for secondary opens. DB::OpenAndCompact opens through OpenAsSecondaryImpl(), so it gets the same check. - CacheReservationHandle gets Resize(), so each collector or summary can change its own part of the shared charge. - util/coding.h adds GetCanonicalVarint32/64, which reject overlong varint encodings. The new formats use them, so each value has one encoding. - BlobReferenceSstSummary is the in-memory list of an SST's groups and totals that Finish() can return. It has no caller yet: the table builder passes a null summary output. It is checked and does not change after Finish(), so the finished builder, the SST metadata and the census (its first user, in diff 5) can share it. It is in this diff because Finish() hands the group buffer and its cache charge over to the summary. Adding the summary later would mean rewriting the charge code of Finish() in diff 5, so the charge code is written and tested once, here. Safety boundaries and unsupported cases: - Public API: this diff adds the option `enable_blob_indirection` with the C API functions `rocksdb_options_set_enable_blob_indirection` and `rocksdb_options_get_enable_blob_indirection`. It also inserts a new CacheEntryRole value before kMisc, which changes the numeric value of kMisc and kNumCacheEntryRoles. - With the option off, new blob values and SSTs are written in the legacy form. Legacy blob indexes stay readable. - One behavior change applies with the option off too: UpdateBoundaries, BlobGarbageMeter and the compaction iterator fully validate wide-column entities with blob columns and their blob indexes (see "Separate change" above). - Exact reference metadata has one format (version 1) with a fixed 72-byte descriptor. The decoders reject any other version as Corruption. - The metadata blocks are protected by the table's block checksum only. With checksum = kNoChecksum, corrupt metadata is detected only when it breaks the format or order checks. - The index decoder checks each decoded count against the remaining input size before it reserves memory. So a small corrupt index cannot make it allocate a large vector. - Inlined-TTL and same-file blob indexes are not collected and cannot be relocated. Encoding, count and checksum errors return an error; a bad reference is never skipped. - Option gate (see Terms): outside tests no DB writes data in the new format. The gate checks only the option value, never persisted state. Diff 11 removes it, once data written with the option stays readable through the end of the stack and the safety checks for turning the option on are in place. - Downgrading the binary after data was written with the option on is not supported. Validation and known limitations: - Unit tests cover the BlobIndex encodings and their rejection of bad input, the metadata layout and every corruption check, the collector (sorted path, unsorted fallback, memory charge), the flush integration, the option checks and option gate, wide-column validation, and canonical varints. See the Test Plan. - The collection buffer is not spilled to disk. One SST with very many blob references needs a matching amount of charged memory: the per-reference amount given under "Collection and memory", plus up to one partly used 512-byte chunk per group. While a fallback group is sorted, it also needs 16 bytes of sort key per reference and a second copy of its entries. For a test group of 200K references with every other one out of order, the test comment estimates about 44 bytes per reference, and the test checks that the peak charge is at most 56 bytes per reference. - Flush still writes a single output file. Until diff 7, one compaction output collects every reference it gets, charged to the block cache. Diff 7 cuts compaction outputs (including L0) at a reference cap and keeps each user key's history in one output. With the option gate, a large output is reachable only through the test hook. Stack context (same in diffs 2-11, except that diffs 8-11 add the diff 8 route rule to the install line; "<== this diff" marks this diff's part): Goal: remove garbage from blob files without rewriting any SST. A value gets a stable identity when it is first written. GC copies the live values of whole files into a relocation file and installs routes in the MANIFEST. Readers follow the routes. SSTs are not changed. How a read finds a value (P is the relocation file that holds the value now): ``` SST entry indirect BlobIndex { generation number, origin, original offset, size, CRCs } | v Version file table: generation number -> live file, or route old file -> P | | live file route found | | v v blob file: read at original offset relocation file P (always one hop): relocation file: look up key look up key (origin, original offset) (origin, original offset) ``` Lifecycle of one value (in this example, F is the blob file where the value is first written, and P is the relocation file that GC writes): ``` write new value goes to blob file F; index: origin = F, generation number = F | census per generation number: the SSTs and references that use it; | per physical file: live bytes, ranked by garbage ratio GC job copy the live values of whole input files (stored bytes, checked | with the stored CRC) into relocation file P, keyed by | (origin, original offset) install one MANIFEST edit: add P, route each input generation number still | in use to P, re-point the routes into the inputs to P, and remove | each input (blob file retirement for a blob file, relocation | file deletion for a relocation file) flatten compaction sets generation number = P in the SSTs it rewrites | route route retirement: route deletion records remove the routes that delete no SST uses ``` Stack map: | Diff | Adds | |---|---| | 1 | Finalize blob metadata outside the DB mutex (prep) | | 2 | Indirect BlobIndex (kIndirectBlob) and exact SST reference metadata <== this diff | | 3 | Relocation file format and stable read path | | 4 | MANIFEST routes and Version ownership | | 5 | Exact reverse reference census | | 6 | Standalone relocation GC (one input file) | | 7 | Pack whole input files; reclaim relocation files | | 8 | Flatten generation numbers; route retirement | | 9 | Remote compaction validation | | 10 | Dynamic options and operations | | 11 | RepairDB rebuilds relocation state; option gate removed | | 12 | db_stress and crash test support | Stack context: terms for later diffs: | Term | Meaning | |---|---| | census | Background count, per generation number, of the SSTs and references that use it, plus live bytes per physical file (diff 5) | | indirect origin | Normal blob file written with the option on; it is the origin of unflattened references | | flatten | Compaction rewrites the generation number to the relocation file that holds the value, so old routes can go. Only flatten makes generation number != origin (diff 8) | | blob file retirement | MANIFEST record (kBlobFileRetirement) that removes a collected blob file from the Version (diff 4) | | route deletion | MANIFEST record that removes a route that no SST uses. Route retirement is the background job that writes these records (diff 8) | Tracking issue: meta#15301 Differential Revision: D124652097
Summary: Pull Request resolved: meta#15540 Why the change is required: The goal of this stack is a blob GC that does not depend on SSTs: it removes garbage from blob files without rewriting any SST. Such a GC needs two things that RocksDB does not have: - A value identity that does not change when the value moves to another file. - A checked list of the blob references that each SST keeps. The list must include older internal versions of a key that the SST still keeps (for example for snapshots), and the blob columns of wide-column entities. This diff adds both. Later diffs build the census (an in-memory reverse index from generation number to the SSTs that use it), GC and repair on top of this list. This diff in one paragraph: With the new column family option enable_blob_indirection on, the blob file writer gives each new value an indirect BlobIndex (type 3, kIndirectBlob). It stores the value's stable identity (origin file, original offset) and its CRCs (an uncompressed value stores only the CRC of the uncompressed value; a compressed value also stores the CRC of its stored bytes). Also with the option on, the block-based table builder collects every blob reference that the SST keeps. It writes them as "exact reference metadata": data blocks, one reference index block, and a 72-byte descriptor table property. In this diff only the table builder writes this metadata, and only tests read it. A temporary option gate (see "option gate" under Terms) keeps the option off outside tests. Separately, wide-column entities with blob columns are now fully checked before their blob indexes are used, also with the option off. Parts of this diff: 1. Indirect BlobIndex (core): types 3 and 4 in db/blob/blob_index.h, and the blob file writer that writes type 3. 2. Exact reference metadata format and decoders (core): db/blob/blob_reference_metadata.{h,cc}. 3. Collector and table builder integration (core): BlobReferenceMetadataCollector, its block cache charge, and BlockBasedTableBuilder. 4. Option enable_blob_indirection and the option gate. Tests need it to turn the feature on. You can skip this part on a first read. 5. Wide-column validation (WideColumnSerialization::ForEachBlobIndex). The table builder uses it to give the blob columns of each entity to the collector. It also changes behavior with the option off (see New behavior). You can skip this part on a first read. 6. Small helpers: canonical varints (util/coding.h) and CacheReservationHandle::Resize(). You can skip these on a first read. How to read this diff: 1. Overview: read "Why" and "This diff in one paragraph" above, and "Terms used in this diff" right below. Then read the top comment of db/blob/blob_reference_metadata.h (terms, layout diagrams, run and entry layout, index layout). 2. Indirect BlobIndex (db/blob/blob_index.h) and canonical varints (util/coding.h GetCanonicalVarint32/64): read the overview comment and the kIndirectBlob / kIndirectFlattenedBlob code. Every later part uses these fields: origin, original offset, generation number, logical CRC and stored CRC. The new formats read varints only in the canonical form. 3. Blob file writer (db/blob/blob_file_builder.cc, blob_log_format.cc, blob_log_writer.cc): see how a new value gets a type-3 index and how its CRC is computed once. 4. Format types and decoders: the structs in db/blob/blob_reference_metadata.h, then in the .cc the constants, GapBase, DecodeEntryBody, DecodeRunEntriesAt, DecodeBlobReferenceIndex and BlobReferenceGroupDecoder. They use only BlobIndex and the canonical varints. Learn the on-disk format here, before you read the writer. 5. Collector and its cache charge: first read the new CacheEntryRole kBlobReferenceMetadataBuildingBuffer (include/rocksdb/cache.h) and CacheReservationHandle::Resize (cache/cache_reservation_manager.h). Then read BlobReferenceMetadataCollector in this order: the class comment, Add, AppendSorted, BuildAndWriteDataBlocks, EncodeIndex, FinishOnce. Then read the unsorted fallback (AppendUnsorted, SortUnsortedGroup, SortAndEncodeGroup). 6. Wide-column change (db/wide/wide_column_serialization.cc, ForEachBlobIndex and WalkV2Columns): it gives every blob index of an entity to a callback after it checks the whole entity. The table builder uses it to feed the collector. UpdateBoundaries, BlobGarbageMeter and the compaction iterator also use it, with the option on or off. 7. Option: enable_blob_indirection (include/rocksdb/advanced_options.h, options/cf_options.cc), and the option checks and gate in db/column_family.cc (ValidateOptions, TEST_SetAllowBlobIndirection), db/db_impl/db_impl_open.cc and db_impl_secondary.cc (the FIFO check for opens that skip ValidateOptions). 8. Table integration: table/block_based/block_based_table_factory.cc (the shared reservation manager), then block_based_table_builder.cc (Rep constructor, Add, WriteBlobReferenceMetadata, EstimatedTailSize). See where the collector is created, fed and finished. block_based_table_reader.cc maps the new block name to a block type. Terms used in this diff: Stack terms: | Term | Meaning | |---|---| | origin | Blob file where the value was first written | | original offset | Value offset in the origin file | | stable identity | (origin, original offset). Never changes, even when the value moves | | generation number | The file number that a reference uses to find its value. A new value has generation number == origin. When GC moves the value, the reference keeps its generation number, and a route sends the read to the relocation file | | indirect BlobIndex | A kIndirectBlob (type 3) or kIndirectFlattenedBlob (type 4) index. It stores the stable identity, so the value can move without an SST rewrite | | unflattened reference | Index with generation number == origin: the value was not moved, or it was moved but no compaction has rewritten its reference yet (see "flatten" in "Stack context: terms for later diffs") | | flattened reference | Index with generation number != origin (type 4). Only flatten (diff 8) writes it; this diff defines the encoding | | route | Maps an old file number (a file that GC has removed from the Version) to the relocation file that now holds its live values. One hop: a target is never itself routed. Stored in the MANIFEST (diff 4) | | relocation file | Block-based table keyed by (origin, original offset) that holds copied values. Each value is the stored bytes plus a 5-byte trailer (kSimpleGen2BlobTrailerSize) | | logical CRC | CRC32C of the uncompressed value (BlobIndex::stable_checksum()) | | stored CRC | CRC32C of the stored bytes plus the compression byte | | physical file | A file that holds value bytes: a normal blob file or a relocation file. A generation number can name a physical file, or it can be only the key of a route | | exact reference metadata | Per-SST sorted list of every blob reference the SST keeps, written when the SST is built. References with the same encoding class, generation number and origin form a group. The list is stored as data blocks, with an index of the groups. An SST that has it is an exact SST | | option gate | Until diff 11, ColumnFamilyData::ValidateOptions returns NotSupported for enable_blob_indirection on a read-write open, CreateColumnFamily and SetOptions, unless a test calls ColumnFamilyData::TEST_SetAllowBlobIndirection(true). It keeps the option off in production while the stack is not complete | Terms of this diff (all defined at the top of db/blob/blob_reference_metadata.h): | Term | Meaning | |---|---| | encoding class (class) | kDirect (a legacy kBlob or kBlobTTL index), unflattened, or flattened | | reference group (group) | All references of one SST with the same (encoding class, generation number, origin) | | run | The part of one group inside one data block | | record bytes | The bytes the value's record takes in the file named by its generation number | | gap | How far a record is from where it would be if the records were back to back. A gap of 0 means back to back | | descriptor | The 72-byte table property that points to the reference index block and holds the totals, the CF ID, the SST file number and the relocation-state fingerprint | | relocation-state fingerprint | 128-bit hash of the routes and relocation files of the Version that an SST was built from (fields route_snapshot_fingerprint_*). Written as 0 in this diff | | legacy | Written the way RocksDB writes it without blob indirection | Previous behavior: - A BlobIndex named only a physical blob file and an offset. A value could not move to another file without rewriting every SST that referenced it. - An SST had no metadata that listed its blob references. The only way to find them was to scan its user keys. - These readers get the blob indexes of a wide-column entity through ForEachBlobFileNumber: UpdateBoundaries, BlobGarbageMeter, the compaction iterator, WideColumnSerialization::RemoveBlobColumns, and the referenced_blob_bytes table property count in BlockBasedTableBuilder::Add. ForEachBlobFileNumber decoded each blob index, but it did not check the rest of the entity framing (name order, section sizes, exact input size), and it could call the callback before it found a later error. New behavior: Separate change (option on or off): - The same readers (UpdateBoundaries, BlobGarbageMeter, the compaction iterator, RemoveBlobColumns, and the referenced_blob_bytes count in BlockBasedTableBuilder::Add) use WideColumnSerialization::ForEachBlobIndex, with the option on or off. With the option on, the table builder also uses it to give the blob columns of each entity to the collector. For an entity with blob columns, it checks the full framing, then decodes and validates every blob index before any callback runs. So an entity with a malformed framing now fails the flush or compaction, even with the option off. The referenced_blob_bytes count itself does not fail the build on a malformed entity. It removes that property from the SST, as in base. Option and option gate: - This diff adds a new column family option, `enable_blob_indirection`. It is off by default, and SetOptions can change it. - The option gate (see Terms) keeps the option off outside tests until the stack is complete. ColumnFamilyData::ValidateOptions rejects the option with NotSupported on a read-write open, CreateColumnFamily and SetOptions. Read-only opens (DBImplReadOnly and CompactedDBImpl) and secondary opens (including DB::OpenAndCompact) do not run ValidateOptions; they check only the FIFO rule (see Implementation). Only tests can turn the option on, through the internal ColumnFamilyData::TEST_SetAllowBlobIndirection(). DBTestBase sets it. - When the option is on, ValidateOptions also requires enable_blob_files, the block-based table format, no blob direct write, and a compaction style other than FIFO. Blob file writer (option on): - The blob file writer encodes each new value with a stable identity (origin file, original offset) and two checksums: - the logical CRC: the CRC32C of the uncompressed value. - the stored CRC: the CRC32C of the exact stored (maybe compressed) bytes, extended with the compression byte. - For an uncompressed value the stored CRC is not written, because the reader can compute it from the logical CRC. - The checksums have no version field of their own. The BlobIndex type byte gives the format version: the two new types, 3 (kIndirectBlob) and 4 (kIndirectFlattenedBlob, see Implementation), are BlobIndex::kIndirectBlobFormatVersion (1). A later format uses new type values. - The read path does not check these checksums yet. The check starts in diff 3. Exact reference metadata (option on): 1. What: a sorted list of every blob reference that the SST keeps, in plain blob values and in the blob columns of wide-column entities. 2. Where: data blocks (target 16 KB each before compression; small blocks waste less I/O, because GC reads a whole block to get a few groups) and one reference index block, written after the SST data blocks and before the SST tail, plus a 72-byte descriptor table property. 3. Who reads it: only tests in this diff. The census, GC and RepairDB read it in later diffs. 4. Details: the top comment of db/blob/blob_reference_metadata.h has the full format with diagrams, the record-bytes rules and the gap rule. - Not collected: SstFileWriter output (it has no DB file number) and any output whose CF is unknown (it has no CF ID). The descriptor needs both values, so these outputs stay in the legacy form. Checks on the metadata: - A repeated or overlapping reference in one group fails the SST build with Corruption. Readers check every count, order, size and total against the index and the descriptor, so corrupt metadata returns Corruption. The writer never makes a data block larger than kTargetDataBlockSize (16 KB), and readers accept up to kMaxDataBlockSize (64 KB). Details: "Readers" and "Integrity" at the top of db/blob/blob_reference_metadata.h, and BlobReferenceGroupDecoder. Collection and memory: - Each reference is encoded when it is added, in the on-disk entry encoding, into its group's chain of 512-byte chunks. Only a group that takes the unsorted fallback is encoded a second time, in Finish(). Inside a group, references normally arrive in rising offset order, so no reference is sorted; at the end only the groups are sorted. A group that gets a lower offset takes the unsorted fallback. The BlobReferenceMetadataCollector class comment explains the chunks, the chunk size and the fallback. - The collector keeps every reference of the SST in memory until the SST is finished, and charges this memory to the block cache under a new CacheEntryRole, kBlobReferenceMetadataBuildingBuffer. The measured cost is about 8.3 to 9 bytes per reference for uncompressed values and 12.8 to 13.5 bytes for compressed values. "Memory usage" and "Memory charge" in the BlobReferenceMetadataCollector class comment give the measurement setup and the charge rule. - There is no fixed limit on references. The collector fails with MemoryLimit above 2^20 groups or 2^20 data blocks in one SST; these are persisted format limits. - There is no charge when the table has no block cache or when charging for this role is disabled. Each block-based table factory holds one cache reservation manager for this role, shared by its builders and the SST summaries. If the charge fails, the SST build fails, so the flush or compaction fails with a background error. The buffers and their charge are released when the SST is finished or abandoned. Implementation: - BlobIndex adds two indirect encodings. The legacy encodings do not change. - Type 3, kIndirectBlob (unflattened, generation number == origin), stores the origin, the original offset, the stored size, the compression type, the logical CRC and, for compressed values, the stored CRC. It does not store the generation number. This saves the size of one varint file number per reference (about 3 to 4 bytes for typical file numbers; an estimate, not measured). - Type 4, kIndirectFlattenedBlob (flattened), stores the generation number first, then the same fields. Its generation number must be above the origin. The decoder rejects type 4 with generation number <= origin, so each reference has one encoding. - The encoders take the fields as one BlobIndex::IndirectBlobFields, set by field name, so values of the same type (file number, offset and size; the two CRCs) cannot be swapped. EncodeIndirectBlob() takes the generation number as a separate argument; EncodeUnflattenedIndirectBlob() uses the origin, so the common path cannot leave the generation number unset. - The new files blob_reference_metadata.{h,cc} define the exact reference metadata. The files also contain: - BlobReferenceMetadataCollector, the collector described above. It takes the SST identity (CF ID, SST file number and the relocation-state fingerprint) as one BlobReferenceSstIdentity, set by field name. - BlobReferenceGroupKey, the key of a group: (encoding class, generation number, origin). It has the order, the hash and the validity check that the collector map, the sort, the decoder and FindBlobReferenceGroup() use. BlobReference and BlobReferenceGroup keep their flat fields and return the key through group_key(). - The index codec and the descriptor table property. - BlobReferenceGroupDecoder, the only decoder of reference runs. It reads a group one data block at a time and holds every corruption check of a group walk. Readers that visit one group (found by a binary search of the index) use it, and so do readers that visit every block in order. - Callers in this diff: only the table builder runs the collector. The decoders (DecodeBlobReferenceIndex, FindBlobReferenceGroup, BlobReferenceGroupDecoder, ForEachBlobReference*) are used only by tests. kMaxDecodedBlobReferenceBytes is used only in compile-time checks of the count limits. The census (diff 5) and GC (diffs 6 and 7) are their first production users. The census is also the first user of BlobReferenceSstSummary (see the last Implementation bullet). - Descriptor: the CF ID and the SST file number are the repair fields. They let RepairDB tell which column family and SST the list belongs to without a MANIFEST. The decoder rejects a descriptor whose repair fields are not set (unknown CF ID or SST file number 0). The descriptor also has a 128-bit relocation-state fingerprint (the route_snapshot_fingerprint fields). Remote compaction uses it in diff 9 to check that an SST built elsewhere used the same routes as the job that asked for it. In this diff the table builder always writes 0, and DecodeFrom() does not check it. The field is in the format now, so the descriptor keeps one format version (1) through the stack. While the option gate is on, no data in this format exists outside tests. - The collector validates each BlobIndex once, when it is added. Its Finish() writes the data blocks and then the reference index block through one block writer, and fills the descriptor, including the index handle. Destroying the collector frees its memory and then its charge, also when the build is abandoned. - BlockBasedTableBuilder creates the collector with the CF ID and SST file number from TableBuilderOptions. It passes the size of each entry's user key to the collector, so the collector can compute the exact size of each blob record. It adds the collector's EstimatedEncodedSize() to its tail size estimate: the blocks are written before the tail, but output file size planning uses this estimate. The reference blocks are written with BlockType::kUserDefinedIndex, which keeps their contents as opaque bytes. Readers read them without the block cache, so prepopulate_block_cache does not warm them into the block cache (WriteMaybeCompressedBlock() with warm_cache false). - With the option on, the blob file builder computes the CRC32C of each stored value once. The blob log record CRC combines it with the key CRC (crc32c::Crc32cCombine). The stored CRC extends it with the compression byte. For an uncompressed value it is also the logical CRC. The bytes written do not change. - WideColumnSerialization::ForEachBlobIndex replaces ForEachBlobFileNumber. It reads the entity header and column types once. For a V1 entity (no column types and no blob columns), it returns right after the header. For a V2 entity without blob columns, it returns right after the type scan. It shares one V2 column walker (WalkV2Columns) with Deserialize(), so the entity is parsed in one pass. - Read-only opens (DBImplReadOnly, CompactedDBImpl) and secondary opens skip ValidateOptions, so they check the compaction style against the option once per column family: in DBImpl::Recover() for read-only opens, and at the start of OpenAsSecondaryImpl() for secondary opens. DB::OpenAndCompact opens through OpenAsSecondaryImpl(), so it gets the same check. - CacheReservationHandle gets Resize(), so each collector or summary can change its own part of the shared charge. - util/coding.h adds GetCanonicalVarint32/64, which reject overlong varint encodings. The new formats use them, so each value has one encoding. - BlobReferenceSstSummary is the in-memory list of an SST's groups and totals that Finish() can return. It has no caller yet: the table builder passes a null summary output. It is checked and does not change after Finish(), so the finished builder, the SST metadata and the census (its first user, in diff 5) can share it. It is in this diff because Finish() hands the group buffer and its cache charge over to the summary. Adding the summary later would mean rewriting the charge code of Finish() in diff 5, so the charge code is written and tested once, here. Safety boundaries and unsupported cases: - Public API: this diff adds the option `enable_blob_indirection` with the C API functions `rocksdb_options_set_enable_blob_indirection` and `rocksdb_options_get_enable_blob_indirection`. It also inserts a new CacheEntryRole value before kMisc, which changes the numeric value of kMisc and kNumCacheEntryRoles. - With the option off, new blob values and SSTs are written in the legacy form. Legacy blob indexes stay readable. - One behavior change applies with the option off too: UpdateBoundaries, BlobGarbageMeter and the compaction iterator fully validate wide-column entities with blob columns and their blob indexes (see "Separate change" above). - Exact reference metadata has one format (version 1) with a fixed 72-byte descriptor. The decoders reject any other version as Corruption. - The metadata blocks are protected by the table's block checksum only. With checksum = kNoChecksum, corrupt metadata is detected only when it breaks the format or order checks. - The index decoder checks each decoded count against the remaining input size before it reserves memory. So a small corrupt index cannot make it allocate a large vector. - Inlined-TTL and same-file blob indexes are not collected and cannot be relocated. Encoding, count and checksum errors return an error; a bad reference is never skipped. - Option gate (see Terms): outside tests no DB writes data in the new format. The gate checks only the option value, never persisted state. Diff 11 removes it, once data written with the option stays readable through the end of the stack and the safety checks for turning the option on are in place. - Downgrading the binary after data was written with the option on is not supported. Validation and known limitations: - Unit tests cover the BlobIndex encodings and their rejection of bad input, the metadata layout and every corruption check, the collector (sorted path, unsorted fallback, memory charge), the flush integration, the option checks and option gate, wide-column validation, and canonical varints. See the Test Plan. - The collection buffer is not spilled to disk. One SST with very many blob references needs a matching amount of charged memory: the per-reference amount given under "Collection and memory", plus up to one partly used 512-byte chunk per group. While a fallback group is sorted, it also needs 16 bytes of sort key per reference and a second copy of its entries. For a test group of 200K references with every other one out of order, the test comment estimates about 44 bytes per reference, and the test checks that the peak charge is at most 56 bytes per reference. - Flush still writes a single output file. Until diff 7, one compaction output collects every reference it gets, charged to the block cache. Diff 7 cuts compaction outputs (including L0) at a reference cap and keeps each user key's history in one output. With the option gate, a large output is reachable only through the test hook. Stack context (same in diffs 2-11, except that diffs 8-11 add the diff 8 route rule to the install line; "<== this diff" marks this diff's part): Goal: remove garbage from blob files without rewriting any SST. A value gets a stable identity when it is first written. GC copies the live values of whole files into a relocation file and installs routes in the MANIFEST. Readers follow the routes. SSTs are not changed. How a read finds a value (P is the relocation file that holds the value now): ``` SST entry indirect BlobIndex { generation number, origin, original offset, size, CRCs } | v Version file table: generation number -> live file, or route old file -> P | | live file route found | | v v blob file: read at original offset relocation file P (always one hop): relocation file: look up key look up key (origin, original offset) (origin, original offset) ``` Lifecycle of one value (in this example, F is the blob file where the value is first written, and P is the relocation file that GC writes): ``` write new value goes to blob file F; index: origin = F, generation number = F | census per generation number: the SSTs and references that use it; | per physical file: live bytes, ranked by garbage ratio GC job copy the live values of whole input files (stored bytes, checked | with the stored CRC) into relocation file P, keyed by | (origin, original offset) install one MANIFEST edit: add P, route each input generation number still | in use to P, re-point the routes into the inputs to P, and remove | each input (blob file retirement for a blob file, relocation | file deletion for a relocation file) flatten compaction sets generation number = P in the SSTs it rewrites | route route retirement: route deletion records remove the routes that delete no SST uses ``` Stack map: | Diff | Adds | |---|---| | 1 | Finalize blob metadata outside the DB mutex (prep) | | 2 | Indirect BlobIndex (kIndirectBlob) and exact SST reference metadata <== this diff | | 3 | Relocation file format and stable read path | | 4 | MANIFEST routes and Version ownership | | 5 | Exact reverse reference census | | 6 | Standalone relocation GC (one input file) | | 7 | Pack whole input files; reclaim relocation files | | 8 | Flatten generation numbers; route retirement | | 9 | Remote compaction validation | | 10 | Dynamic options and operations | | 11 | RepairDB rebuilds relocation state; option gate removed | | 12 | db_stress and crash test support | Stack context: terms for later diffs: | Term | Meaning | |---|---| | census | Background count, per generation number, of the SSTs and references that use it, plus live bytes per physical file (diff 5) | | indirect origin | Normal blob file written with the option on; it is the origin of unflattened references | | flatten | Compaction rewrites the generation number to the relocation file that holds the value, so old routes can go. Only flatten makes generation number != origin (diff 8) | | blob file retirement | MANIFEST record (kBlobFileRetirement) that removes a collected blob file from the Version (diff 4) | | route deletion | MANIFEST record that removes a route that no SST uses. Route retirement is the background job that writes these records (diff 8) | Tracking issue: meta#15301 Differential Revision: D124652097
Summary: Pull Request resolved: meta#15540 Why the change is required: The goal of this stack is a blob GC that does not depend on SSTs: it removes garbage from blob files without rewriting any SST. Such a GC needs two things that RocksDB does not have: - A value identity that does not change when the value moves to another file. - A checked list of the blob references that each SST keeps. The list must include older internal versions of a key that the SST still keeps (for example for snapshots), and the blob columns of wide-column entities. This diff adds both. Later diffs build the census (an in-memory reverse index from generation number to the SSTs that use it), GC and repair on top of this list. This diff in one paragraph: With the new column family option enable_blob_indirection on, the blob file writer gives each new value an indirect BlobIndex (type 3, kIndirectBlob). It stores the value's stable identity (origin file, original offset) and its CRCs (an uncompressed value stores only the CRC of the uncompressed value; a compressed value also stores the CRC of its stored bytes). Also with the option on, the block-based table builder collects every blob reference that the SST keeps. It writes them as "exact reference metadata": data blocks, one reference index block, and a 72-byte descriptor table property. In this diff only the table builder writes this metadata, and only tests read it. A temporary option gate (see "option gate" under Terms) keeps the option off outside tests. Separately, wide-column entities with blob columns are now fully checked before their blob indexes are used, also with the option off. Parts of this diff: 1. Indirect BlobIndex (core): types 3 and 4 in db/blob/blob_index.h, and the blob file writer that writes type 3. 2. Exact reference metadata format and decoders (core): db/blob/blob_reference_metadata.{h,cc}. 3. Collector and table builder integration (core): BlobReferenceMetadataCollector, its block cache charge, and BlockBasedTableBuilder. 4. Option enable_blob_indirection and the option gate. Tests need it to turn the feature on. You can skip this part on a first read. 5. Wide-column validation (WideColumnSerialization::ForEachBlobIndex). The table builder uses it to give the blob columns of each entity to the collector. It also changes behavior with the option off (see New behavior). You can skip this part on a first read. 6. Small helpers: canonical varints (util/coding.h) and CacheReservationHandle::Resize(). You can skip these on a first read. How to read this diff: 1. Overview: read "Why" and "This diff in one paragraph" above, and "Terms used in this diff" right below. Then read the top comment of db/blob/blob_reference_metadata.h (terms, layout diagrams, run and entry layout, index layout). 2. Indirect BlobIndex (db/blob/blob_index.h) and canonical varints (util/coding.h GetCanonicalVarint32/64): read the overview comment and the kIndirectBlob / kIndirectFlattenedBlob code. Every later part uses these fields: origin, original offset, generation number, logical CRC and stored CRC. The new formats read varints only in the canonical form. 3. Blob file writer (db/blob/blob_file_builder.cc, blob_log_format.cc, blob_log_writer.cc): see how a new value gets a type-3 index and how its CRC is computed once. 4. Format types and decoders: the structs in db/blob/blob_reference_metadata.h, then in the .cc the constants, GapBase, DecodeEntryBody, DecodeRunEntriesAt, DecodeBlobReferenceIndex and BlobReferenceGroupDecoder. They use only BlobIndex and the canonical varints. Learn the on-disk format here, before you read the writer. 5. Collector and its cache charge: first read the new CacheEntryRole kBlobReferenceMetadataBuildingBuffer (include/rocksdb/cache.h) and CacheReservationHandle::Resize (cache/cache_reservation_manager.h). Then read BlobReferenceMetadataCollector in this order: the class comment, Add, AppendSorted, BuildAndWriteDataBlocks, EncodeIndex, FinishOnce. Then read the unsorted fallback (AppendUnsorted, SortUnsortedGroup, SortAndEncodeGroup). 6. Wide-column change (db/wide/wide_column_serialization.cc, ForEachBlobIndex and WalkV2Columns): it gives every blob index of an entity to a callback after it checks the whole entity. The table builder uses it to feed the collector. UpdateBoundaries, BlobGarbageMeter and the compaction iterator also use it, with the option on or off. 7. Option: enable_blob_indirection (include/rocksdb/advanced_options.h, options/cf_options.cc), and the option checks and gate in db/column_family.cc (ValidateOptions, TEST_SetAllowBlobIndirection), db/db_impl/db_impl_open.cc and db_impl_secondary.cc (the FIFO check for opens that skip ValidateOptions). 8. Table integration: table/block_based/block_based_table_factory.cc (the shared reservation manager), then block_based_table_builder.cc (Rep constructor, Add, WriteBlobReferenceMetadata, EstimatedTailSize). See where the collector is created, fed and finished. block_based_table_reader.cc maps the new block name to a block type. Terms used in this diff: Stack terms: | Term | Meaning | |---|---| | origin | Blob file where the value was first written | | original offset | Value offset in the origin file | | stable identity | (origin, original offset). Never changes, even when the value moves | | generation number | The file number that a reference uses to find its value. A new value has generation number == origin. When GC moves the value, the reference keeps its generation number, and a route sends the read to the relocation file | | indirect BlobIndex | A kIndirectBlob (type 3) or kIndirectFlattenedBlob (type 4) index. It stores the stable identity, so the value can move without an SST rewrite | | unflattened reference | Index with generation number == origin: the value was not moved, or it was moved but no compaction has rewritten its reference yet (see "flatten" in "Stack context: terms for later diffs") | | flattened reference | Index with generation number != origin (type 4). Only flatten (diff 8) writes it; this diff defines the encoding | | route | Maps an old file number (a file that GC has removed from the Version) to the relocation file that now holds its live values. One hop: a target is never itself routed. Stored in the MANIFEST (diff 4) | | relocation file | Block-based table keyed by (origin, original offset) that holds copied values. Each value is the stored bytes plus a 5-byte trailer (kSimpleGen2BlobTrailerSize) | | logical CRC | CRC32C of the uncompressed value (BlobIndex::stable_checksum()) | | stored CRC | CRC32C of the stored bytes plus the compression byte | | physical file | A file that holds value bytes: a normal blob file or a relocation file. A generation number can name a physical file, or it can be only the key of a route | | exact reference metadata | Per-SST sorted list of every blob reference the SST keeps, written when the SST is built. References with the same encoding class, generation number and origin form a group. The list is stored as data blocks, with an index of the groups. An SST that has it is an exact SST | | option gate | Until diff 11, ColumnFamilyData::ValidateOptions returns NotSupported for enable_blob_indirection on a read-write open, CreateColumnFamily and SetOptions, unless a test calls ColumnFamilyData::TEST_SetAllowBlobIndirection(true). It keeps the option off in production while the stack is not complete | Terms of this diff (all defined at the top of db/blob/blob_reference_metadata.h): | Term | Meaning | |---|---| | encoding class (class) | kDirect (a legacy kBlob or kBlobTTL index), unflattened, or flattened | | reference group (group) | All references of one SST with the same (encoding class, generation number, origin) | | run | The part of one group inside one data block | | record bytes | The bytes the value's record takes in the file named by its generation number | | gap | How far a record is from where it would be if the records were back to back. A gap of 0 means back to back | | descriptor | The 72-byte table property that points to the reference index block and holds the totals, the CF ID, the SST file number and the relocation-state fingerprint | | relocation-state fingerprint | 128-bit hash of the routes and relocation files of the Version that an SST was built from (fields route_snapshot_fingerprint_*). Written as 0 in this diff | | legacy | Written the way RocksDB writes it without blob indirection | Previous behavior: - A BlobIndex named only a physical blob file and an offset. A value could not move to another file without rewriting every SST that referenced it. - An SST had no metadata that listed its blob references. The only way to find them was to scan its user keys. - These readers get the blob indexes of a wide-column entity through ForEachBlobFileNumber: UpdateBoundaries, BlobGarbageMeter, the compaction iterator, WideColumnSerialization::RemoveBlobColumns, and the referenced_blob_bytes table property count in BlockBasedTableBuilder::Add. ForEachBlobFileNumber decoded each blob index, but it did not check the rest of the entity framing (name order, section sizes, exact input size), and it could call the callback before it found a later error. New behavior: Separate change (option on or off): - The same readers (UpdateBoundaries, BlobGarbageMeter, the compaction iterator, RemoveBlobColumns, and the referenced_blob_bytes count in BlockBasedTableBuilder::Add) use WideColumnSerialization::ForEachBlobIndex, with the option on or off. With the option on, the table builder also uses it to give the blob columns of each entity to the collector. For an entity with blob columns, it checks the full framing, then decodes and validates every blob index before any callback runs. So an entity with a malformed framing now fails the flush or compaction, even with the option off. The referenced_blob_bytes count itself does not fail the build on a malformed entity. It removes that property from the SST, as in base. Option and option gate: - This diff adds a new column family option, `enable_blob_indirection`. It is off by default, and SetOptions can change it. - The option gate (see Terms) keeps the option off outside tests until the stack is complete. ColumnFamilyData::ValidateOptions rejects the option with NotSupported on a read-write open, CreateColumnFamily and SetOptions. Read-only opens (DBImplReadOnly and CompactedDBImpl) and secondary opens (including DB::OpenAndCompact) do not run ValidateOptions; they check only the FIFO rule (see Implementation). Only tests can turn the option on, through the internal ColumnFamilyData::TEST_SetAllowBlobIndirection(). DBTestBase sets it. - When the option is on, ValidateOptions also requires enable_blob_files, the block-based table format, no blob direct write, and a compaction style other than FIFO. Blob file writer (option on): - The blob file writer encodes each new value with a stable identity (origin file, original offset) and two checksums: - the logical CRC: the CRC32C of the uncompressed value. - the stored CRC: the CRC32C of the exact stored (maybe compressed) bytes, extended with the compression byte. - For an uncompressed value the stored CRC is not written, because the reader can compute it from the logical CRC. - The checksums have no version field of their own. The BlobIndex type byte gives the format version: the two new types, 3 (kIndirectBlob) and 4 (kIndirectFlattenedBlob, see Implementation), are BlobIndex::kIndirectBlobFormatVersion (1). A later format uses new type values. - The read path does not check these checksums yet. The check starts in diff 3. Exact reference metadata (option on): 1. What: a sorted list of every blob reference that the SST keeps, in plain blob values and in the blob columns of wide-column entities. 2. Where: data blocks (target 16 KB each before compression; small blocks waste less I/O, because GC reads a whole block to get a few groups) and one reference index block, written after the SST data blocks and before the SST tail, plus a 72-byte descriptor table property. 3. Who reads it: only tests in this diff. The census, GC and RepairDB read it in later diffs. 4. Details: the top comment of db/blob/blob_reference_metadata.h has the full format with diagrams, the record-bytes rules and the gap rule. - Not collected: SstFileWriter output (it has no DB file number) and any output whose CF is unknown (it has no CF ID). The descriptor needs both values, so these outputs stay in the legacy form. Checks on the metadata: - A repeated or overlapping reference in one group fails the SST build with Corruption. Readers check every count, order, size and total against the index and the descriptor, so corrupt metadata returns Corruption. The writer never makes a data block larger than kTargetDataBlockSize (16 KB), and readers accept up to kMaxDataBlockSize (64 KB). Details: "Readers" and "Integrity" at the top of db/blob/blob_reference_metadata.h, and BlobReferenceGroupDecoder. Collection and memory: - Each reference is encoded when it is added, in the on-disk entry encoding, into its group's chain of 512-byte chunks. Only a group that takes the unsorted fallback is encoded a second time, in Finish(). Inside a group, references normally arrive in rising offset order, so no reference is sorted; at the end only the groups are sorted. A group that gets a lower offset takes the unsorted fallback. The BlobReferenceMetadataCollector class comment explains the chunks, the chunk size and the fallback. - The collector keeps every reference of the SST in memory until the SST is finished, and charges this memory to the block cache under a new CacheEntryRole, kBlobReferenceMetadataBuildingBuffer. The measured cost is about 8.3 to 9 bytes per reference for uncompressed values and 12.8 to 13.5 bytes for compressed values. "Memory usage" and "Memory charge" in the BlobReferenceMetadataCollector class comment give the measurement setup and the charge rule. - There is no fixed limit on references. The collector fails with MemoryLimit above 2^20 groups or 2^20 data blocks in one SST; these are persisted format limits. - There is no charge when the table has no block cache or when charging for this role is disabled. Each block-based table factory holds one cache reservation manager for this role, shared by its builders and the SST summaries. If the charge fails, the SST build fails, so the flush or compaction fails with a background error. The buffers and their charge are released when the SST is finished or abandoned. Implementation: - BlobIndex adds two indirect encodings. The legacy encodings do not change. - Type 3, kIndirectBlob (unflattened, generation number == origin), stores the origin, the original offset, the stored size, the compression type, the logical CRC and, for compressed values, the stored CRC. It does not store the generation number. This saves the size of one varint file number per reference (about 3 to 4 bytes for typical file numbers; an estimate, not measured). - Type 4, kIndirectFlattenedBlob (flattened), stores the generation number first, then the same fields. Its generation number must be above the origin. The decoder rejects type 4 with generation number <= origin, so each reference has one encoding. - The encoders take the fields as one BlobIndex::IndirectBlobFields, set by field name, so values of the same type (file number, offset and size; the two CRCs) cannot be swapped. EncodeIndirectBlob() takes the generation number as a separate argument; EncodeUnflattenedIndirectBlob() uses the origin, so the common path cannot leave the generation number unset. - The new files blob_reference_metadata.{h,cc} define the exact reference metadata. The files also contain: - BlobReferenceMetadataCollector, the collector described above. It takes the SST identity (CF ID, SST file number and the relocation-state fingerprint) as one BlobReferenceSstIdentity, set by field name. - BlobReferenceGroupKey, the key of a group: (encoding class, generation number, origin). It has the order, the hash and the validity check that the collector map, the sort, the decoder and FindBlobReferenceGroup() use. BlobReference and BlobReferenceGroup keep their flat fields and return the key through group_key(). - The index codec and the descriptor table property. - BlobReferenceGroupDecoder, the only decoder of reference runs. It reads a group one data block at a time and holds every corruption check of a group walk. Readers that visit one group (found by a binary search of the index) use it, and so do readers that visit every block in order. - Callers in this diff: only the table builder runs the collector. The decoders (DecodeBlobReferenceIndex, FindBlobReferenceGroup, BlobReferenceGroupDecoder, ForEachBlobReference*) are used only by tests. kMaxDecodedBlobReferenceBytes is used only in compile-time checks of the count limits. The census (diff 5) and GC (diffs 6 and 7) are their first production users. The census is also the first user of BlobReferenceSstSummary (see the last Implementation bullet). - Descriptor: the CF ID and the SST file number are the repair fields. They let RepairDB tell which column family and SST the list belongs to without a MANIFEST. The decoder rejects a descriptor whose repair fields are not set (unknown CF ID or SST file number 0). The descriptor also has a 128-bit relocation-state fingerprint (the route_snapshot_fingerprint fields). Remote compaction uses it in diff 9 to check that an SST built elsewhere used the same routes as the job that asked for it. In this diff the table builder always writes 0, and DecodeFrom() does not check it. The field is in the format now, so the descriptor keeps one format version (1) through the stack. While the option gate is on, no data in this format exists outside tests. - The collector validates each BlobIndex once, when it is added. Its Finish() writes the data blocks and then the reference index block through one block writer, and fills the descriptor, including the index handle. Destroying the collector frees its memory and then its charge, also when the build is abandoned. - BlockBasedTableBuilder creates the collector with the CF ID and SST file number from TableBuilderOptions. It passes the size of each entry's user key to the collector, so the collector can compute the exact size of each blob record. It adds the collector's EstimatedEncodedSize() to its tail size estimate: the blocks are written before the tail, but output file size planning uses this estimate. The reference blocks are written with BlockType::kUserDefinedIndex, which keeps their contents as opaque bytes. Readers read them without the block cache, so prepopulate_block_cache does not warm them into the block cache (WriteMaybeCompressedBlock() with warm_cache false). - With the option on, the blob file builder computes the CRC32C of each stored value once. The blob log record CRC combines it with the key CRC (crc32c::Crc32cCombine). The stored CRC extends it with the compression byte. For an uncompressed value it is also the logical CRC. The bytes written do not change. - WideColumnSerialization::ForEachBlobIndex replaces ForEachBlobFileNumber. It reads the entity header and column types once. For a V1 entity (no column types and no blob columns), it returns right after the header. For a V2 entity without blob columns, it returns right after the type scan. It shares one V2 column walker (WalkV2Columns) with Deserialize(), so the entity is parsed in one pass. - Read-only opens (DBImplReadOnly, CompactedDBImpl) and secondary opens skip ValidateOptions, so they check the compaction style against the option once per column family: in DBImpl::Recover() for read-only opens, and at the start of OpenAsSecondaryImpl() for secondary opens. DB::OpenAndCompact opens through OpenAsSecondaryImpl(), so it gets the same check. - CacheReservationHandle gets Resize(), so each collector or summary can change its own part of the shared charge. - util/coding.h adds GetCanonicalVarint32/64, which reject overlong varint encodings. The new formats use them, so each value has one encoding. - BlobReferenceSstSummary is the in-memory list of an SST's groups and totals that Finish() can return. It has no caller yet: the table builder passes a null summary output. It is checked and does not change after Finish(), so the finished builder, the SST metadata and the census (its first user, in diff 5) can share it. It is in this diff because Finish() hands the group buffer and its cache charge over to the summary. Adding the summary later would mean rewriting the charge code of Finish() in diff 5, so the charge code is written and tested once, here. Safety boundaries and unsupported cases: - Public API: this diff adds the option `enable_blob_indirection` with the C API functions `rocksdb_options_set_enable_blob_indirection` and `rocksdb_options_get_enable_blob_indirection`. It also inserts a new CacheEntryRole value before kMisc, which changes the numeric value of kMisc and kNumCacheEntryRoles. - With the option off, new blob values and SSTs are written in the legacy form. Legacy blob indexes stay readable. - One behavior change applies with the option off too: UpdateBoundaries, BlobGarbageMeter and the compaction iterator fully validate wide-column entities with blob columns and their blob indexes (see "Separate change" above). - Exact reference metadata has one format (version 1) with a fixed 72-byte descriptor. The decoders reject any other version as Corruption. - The metadata blocks are protected by the table's block checksum only. With checksum = kNoChecksum, corrupt metadata is detected only when it breaks the format or order checks. - The index decoder checks each decoded count against the remaining input size before it reserves memory. So a small corrupt index cannot make it allocate a large vector. - Inlined-TTL and same-file blob indexes are not collected and cannot be relocated. Encoding, count and checksum errors return an error; a bad reference is never skipped. - Option gate (see Terms): outside tests no DB writes data in the new format. The gate checks only the option value, never persisted state. Diff 11 removes it, once data written with the option stays readable through the end of the stack and the safety checks for turning the option on are in place. - Downgrading the binary after data was written with the option on is not supported. Validation and known limitations: - Unit tests cover the BlobIndex encodings and their rejection of bad input, the metadata layout and every corruption check, the collector (sorted path, unsorted fallback, memory charge), the flush integration, the option checks and option gate, wide-column validation, and canonical varints. See the Test Plan. - The collection buffer is not spilled to disk. One SST with very many blob references needs a matching amount of charged memory: the per-reference amount given under "Collection and memory", plus up to one partly used 512-byte chunk per group. While a fallback group is sorted, it also needs 16 bytes of sort key per reference and a second copy of its entries. For a test group of 200K references with every other one out of order, the test comment estimates about 44 bytes per reference, and the test checks that the peak charge is at most 56 bytes per reference. - Flush still writes a single output file. Until diff 7, one compaction output collects every reference it gets, charged to the block cache. Diff 7 cuts compaction outputs (including L0) at a reference cap and keeps each user key's history in one output. With the option gate, a large output is reachable only through the test hook. Stack context (same in diffs 2-11, except that diffs 8-11 add the diff 8 route rule to the install line; "<== this diff" marks this diff's part): Goal: remove garbage from blob files without rewriting any SST. A value gets a stable identity when it is first written. GC copies the live values of whole files into a relocation file and installs routes in the MANIFEST. Readers follow the routes. SSTs are not changed. How a read finds a value (P is the relocation file that holds the value now): ``` SST entry indirect BlobIndex { generation number, origin, original offset, size, CRCs } | v Version file table: generation number -> live file, or route old file -> P | | live file route found | | v v blob file: read at original offset relocation file P (always one hop): relocation file: look up key look up key (origin, original offset) (origin, original offset) ``` Lifecycle of one value (in this example, F is the blob file where the value is first written, and P is the relocation file that GC writes): ``` write new value goes to blob file F; index: origin = F, generation number = F | census per generation number: the SSTs and references that use it; | per physical file: live bytes, ranked by garbage ratio GC job copy the live values of whole input files (stored bytes, checked | with the stored CRC) into relocation file P, keyed by | (origin, original offset) install one MANIFEST edit: add P, route each input generation number still | in use to P, re-point the routes into the inputs to P, and remove | each input (blob file retirement for a blob file, relocation | file deletion for a relocation file) flatten compaction sets generation number = P in the SSTs it rewrites | route route retirement: route deletion records remove the routes that delete no SST uses ``` Stack map: | Diff | Adds | |---|---| | 1 | Finalize blob metadata outside the DB mutex (prep) | | 2 | Indirect BlobIndex (kIndirectBlob) and exact SST reference metadata <== this diff | | 3 | Relocation file format and stable read path | | 4 | MANIFEST routes and Version ownership | | 5 | Exact reverse reference census | | 6 | Standalone relocation GC (one input file) | | 7 | Pack whole input files; reclaim relocation files | | 8 | Flatten generation numbers; route retirement | | 9 | Remote compaction validation | | 10 | Dynamic options and operations | | 11 | RepairDB rebuilds relocation state; option gate removed | | 12 | db_stress and crash test support | Stack context: terms for later diffs: | Term | Meaning | |---|---| | census | Background count, per generation number, of the SSTs and references that use it, plus live bytes per physical file (diff 5) | | indirect origin | Normal blob file written with the option on; it is the origin of unflattened references | | flatten | Compaction rewrites the generation number to the relocation file that holds the value, so old routes can go. Only flatten makes generation number != origin (diff 8) | | blob file retirement | MANIFEST record (kBlobFileRetirement) that removes a collected blob file from the Version (diff 4) | | route deletion | MANIFEST record that removes a route that no SST uses. Route retirement is the background job that writes these records (diff 8) | Tracking issue: meta#15301 Differential Revision: D124652097
Summary: Pull Request resolved: meta#15540 Why the change is required: The goal of this stack is a blob GC that does not depend on SSTs: it removes garbage from blob files without rewriting any SST. Such a GC needs two things that RocksDB does not have: - A value identity that does not change when the value moves to another file. - A checked list of the blob references that each SST keeps. The list must include older internal versions of a key that the SST still keeps (for example for snapshots), and the blob columns of wide-column entities. This diff adds both. Later diffs build the census (an in-memory reverse index from generation number to the SSTs that use it), GC and repair on top of this list. This diff in one paragraph: With the new column family option enable_blob_indirection on, the blob file writer gives each new value an indirect BlobIndex (type 3, kIndirectBlob). It stores the value's stable identity (origin file, original offset) and its CRCs (an uncompressed value stores only the CRC of the uncompressed value; a compressed value also stores the CRC of its stored bytes). Also with the option on, the block-based table builder collects every blob reference that the SST keeps. It writes them as "exact reference metadata": data blocks, one reference index block, and a 72-byte descriptor table property. In this diff only the table builder writes this metadata, and only tests read it. A temporary option gate (see "option gate" under Terms) keeps the option off outside tests. Separately, wide-column entities with blob columns are now fully checked before their blob indexes are used, also with the option off. Parts of this diff: 1. Indirect BlobIndex (core): types 3 and 4 in db/blob/blob_index.h, and the blob file writer that writes type 3. 2. Exact reference metadata format and decoders (core): db/blob/blob_reference_metadata.{h,cc}. 3. Collector and table builder integration (core): BlobReferenceMetadataCollector, its block cache charge, and BlockBasedTableBuilder. 4. Option enable_blob_indirection and the option gate. Tests need it to turn the feature on. You can skip this part on a first read. 5. Wide-column validation (WideColumnSerialization::ForEachBlobIndex). The table builder uses it to give the blob columns of each entity to the collector. It also changes behavior with the option off (see New behavior). You can skip this part on a first read. 6. Small helpers: canonical varints (util/coding.h) and CacheReservationHandle::Resize(). You can skip these on a first read. How to read this diff: 1. Overview: read "Why" and "This diff in one paragraph" above, and "Terms used in this diff" right below. Then read the top comment of db/blob/blob_reference_metadata.h (terms, layout diagrams, run and entry layout, index layout). 2. Indirect BlobIndex (db/blob/blob_index.h) and canonical varints (util/coding.h GetCanonicalVarint32/64): read the overview comment and the kIndirectBlob / kIndirectFlattenedBlob code. Every later part uses these fields: origin, original offset, generation number, logical CRC and stored CRC. The new formats read varints only in the canonical form. 3. Blob file writer (db/blob/blob_file_builder.cc, blob_log_format.cc, blob_log_writer.cc): see how a new value gets a type-3 index and how its CRC is computed once. 4. Format types and decoders: the structs in db/blob/blob_reference_metadata.h, then in the .cc the constants, GapBase, DecodeEntryBody, DecodeRunEntriesAt, DecodeBlobReferenceIndex and BlobReferenceGroupDecoder. They use only BlobIndex and the canonical varints. Learn the on-disk format here, before you read the writer. 5. Collector and its cache charge: first read the new CacheEntryRole kBlobReferenceMetadataBuildingBuffer (include/rocksdb/cache.h) and CacheReservationHandle::Resize (cache/cache_reservation_manager.h). Then read BlobReferenceMetadataCollector in this order: the class comment, Add, AppendSorted, BuildAndWriteDataBlocks, EncodeIndex, FinishOnce. Then read the unsorted fallback (AppendUnsorted, SortUnsortedGroup, SortAndEncodeGroup). 6. Wide-column change (db/wide/wide_column_serialization.cc, ForEachBlobIndex and WalkV2Columns): it gives every blob index of an entity to a callback after it checks the whole entity. The table builder uses it to feed the collector. UpdateBoundaries, BlobGarbageMeter and the compaction iterator also use it, with the option on or off. 7. Option: enable_blob_indirection (include/rocksdb/advanced_options.h, options/cf_options.cc), and the option checks and gate in db/column_family.cc (ValidateOptions, TEST_SetAllowBlobIndirection), db/db_impl/db_impl_open.cc and db_impl_secondary.cc (the FIFO check for opens that skip ValidateOptions). 8. Table integration: table/block_based/block_based_table_factory.cc (the shared reservation manager), then block_based_table_builder.cc (Rep constructor, Add, WriteBlobReferenceMetadata, EstimatedTailSize). See where the collector is created, fed and finished. block_based_table_reader.cc maps the new block name to a block type. Terms used in this diff: Stack terms: | Term | Meaning | |---|---| | origin | Blob file where the value was first written | | original offset | Value offset in the origin file | | stable identity | (origin, original offset). Never changes, even when the value moves | | generation number | The file number that a reference uses to find its value. A new value has generation number == origin. When GC moves the value, the reference keeps its generation number, and a route sends the read to the relocation file | | indirect BlobIndex | A kIndirectBlob (type 3) or kIndirectFlattenedBlob (type 4) index. It stores the stable identity, so the value can move without an SST rewrite | | unflattened reference | Index with generation number == origin: the value was not moved, or it was moved but no compaction has rewritten its reference yet (see "flatten" in "Stack context: terms for later diffs") | | flattened reference | Index with generation number != origin (type 4). Only flatten (diff 8) writes it; this diff defines the encoding | | route | Maps an old file number (a file that GC has removed from the Version) to the relocation file that now holds its live values. One hop: a target is never itself routed. Stored in the MANIFEST (diff 4) | | relocation file | Block-based table keyed by (origin, original offset) that holds copied values. Each value is the stored bytes plus a 5-byte trailer (kSimpleGen2BlobTrailerSize) | | logical CRC | CRC32C of the uncompressed value (BlobIndex::stable_checksum()) | | stored CRC | CRC32C of the stored bytes plus the compression byte | | physical file | A file that holds value bytes: a normal blob file or a relocation file. A generation number can name a physical file, or it can be only the key of a route | | exact reference metadata | Per-SST sorted list of every blob reference the SST keeps, written when the SST is built. References with the same encoding class, generation number and origin form a group. The list is stored as data blocks, with an index of the groups. An SST that has it is an exact SST | | option gate | Until diff 11, ColumnFamilyData::ValidateOptions returns NotSupported for enable_blob_indirection on a read-write open, CreateColumnFamily and SetOptions, unless a test calls ColumnFamilyData::TEST_SetAllowBlobIndirection(true). It keeps the option off in production while the stack is not complete | Terms of this diff (all defined at the top of db/blob/blob_reference_metadata.h): | Term | Meaning | |---|---| | encoding class (class) | kDirect (a legacy kBlob or kBlobTTL index), unflattened, or flattened | | reference group (group) | All references of one SST with the same (encoding class, generation number, origin) | | run | The part of one group inside one data block | | record bytes | The bytes the value's record takes in the file named by its generation number | | gap | How far a record is from where it would be if the records were back to back. A gap of 0 means back to back | | descriptor | The 72-byte table property that points to the reference index block and holds the totals, the CF ID, the SST file number and the relocation-state fingerprint | | relocation-state fingerprint | 128-bit hash of the routes and relocation files of the Version that an SST was built from (fields route_snapshot_fingerprint_*). Written as 0 in this diff | | legacy | Written the way RocksDB writes it without blob indirection | Previous behavior: - A BlobIndex named only a physical blob file and an offset. A value could not move to another file without rewriting every SST that referenced it. - An SST had no metadata that listed its blob references. The only way to find them was to scan its user keys. - These readers get the blob indexes of a wide-column entity through ForEachBlobFileNumber: UpdateBoundaries, BlobGarbageMeter, the compaction iterator, WideColumnSerialization::RemoveBlobColumns, and the referenced_blob_bytes table property count in BlockBasedTableBuilder::Add. ForEachBlobFileNumber decoded each blob index, but it did not check the rest of the entity framing (name order, section sizes, exact input size), and it could call the callback before it found a later error. New behavior: Separate change (option on or off): - The same readers (UpdateBoundaries, BlobGarbageMeter, the compaction iterator, RemoveBlobColumns, and the referenced_blob_bytes count in BlockBasedTableBuilder::Add) use WideColumnSerialization::ForEachBlobIndex, with the option on or off. With the option on, the table builder also uses it to give the blob columns of each entity to the collector. For an entity with blob columns, it checks the full framing, then decodes and validates every blob index before any callback runs. So an entity with a malformed framing now fails the flush or compaction, even with the option off. The referenced_blob_bytes count itself does not fail the build on a malformed entity. It removes that property from the SST, as in base. Option and option gate: - This diff adds a new column family option, `enable_blob_indirection`. It is off by default, and SetOptions can change it. - The option gate (see Terms) keeps the option off outside tests until the stack is complete. ColumnFamilyData::ValidateOptions rejects the option with NotSupported on a read-write open, CreateColumnFamily and SetOptions. Read-only opens (DBImplReadOnly and CompactedDBImpl) and secondary opens (including DB::OpenAndCompact) do not run ValidateOptions; they check only the FIFO rule (see Implementation). Only tests can turn the option on, through the internal ColumnFamilyData::TEST_SetAllowBlobIndirection(). DBTestBase sets it. - When the option is on, ValidateOptions also requires enable_blob_files, the block-based table format, no blob direct write, and a compaction style other than FIFO. Blob file writer (option on): - The blob file writer encodes each new value with a stable identity (origin file, original offset) and two checksums: - the logical CRC: the CRC32C of the uncompressed value. - the stored CRC: the CRC32C of the exact stored (maybe compressed) bytes, extended with the compression byte. - For an uncompressed value the stored CRC is not written, because the reader can compute it from the logical CRC. - The checksums have no version field of their own. The BlobIndex type byte gives the format version: the two new types, 3 (kIndirectBlob) and 4 (kIndirectFlattenedBlob, see Implementation), are BlobIndex::kIndirectBlobFormatVersion (1). A later format uses new type values. - The read path does not check these checksums yet. The check starts in diff 3. Exact reference metadata (option on): 1. What: a sorted list of every blob reference that the SST keeps, in plain blob values and in the blob columns of wide-column entities. 2. Where: data blocks (target 16 KB each before compression; small blocks waste less I/O, because GC reads a whole block to get a few groups) and one reference index block, written after the SST data blocks and before the SST tail, plus a 72-byte descriptor table property. 3. Who reads it: only tests in this diff. The census, GC and RepairDB read it in later diffs. 4. Details: the top comment of db/blob/blob_reference_metadata.h has the full format with diagrams, the record-bytes rules and the gap rule. - Not collected: SstFileWriter output (it has no DB file number) and any output whose CF is unknown (it has no CF ID). The descriptor needs both values, so these outputs stay in the legacy form. Checks on the metadata: - A repeated or overlapping reference in one group fails the SST build with Corruption. Readers check every count, order, size and total against the index and the descriptor, so corrupt metadata returns Corruption. The writer never makes a data block larger than kTargetDataBlockSize (16 KB), and readers accept up to kMaxDataBlockSize (64 KB). Details: "Readers" and "Integrity" at the top of db/blob/blob_reference_metadata.h, and BlobReferenceGroupDecoder. Collection and memory: - Each reference is encoded when it is added, in the on-disk entry encoding, into its group's chain of 512-byte chunks. Only a group that takes the unsorted fallback is encoded a second time, in Finish(). Inside a group, references normally arrive in rising offset order, so no reference is sorted; at the end only the groups are sorted. A group that gets a lower offset takes the unsorted fallback. The BlobReferenceMetadataCollector class comment explains the chunks, the chunk size and the fallback. - The collector keeps every reference of the SST in memory until the SST is finished, and charges this memory to the block cache under a new CacheEntryRole, kBlobReferenceMetadataBuildingBuffer. The measured cost is about 8.3 to 9 bytes per reference for uncompressed values and 12.8 to 13.5 bytes for compressed values. "Memory usage" and "Memory charge" in the BlobReferenceMetadataCollector class comment give the measurement setup and the charge rule. - There is no fixed limit on references. The collector fails with MemoryLimit above 2^20 groups or 2^20 data blocks in one SST; these are persisted format limits. - There is no charge when the table has no block cache or when charging for this role is disabled. Each block-based table factory holds one cache reservation manager for this role, shared by its builders and the SST summaries. If the charge fails, the SST build fails, so the flush or compaction fails with a background error. The buffers and their charge are released when the SST is finished or abandoned. Implementation: - BlobIndex adds two indirect encodings. The legacy encodings do not change. - Type 3, kIndirectBlob (unflattened, generation number == origin), stores the origin, the original offset, the stored size, the compression type, the logical CRC and, for compressed values, the stored CRC. It does not store the generation number. This saves the size of one varint file number per reference (about 3 to 4 bytes for typical file numbers; an estimate, not measured). - Type 4, kIndirectFlattenedBlob (flattened), stores the generation number first, then the same fields. Its generation number must be above the origin. The decoder rejects type 4 with generation number <= origin, so each reference has one encoding. - The encoders take the fields as one BlobIndex::IndirectBlobFields, set by field name, so values of the same type (file number, offset and size; the two CRCs) cannot be swapped. EncodeIndirectBlob() takes the generation number as a separate argument; EncodeUnflattenedIndirectBlob() uses the origin, so the common path cannot leave the generation number unset. - The new files blob_reference_metadata.{h,cc} define the exact reference metadata. The files also contain: - BlobReferenceMetadataCollector, the collector described above. It takes the SST identity (CF ID, SST file number and the relocation-state fingerprint) as one BlobReferenceSstIdentity, set by field name. - BlobReferenceGroupKey, the key of a group: (encoding class, generation number, origin). It has the order, the hash and the validity check that the collector map, the sort, the decoder and FindBlobReferenceGroup() use. BlobReference and BlobReferenceGroup keep their flat fields and return the key through group_key(). - The index codec and the descriptor table property. - BlobReferenceGroupDecoder, the only decoder of reference runs. It reads a group one data block at a time and holds every corruption check of a group walk. Readers that visit one group (found by a binary search of the index) use it, and so do readers that visit every block in order. - Callers in this diff: only the table builder runs the collector. The decoders (DecodeBlobReferenceIndex, FindBlobReferenceGroup, BlobReferenceGroupDecoder, ForEachBlobReference*) are used only by tests. kMaxDecodedBlobReferenceBytes is used only in compile-time checks of the count limits. The census (diff 5) and GC (diffs 6 and 7) are their first production users. The census is also the first user of BlobReferenceSstSummary (see the last Implementation bullet). - Descriptor: the CF ID and the SST file number are the repair fields. They let RepairDB tell which column family and SST the list belongs to without a MANIFEST. The decoder rejects a descriptor whose repair fields are not set (unknown CF ID or SST file number 0). The descriptor also has a 128-bit relocation-state fingerprint (the route_snapshot_fingerprint fields). Remote compaction uses it in diff 9 to check that an SST built elsewhere used the same routes as the job that asked for it. In this diff the table builder always writes 0, and DecodeFrom() does not check it. The field is in the format now, so the descriptor keeps one format version (1) through the stack. While the option gate is on, no data in this format exists outside tests. - The collector validates each BlobIndex once, when it is added. Its Finish() writes the data blocks and then the reference index block through one block writer, and fills the descriptor, including the index handle. Destroying the collector frees its memory and then its charge, also when the build is abandoned. - BlockBasedTableBuilder creates the collector with the CF ID and SST file number from TableBuilderOptions. It passes the size of each entry's user key to the collector, so the collector can compute the exact size of each blob record. It adds the collector's EstimatedEncodedSize() to its tail size estimate: the blocks are written before the tail, but output file size planning uses this estimate. The reference blocks are written with BlockType::kUserDefinedIndex, which keeps their contents as opaque bytes. Readers read them without the block cache, so prepopulate_block_cache does not warm them into the block cache (WriteMaybeCompressedBlock() with warm_cache false). - With the option on, the blob file builder computes the CRC32C of each stored value once. The blob log record CRC combines it with the key CRC (crc32c::Crc32cCombine). The stored CRC extends it with the compression byte. For an uncompressed value it is also the logical CRC. The bytes written do not change. - WideColumnSerialization::ForEachBlobIndex replaces ForEachBlobFileNumber. It reads the entity header and column types once. For a V1 entity (no column types and no blob columns), it returns right after the header. For a V2 entity without blob columns, it returns right after the type scan. It shares one V2 column walker (WalkV2Columns) with Deserialize(), so the entity is parsed in one pass. - Read-only opens (DBImplReadOnly, CompactedDBImpl) and secondary opens skip ValidateOptions, so they check the compaction style against the option once per column family: in DBImpl::Recover() for read-only opens, and at the start of OpenAsSecondaryImpl() for secondary opens. DB::OpenAndCompact opens through OpenAsSecondaryImpl(), so it gets the same check. - CacheReservationHandle gets Resize(), so each collector or summary can change its own part of the shared charge. - util/coding.h adds GetCanonicalVarint32/64, which reject overlong varint encodings. The new formats use them, so each value has one encoding. - BlobReferenceSstSummary is the in-memory list of an SST's groups and totals that Finish() can return. It has no caller yet: the table builder passes a null summary output. It is checked and does not change after Finish(), so the finished builder, the SST metadata and the census (its first user, in diff 5) can share it. It is in this diff because Finish() hands the group buffer and its cache charge over to the summary. Adding the summary later would mean rewriting the charge code of Finish() in diff 5, so the charge code is written and tested once, here. Safety boundaries and unsupported cases: - Public API: this diff adds the option `enable_blob_indirection` with the C API functions `rocksdb_options_set_enable_blob_indirection` and `rocksdb_options_get_enable_blob_indirection`. It also inserts a new CacheEntryRole value before kMisc, which changes the numeric value of kMisc and kNumCacheEntryRoles. - With the option off, new blob values and SSTs are written in the legacy form. Legacy blob indexes stay readable. - One behavior change applies with the option off too: UpdateBoundaries, BlobGarbageMeter and the compaction iterator fully validate wide-column entities with blob columns and their blob indexes (see "Separate change" above). - Exact reference metadata has one format (version 1) with a fixed 72-byte descriptor. The decoders reject any other version as Corruption. - The metadata blocks are protected by the table's block checksum only. With checksum = kNoChecksum, corrupt metadata is detected only when it breaks the format or order checks. - The index decoder checks each decoded count against the remaining input size before it reserves memory. So a small corrupt index cannot make it allocate a large vector. - Inlined-TTL and same-file blob indexes are not collected and cannot be relocated. Encoding, count and checksum errors return an error; a bad reference is never skipped. - Option gate (see Terms): outside tests no DB writes data in the new format. The gate checks only the option value, never persisted state. Diff 11 removes it, once data written with the option stays readable through the end of the stack and the safety checks for turning the option on are in place. - Downgrading the binary after data was written with the option on is not supported. Validation and known limitations: - Unit tests cover the BlobIndex encodings and their rejection of bad input, the metadata layout and every corruption check, the collector (sorted path, unsorted fallback, memory charge), the flush integration, the option checks and option gate, wide-column validation, and canonical varints. See the Test Plan. - The collection buffer is not spilled to disk. One SST with very many blob references needs a matching amount of charged memory: the per-reference amount given under "Collection and memory", plus up to one partly used 512-byte chunk per group. While a fallback group is sorted, it also needs 16 bytes of sort key per reference and a second copy of its entries. For a test group of 200K references with every other one out of order, the test comment estimates about 44 bytes per reference, and the test checks that the peak charge is at most 56 bytes per reference. - Flush still writes a single output file. Until diff 7, one compaction output collects every reference it gets, charged to the block cache. Diff 7 cuts compaction outputs (including L0) at a reference cap and keeps each user key's history in one output. With the option gate, a large output is reachable only through the test hook. Stack context (same in diffs 2-11, except that diffs 8-11 add the diff 8 route rule to the install line; "<== this diff" marks this diff's part): Goal: remove garbage from blob files without rewriting any SST. A value gets a stable identity when it is first written. GC copies the live values of whole files into a relocation file and installs routes in the MANIFEST. Readers follow the routes. SSTs are not changed. How a read finds a value (P is the relocation file that holds the value now): ``` SST entry indirect BlobIndex { generation number, origin, original offset, size, CRCs } | v Version file table: generation number -> live file, or route old file -> P | | live file route found | | v v blob file: read at original offset relocation file P (always one hop): relocation file: look up key look up key (origin, original offset) (origin, original offset) ``` Lifecycle of one value (in this example, F is the blob file where the value is first written, and P is the relocation file that GC writes): ``` write new value goes to blob file F; index: origin = F, generation number = F | census per generation number: the SSTs and references that use it; | per physical file: live bytes, ranked by garbage ratio GC job copy the live values of whole input files (stored bytes, checked | with the stored CRC) into relocation file P, keyed by | (origin, original offset) install one MANIFEST edit: add P, route each input generation number still | in use to P, re-point the routes into the inputs to P, and remove | each input (blob file retirement for a blob file, relocation | file deletion for a relocation file) flatten compaction sets generation number = P in the SSTs it rewrites | route route retirement: route deletion records remove the routes that delete no SST uses ``` Stack map: | Diff | Adds | |---|---| | 1 | Finalize blob metadata outside the DB mutex (prep) | | 2 | Indirect BlobIndex (kIndirectBlob) and exact SST reference metadata <== this diff | | 3 | Relocation file format and stable read path | | 4 | MANIFEST routes and Version ownership | | 5 | Exact reverse reference census | | 6 | Standalone relocation GC (one input file) | | 7 | Pack whole input files; reclaim relocation files | | 8 | Flatten generation numbers; route retirement | | 9 | Remote compaction validation | | 10 | Dynamic options and operations | | 11 | RepairDB rebuilds relocation state; option gate removed | | 12 | db_stress and crash test support | Stack context: terms for later diffs: | Term | Meaning | |---|---| | census | Background count, per generation number, of the SSTs and references that use it, plus live bytes per physical file (diff 5) | | indirect origin | Normal blob file written with the option on; it is the origin of unflattened references | | flatten | Compaction rewrites the generation number to the relocation file that holds the value, so old routes can go. Only flatten makes generation number != origin (diff 8) | | blob file retirement | MANIFEST record (kBlobFileRetirement) that removes a collected blob file from the Version (diff 4) | | route deletion | MANIFEST record that removes a route that no SST uses. Route retirement is the background job that writes these records (diff 8) | Tracking issue: meta#15301 Differential Revision: D124652097
Summary: Pull Request resolved: meta#15540 Why the change is required: The goal of this stack is a blob GC that does not depend on SSTs: it removes garbage from blob files without rewriting any SST. Such a GC needs two things that RocksDB does not have: - A value identity that does not change when the value moves to another file. - A checked list of the blob references that each SST keeps. The list must include older internal versions of a key that the SST still keeps (for example for snapshots), and the blob columns of wide-column entities. This diff adds both. Later diffs build the census (an in-memory reverse index from generation number to the SSTs that use it), GC and repair on top of this list. This diff in one paragraph: With the new column family option enable_blob_indirection on, the blob file writer gives each new value an indirect BlobIndex (type 3, kIndirectBlob). It stores the value's stable identity (origin file, original offset) and its CRCs (an uncompressed value stores only the CRC of the uncompressed value; a compressed value also stores the CRC of its stored bytes). Also with the option on, the block-based table builder collects every blob reference that the SST keeps. It writes them as "exact reference metadata": data blocks, one reference index block, and a 72-byte descriptor table property. In this diff only the table builder writes this metadata, and only tests read it. A temporary option gate (see "option gate" under Terms) keeps the option off outside tests. Separately, wide-column entities with blob columns are now fully checked before their blob indexes are used, also with the option off. Parts of this diff: 1. Indirect BlobIndex (core): types 3 and 4 in db/blob/blob_index.h, and the blob file writer that writes type 3. 2. Exact reference metadata format and decoders (core): db/blob/blob_reference_metadata.{h,cc}. 3. Collector and table builder integration (core): BlobReferenceMetadataCollector, its block cache charge, and BlockBasedTableBuilder. 4. Option enable_blob_indirection and the option gate. Tests need it to turn the feature on. You can skip this part on a first read. 5. Wide-column validation (WideColumnSerialization::ForEachBlobIndex). The table builder uses it to give the blob columns of each entity to the collector. It also changes behavior with the option off (see New behavior). You can skip this part on a first read. 6. Small helpers: canonical varints (util/coding.h) and CacheReservationHandle::Resize(). You can skip these on a first read. How to read this diff: 1. Overview: read "Why" and "This diff in one paragraph" above, and "Terms used in this diff" right below. Then read the top comment of db/blob/blob_reference_metadata.h (terms, layout diagrams, run and entry layout, index layout). 2. Indirect BlobIndex (db/blob/blob_index.h) and canonical varints (util/coding.h GetCanonicalVarint32/64): read the overview comment and the kIndirectBlob / kIndirectFlattenedBlob code. Every later part uses these fields: origin, original offset, generation number, logical CRC and stored CRC. The new formats read varints only in the canonical form. 3. Blob file writer (db/blob/blob_file_builder.cc, blob_log_format.cc, blob_log_writer.cc): see how a new value gets a type-3 index and how its CRC is computed once. 4. Format types and decoders: the structs in db/blob/blob_reference_metadata.h, then in the .cc the constants, GapBase, DecodeEntryBody, DecodeRunEntriesAt, DecodeBlobReferenceIndex and BlobReferenceGroupDecoder. They use only BlobIndex and the canonical varints. Learn the on-disk format here, before you read the writer. 5. Collector and its cache charge: first read the new CacheEntryRole kBlobReferenceMetadataBuildingBuffer (include/rocksdb/cache.h) and CacheReservationHandle::Resize (cache/cache_reservation_manager.h). Then read BlobReferenceMetadataCollector in this order: the class comment, Add, AppendSorted, BuildAndWriteDataBlocks, EncodeIndex, FinishOnce. Then read the unsorted fallback (AppendUnsorted, SortUnsortedGroup, SortAndEncodeGroup). 6. Wide-column change (db/wide/wide_column_serialization.cc, ForEachBlobIndex and WalkV2Columns): it gives every blob index of an entity to a callback after it checks the whole entity. The table builder uses it to feed the collector. UpdateBoundaries, BlobGarbageMeter and the compaction iterator also use it, with the option on or off. 7. Option: enable_blob_indirection (include/rocksdb/advanced_options.h, options/cf_options.cc), and the option checks and gate in db/column_family.cc (ValidateOptions, TEST_SetAllowBlobIndirection), db/db_impl/db_impl_open.cc and db_impl_secondary.cc (the FIFO check for opens that skip ValidateOptions). 8. Table integration: table/block_based/block_based_table_factory.cc (the shared reservation manager), then block_based_table_builder.cc (Rep constructor, Add, WriteBlobReferenceMetadata, EstimatedTailSize). See where the collector is created, fed and finished. block_based_table_reader.cc maps the new block name to a block type. Terms used in this diff: Stack terms: | Term | Meaning | |---|---| | origin | Blob file where the value was first written | | original offset | Value offset in the origin file | | stable identity | (origin, original offset). Never changes, even when the value moves | | generation number | The file number that a reference uses to find its value. A new value has generation number == origin. When GC moves the value, the reference keeps its generation number, and a route sends the read to the relocation file | | indirect BlobIndex | A kIndirectBlob (type 3) or kIndirectFlattenedBlob (type 4) index. It stores the stable identity, so the value can move without an SST rewrite | | unflattened reference | Index with generation number == origin: the value was not moved, or it was moved but no compaction has rewritten its reference yet (see "flatten" in "Stack context: terms for later diffs") | | flattened reference | Index with generation number != origin (type 4). Only flatten (diff 8) writes it; this diff defines the encoding | | route | Maps an old file number (a file that GC has removed from the Version) to the relocation file that now holds its live values. One hop: a target is never itself routed. Stored in the MANIFEST (diff 4) | | relocation file | Block-based table keyed by (origin, original offset) that holds copied values. Each value is the stored bytes plus a 5-byte trailer (kSimpleGen2BlobTrailerSize) | | logical CRC | CRC32C of the uncompressed value (BlobIndex::stable_checksum()) | | stored CRC | CRC32C of the stored bytes plus the compression byte | | physical file | A file that holds value bytes: a normal blob file or a relocation file. A generation number can name a physical file, or it can be only the key of a route | | exact reference metadata | Per-SST sorted list of every blob reference the SST keeps, written when the SST is built. References with the same encoding class, generation number and origin form a group. The list is stored as data blocks, with an index of the groups. An SST that has it is an exact SST | | option gate | Until diff 11, ColumnFamilyData::ValidateOptions returns NotSupported for enable_blob_indirection on a read-write open, CreateColumnFamily and SetOptions, unless a test calls ColumnFamilyData::TEST_SetAllowBlobIndirection(true). It keeps the option off in production while the stack is not complete | Terms of this diff (all defined at the top of db/blob/blob_reference_metadata.h): | Term | Meaning | |---|---| | encoding class (class) | kDirect (a legacy kBlob or kBlobTTL index), unflattened, or flattened | | reference group (group) | All references of one SST with the same (encoding class, generation number, origin) | | run | The part of one group inside one data block | | record bytes | The bytes the value's record takes in the file named by its generation number | | gap | How far a record is from where it would be if the records were back to back. A gap of 0 means back to back | | descriptor | The 72-byte table property that points to the reference index block and holds the totals, the CF ID, the SST file number and the relocation-state fingerprint | | relocation-state fingerprint | 128-bit hash of the routes and relocation files of the Version that an SST was built from (fields route_snapshot_fingerprint_*). Written as 0 in this diff | | legacy | Written the way RocksDB writes it without blob indirection | Previous behavior: - A BlobIndex named only a physical blob file and an offset. A value could not move to another file without rewriting every SST that referenced it. - An SST had no metadata that listed its blob references. The only way to find them was to scan its user keys. - These readers get the blob indexes of a wide-column entity through ForEachBlobFileNumber: UpdateBoundaries, BlobGarbageMeter, the compaction iterator, WideColumnSerialization::RemoveBlobColumns, and the referenced_blob_bytes table property count in BlockBasedTableBuilder::Add. ForEachBlobFileNumber decoded each blob index, but it did not check the rest of the entity framing (name order, section sizes, exact input size), and it could call the callback before it found a later error. New behavior: Separate change (option on or off): - The same readers (UpdateBoundaries, BlobGarbageMeter, the compaction iterator, RemoveBlobColumns, and the referenced_blob_bytes count in BlockBasedTableBuilder::Add) use WideColumnSerialization::ForEachBlobIndex, with the option on or off. With the option on, the table builder also uses it to give the blob columns of each entity to the collector. For an entity with blob columns, it checks the full framing, then decodes and validates every blob index before any callback runs. So an entity with a malformed framing now fails the flush or compaction, even with the option off. The referenced_blob_bytes count itself does not fail the build on a malformed entity. It removes that property from the SST, as in base. Option and option gate: - This diff adds a new column family option, `enable_blob_indirection`. It is off by default, and SetOptions can change it. - The option gate (see Terms) keeps the option off outside tests until the stack is complete. ColumnFamilyData::ValidateOptions rejects the option with NotSupported on a read-write open, CreateColumnFamily and SetOptions. Read-only opens (DBImplReadOnly and CompactedDBImpl) and secondary opens (including DB::OpenAndCompact) do not run ValidateOptions; they check only the FIFO rule (see Implementation). Only tests can turn the option on, through the internal ColumnFamilyData::TEST_SetAllowBlobIndirection(). DBTestBase sets it. - When the option is on, ValidateOptions also requires enable_blob_files, the block-based table format, no blob direct write, and a compaction style other than FIFO. Blob file writer (option on): - The blob file writer encodes each new value with a stable identity (origin file, original offset) and two checksums: - the logical CRC: the CRC32C of the uncompressed value. - the stored CRC: the CRC32C of the exact stored (maybe compressed) bytes, extended with the compression byte. - For an uncompressed value the stored CRC is not written, because the reader can compute it from the logical CRC. - The checksums have no version field of their own. The BlobIndex type byte gives the format version: the two new types, 3 (kIndirectBlob) and 4 (kIndirectFlattenedBlob, see Implementation), are BlobIndex::kIndirectBlobFormatVersion (1). A later format uses new type values. - The read path does not check these checksums yet. The check starts in diff 3. Exact reference metadata (option on): 1. What: a sorted list of every blob reference that the SST keeps, in plain blob values and in the blob columns of wide-column entities. 2. Where: data blocks (target 16 KB each before compression; small blocks waste less I/O, because GC reads a whole block to get a few groups) and one reference index block, written after the SST data blocks and before the SST tail, plus a 72-byte descriptor table property. 3. Who reads it: only tests in this diff. The census, GC and RepairDB read it in later diffs. 4. Details: the top comment of db/blob/blob_reference_metadata.h has the full format with diagrams, the record-bytes rules and the gap rule. - Not collected: SstFileWriter output (it has no DB file number) and any output whose CF is unknown (it has no CF ID). The descriptor needs both values, so these outputs stay in the legacy form. Checks on the metadata: - A repeated or overlapping reference in one group fails the SST build with Corruption. Readers check every count, order, size and total against the index and the descriptor, so corrupt metadata returns Corruption. The writer never makes a data block larger than kTargetDataBlockSize (16 KB), and readers accept up to kMaxDataBlockSize (64 KB). Details: "Readers" and "Integrity" at the top of db/blob/blob_reference_metadata.h, and BlobReferenceGroupDecoder. Collection and memory: - Each reference is encoded when it is added, in the on-disk entry encoding, into its group's chain of 512-byte chunks. Only a group that takes the unsorted fallback is encoded a second time, in Finish(). Inside a group, references normally arrive in rising offset order, so no reference is sorted; at the end only the groups are sorted. A group that gets a lower offset takes the unsorted fallback. The BlobReferenceMetadataCollector class comment explains the chunks, the chunk size and the fallback. - The collector keeps every reference of the SST in memory until the SST is finished, and charges this memory to the block cache under a new CacheEntryRole, kBlobReferenceMetadataBuildingBuffer. The measured cost is about 8.3 to 9 bytes per reference for uncompressed values and 12.8 to 13.5 bytes for compressed values. "Memory usage" and "Memory charge" in the BlobReferenceMetadataCollector class comment give the measurement setup and the charge rule. - There is no fixed limit on references. The collector fails with MemoryLimit above 2^20 groups or 2^20 data blocks in one SST; these are persisted format limits. - There is no charge when the table has no block cache or when charging for this role is disabled. Each block-based table factory holds one cache reservation manager for this role, shared by its builders and the SST summaries. If the charge fails, the SST build fails, so the flush or compaction fails with a background error. The buffers and their charge are released when the SST is finished or abandoned. Implementation: - BlobIndex adds two indirect encodings. The legacy encodings do not change. - Type 3, kIndirectBlob (unflattened, generation number == origin), stores the origin, the original offset, the stored size, the compression type, the logical CRC and, for compressed values, the stored CRC. It does not store the generation number. This saves the size of one varint file number per reference (about 3 to 4 bytes for typical file numbers; an estimate, not measured). - Type 4, kIndirectFlattenedBlob (flattened), stores the generation number first, then the same fields. Its generation number must be above the origin. The decoder rejects type 4 with generation number <= origin, so each reference has one encoding. - The encoders take the fields as one BlobIndex::IndirectBlobFields, set by field name, so values of the same type (file number, offset and size; the two CRCs) cannot be swapped. EncodeIndirectBlob() takes the generation number as a separate argument; EncodeUnflattenedIndirectBlob() uses the origin, so the common path cannot leave the generation number unset. - The new files blob_reference_metadata.{h,cc} define the exact reference metadata. The files also contain: - BlobReferenceMetadataCollector, the collector described above. It takes the SST identity (CF ID, SST file number and the relocation-state fingerprint) as one BlobReferenceSstIdentity, set by field name. - BlobReferenceGroupKey, the key of a group: (encoding class, generation number, origin). It has the order, the hash and the validity check that the collector map, the sort, the decoder and FindBlobReferenceGroup() use. BlobReference and BlobReferenceGroup keep their flat fields and return the key through group_key(). - The index codec and the descriptor table property. - BlobReferenceGroupDecoder, the only decoder of reference runs. It reads a group one data block at a time and holds every corruption check of a group walk. Readers that visit one group (found by a binary search of the index) use it, and so do readers that visit every block in order. - Callers in this diff: only the table builder runs the collector. The decoders (DecodeBlobReferenceIndex, FindBlobReferenceGroup, BlobReferenceGroupDecoder, ForEachBlobReference*) are used only by tests. kMaxDecodedBlobReferenceBytes is used only in compile-time checks of the count limits. The census (diff 5) and GC (diffs 6 and 7) are their first production users. The census is also the first user of BlobReferenceSstSummary (see the last Implementation bullet). - Descriptor: the CF ID and the SST file number are the repair fields. They let RepairDB tell which column family and SST the list belongs to without a MANIFEST. The decoder rejects a descriptor whose repair fields are not set (unknown CF ID or SST file number 0). The descriptor also has a 128-bit relocation-state fingerprint (the route_snapshot_fingerprint fields). Remote compaction uses it in diff 9 to check that an SST built elsewhere used the same routes as the job that asked for it. In this diff the table builder always writes 0, and DecodeFrom() does not check it. The field is in the format now, so the descriptor keeps one format version (1) through the stack. While the option gate is on, no data in this format exists outside tests. - The collector validates each BlobIndex once, when it is added. Its Finish() writes the data blocks and then the reference index block through one block writer, and fills the descriptor, including the index handle. Destroying the collector frees its memory and then its charge, also when the build is abandoned. - BlockBasedTableBuilder creates the collector with the CF ID and SST file number from TableBuilderOptions. It passes the size of each entry's user key to the collector, so the collector can compute the exact size of each blob record. It adds the collector's EstimatedEncodedSize() to its tail size estimate: the blocks are written before the tail, but output file size planning uses this estimate. The reference blocks are written with BlockType::kUserDefinedIndex, which keeps their contents as opaque bytes. Readers read them without the block cache, so prepopulate_block_cache does not warm them into the block cache (WriteMaybeCompressedBlock() with warm_cache false). - With the option on, the blob file builder computes the CRC32C of each stored value once. The blob log record CRC combines it with the key CRC (crc32c::Crc32cCombine). The stored CRC extends it with the compression byte. For an uncompressed value it is also the logical CRC. The bytes written do not change. - WideColumnSerialization::ForEachBlobIndex replaces ForEachBlobFileNumber. It reads the entity header and column types once. For a V1 entity (no column types and no blob columns), it returns right after the header. For a V2 entity without blob columns, it returns right after the type scan. It shares one V2 column walker (WalkV2Columns) with Deserialize(), so the entity is parsed in one pass. - Read-only opens (DBImplReadOnly, CompactedDBImpl) and secondary opens skip ValidateOptions, so they check the compaction style against the option once per column family: in DBImpl::Recover() for read-only opens, and at the start of OpenAsSecondaryImpl() for secondary opens. DB::OpenAndCompact opens through OpenAsSecondaryImpl(), so it gets the same check. - CacheReservationHandle gets Resize(), so each collector or summary can change its own part of the shared charge. - util/coding.h adds GetCanonicalVarint32/64, which reject overlong varint encodings. The new formats use them, so each value has one encoding. - BlobReferenceSstSummary is the in-memory list of an SST's groups and totals that Finish() can return. It has no caller yet: the table builder passes a null summary output. It is checked and does not change after Finish(), so the finished builder, the SST metadata and the census (its first user, in diff 5) can share it. It is in this diff because Finish() hands the group buffer and its cache charge over to the summary. Adding the summary later would mean rewriting the charge code of Finish() in diff 5, so the charge code is written and tested once, here. Safety boundaries and unsupported cases: - Public API: this diff adds the option `enable_blob_indirection` with the C API functions `rocksdb_options_set_enable_blob_indirection` and `rocksdb_options_get_enable_blob_indirection`. It also inserts a new CacheEntryRole value before kMisc, which changes the numeric value of kMisc and kNumCacheEntryRoles. - With the option off, new blob values and SSTs are written in the legacy form. Legacy blob indexes stay readable. - One behavior change applies with the option off too: UpdateBoundaries, BlobGarbageMeter and the compaction iterator fully validate wide-column entities with blob columns and their blob indexes (see "Separate change" above). - Exact reference metadata has one format (version 1) with a fixed 72-byte descriptor. The decoders reject any other version as Corruption. - The metadata blocks are protected by the table's block checksum only. With checksum = kNoChecksum, corrupt metadata is detected only when it breaks the format or order checks. - The index decoder checks each decoded count against the remaining input size before it reserves memory. So a small corrupt index cannot make it allocate a large vector. - Inlined-TTL and same-file blob indexes are not collected and cannot be relocated. Encoding, count and checksum errors return an error; a bad reference is never skipped. - Option gate (see Terms): outside tests no DB writes data in the new format. The gate checks only the option value, never persisted state. Diff 11 removes it, once data written with the option stays readable through the end of the stack and the safety checks for turning the option on are in place. - Downgrading the binary after data was written with the option on is not supported. Validation and known limitations: - Unit tests cover the BlobIndex encodings and their rejection of bad input, the metadata layout and every corruption check, the collector (sorted path, unsorted fallback, memory charge), the flush integration, the option checks and option gate, wide-column validation, and canonical varints. See the Test Plan. - The collection buffer is not spilled to disk. One SST with very many blob references needs a matching amount of charged memory: the per-reference amount given under "Collection and memory", plus up to one partly used 512-byte chunk per group. While a fallback group is sorted, it also needs 16 bytes of sort key per reference and a second copy of its entries. For a test group of 200K references with every other one out of order, the test comment estimates about 44 bytes per reference, and the test checks that the peak charge is at most 56 bytes per reference. - Flush still writes a single output file. Until diff 7, one compaction output collects every reference it gets, charged to the block cache. Diff 7 cuts compaction outputs (including L0) at a reference cap and keeps each user key's history in one output. With the option gate, a large output is reachable only through the test hook. Stack context (same in diffs 2-11, except that diffs 8-11 add the diff 8 route rule to the install line; "<== this diff" marks this diff's part): Goal: remove garbage from blob files without rewriting any SST. A value gets a stable identity when it is first written. GC copies the live values of whole files into a relocation file and installs routes in the MANIFEST. Readers follow the routes. SSTs are not changed. How a read finds a value (P is the relocation file that holds the value now): ``` SST entry indirect BlobIndex { generation number, origin, original offset, size, CRCs } | v Version file table: generation number -> live file, or route old file -> P | | live file route found | | v v blob file: read at original offset relocation file P (always one hop): relocation file: look up key look up key (origin, original offset) (origin, original offset) ``` Lifecycle of one value (in this example, F is the blob file where the value is first written, and P is the relocation file that GC writes): ``` write new value goes to blob file F; index: origin = F, generation number = F | census per generation number: the SSTs and references that use it; | per physical file: live bytes, ranked by garbage ratio GC job copy the live values of whole input files (stored bytes, checked | with the stored CRC) into relocation file P, keyed by | (origin, original offset) install one MANIFEST edit: add P, route each input generation number still | in use to P, re-point the routes into the inputs to P, and remove | each input (blob file retirement for a blob file, relocation | file deletion for a relocation file) flatten compaction sets generation number = P in the SSTs it rewrites | route route retirement: route deletion records remove the routes that delete no SST uses ``` Stack map: | Diff | Adds | |---|---| | 1 | Finalize blob metadata outside the DB mutex (prep) | | 2 | Indirect BlobIndex (kIndirectBlob) and exact SST reference metadata <== this diff | | 3 | Relocation file format and stable read path | | 4 | MANIFEST routes and Version ownership | | 5 | Exact reverse reference census | | 6 | Standalone relocation GC (one input file) | | 7 | Pack whole input files; reclaim relocation files | | 8 | Flatten generation numbers; route retirement | | 9 | Remote compaction validation | | 10 | Dynamic options and operations | | 11 | RepairDB rebuilds relocation state; option gate removed | | 12 | db_stress and crash test support | Stack context: terms for later diffs: | Term | Meaning | |---|---| | census | Background count, per generation number, of the SSTs and references that use it, plus live bytes per physical file (diff 5) | | indirect origin | Normal blob file written with the option on; it is the origin of unflattened references | | flatten | Compaction rewrites the generation number to the relocation file that holds the value, so old routes can go. Only flatten makes generation number != origin (diff 8) | | blob file retirement | MANIFEST record (kBlobFileRetirement) that removes a collected blob file from the Version (diff 4) | | route deletion | MANIFEST record that removes a route that no SST uses. Route retirement is the background job that writes these records (diff 8) | Tracking issue: meta#15301 Differential Revision: D124652097
Summary: Pull Request resolved: meta#15540 Why the change is required: The goal of this stack is a blob GC that does not depend on SSTs: it removes garbage from blob files without rewriting any SST. Such a GC needs two things that RocksDB does not have: - A value identity that does not change when the value moves to another file. - A checked list of the blob references that each SST keeps. The list must include older internal versions of a key that the SST still keeps (for example for snapshots), and the blob columns of wide-column entities. This diff adds both. Later diffs build the census (an in-memory reverse index from generation number to the SSTs that use it), GC and repair on top of this list. This diff in one paragraph: With the new column family option enable_blob_indirection on, the blob file writer gives each new value an indirect BlobIndex (type 3, kIndirectBlob). It stores the value's stable identity (origin file, original offset) and its CRCs (an uncompressed value stores only the CRC of the uncompressed value; a compressed value also stores the CRC of its stored bytes). Also with the option on, the block-based table builder collects every blob reference that the SST keeps. It writes them as "exact reference metadata": data blocks, one reference index block, and a 72-byte descriptor table property. In this diff only the table builder writes this metadata, and only tests read it. A temporary option gate (see "option gate" under Terms) keeps the option off outside tests. Separately, wide-column entities with blob columns are now fully checked before their blob indexes are used, also with the option off. Parts of this diff: 1. Indirect BlobIndex (core): types 3 and 4 in db/blob/blob_index.h, and the blob file writer that writes type 3. 2. Exact reference metadata format and decoders (core): db/blob/blob_reference_metadata.{h,cc}. 3. Collector and table builder integration (core): BlobReferenceMetadataCollector, its block cache charge, and BlockBasedTableBuilder. 4. Option enable_blob_indirection and the option gate. Tests need it to turn the feature on. You can skip this part on a first read. 5. Wide-column validation (WideColumnSerialization::ForEachBlobIndex). The table builder uses it to give the blob columns of each entity to the collector. It also changes behavior with the option off (see New behavior). You can skip this part on a first read. 6. Small helpers: canonical varints (util/coding.h) and CacheReservationHandle::Resize(). You can skip these on a first read. How to read this diff: 1. Overview: read "Why" and "This diff in one paragraph" above, and "Terms used in this diff" right below. Then read the top comment of db/blob/blob_reference_metadata.h (terms, layout diagrams, run and entry layout, index layout). 2. Indirect BlobIndex (db/blob/blob_index.h) and canonical varints (util/coding.h GetCanonicalVarint32/64): read the overview comment and the kIndirectBlob / kIndirectFlattenedBlob code. Every later part uses these fields: origin, original offset, generation number, logical CRC and stored CRC. The new formats read varints only in the canonical form. 3. Blob file writer (db/blob/blob_file_builder.cc, blob_log_format.cc, blob_log_writer.cc): see how a new value gets a type-3 index and how its CRC is computed once. 4. Format types and decoders: the structs in db/blob/blob_reference_metadata.h, then in the .cc the constants, GapBase, DecodeEntryBody, DecodeRunEntriesAt, DecodeBlobReferenceIndex and BlobReferenceGroupDecoder. They use only BlobIndex and the canonical varints. Learn the on-disk format here, before you read the writer. 5. Collector and its cache charge: first read the new CacheEntryRole kBlobReferenceMetadataBuildingBuffer (include/rocksdb/cache.h) and CacheReservationHandle::Resize (cache/cache_reservation_manager.h). Then read BlobReferenceMetadataCollector in this order: the class comment, Add, AppendSorted, BuildAndWriteDataBlocks, EncodeIndex, FinishOnce. Then read the unsorted fallback (AppendUnsorted, SortUnsortedGroup, SortAndEncodeGroup). 6. Wide-column change (db/wide/wide_column_serialization.cc, ForEachBlobIndex and WalkV2Columns): it gives every blob index of an entity to a callback after it checks the whole entity. The table builder uses it to feed the collector. UpdateBoundaries, BlobGarbageMeter and the compaction iterator also use it, with the option on or off. 7. Option: enable_blob_indirection (include/rocksdb/advanced_options.h, options/cf_options.cc), and the option checks and gate in db/column_family.cc (ValidateOptions, TEST_SetAllowBlobIndirection), db/db_impl/db_impl_open.cc and db_impl_secondary.cc (the FIFO check for opens that skip ValidateOptions). 8. Table integration: table/block_based/block_based_table_factory.cc (the shared reservation manager), then block_based_table_builder.cc (Rep constructor, Add, WriteBlobReferenceMetadata, EstimatedTailSize). See where the collector is created, fed and finished. block_based_table_reader.cc maps the new block name to a block type. Terms used in this diff: Stack terms: | Term | Meaning | |---|---| | origin | Blob file where the value was first written | | original offset | Value offset in the origin file | | stable identity | (origin, original offset). Never changes, even when the value moves | | generation number | The file number that a reference uses to find its value. A new value has generation number == origin. When GC moves the value, the reference keeps its generation number, and a route sends the read to the relocation file | | indirect BlobIndex | A kIndirectBlob (type 3) or kIndirectFlattenedBlob (type 4) index. It stores the stable identity, so the value can move without an SST rewrite | | unflattened reference | Index with generation number == origin: the value was not moved, or it was moved but no compaction has rewritten its reference yet (see "flatten" in "Stack context: terms for later diffs") | | flattened reference | Index with generation number != origin (type 4). Only flatten (diff 8) writes it; this diff defines the encoding | | route | Maps an old file number (a file that GC has removed from the Version) to the relocation file that now holds its live values. One hop: a target is never itself routed. Stored in the MANIFEST (diff 4) | | relocation file | Block-based table keyed by (origin, original offset) that holds copied values. Each value is the stored bytes plus a 5-byte trailer (kSimpleGen2BlobTrailerSize) | | logical CRC | CRC32C of the uncompressed value (BlobIndex::stable_checksum()) | | stored CRC | CRC32C of the stored bytes plus the compression byte | | physical file | A file that holds value bytes: a normal blob file or a relocation file. A generation number can name a physical file, or it can be only the key of a route | | exact reference metadata | Per-SST sorted list of every blob reference the SST keeps, written when the SST is built. References with the same encoding class, generation number and origin form a group. The list is stored as data blocks, with an index of the groups. An SST that has it is an exact SST | | option gate | Until diff 11, ColumnFamilyData::ValidateOptions returns NotSupported for enable_blob_indirection on a read-write open, CreateColumnFamily and SetOptions, unless a test calls ColumnFamilyData::TEST_SetAllowBlobIndirection(true). It keeps the option off in production while the stack is not complete | Terms of this diff (all defined at the top of db/blob/blob_reference_metadata.h): | Term | Meaning | |---|---| | encoding class (class) | kDirect (a legacy kBlob or kBlobTTL index), unflattened, or flattened | | reference group (group) | All references of one SST with the same (encoding class, generation number, origin) | | run | The part of one group inside one data block | | record bytes | The bytes the value's record takes in the file named by its generation number | | gap | How far a record is from where it would be if the records were back to back. A gap of 0 means back to back | | descriptor | The 72-byte table property that points to the reference index block and holds the totals, the CF ID, the SST file number and the relocation-state fingerprint | | relocation-state fingerprint | 128-bit hash of the routes and relocation files of the Version that an SST was built from (fields route_snapshot_fingerprint_*). Written as 0 in this diff | | legacy | Written the way RocksDB writes it without blob indirection | Previous behavior: - A BlobIndex named only a physical blob file and an offset. A value could not move to another file without rewriting every SST that referenced it. - An SST had no metadata that listed its blob references. The only way to find them was to scan its user keys. - These readers get the blob indexes of a wide-column entity through ForEachBlobFileNumber: UpdateBoundaries, BlobGarbageMeter, the compaction iterator, WideColumnSerialization::RemoveBlobColumns, and the referenced_blob_bytes table property count in BlockBasedTableBuilder::Add. ForEachBlobFileNumber decoded each blob index, but it did not check the rest of the entity framing (name order, section sizes, exact input size), and it could call the callback before it found a later error. New behavior: Separate change (option on or off): - The same readers (UpdateBoundaries, BlobGarbageMeter, the compaction iterator, RemoveBlobColumns, and the referenced_blob_bytes count in BlockBasedTableBuilder::Add) use WideColumnSerialization::ForEachBlobIndex, with the option on or off. With the option on, the table builder also uses it to give the blob columns of each entity to the collector. For an entity with blob columns, it checks the full framing, then decodes and validates every blob index before any callback runs. So an entity with a malformed framing now fails the flush or compaction, even with the option off. The referenced_blob_bytes count itself does not fail the build on a malformed entity. It removes that property from the SST, as in base. Option and option gate: - This diff adds a new column family option, `enable_blob_indirection`. It is off by default, and SetOptions can change it. - The option gate (see Terms) keeps the option off outside tests until the stack is complete. ColumnFamilyData::ValidateOptions rejects the option with NotSupported on a read-write open, CreateColumnFamily and SetOptions. Read-only opens (DBImplReadOnly and CompactedDBImpl) and secondary opens (including DB::OpenAndCompact) do not run ValidateOptions; they check only the FIFO rule (see Implementation). Only tests can turn the option on, through the internal ColumnFamilyData::TEST_SetAllowBlobIndirection(). DBTestBase sets it. - When the option is on, ValidateOptions also requires enable_blob_files, the block-based table format, no blob direct write, and a compaction style other than FIFO. Blob file writer (option on): - The blob file writer encodes each new value with a stable identity (origin file, original offset) and two checksums: - the logical CRC: the CRC32C of the uncompressed value. - the stored CRC: the CRC32C of the exact stored (maybe compressed) bytes, extended with the compression byte. - For an uncompressed value the stored CRC is not written, because the reader can compute it from the logical CRC. - The checksums have no version field of their own. The BlobIndex type byte gives the format version: the two new types, 3 (kIndirectBlob) and 4 (kIndirectFlattenedBlob, see Implementation), are BlobIndex::kIndirectBlobFormatVersion (1). A later format uses new type values. - The read path does not check these checksums yet. The check starts in diff 3. Exact reference metadata (option on): 1. What: a sorted list of every blob reference that the SST keeps, in plain blob values and in the blob columns of wide-column entities. 2. Where: data blocks (target 16 KB each before compression; small blocks waste less I/O, because GC reads a whole block to get a few groups) and one reference index block, written after the SST data blocks and before the SST tail, plus a 72-byte descriptor table property. 3. Who reads it: only tests in this diff. The census, GC and RepairDB read it in later diffs. 4. Details: the top comment of db/blob/blob_reference_metadata.h has the full format with diagrams, the record-bytes rules and the gap rule. - Not collected: SstFileWriter output (it has no DB file number) and any output whose CF is unknown (it has no CF ID). The descriptor needs both values, so these outputs stay in the legacy form. Checks on the metadata: - A repeated or overlapping reference in one group fails the SST build with Corruption. Readers check every count, order, size and total against the index and the descriptor, so corrupt metadata returns Corruption. The writer never makes a data block larger than kTargetDataBlockSize (16 KB), and readers accept up to kMaxDataBlockSize (64 KB). Details: "Readers" and "Integrity" at the top of db/blob/blob_reference_metadata.h, and BlobReferenceGroupDecoder. Collection and memory: - Each reference is encoded when it is added, in the on-disk entry encoding, into its group's chain of 512-byte chunks. Only a group that takes the unsorted fallback is encoded a second time, in Finish(). Inside a group, references normally arrive in rising offset order, so no reference is sorted; at the end only the groups are sorted. A group that gets a lower offset takes the unsorted fallback. The BlobReferenceMetadataCollector class comment explains the chunks, the chunk size and the fallback. - The collector keeps every reference of the SST in memory until the SST is finished, and charges this memory to the block cache under a new CacheEntryRole, kBlobReferenceMetadataBuildingBuffer. The measured cost is about 8.3 to 9 bytes per reference for uncompressed values and 12.8 to 13.5 bytes for compressed values. "Memory usage" and "Memory charge" in the BlobReferenceMetadataCollector class comment give the measurement setup and the charge rule. - There is no fixed limit on references. The collector fails with MemoryLimit above 2^20 groups or 2^20 data blocks in one SST; these are persisted format limits. - There is no charge when the table has no block cache or when charging for this role is disabled. Each block-based table factory holds one cache reservation manager for this role, shared by its builders and the SST summaries. If the charge fails, the SST build fails, so the flush or compaction fails with a background error. The buffers and their charge are released when the SST is finished or abandoned. Implementation: - BlobIndex adds two indirect encodings. The legacy encodings do not change. - Type 3, kIndirectBlob (unflattened, generation number == origin), stores the origin, the original offset, the stored size, the compression type, the logical CRC and, for compressed values, the stored CRC. It does not store the generation number. This saves the size of one varint file number per reference (about 3 to 4 bytes for typical file numbers; an estimate, not measured). - Type 4, kIndirectFlattenedBlob (flattened), stores the generation number first, then the same fields. Its generation number must be above the origin. The decoder rejects type 4 with generation number <= origin, so each reference has one encoding. - The encoders take the fields as one BlobIndex::IndirectBlobFields, set by field name, so values of the same type (file number, offset and size; the two CRCs) cannot be swapped. EncodeIndirectBlob() takes the generation number as a separate argument; EncodeUnflattenedIndirectBlob() uses the origin, so the common path cannot leave the generation number unset. - The new files blob_reference_metadata.{h,cc} define the exact reference metadata. The files also contain: - BlobReferenceMetadataCollector, the collector described above. It takes the SST identity (CF ID, SST file number and the relocation-state fingerprint) as one BlobReferenceSstIdentity, set by field name. - BlobReferenceGroupKey, the key of a group: (encoding class, generation number, origin). It has the order, the hash and the validity check that the collector map, the sort, the decoder and FindBlobReferenceGroup() use. BlobReference and BlobReferenceGroup keep their flat fields and return the key through group_key(). - The index codec and the descriptor table property. - BlobReferenceGroupDecoder, the only decoder of reference runs. It reads a group one data block at a time and holds every corruption check of a group walk. Readers that visit one group (found by a binary search of the index) use it, and so do readers that visit every block in order. - Callers in this diff: only the table builder runs the collector. The decoders (DecodeBlobReferenceIndex, FindBlobReferenceGroup, BlobReferenceGroupDecoder, ForEachBlobReference*) are used only by tests. kMaxDecodedBlobReferenceBytes is used only in compile-time checks of the count limits. The census (diff 5) and GC (diffs 6 and 7) are their first production users. The census is also the first user of BlobReferenceSstSummary (see the last Implementation bullet). - Descriptor: the CF ID and the SST file number are the repair fields. They let RepairDB tell which column family and SST the list belongs to without a MANIFEST. The decoder rejects a descriptor whose repair fields are not set (unknown CF ID or SST file number 0). The descriptor also has a 128-bit relocation-state fingerprint (the route_snapshot_fingerprint fields). Remote compaction uses it in diff 9 to check that an SST built elsewhere used the same routes as the job that asked for it. In this diff the table builder always writes 0, and DecodeFrom() does not check it. The field is in the format now, so the descriptor keeps one format version (1) through the stack. While the option gate is on, no data in this format exists outside tests. - The collector validates each BlobIndex once, when it is added. Its Finish() writes the data blocks and then the reference index block through one block writer, and fills the descriptor, including the index handle. Destroying the collector frees its memory and then its charge, also when the build is abandoned. - BlockBasedTableBuilder creates the collector with the CF ID and SST file number from TableBuilderOptions. It passes the size of each entry's user key to the collector, so the collector can compute the exact size of each blob record. It adds the collector's EstimatedEncodedSize() to its tail size estimate: the blocks are written before the tail, but output file size planning uses this estimate. The reference blocks are written with BlockType::kUserDefinedIndex, which keeps their contents as opaque bytes. Readers read them without the block cache, so prepopulate_block_cache does not warm them into the block cache (WriteMaybeCompressedBlock() with warm_cache false). - With the option on, the blob file builder computes the CRC32C of each stored value once. The blob log record CRC combines it with the key CRC (crc32c::Crc32cCombine). The stored CRC extends it with the compression byte. For an uncompressed value it is also the logical CRC. The bytes written do not change. - WideColumnSerialization::ForEachBlobIndex replaces ForEachBlobFileNumber. It reads the entity header and column types once. For a V1 entity (no column types and no blob columns), it returns right after the header. For a V2 entity without blob columns, it returns right after the type scan. It shares one V2 column walker (WalkV2Columns) with Deserialize(), so the entity is parsed in one pass. - Read-only opens (DBImplReadOnly, CompactedDBImpl) and secondary opens skip ValidateOptions, so they check the compaction style against the option once per column family: in DBImpl::Recover() for read-only opens, and at the start of OpenAsSecondaryImpl() for secondary opens. DB::OpenAndCompact opens through OpenAsSecondaryImpl(), so it gets the same check. - CacheReservationHandle gets Resize(), so each collector or summary can change its own part of the shared charge. - util/coding.h adds GetCanonicalVarint32/64, which reject overlong varint encodings. The new formats use them, so each value has one encoding. - BlobReferenceSstSummary is the in-memory list of an SST's groups and totals that Finish() can return. It has no caller yet: the table builder passes a null summary output. It is checked and does not change after Finish(), so the finished builder, the SST metadata and the census (its first user, in diff 5) can share it. It is in this diff because Finish() hands the group buffer and its cache charge over to the summary. Adding the summary later would mean rewriting the charge code of Finish() in diff 5, so the charge code is written and tested once, here. Safety boundaries and unsupported cases: - Public API: this diff adds the option `enable_blob_indirection` with the C API functions `rocksdb_options_set_enable_blob_indirection` and `rocksdb_options_get_enable_blob_indirection`. It also inserts a new CacheEntryRole value before kMisc, which changes the numeric value of kMisc and kNumCacheEntryRoles. - With the option off, new blob values and SSTs are written in the legacy form. Legacy blob indexes stay readable. - One behavior change applies with the option off too: UpdateBoundaries, BlobGarbageMeter and the compaction iterator fully validate wide-column entities with blob columns and their blob indexes (see "Separate change" above). - Exact reference metadata has one format (version 1) with a fixed 72-byte descriptor. The decoders reject any other version as Corruption. - The metadata blocks are protected by the table's block checksum only. With checksum = kNoChecksum, corrupt metadata is detected only when it breaks the format or order checks. - The index decoder checks each decoded count against the remaining input size before it reserves memory. So a small corrupt index cannot make it allocate a large vector. - Inlined-TTL and same-file blob indexes are not collected and cannot be relocated. Encoding, count and checksum errors return an error; a bad reference is never skipped. - Option gate (see Terms): outside tests no DB writes data in the new format. The gate checks only the option value, never persisted state. Diff 11 removes it, once data written with the option stays readable through the end of the stack and the safety checks for turning the option on are in place. - Downgrading the binary after data was written with the option on is not supported. Validation and known limitations: - Unit tests cover the BlobIndex encodings and their rejection of bad input, the metadata layout and every corruption check, the collector (sorted path, unsorted fallback, memory charge), the flush integration, the option checks and option gate, wide-column validation, and canonical varints. See the Test Plan. - The collection buffer is not spilled to disk. One SST with very many blob references needs a matching amount of charged memory: the per-reference amount given under "Collection and memory", plus up to one partly used 512-byte chunk per group. While a fallback group is sorted, it also needs 16 bytes of sort key per reference and a second copy of its entries. For a test group of 200K references with every other one out of order, the test comment estimates about 44 bytes per reference, and the test checks that the peak charge is at most 56 bytes per reference. - Flush still writes a single output file. Until diff 7, one compaction output collects every reference it gets, charged to the block cache. Diff 7 cuts compaction outputs (including L0) at a reference cap and keeps each user key's history in one output. With the option gate, a large output is reachable only through the test hook. Stack context (same in diffs 2-11, except that diffs 8-11 add the diff 8 route rule to the install line; "<== this diff" marks this diff's part): Goal: remove garbage from blob files without rewriting any SST. A value gets a stable identity when it is first written. GC copies the live values of whole files into a relocation file and installs routes in the MANIFEST. Readers follow the routes. SSTs are not changed. How a read finds a value (P is the relocation file that holds the value now): ``` SST entry indirect BlobIndex { generation number, origin, original offset, size, CRCs } | v Version file table: generation number -> live file, or route old file -> P | | live file route found | | v v blob file: read at original offset relocation file P (always one hop): relocation file: look up key look up key (origin, original offset) (origin, original offset) ``` Lifecycle of one value (in this example, F is the blob file where the value is first written, and P is the relocation file that GC writes): ``` write new value goes to blob file F; index: origin = F, generation number = F | census per generation number: the SSTs and references that use it; | per physical file: live bytes, ranked by garbage ratio GC job copy the live values of whole input files (stored bytes, checked | with the stored CRC) into relocation file P, keyed by | (origin, original offset) install one MANIFEST edit: add P, route each input generation number still | in use to P, re-point the routes into the inputs to P, and remove | each input (blob file retirement for a blob file, relocation | file deletion for a relocation file) flatten compaction sets generation number = P in the SSTs it rewrites | route route retirement: route deletion records remove the routes that delete no SST uses ``` Stack map: | Diff | Adds | |---|---| | 1 | Finalize blob metadata outside the DB mutex (prep) | | 2 | Indirect BlobIndex (kIndirectBlob) and exact SST reference metadata <== this diff | | 3 | Relocation file format and stable read path | | 4 | MANIFEST routes and Version ownership | | 5 | Exact reverse reference census | | 6 | Standalone relocation GC (one input file) | | 7 | Pack whole input files; reclaim relocation files | | 8 | Flatten generation numbers; route retirement | | 9 | Remote compaction validation | | 10 | Dynamic options and operations | | 11 | RepairDB rebuilds relocation state; option gate removed | | 12 | db_stress and crash test support | Stack context: terms for later diffs: | Term | Meaning | |---|---| | census | Background count, per generation number, of the SSTs and references that use it, plus live bytes per physical file (diff 5) | | indirect origin | Normal blob file written with the option on; it is the origin of unflattened references | | flatten | Compaction rewrites the generation number to the relocation file that holds the value, so old routes can go. Only flatten makes generation number != origin (diff 8) | | blob file retirement | MANIFEST record (kBlobFileRetirement) that removes a collected blob file from the Version (diff 4) | | route deletion | MANIFEST record that removes a route that no SST uses. Route retirement is the background job that writes these records (diff 8) | Tracking issue: meta#15301 Differential Revision: D124652097
Summary: Pull Request resolved: meta#15540 Why the change is required: The goal of this stack is a blob GC that does not depend on SSTs: it removes garbage from blob files without rewriting any SST. Such a GC needs two things that RocksDB does not have: - A value identity that does not change when the value moves to another file. - A checked list of the blob references that each SST keeps. The list must include older internal versions of a key that the SST still keeps (for example for snapshots), and the blob columns of wide-column entities. This diff adds both. Later diffs build the census (an in-memory reverse index from generation number to the SSTs that use it), GC and repair on top of this list. This diff in one paragraph: With the new column family option enable_blob_indirection on, the blob file writer gives each new value an indirect BlobIndex (type 3, kIndirectBlob). It stores the value's stable identity (origin file, original offset) and its CRCs (an uncompressed value stores only the CRC of the uncompressed value; a compressed value also stores the CRC of its stored bytes). Also with the option on, the block-based table builder collects every blob reference that the SST keeps. It writes them as "exact reference metadata": data blocks, one reference index block, and a 72-byte descriptor table property. In this diff only the table builder writes this metadata, and only tests read it. A temporary option gate (see "option gate" under Terms) keeps the option off outside tests. Separately, wide-column entities with blob columns are now fully checked before their blob indexes are used, also with the option off. Parts of this diff: 1. Indirect BlobIndex (core): types 3 and 4 in db/blob/blob_index.h, and the blob file writer that writes type 3. 2. Exact reference metadata format and decoders (core): db/blob/blob_reference_metadata.{h,cc}. 3. Collector and table builder integration (core): BlobReferenceMetadataCollector, its block cache charge, and BlockBasedTableBuilder. 4. Option enable_blob_indirection and the option gate. Tests need it to turn the feature on. You can skip this part on a first read. 5. Wide-column validation (WideColumnSerialization::ForEachBlobIndex). The table builder uses it to give the blob columns of each entity to the collector. It also changes behavior with the option off (see New behavior). You can skip this part on a first read. 6. Small helpers: canonical varints (util/coding.h) and CacheReservationHandle::Resize(). You can skip these on a first read. How to read this diff: 1. Overview: read "Why" and "This diff in one paragraph" above, and "Terms used in this diff" right below. Then read the top comment of db/blob/blob_reference_metadata.h (terms, layout diagrams, run and entry layout, index layout). 2. Indirect BlobIndex (db/blob/blob_index.h) and canonical varints (util/coding.h GetCanonicalVarint32/64): read the overview comment and the kIndirectBlob / kIndirectFlattenedBlob code. Every later part uses these fields: origin, original offset, generation number, logical CRC and stored CRC. The new formats read varints only in the canonical form. 3. Blob file writer (db/blob/blob_file_builder.cc, blob_log_format.cc, blob_log_writer.cc): see how a new value gets a type-3 index and how its CRC is computed once. 4. Format types and decoders: the structs in db/blob/blob_reference_metadata.h, then in the .cc the constants, GapBase, DecodeEntryBody, DecodeRunEntriesAt, DecodeBlobReferenceIndex and BlobReferenceGroupDecoder. They use only BlobIndex and the canonical varints. Learn the on-disk format here, before you read the writer. 5. Collector and its cache charge: first read the new CacheEntryRole kBlobReferenceMetadataBuildingBuffer (include/rocksdb/cache.h) and CacheReservationHandle::Resize (cache/cache_reservation_manager.h). Then read BlobReferenceMetadataCollector in this order: the class comment, Add, AppendSorted, BuildAndWriteDataBlocks, EncodeIndex, FinishOnce. Then read the unsorted fallback (AppendUnsorted, SortUnsortedGroup, SortAndEncodeGroup). 6. Wide-column change (db/wide/wide_column_serialization.cc, ForEachBlobIndex and WalkV2Columns): it gives every blob index of an entity to a callback after it checks the whole entity. The table builder uses it to feed the collector. UpdateBoundaries, BlobGarbageMeter and the compaction iterator also use it, with the option on or off. 7. Option: enable_blob_indirection (include/rocksdb/advanced_options.h, options/cf_options.cc), and the option checks and gate in db/column_family.cc (ValidateOptions, TEST_SetAllowBlobIndirection), db/db_impl/db_impl_open.cc and db_impl_secondary.cc (the FIFO check for opens that skip ValidateOptions). 8. Table integration: table/block_based/block_based_table_factory.cc (the shared reservation manager), then block_based_table_builder.cc (Rep constructor, Add, WriteBlobReferenceMetadata, EstimatedTailSize). See where the collector is created, fed and finished. block_based_table_reader.cc maps the new block name to a block type. Terms used in this diff: Stack terms: | Term | Meaning | |---|---| | origin | Blob file where the value was first written | | original offset | Value offset in the origin file | | stable identity | (origin, original offset). Never changes, even when the value moves | | generation number | The file number that a reference uses to find its value. A new value has generation number == origin. When GC moves the value, the reference keeps its generation number, and a route sends the read to the relocation file | | indirect BlobIndex | A kIndirectBlob (type 3) or kIndirectFlattenedBlob (type 4) index. It stores the stable identity, so the value can move without an SST rewrite | | unflattened reference | Index with generation number == origin: the value was not moved, or it was moved but no compaction has rewritten its reference yet (see "flatten" in "Stack context: terms for later diffs") | | flattened reference | Index with generation number != origin (type 4). Only flatten (diff 8) writes it; this diff defines the encoding | | route | Maps an old file number (a file that GC has removed from the Version) to the relocation file that now holds its live values. One hop: a target is never itself routed. Stored in the MANIFEST (diff 4) | | relocation file | Block-based table keyed by (origin, original offset) that holds copied values. Each value is the stored bytes plus a 5-byte trailer (kSimpleGen2BlobTrailerSize) | | logical CRC | CRC32C of the uncompressed value (BlobIndex::stable_checksum()) | | stored CRC | CRC32C of the stored bytes plus the compression byte | | physical file | A file that holds value bytes: a normal blob file or a relocation file. A generation number can name a physical file, or it can be only the key of a route | | exact reference metadata | Per-SST sorted list of every blob reference the SST keeps, written when the SST is built. References with the same encoding class, generation number and origin form a group. The list is stored as data blocks, with an index of the groups. An SST that has it is an exact SST | | option gate | Until diff 11, ColumnFamilyData::ValidateOptions returns NotSupported for enable_blob_indirection on a read-write open, CreateColumnFamily and SetOptions, unless a test calls ColumnFamilyData::TEST_SetAllowBlobIndirection(true). It keeps the option off in production while the stack is not complete | Terms of this diff (all defined at the top of db/blob/blob_reference_metadata.h): | Term | Meaning | |---|---| | encoding class (class) | kDirect (a legacy kBlob or kBlobTTL index), unflattened, or flattened | | reference group (group) | All references of one SST with the same (encoding class, generation number, origin) | | run | The part of one group inside one data block | | record bytes | The bytes the value's record takes in the file named by its generation number | | gap | How far a record is from where it would be if the records were back to back. A gap of 0 means back to back | | descriptor | The 72-byte table property that points to the reference index block and holds the totals, the CF ID, the SST file number and the relocation-state fingerprint | | relocation-state fingerprint | 128-bit hash of the routes and relocation files of the Version that an SST was built from (fields route_snapshot_fingerprint_*). Written as 0 in this diff | | legacy | Written the way RocksDB writes it without blob indirection | Previous behavior: - A BlobIndex named only a physical blob file and an offset. A value could not move to another file without rewriting every SST that referenced it. - An SST had no metadata that listed its blob references. The only way to find them was to scan its user keys. - These readers get the blob indexes of a wide-column entity through ForEachBlobFileNumber: UpdateBoundaries, BlobGarbageMeter, the compaction iterator, WideColumnSerialization::RemoveBlobColumns, and the referenced_blob_bytes table property count in BlockBasedTableBuilder::Add. ForEachBlobFileNumber decoded each blob index, but it did not check the rest of the entity framing (name order, section sizes, exact input size), and it could call the callback before it found a later error. New behavior: Separate change (option on or off): - The same readers (UpdateBoundaries, BlobGarbageMeter, the compaction iterator, RemoveBlobColumns, and the referenced_blob_bytes count in BlockBasedTableBuilder::Add) use WideColumnSerialization::ForEachBlobIndex, with the option on or off. With the option on, the table builder also uses it to give the blob columns of each entity to the collector. For an entity with blob columns, it checks the full framing, then decodes and validates every blob index before any callback runs. So an entity with a malformed framing now fails the flush or compaction, even with the option off. The referenced_blob_bytes count itself does not fail the build on a malformed entity. It removes that property from the SST, as in base. Option and option gate: - This diff adds a new column family option, `enable_blob_indirection`. It is off by default, and SetOptions can change it. - The option gate (see Terms) keeps the option off outside tests until the stack is complete. ColumnFamilyData::ValidateOptions rejects the option with NotSupported on a read-write open, CreateColumnFamily and SetOptions. Read-only opens (DBImplReadOnly and CompactedDBImpl) and secondary opens (including DB::OpenAndCompact) do not run ValidateOptions; they check only the FIFO rule (see Implementation). Only tests can turn the option on, through the internal ColumnFamilyData::TEST_SetAllowBlobIndirection(). DBTestBase sets it. - When the option is on, ValidateOptions also requires enable_blob_files, the block-based table format, no blob direct write, and a compaction style other than FIFO. Blob file writer (option on): - The blob file writer encodes each new value with a stable identity (origin file, original offset) and two checksums: - the logical CRC: the CRC32C of the uncompressed value. - the stored CRC: the CRC32C of the exact stored (maybe compressed) bytes, extended with the compression byte. - For an uncompressed value the stored CRC is not written, because the reader can compute it from the logical CRC. - The checksums have no version field of their own. The BlobIndex type byte gives the format version: the two new types, 3 (kIndirectBlob) and 4 (kIndirectFlattenedBlob, see Implementation), are BlobIndex::kIndirectBlobFormatVersion (1). A later format uses new type values. - The read path does not check these checksums yet. The check starts in diff 3. Exact reference metadata (option on): 1. What: a sorted list of every blob reference that the SST keeps, in plain blob values and in the blob columns of wide-column entities. 2. Where: data blocks (target 16 KB each before compression; small blocks waste less I/O, because GC reads a whole block to get a few groups) and one reference index block, written after the SST data blocks and before the SST tail, plus a 72-byte descriptor table property. 3. Who reads it: only tests in this diff. The census, GC and RepairDB read it in later diffs. 4. Details: the top comment of db/blob/blob_reference_metadata.h has the full format with diagrams, the record-bytes rules and the gap rule. - Not collected: SstFileWriter output (it has no DB file number) and any output whose CF is unknown (it has no CF ID). The descriptor needs both values, so these outputs stay in the legacy form. Checks on the metadata: - A repeated or overlapping reference in one group fails the SST build with Corruption. Readers check every count, order, size and total against the index and the descriptor, so corrupt metadata returns Corruption. The writer never makes a data block larger than kTargetDataBlockSize (16 KB), and readers accept up to kMaxDataBlockSize (64 KB). Details: "Readers" and "Integrity" at the top of db/blob/blob_reference_metadata.h, and BlobReferenceGroupDecoder. Collection and memory: - Each reference is encoded when it is added, in the on-disk entry encoding, into its group's chain of 512-byte chunks. Only a group that takes the unsorted fallback is encoded a second time, in Finish(). Inside a group, references normally arrive in rising offset order, so no reference is sorted; at the end only the groups are sorted. A group that gets a lower offset takes the unsorted fallback. The BlobReferenceMetadataCollector class comment explains the chunks, the chunk size and the fallback. - The collector keeps every reference of the SST in memory until the SST is finished, and charges this memory to the block cache under a new CacheEntryRole, kBlobReferenceMetadataBuildingBuffer. The measured cost is about 8.3 to 9 bytes per reference for uncompressed values and 12.8 to 13.5 bytes for compressed values. "Memory usage" and "Memory charge" in the BlobReferenceMetadataCollector class comment give the measurement setup and the charge rule. - There is no fixed limit on references. The collector fails with MemoryLimit above 2^20 groups or 2^20 data blocks in one SST; these are persisted format limits. - There is no charge when the table has no block cache or when charging for this role is disabled. Each block-based table factory holds one cache reservation manager for this role, shared by its builders and the SST summaries. If the charge fails, the SST build fails, so the flush or compaction fails with a background error. The buffers and their charge are released when the SST is finished or abandoned. Implementation: - BlobIndex adds two indirect encodings. The legacy encodings do not change. - Type 3, kIndirectBlob (unflattened, generation number == origin), stores the origin, the original offset, the stored size, the compression type, the logical CRC and, for compressed values, the stored CRC. It does not store the generation number. This saves the size of one varint file number per reference (about 3 to 4 bytes for typical file numbers; an estimate, not measured). - Type 4, kIndirectFlattenedBlob (flattened), stores the generation number first, then the same fields. Its generation number must be above the origin. The decoder rejects type 4 with generation number <= origin, so each reference has one encoding. - The encoders take the fields as one BlobIndex::IndirectBlobFields, set by field name, so values of the same type (file number, offset and size; the two CRCs) cannot be swapped. EncodeIndirectBlob() takes the generation number as a separate argument; EncodeUnflattenedIndirectBlob() uses the origin, so the common path cannot leave the generation number unset. - The new files blob_reference_metadata.{h,cc} define the exact reference metadata. The files also contain: - BlobReferenceMetadataCollector, the collector described above. It takes the SST identity (CF ID, SST file number and the relocation-state fingerprint) as one BlobReferenceSstIdentity, set by field name. - BlobReferenceGroupKey, the key of a group: (encoding class, generation number, origin). It has the order, the hash and the validity check that the collector map, the sort, the decoder and FindBlobReferenceGroup() use. BlobReference and BlobReferenceGroup keep their flat fields and return the key through group_key(). - The index codec and the descriptor table property. - BlobReferenceGroupDecoder, the only decoder of reference runs. It reads a group one data block at a time and holds every corruption check of a group walk. Readers that visit one group (found by a binary search of the index) use it, and so do readers that visit every block in order. - Callers in this diff: only the table builder runs the collector. The decoders (DecodeBlobReferenceIndex, FindBlobReferenceGroup, BlobReferenceGroupDecoder, ForEachBlobReference*) are used only by tests. kMaxDecodedBlobReferenceBytes is used only in compile-time checks of the count limits. The census (diff 5) and GC (diffs 6 and 7) are their first production users. The census is also the first user of BlobReferenceSstSummary (see the last Implementation bullet). - Descriptor: the CF ID and the SST file number are the repair fields. They let RepairDB tell which column family and SST the list belongs to without a MANIFEST. The decoder rejects a descriptor whose repair fields are not set (unknown CF ID or SST file number 0). The descriptor also has a 128-bit relocation-state fingerprint (the route_snapshot_fingerprint fields). Remote compaction uses it in diff 9 to check that an SST built elsewhere used the same routes as the job that asked for it. In this diff the table builder always writes 0, and DecodeFrom() does not check it. The field is in the format now, so the descriptor keeps one format version (1) through the stack. While the option gate is on, no data in this format exists outside tests. - The collector validates each BlobIndex once, when it is added. Its Finish() writes the data blocks and then the reference index block through one block writer, and fills the descriptor, including the index handle. Destroying the collector frees its memory and then its charge, also when the build is abandoned. - BlockBasedTableBuilder creates the collector with the CF ID and SST file number from TableBuilderOptions. It passes the size of each entry's user key to the collector, so the collector can compute the exact size of each blob record. It adds the collector's EstimatedEncodedSize() to its tail size estimate: the blocks are written before the tail, but output file size planning uses this estimate. The reference blocks are written with BlockType::kUserDefinedIndex, which keeps their contents as opaque bytes. Readers read them without the block cache, so prepopulate_block_cache does not warm them into the block cache (WriteMaybeCompressedBlock() with warm_cache false). - With the option on, the blob file builder computes the CRC32C of each stored value once. The blob log record CRC combines it with the key CRC (crc32c::Crc32cCombine). The stored CRC extends it with the compression byte. For an uncompressed value it is also the logical CRC. The bytes written do not change. - WideColumnSerialization::ForEachBlobIndex replaces ForEachBlobFileNumber. It reads the entity header and column types once. For a V1 entity (no column types and no blob columns), it returns right after the header. For a V2 entity without blob columns, it returns right after the type scan. It shares one V2 column walker (WalkV2Columns) with Deserialize(), so the entity is parsed in one pass. - Read-only opens (DBImplReadOnly, CompactedDBImpl) and secondary opens skip ValidateOptions, so they check the compaction style against the option once per column family: in DBImpl::Recover() for read-only opens, and at the start of OpenAsSecondaryImpl() for secondary opens. DB::OpenAndCompact opens through OpenAsSecondaryImpl(), so it gets the same check. - CacheReservationHandle gets Resize(), so each collector or summary can change its own part of the shared charge. - util/coding.h adds GetCanonicalVarint32/64, which reject overlong varint encodings. The new formats use them, so each value has one encoding. - BlobReferenceSstSummary is the in-memory list of an SST's groups and totals that Finish() can return. It has no caller yet: the table builder passes a null summary output. It is checked and does not change after Finish(), so the finished builder, the SST metadata and the census (its first user, in diff 5) can share it. It is in this diff because Finish() hands the group buffer and its cache charge over to the summary. Adding the summary later would mean rewriting the charge code of Finish() in diff 5, so the charge code is written and tested once, here. Safety boundaries and unsupported cases: - Public API: this diff adds the option `enable_blob_indirection` with the C API functions `rocksdb_options_set_enable_blob_indirection` and `rocksdb_options_get_enable_blob_indirection`. It also inserts a new CacheEntryRole value before kMisc, which changes the numeric value of kMisc and kNumCacheEntryRoles. - With the option off, new blob values and SSTs are written in the legacy form. Legacy blob indexes stay readable. - One behavior change applies with the option off too: UpdateBoundaries, BlobGarbageMeter and the compaction iterator fully validate wide-column entities with blob columns and their blob indexes (see "Separate change" above). - Exact reference metadata has one format (version 1) with a fixed 72-byte descriptor. The decoders reject any other version as Corruption. - The metadata blocks are protected by the table's block checksum only. With checksum = kNoChecksum, corrupt metadata is detected only when it breaks the format or order checks. - The index decoder checks each decoded count against the remaining input size before it reserves memory. So a small corrupt index cannot make it allocate a large vector. - Inlined-TTL and same-file blob indexes are not collected and cannot be relocated. Encoding, count and checksum errors return an error; a bad reference is never skipped. - Option gate (see Terms): outside tests no DB writes data in the new format. The gate checks only the option value, never persisted state. Diff 11 removes it, once data written with the option stays readable through the end of the stack and the safety checks for turning the option on are in place. - Downgrading the binary after data was written with the option on is not supported. Validation and known limitations: - Unit tests cover the BlobIndex encodings and their rejection of bad input, the metadata layout and every corruption check, the collector (sorted path, unsorted fallback, memory charge), the flush integration, the option checks and option gate, wide-column validation, and canonical varints. See the Test Plan. - The collection buffer is not spilled to disk. One SST with very many blob references needs a matching amount of charged memory: the per-reference amount given under "Collection and memory", plus up to one partly used 512-byte chunk per group. While a fallback group is sorted, it also needs 16 bytes of sort key per reference and a second copy of its entries. For a test group of 200K references with every other one out of order, the test comment estimates about 44 bytes per reference, and the test checks that the peak charge is at most 56 bytes per reference. - Flush still writes a single output file. Until diff 7, one compaction output collects every reference it gets, charged to the block cache. Diff 7 cuts compaction outputs (including L0) at a reference cap and keeps each user key's history in one output. With the option gate, a large output is reachable only through the test hook. Stack context (same in diffs 2-11, except that diffs 8-11 add the diff 8 route rule to the install line; "<== this diff" marks this diff's part): Goal: remove garbage from blob files without rewriting any SST. A value gets a stable identity when it is first written. GC copies the live values of whole files into a relocation file and installs routes in the MANIFEST. Readers follow the routes. SSTs are not changed. How a read finds a value (P is the relocation file that holds the value now): ``` SST entry indirect BlobIndex { generation number, origin, original offset, size, CRCs } | v Version file table: generation number -> live file, or route old file -> P | | live file route found | | v v blob file: read at original offset relocation file P (always one hop): relocation file: look up key look up key (origin, original offset) (origin, original offset) ``` Lifecycle of one value (in this example, F is the blob file where the value is first written, and P is the relocation file that GC writes): ``` write new value goes to blob file F; index: origin = F, generation number = F | census per generation number: the SSTs and references that use it; | per physical file: live bytes, ranked by garbage ratio GC job copy the live values of whole input files (stored bytes, checked | with the stored CRC) into relocation file P, keyed by | (origin, original offset) install one MANIFEST edit: add P, route each input generation number still | in use to P, re-point the routes into the inputs to P, and remove | each input (blob file retirement for a blob file, relocation | file deletion for a relocation file) flatten compaction sets generation number = P in the SSTs it rewrites | route route retirement: route deletion records remove the routes that delete no SST uses ``` Stack map: | Diff | Adds | |---|---| | 1 | Finalize blob metadata outside the DB mutex (prep) | | 2 | Indirect BlobIndex (kIndirectBlob) and exact SST reference metadata <== this diff | | 3 | Relocation file format and stable read path | | 4 | MANIFEST routes and Version ownership | | 5 | Exact reverse reference census | | 6 | Standalone relocation GC (one input file) | | 7 | Pack whole input files; reclaim relocation files | | 8 | Flatten generation numbers; route retirement | | 9 | Remote compaction validation | | 10 | Dynamic options and operations | | 11 | RepairDB rebuilds relocation state; option gate removed | | 12 | db_stress and crash test support | Stack context: terms for later diffs: | Term | Meaning | |---|---| | census | Background count, per generation number, of the SSTs and references that use it, plus live bytes per physical file (diff 5) | | indirect origin | Normal blob file written with the option on; it is the origin of unflattened references | | flatten | Compaction rewrites the generation number to the relocation file that holds the value, so old routes can go. Only flatten makes generation number != origin (diff 8) | | blob file retirement | MANIFEST record (kBlobFileRetirement) that removes a collected blob file from the Version (diff 4) | | route deletion | MANIFEST record that removes a route that no SST uses. Route retirement is the background job that writes these records (diff 8) | Tracking issue: meta#15301 Differential Revision: D124652097
Summary: Pull Request resolved: meta#15540 Why the change is required: The goal of this stack is a blob GC that does not depend on SSTs: it removes garbage from blob files without rewriting any SST. Such a GC needs two things that RocksDB does not have: - A value identity that does not change when the value moves to another file. - A checked list of the blob references that each SST keeps. The list must include older internal versions of a key that the SST still keeps (for example for snapshots), and the blob columns of wide-column entities. This diff adds both. Later diffs build the census (an in-memory reverse index from generation number to the SSTs that use it), GC and repair on top of this list. This diff in one paragraph: With the new column family option enable_blob_indirection on, the blob file writer gives each new value an indirect BlobIndex (type 3, kIndirectBlob). It stores the value's stable identity (origin file, original offset) and its CRCs (an uncompressed value stores only the CRC of the uncompressed value; a compressed value also stores the CRC of its stored bytes). Also with the option on, the block-based table builder collects every blob reference that the SST keeps. It writes them as "exact reference metadata": data blocks, one reference index block, and a 72-byte descriptor table property. In this diff only the table builder writes this metadata, and only tests read it. A temporary option gate (see "option gate" under Terms) keeps the option off outside tests. Separately, wide-column entities with blob columns are now fully checked before their blob indexes are used, also with the option off. Parts of this diff: 1. Indirect BlobIndex (core): types 3 and 4 in db/blob/blob_index.h, and the blob file writer that writes type 3. 2. Exact reference metadata format and decoders (core): db/blob/blob_reference_metadata.{h,cc}. 3. Collector and table builder integration (core): BlobReferenceMetadataCollector, its block cache charge, and BlockBasedTableBuilder. 4. Option enable_blob_indirection and the option gate. Tests need it to turn the feature on. You can skip this part on a first read. 5. Wide-column validation (WideColumnSerialization::ForEachBlobIndex). The table builder uses it to give the blob columns of each entity to the collector. It also changes behavior with the option off (see New behavior). You can skip this part on a first read. 6. Small helpers: canonical varints (util/coding.h) and CacheReservationHandle::Resize(). You can skip these on a first read. How to read this diff: 1. Overview: read "Why" and "This diff in one paragraph" above, and "Terms used in this diff" right below. Then read the top comment of db/blob/blob_reference_metadata.h (terms, layout diagrams, run and entry layout, index layout). 2. Indirect BlobIndex (db/blob/blob_index.h) and canonical varints (util/coding.h GetCanonicalVarint32/64): read the overview comment and the kIndirectBlob / kIndirectFlattenedBlob code. Every later part uses these fields: origin, original offset, generation number, logical CRC and stored CRC. The new formats read varints only in the canonical form. 3. Blob file writer (db/blob/blob_file_builder.cc, blob_log_format.cc, blob_log_writer.cc): see how a new value gets a type-3 index and how its CRC is computed once. 4. Format types and decoders: the structs in db/blob/blob_reference_metadata.h, then in the .cc the constants, GapBase, DecodeEntryBody, DecodeRunEntriesAt, DecodeBlobReferenceIndex and BlobReferenceGroupDecoder. They use only BlobIndex and the canonical varints. Learn the on-disk format here, before you read the writer. 5. Collector and its cache charge: first read the new CacheEntryRole kBlobReferenceMetadataBuildingBuffer (include/rocksdb/cache.h) and CacheReservationHandle::Resize (cache/cache_reservation_manager.h). Then read BlobReferenceMetadataCollector in this order: the class comment, Add, AppendSorted, BuildAndWriteDataBlocks, EncodeIndex, FinishOnce. Then read the unsorted fallback (AppendUnsorted, SortUnsortedGroup, SortAndEncodeGroup). 6. Wide-column change (db/wide/wide_column_serialization.cc, ForEachBlobIndex and WalkV2Columns): it gives every blob index of an entity to a callback after it checks the whole entity. The table builder uses it to feed the collector. UpdateBoundaries, BlobGarbageMeter and the compaction iterator also use it, with the option on or off. 7. Option: enable_blob_indirection (include/rocksdb/advanced_options.h, options/cf_options.cc), and the option checks and gate in db/column_family.cc (ValidateOptions, TEST_SetAllowBlobIndirection), db/db_impl/db_impl_open.cc and db_impl_secondary.cc (the FIFO check for opens that skip ValidateOptions). 8. Table integration: table/block_based/block_based_table_factory.cc (the shared reservation manager), then block_based_table_builder.cc (Rep constructor, Add, WriteBlobReferenceMetadata, EstimatedTailSize). See where the collector is created, fed and finished. block_based_table_reader.cc maps the new block name to a block type. Terms used in this diff: Stack terms: | Term | Meaning | |---|---| | origin | Blob file where the value was first written | | original offset | Value offset in the origin file | | stable identity | (origin, original offset). Never changes, even when the value moves | | generation number | The file number that a reference uses to find its value. A new value has generation number == origin. When GC moves the value, the reference keeps its generation number, and a route sends the read to the relocation file | | indirect BlobIndex | A kIndirectBlob (type 3) or kIndirectFlattenedBlob (type 4) index. It stores the stable identity, so the value can move without an SST rewrite | | unflattened reference | Index with generation number == origin: the value was not moved, or it was moved but no compaction has rewritten its reference yet (see "flatten" in "Stack context: terms for later diffs") | | flattened reference | Index with generation number != origin (type 4). Only flatten (diff 8) writes it; this diff defines the encoding | | route | Maps an old file number (a file that GC has removed from the Version) to the relocation file that now holds its live values. One hop: a target is never itself routed. Stored in the MANIFEST (diff 4) | | relocation file | Block-based table keyed by (origin, original offset) that holds copied values. Each value is the stored bytes plus a 5-byte trailer (kSimpleGen2BlobTrailerSize) | | logical CRC | CRC32C of the uncompressed value (BlobIndex::stable_checksum()) | | stored CRC | CRC32C of the stored bytes plus the compression byte | | physical file | A file that holds value bytes: a normal blob file or a relocation file. A generation number can name a physical file, or it can be only the key of a route | | exact reference metadata | Per-SST sorted list of every blob reference the SST keeps, written when the SST is built. References with the same encoding class, generation number and origin form a group. The list is stored as data blocks, with an index of the groups. An SST that has it is an exact SST | | option gate | Until diff 11, ColumnFamilyData::ValidateOptions returns NotSupported for enable_blob_indirection on a read-write open, CreateColumnFamily and SetOptions, unless a test calls ColumnFamilyData::TEST_SetAllowBlobIndirection(true). It keeps the option off in production while the stack is not complete | Terms of this diff (all defined at the top of db/blob/blob_reference_metadata.h): | Term | Meaning | |---|---| | encoding class (class) | kDirect (a legacy kBlob or kBlobTTL index), unflattened, or flattened | | reference group (group) | All references of one SST with the same (encoding class, generation number, origin) | | run | The part of one group inside one data block | | record bytes | The bytes the value's record takes in the file named by its generation number | | gap | How far a record is from where it would be if the records were back to back. A gap of 0 means back to back | | descriptor | The 72-byte table property that points to the reference index block and holds the totals, the CF ID, the SST file number and the relocation-state fingerprint | | relocation-state fingerprint | 128-bit hash of the routes and relocation files of the Version that an SST was built from (fields route_snapshot_fingerprint_*). Written as 0 in this diff | | legacy | Written the way RocksDB writes it without blob indirection | Previous behavior: - A BlobIndex named only a physical blob file and an offset. A value could not move to another file without rewriting every SST that referenced it. - An SST had no metadata that listed its blob references. The only way to find them was to scan its user keys. - These readers get the blob indexes of a wide-column entity through ForEachBlobFileNumber: UpdateBoundaries, BlobGarbageMeter, the compaction iterator, WideColumnSerialization::RemoveBlobColumns, and the referenced_blob_bytes table property count in BlockBasedTableBuilder::Add. ForEachBlobFileNumber decoded each blob index, but it did not check the rest of the entity framing (name order, section sizes, exact input size), and it could call the callback before it found a later error. New behavior: Separate change (option on or off): - The same readers (UpdateBoundaries, BlobGarbageMeter, the compaction iterator, RemoveBlobColumns, and the referenced_blob_bytes count in BlockBasedTableBuilder::Add) use WideColumnSerialization::ForEachBlobIndex, with the option on or off. With the option on, the table builder also uses it to give the blob columns of each entity to the collector. For an entity with blob columns, it checks the full framing, then decodes and validates every blob index before any callback runs. So an entity with a malformed framing now fails the flush or compaction, even with the option off. The referenced_blob_bytes count itself does not fail the build on a malformed entity. It removes that property from the SST, as in base. Option and option gate: - This diff adds a new column family option, `enable_blob_indirection`. It is off by default, and SetOptions can change it. - The option gate (see Terms) keeps the option off outside tests until the stack is complete. ColumnFamilyData::ValidateOptions rejects the option with NotSupported on a read-write open, CreateColumnFamily and SetOptions. Read-only opens (DBImplReadOnly and CompactedDBImpl) and secondary opens (including DB::OpenAndCompact) do not run ValidateOptions; they check only the FIFO rule (see Implementation). Only tests can turn the option on, through the internal ColumnFamilyData::TEST_SetAllowBlobIndirection(). DBTestBase sets it. - When the option is on, ValidateOptions also requires enable_blob_files, the block-based table format, no blob direct write, and a compaction style other than FIFO. Blob file writer (option on): - The blob file writer encodes each new value with a stable identity (origin file, original offset) and two checksums: - the logical CRC: the CRC32C of the uncompressed value. - the stored CRC: the CRC32C of the exact stored (maybe compressed) bytes, extended with the compression byte. - For an uncompressed value the stored CRC is not written, because the reader can compute it from the logical CRC. - The checksums have no version field of their own. The BlobIndex type byte gives the format version: the two new types, 3 (kIndirectBlob) and 4 (kIndirectFlattenedBlob, see Implementation), are BlobIndex::kIndirectBlobFormatVersion (1). A later format uses new type values. - The read path does not check these checksums yet. The check starts in diff 3. Exact reference metadata (option on): 1. What: a sorted list of every blob reference that the SST keeps, in plain blob values and in the blob columns of wide-column entities. 2. Where: data blocks (target 16 KB each before compression; small blocks waste less I/O, because GC reads a whole block to get a few groups) and one reference index block, written after the SST data blocks and before the SST tail, plus a 72-byte descriptor table property. 3. Who reads it: only tests in this diff. The census, GC and RepairDB read it in later diffs. 4. Details: the top comment of db/blob/blob_reference_metadata.h has the full format with diagrams, the record-bytes rules and the gap rule. - Not collected: SstFileWriter output (it has no DB file number) and any output whose CF is unknown (it has no CF ID). The descriptor needs both values, so these outputs stay in the legacy form. Checks on the metadata: - A repeated or overlapping reference in one group fails the SST build with Corruption. Readers check every count, order, size and total against the index and the descriptor, so corrupt metadata returns Corruption. The writer never makes a data block larger than kTargetDataBlockSize (16 KB), and readers accept up to kMaxDataBlockSize (64 KB). Details: "Readers" and "Integrity" at the top of db/blob/blob_reference_metadata.h, and BlobReferenceGroupDecoder. Collection and memory: - Each reference is encoded when it is added, in the on-disk entry encoding, into its group's chain of 512-byte chunks. Only a group that takes the unsorted fallback is encoded a second time, in Finish(). Inside a group, references normally arrive in rising offset order, so no reference is sorted; at the end only the groups are sorted. A group that gets a lower offset takes the unsorted fallback. The BlobReferenceMetadataCollector class comment explains the chunks, the chunk size and the fallback. - The collector keeps every reference of the SST in memory until the SST is finished, and charges this memory to the block cache under a new CacheEntryRole, kBlobReferenceMetadataBuildingBuffer. The measured cost is about 8.3 to 9 bytes per reference for uncompressed values and 12.8 to 13.5 bytes for compressed values. "Memory usage" and "Memory charge" in the BlobReferenceMetadataCollector class comment give the measurement setup and the charge rule. - There is no fixed limit on references. The collector fails with MemoryLimit above 2^20 groups or 2^20 data blocks in one SST; these are persisted format limits. - There is no charge when the table has no block cache or when charging for this role is disabled. Each block-based table factory holds one cache reservation manager for this role, shared by its builders and the SST summaries. If the charge fails, the SST build fails, so the flush or compaction fails with a background error. The buffers and their charge are released when the SST is finished or abandoned. Implementation: - BlobIndex adds two indirect encodings. The legacy encodings do not change. - Type 3, kIndirectBlob (unflattened, generation number == origin), stores the origin, the original offset, the stored size, the compression type, the logical CRC and, for compressed values, the stored CRC. It does not store the generation number. This saves the size of one varint file number per reference (about 3 to 4 bytes for typical file numbers; an estimate, not measured). - Type 4, kIndirectFlattenedBlob (flattened), stores the generation number first, then the same fields. Its generation number must be above the origin. The decoder rejects type 4 with generation number <= origin, so each reference has one encoding. - The encoders take the fields as one BlobIndex::IndirectBlobFields, set by field name, so values of the same type (file number, offset and size; the two CRCs) cannot be swapped. EncodeIndirectBlob() takes the generation number as a separate argument; EncodeUnflattenedIndirectBlob() uses the origin, so the common path cannot leave the generation number unset. - The new files blob_reference_metadata.{h,cc} define the exact reference metadata. The files also contain: - BlobReferenceMetadataCollector, the collector described above. It takes the SST identity (CF ID, SST file number and the relocation-state fingerprint) as one BlobReferenceSstIdentity, set by field name. - BlobReferenceGroupKey, the key of a group: (encoding class, generation number, origin). It has the order, the hash and the validity check that the collector map, the sort, the decoder and FindBlobReferenceGroup() use. BlobReference and BlobReferenceGroup keep their flat fields and return the key through group_key(). - The index codec and the descriptor table property. - BlobReferenceGroupDecoder, the only decoder of reference runs. It reads a group one data block at a time and holds every corruption check of a group walk. Readers that visit one group (found by a binary search of the index) use it, and so do readers that visit every block in order. - Callers in this diff: only the table builder runs the collector. The decoders (DecodeBlobReferenceIndex, FindBlobReferenceGroup, BlobReferenceGroupDecoder, ForEachBlobReference*) are used only by tests. kMaxDecodedBlobReferenceBytes is used only in compile-time checks of the count limits. The census (diff 5) and GC (diffs 6 and 7) are their first production users. The census is also the first user of BlobReferenceSstSummary (see the last Implementation bullet). - Descriptor: the CF ID and the SST file number are the repair fields. They let RepairDB tell which column family and SST the list belongs to without a MANIFEST. The decoder rejects a descriptor whose repair fields are not set (unknown CF ID or SST file number 0). The descriptor also has a 128-bit relocation-state fingerprint (the route_snapshot_fingerprint fields). Remote compaction uses it in diff 9 to check that an SST built elsewhere used the same routes as the job that asked for it. In this diff the table builder always writes 0, and DecodeFrom() does not check it. The field is in the format now, so the descriptor keeps one format version (1) through the stack. While the option gate is on, no data in this format exists outside tests. - The collector validates each BlobIndex once, when it is added. Its Finish() writes the data blocks and then the reference index block through one block writer, and fills the descriptor, including the index handle. Destroying the collector frees its memory and then its charge, also when the build is abandoned. - BlockBasedTableBuilder creates the collector with the CF ID and SST file number from TableBuilderOptions. It passes the size of each entry's user key to the collector, so the collector can compute the exact size of each blob record. It adds the collector's EstimatedEncodedSize() to its tail size estimate: the blocks are written before the tail, but output file size planning uses this estimate. The reference blocks are written with BlockType::kUserDefinedIndex, which keeps their contents as opaque bytes. Readers read them without the block cache, so prepopulate_block_cache does not warm them into the block cache (WriteMaybeCompressedBlock() with warm_cache false). - With the option on, the blob file builder computes the CRC32C of each stored value once. The blob log record CRC combines it with the key CRC (crc32c::Crc32cCombine). The stored CRC extends it with the compression byte. For an uncompressed value it is also the logical CRC. The bytes written do not change. - WideColumnSerialization::ForEachBlobIndex replaces ForEachBlobFileNumber. It reads the entity header and column types once. For a V1 entity (no column types and no blob columns), it returns right after the header. For a V2 entity without blob columns, it returns right after the type scan. It shares one V2 column walker (WalkV2Columns) with Deserialize(), so the entity is parsed in one pass. - Read-only opens (DBImplReadOnly, CompactedDBImpl) and secondary opens skip ValidateOptions, so they check the compaction style against the option once per column family: in DBImpl::Recover() for read-only opens, and at the start of OpenAsSecondaryImpl() for secondary opens. DB::OpenAndCompact opens through OpenAsSecondaryImpl(), so it gets the same check. - CacheReservationHandle gets Resize(), so each collector or summary can change its own part of the shared charge. - util/coding.h adds GetCanonicalVarint32/64, which reject overlong varint encodings. The new formats use them, so each value has one encoding. - BlobReferenceSstSummary is the in-memory list of an SST's groups and totals that Finish() can return. It has no caller yet: the table builder passes a null summary output. It is checked and does not change after Finish(), so the finished builder, the SST metadata and the census (its first user, in diff 5) can share it. It is in this diff because Finish() hands the group buffer and its cache charge over to the summary. Adding the summary later would mean rewriting the charge code of Finish() in diff 5, so the charge code is written and tested once, here. Safety boundaries and unsupported cases: - Public API: this diff adds the option `enable_blob_indirection` with the C API functions `rocksdb_options_set_enable_blob_indirection` and `rocksdb_options_get_enable_blob_indirection`. It also inserts a new CacheEntryRole value before kMisc, which changes the numeric value of kMisc and kNumCacheEntryRoles. - With the option off, new blob values and SSTs are written in the legacy form. Legacy blob indexes stay readable. - One behavior change applies with the option off too: UpdateBoundaries, BlobGarbageMeter and the compaction iterator fully validate wide-column entities with blob columns and their blob indexes (see "Separate change" above). - Exact reference metadata has one format (version 1) with a fixed 72-byte descriptor. The decoders reject any other version as Corruption. - The metadata blocks are protected by the table's block checksum only. With checksum = kNoChecksum, corrupt metadata is detected only when it breaks the format or order checks. - The index decoder checks each decoded count against the remaining input size before it reserves memory. So a small corrupt index cannot make it allocate a large vector. - Inlined-TTL and same-file blob indexes are not collected and cannot be relocated. Encoding, count and checksum errors return an error; a bad reference is never skipped. - Option gate (see Terms): outside tests no DB writes data in the new format. The gate checks only the option value, never persisted state. Diff 11 removes it, once data written with the option stays readable through the end of the stack and the safety checks for turning the option on are in place. - Downgrading the binary after data was written with the option on is not supported. Validation and known limitations: - Unit tests cover the BlobIndex encodings and their rejection of bad input, the metadata layout and every corruption check, the collector (sorted path, unsorted fallback, memory charge), the flush integration, the option checks and option gate, wide-column validation, and canonical varints. See the Test Plan. - The collection buffer is not spilled to disk. One SST with very many blob references needs a matching amount of charged memory: the per-reference amount given under "Collection and memory", plus up to one partly used 512-byte chunk per group. While a fallback group is sorted, it also needs 16 bytes of sort key per reference and a second copy of its entries. For a test group of 200K references with every other one out of order, the test comment estimates about 44 bytes per reference, and the test checks that the peak charge is at most 56 bytes per reference. - Flush still writes a single output file. Until diff 7, one compaction output collects every reference it gets, charged to the block cache. Diff 7 cuts compaction outputs (including L0) at a reference cap and keeps each user key's history in one output. With the option gate, a large output is reachable only through the test hook. Stack context (same in diffs 2-11, except that diffs 8-11 add the diff 8 route rule to the install line; "<== this diff" marks this diff's part): Goal: remove garbage from blob files without rewriting any SST. A value gets a stable identity when it is first written. GC copies the live values of whole files into a relocation file and installs routes in the MANIFEST. Readers follow the routes. SSTs are not changed. How a read finds a value (P is the relocation file that holds the value now): ``` SST entry indirect BlobIndex { generation number, origin, original offset, size, CRCs } | v Version file table: generation number -> live file, or route old file -> P | | live file route found | | v v blob file: read at original offset relocation file P (always one hop): relocation file: look up key look up key (origin, original offset) (origin, original offset) ``` Lifecycle of one value (in this example, F is the blob file where the value is first written, and P is the relocation file that GC writes): ``` write new value goes to blob file F; index: origin = F, generation number = F | census per generation number: the SSTs and references that use it; | per physical file: live bytes, ranked by garbage ratio GC job copy the live values of whole input files (stored bytes, checked | with the stored CRC) into relocation file P, keyed by | (origin, original offset) install one MANIFEST edit: add P, route each input generation number still | in use to P, re-point the routes into the inputs to P, and remove | each input (blob file retirement for a blob file, relocation | file deletion for a relocation file) flatten compaction sets generation number = P in the SSTs it rewrites | route route retirement: route deletion records remove the routes that delete no SST uses ``` Stack map: | Diff | Adds | |---|---| | 1 | Finalize blob metadata outside the DB mutex (prep) | | 2 | Indirect BlobIndex (kIndirectBlob) and exact SST reference metadata <== this diff | | 3 | Relocation file format and stable read path | | 4 | MANIFEST routes and Version ownership | | 5 | Exact reverse reference census | | 6 | Standalone relocation GC (one input file) | | 7 | Pack whole input files; reclaim relocation files | | 8 | Flatten generation numbers; route retirement | | 9 | Remote compaction validation | | 10 | Dynamic options and operations | | 11 | RepairDB rebuilds relocation state; option gate removed | | 12 | db_stress and crash test support | Stack context: terms for later diffs: | Term | Meaning | |---|---| | census | Background count, per generation number, of the SSTs and references that use it, plus live bytes per physical file (diff 5) | | indirect origin | Normal blob file written with the option on; it is the origin of unflattened references | | flatten | Compaction rewrites the generation number to the relocation file that holds the value, so old routes can go. Only flatten makes generation number != origin (diff 8) | | blob file retirement | MANIFEST record (kBlobFileRetirement) that removes a collected blob file from the Version (diff 4) | | route deletion | MANIFEST record that removes a route that no SST uses. Route retirement is the background job that writes these records (diff 8) | Tracking issue: meta#15301 Differential Revision: D124652097
Summary: Pull Request resolved: meta#15540 Why the change is required: The goal of this stack is a blob GC that does not depend on SSTs: it removes garbage from blob files without rewriting any SST. Such a GC needs two things that RocksDB does not have: - A value identity that does not change when the value moves to another file. - A checked list of the blob references that each SST keeps. The list must include older internal versions of a key that the SST still keeps (for example for snapshots), and the blob columns of wide-column entities. This diff adds both. Later diffs build the census (an in-memory reverse index from generation number to the SSTs that use it), GC and repair on top of this list. This diff in one paragraph: With the new column family option enable_blob_indirection on, the blob file writer gives each new value an indirect BlobIndex (type 3, kIndirectBlob). It stores the value's stable identity (origin file, original offset) and its CRCs (an uncompressed value stores only the CRC of the uncompressed value; a compressed value also stores the CRC of its stored bytes). Also with the option on, the block-based table builder collects every blob reference that the SST keeps. It writes them as "exact reference metadata": data blocks, one reference index block, and a 72-byte descriptor table property. In this diff only the table builder writes this metadata, and only tests read it. A temporary option gate (see "option gate" under Terms) keeps the option off outside tests. Separately, wide-column entities with blob columns are now fully checked before their blob indexes are used, also with the option off. Parts of this diff: 1. Indirect BlobIndex (core): types 3 and 4 in db/blob/blob_index.h, and the blob file writer that writes type 3. 2. Exact reference metadata format and decoders (core): db/blob/blob_reference_metadata.{h,cc}. 3. Collector and table builder integration (core): BlobReferenceMetadataCollector, its block cache charge, and BlockBasedTableBuilder. 4. Option enable_blob_indirection and the option gate. Tests need it to turn the feature on. You can skip this part on a first read. 5. Wide-column validation (WideColumnSerialization::ForEachBlobIndex). The table builder uses it to give the blob columns of each entity to the collector. It also changes behavior with the option off (see New behavior). You can skip this part on a first read. 6. Small helpers: canonical varints (util/coding.h) and CacheReservationHandle::Resize(). You can skip these on a first read. How to read this diff: 1. Overview: read "Why" and "This diff in one paragraph" above, and "Terms used in this diff" right below. Then read the top comment of db/blob/blob_reference_metadata.h (terms, layout diagrams, run and entry layout, index layout). 2. Indirect BlobIndex (db/blob/blob_index.h) and canonical varints (util/coding.h GetCanonicalVarint32/64): read the overview comment and the kIndirectBlob / kIndirectFlattenedBlob code. Every later part uses these fields: origin, original offset, generation number, logical CRC and stored CRC. The new formats read varints only in the canonical form. 3. Blob file writer (db/blob/blob_file_builder.cc, blob_log_format.cc, blob_log_writer.cc): see how a new value gets a type-3 index and how its CRC is computed once. 4. Format types and decoders: the structs in db/blob/blob_reference_metadata.h, then in the .cc the constants, GapBase, DecodeEntryBody, DecodeRunEntriesAt, DecodeBlobReferenceIndex and BlobReferenceGroupDecoder. They use only BlobIndex and the canonical varints. Learn the on-disk format here, before you read the writer. 5. Collector and its cache charge: first read the new CacheEntryRole kBlobReferenceMetadataBuildingBuffer (include/rocksdb/cache.h) and CacheReservationHandle::Resize (cache/cache_reservation_manager.h). Then read BlobReferenceMetadataCollector in this order: the class comment, Add, AppendSorted, BuildAndWriteDataBlocks, EncodeIndex, FinishOnce. Then read the unsorted fallback (AppendUnsorted, SortUnsortedGroup, SortAndEncodeGroup). 6. Wide-column change (db/wide/wide_column_serialization.cc, ForEachBlobIndex and WalkV2Columns): it gives every blob index of an entity to a callback after it checks the whole entity. The table builder uses it to feed the collector. UpdateBoundaries, BlobGarbageMeter and the compaction iterator also use it, with the option on or off. 7. Option: enable_blob_indirection (include/rocksdb/advanced_options.h, options/cf_options.cc), and the option checks and gate in db/column_family.cc (ValidateOptions, TEST_SetAllowBlobIndirection), db/db_impl/db_impl_open.cc and db_impl_secondary.cc (the FIFO check for opens that skip ValidateOptions). 8. Table integration: table/block_based/block_based_table_factory.cc (the shared reservation manager), then block_based_table_builder.cc (Rep constructor, Add, WriteBlobReferenceMetadata, EstimatedTailSize). See where the collector is created, fed and finished. block_based_table_reader.cc maps the new block name to a block type. Terms used in this diff: Stack terms: | Term | Meaning | |---|---| | origin | Blob file where the value was first written | | original offset | Value offset in the origin file | | stable identity | (origin, original offset). Never changes, even when the value moves | | generation number | The file number that a reference uses to find its value. A new value has generation number == origin. When GC moves the value, the reference keeps its generation number, and a route sends the read to the relocation file | | indirect BlobIndex | A kIndirectBlob (type 3) or kIndirectFlattenedBlob (type 4) index. It stores the stable identity, so the value can move without an SST rewrite | | unflattened reference | Index with generation number == origin: the value was not moved, or it was moved but no compaction has rewritten its reference yet (see "flatten" in "Stack context: terms for later diffs") | | flattened reference | Index with generation number != origin (type 4). Only flatten (diff 8) writes it; this diff defines the encoding | | route | Maps an old file number (a file that GC has removed from the Version) to the relocation file that now holds its live values. One hop: a target is never itself routed. Stored in the MANIFEST (diff 4) | | relocation file | Block-based table keyed by (origin, original offset) that holds copied values. Each value is the stored bytes plus a 5-byte trailer (kSimpleGen2BlobTrailerSize) | | logical CRC | CRC32C of the uncompressed value (BlobIndex::stable_checksum()) | | stored CRC | CRC32C of the stored bytes plus the compression byte | | physical file | A file that holds value bytes: a normal blob file or a relocation file. A generation number can name a physical file, or it can be only the key of a route | | exact reference metadata | Per-SST sorted list of every blob reference the SST keeps, written when the SST is built. References with the same encoding class, generation number and origin form a group. The list is stored as data blocks, with an index of the groups. An SST that has it is an exact SST | | option gate | Until diff 11, ColumnFamilyData::ValidateOptions returns NotSupported for enable_blob_indirection on a read-write open, CreateColumnFamily and SetOptions, unless a test calls ColumnFamilyData::TEST_SetAllowBlobIndirection(true). It keeps the option off in production while the stack is not complete | Terms of this diff (all defined at the top of db/blob/blob_reference_metadata.h): | Term | Meaning | |---|---| | encoding class (class) | kDirect (a legacy kBlob or kBlobTTL index), unflattened, or flattened | | reference group (group) | All references of one SST with the same (encoding class, generation number, origin) | | run | The part of one group inside one data block | | record bytes | The bytes the value's record takes in the file named by its generation number | | gap | How far a record is from where it would be if the records were back to back. A gap of 0 means back to back | | descriptor | The 72-byte table property that points to the reference index block and holds the totals, the CF ID, the SST file number and the relocation-state fingerprint | | relocation-state fingerprint | 128-bit hash of the routes and relocation files of the Version that an SST was built from (fields route_snapshot_fingerprint_*). Written as 0 in this diff | | legacy | Written the way RocksDB writes it without blob indirection | Previous behavior: - A BlobIndex named only a physical blob file and an offset. A value could not move to another file without rewriting every SST that referenced it. - An SST had no metadata that listed its blob references. The only way to find them was to scan its user keys. - These readers get the blob indexes of a wide-column entity through ForEachBlobFileNumber: UpdateBoundaries, BlobGarbageMeter, the compaction iterator, WideColumnSerialization::RemoveBlobColumns, and the referenced_blob_bytes table property count in BlockBasedTableBuilder::Add. ForEachBlobFileNumber decoded each blob index, but it did not check the rest of the entity framing (name order, section sizes, exact input size), and it could call the callback before it found a later error. New behavior: Separate change (option on or off): - The same readers (UpdateBoundaries, BlobGarbageMeter, the compaction iterator, RemoveBlobColumns, and the referenced_blob_bytes count in BlockBasedTableBuilder::Add) use WideColumnSerialization::ForEachBlobIndex, with the option on or off. With the option on, the table builder also uses it to give the blob columns of each entity to the collector. For an entity with blob columns, it checks the full framing, then decodes and validates every blob index before any callback runs. So an entity with a malformed framing now fails the flush or compaction, even with the option off. The referenced_blob_bytes count itself does not fail the build on a malformed entity. It removes that property from the SST, as in base. Option and option gate: - This diff adds a new column family option, `enable_blob_indirection`. It is off by default, and SetOptions can change it. - The option gate (see Terms) keeps the option off outside tests until the stack is complete. ColumnFamilyData::ValidateOptions rejects the option with NotSupported on a read-write open, CreateColumnFamily and SetOptions. Read-only opens (DBImplReadOnly and CompactedDBImpl) and secondary opens (including DB::OpenAndCompact) do not run ValidateOptions; they check only the FIFO rule (see Implementation). Only tests can turn the option on, through the internal ColumnFamilyData::TEST_SetAllowBlobIndirection(). DBTestBase sets it. - When the option is on, ValidateOptions also requires enable_blob_files, the block-based table format, no blob direct write, and a compaction style other than FIFO. Blob file writer (option on): - The blob file writer encodes each new value with a stable identity (origin file, original offset) and two checksums: - the logical CRC: the CRC32C of the uncompressed value. - the stored CRC: the CRC32C of the exact stored (maybe compressed) bytes, extended with the compression byte. - For an uncompressed value the stored CRC is not written, because the reader can compute it from the logical CRC. - The checksums have no version field of their own. The BlobIndex type byte gives the format version: the two new types, 3 (kIndirectBlob) and 4 (kIndirectFlattenedBlob, see Implementation), are BlobIndex::kIndirectBlobFormatVersion (1). A later format uses new type values. - The read path does not check these checksums yet. The check starts in diff 3. Exact reference metadata (option on): 1. What: a sorted list of every blob reference that the SST keeps, in plain blob values and in the blob columns of wide-column entities. 2. Where: data blocks (target 16 KB each before compression; small blocks waste less I/O, because GC reads a whole block to get a few groups) and one reference index block, written after the SST data blocks and before the SST tail, plus a 72-byte descriptor table property. 3. Who reads it: only tests in this diff. The census, GC and RepairDB read it in later diffs. 4. Details: the top comment of db/blob/blob_reference_metadata.h has the full format with diagrams, the record-bytes rules and the gap rule. - Not collected: SstFileWriter output (it has no DB file number) and any output whose CF is unknown (it has no CF ID). The descriptor needs both values, so these outputs stay in the legacy form. Checks on the metadata: - A repeated or overlapping reference in one group fails the SST build with Corruption. Readers check every count, order, size and total against the index and the descriptor, so corrupt metadata returns Corruption. The writer never makes a data block larger than kTargetDataBlockSize (16 KB), and readers accept up to kMaxDataBlockSize (64 KB). Details: "Readers" and "Integrity" at the top of db/blob/blob_reference_metadata.h, and BlobReferenceGroupDecoder. Collection and memory: - Each reference is encoded when it is added, in the on-disk entry encoding, into its group's chain of 512-byte chunks. Only a group that takes the unsorted fallback is encoded a second time, in Finish(). Inside a group, references normally arrive in rising offset order, so no reference is sorted; at the end only the groups are sorted. A group that gets a lower offset takes the unsorted fallback. The BlobReferenceMetadataCollector class comment explains the chunks, the chunk size and the fallback. - The collector keeps every reference of the SST in memory until the SST is finished, and charges this memory to the block cache under a new CacheEntryRole, kBlobReferenceMetadataBuildingBuffer. The measured cost is about 8.3 to 9 bytes per reference for uncompressed values and 12.8 to 13.5 bytes for compressed values. "Memory usage" and "Memory charge" in the BlobReferenceMetadataCollector class comment give the measurement setup and the charge rule. - There is no fixed limit on references. The collector fails with MemoryLimit above 2^20 groups or 2^20 data blocks in one SST; these are persisted format limits. - There is no charge when the table has no block cache or when charging for this role is disabled. Each block-based table factory holds one cache reservation manager for this role, shared by its builders and the SST summaries. If the charge fails, the SST build fails, so the flush or compaction fails with a background error. The buffers and their charge are released when the SST is finished or abandoned. Implementation: - BlobIndex adds two indirect encodings. The legacy encodings do not change. - Type 3, kIndirectBlob (unflattened, generation number == origin), stores the origin, the original offset, the stored size, the compression type, the logical CRC and, for compressed values, the stored CRC. It does not store the generation number. This saves the size of one varint file number per reference (about 3 to 4 bytes for typical file numbers; an estimate, not measured). - Type 4, kIndirectFlattenedBlob (flattened), stores the generation number first, then the same fields. Its generation number must be above the origin. The decoder rejects type 4 with generation number <= origin, so each reference has one encoding. - The encoders take the fields as one BlobIndex::IndirectBlobFields, set by field name, so values of the same type (file number, offset and size; the two CRCs) cannot be swapped. EncodeIndirectBlob() takes the generation number as a separate argument; EncodeUnflattenedIndirectBlob() uses the origin, so the common path cannot leave the generation number unset. - The new files blob_reference_metadata.{h,cc} define the exact reference metadata. The files also contain: - BlobReferenceMetadataCollector, the collector described above. It takes the SST identity (CF ID, SST file number and the relocation-state fingerprint) as one BlobReferenceSstIdentity, set by field name. - BlobReferenceGroupKey, the key of a group: (encoding class, generation number, origin). It has the order, the hash and the validity check that the collector map, the sort, the decoder and FindBlobReferenceGroup() use. BlobReference and BlobReferenceGroup keep their flat fields and return the key through group_key(). - The index codec and the descriptor table property. - BlobReferenceGroupDecoder, the only decoder of reference runs. It reads a group one data block at a time and holds every corruption check of a group walk. Readers that visit one group (found by a binary search of the index) use it, and so do readers that visit every block in order. - Callers in this diff: only the table builder runs the collector. The decoders (DecodeBlobReferenceIndex, FindBlobReferenceGroup, BlobReferenceGroupDecoder, ForEachBlobReference*) are used only by tests. kMaxDecodedBlobReferenceBytes is used only in compile-time checks of the count limits. The census (diff 5) and GC (diffs 6 and 7) are their first production users. The census is also the first user of BlobReferenceSstSummary (see the last Implementation bullet). - Descriptor: the CF ID and the SST file number are the repair fields. They let RepairDB tell which column family and SST the list belongs to without a MANIFEST. The decoder rejects a descriptor whose repair fields are not set (unknown CF ID or SST file number 0). The descriptor also has a 128-bit relocation-state fingerprint (the route_snapshot_fingerprint fields). Remote compaction uses it in diff 9 to check that an SST built elsewhere used the same routes as the job that asked for it. In this diff the table builder always writes 0, and DecodeFrom() does not check it. The field is in the format now, so the descriptor keeps one format version (1) through the stack. While the option gate is on, no data in this format exists outside tests. - The collector validates each BlobIndex once, when it is added. Its Finish() writes the data blocks and then the reference index block through one block writer, and fills the descriptor, including the index handle. Destroying the collector frees its memory and then its charge, also when the build is abandoned. - BlockBasedTableBuilder creates the collector with the CF ID and SST file number from TableBuilderOptions. It passes the size of each entry's user key to the collector, so the collector can compute the exact size of each blob record. It adds the collector's EstimatedEncodedSize() to its tail size estimate: the blocks are written before the tail, but output file size planning uses this estimate. The reference blocks are written with BlockType::kUserDefinedIndex, which keeps their contents as opaque bytes. Readers read them without the block cache, so prepopulate_block_cache does not warm them into the block cache (WriteMaybeCompressedBlock() with warm_cache false). - With the option on, the blob file builder computes the CRC32C of each stored value once. The blob log record CRC combines it with the key CRC (crc32c::Crc32cCombine). The stored CRC extends it with the compression byte. For an uncompressed value it is also the logical CRC. The bytes written do not change. - WideColumnSerialization::ForEachBlobIndex replaces ForEachBlobFileNumber. It reads the entity header and column types once. For a V1 entity (no column types and no blob columns), it returns right after the header. For a V2 entity without blob columns, it returns right after the type scan. It shares one V2 column walker (WalkV2Columns) with Deserialize(), so the entity is parsed in one pass. - Read-only opens (DBImplReadOnly, CompactedDBImpl) and secondary opens skip ValidateOptions, so they check the compaction style against the option once per column family: in DBImpl::Recover() for read-only opens, and at the start of OpenAsSecondaryImpl() for secondary opens. DB::OpenAndCompact opens through OpenAsSecondaryImpl(), so it gets the same check. - CacheReservationHandle gets Resize(), so each collector or summary can change its own part of the shared charge. - util/coding.h adds GetCanonicalVarint32/64, which reject overlong varint encodings. The new formats use them, so each value has one encoding. - BlobReferenceSstSummary is the in-memory list of an SST's groups and totals that Finish() can return. It has no caller yet: the table builder passes a null summary output. It is checked and does not change after Finish(), so the finished builder, the SST metadata and the census (its first user, in diff 5) can share it. It is in this diff because Finish() hands the group buffer and its cache charge over to the summary. Adding the summary later would mean rewriting the charge code of Finish() in diff 5, so the charge code is written and tested once, here. Safety boundaries and unsupported cases: - Public API: this diff adds the option `enable_blob_indirection` with the C API functions `rocksdb_options_set_enable_blob_indirection` and `rocksdb_options_get_enable_blob_indirection`. It also inserts a new CacheEntryRole value before kMisc, which changes the numeric value of kMisc and kNumCacheEntryRoles. - With the option off, new blob values and SSTs are written in the legacy form. Legacy blob indexes stay readable. - One behavior change applies with the option off too: UpdateBoundaries, BlobGarbageMeter and the compaction iterator fully validate wide-column entities with blob columns and their blob indexes (see "Separate change" above). - Exact reference metadata has one format (version 1) with a fixed 72-byte descriptor. The decoders reject any other version as Corruption. - The metadata blocks are protected by the table's block checksum only. With checksum = kNoChecksum, corrupt metadata is detected only when it breaks the format or order checks. - The index decoder checks each decoded count against the remaining input size before it reserves memory. So a small corrupt index cannot make it allocate a large vector. - Inlined-TTL and same-file blob indexes are not collected and cannot be relocated. Encoding, count and checksum errors return an error; a bad reference is never skipped. - Option gate (see Terms): outside tests no DB writes data in the new format. The gate checks only the option value, never persisted state. Diff 11 removes it, once data written with the option stays readable through the end of the stack and the safety checks for turning the option on are in place. - Downgrading the binary after data was written with the option on is not supported. Validation and known limitations: - Unit tests cover the BlobIndex encodings and their rejection of bad input, the metadata layout and every corruption check, the collector (sorted path, unsorted fallback, memory charge), the flush integration, the option checks and option gate, wide-column validation, and canonical varints. See the Test Plan. - The collection buffer is not spilled to disk. One SST with very many blob references needs a matching amount of charged memory: the per-reference amount given under "Collection and memory", plus up to one partly used 512-byte chunk per group. While a fallback group is sorted, it also needs 16 bytes of sort key per reference and a second copy of its entries. For a test group of 200K references with every other one out of order, the test comment estimates about 44 bytes per reference, and the test checks that the peak charge is at most 56 bytes per reference. - Flush still writes a single output file. Until diff 7, one compaction output collects every reference it gets, charged to the block cache. Diff 7 cuts compaction outputs (including L0) at a reference cap and keeps each user key's history in one output. With the option gate, a large output is reachable only through the test hook. Stack context (same in diffs 2-11, except that diffs 8-11 add the diff 8 route rule to the install line; "<== this diff" marks this diff's part): Goal: remove garbage from blob files without rewriting any SST. A value gets a stable identity when it is first written. GC copies the live values of whole files into a relocation file and installs routes in the MANIFEST. Readers follow the routes. SSTs are not changed. How a read finds a value (P is the relocation file that holds the value now): ``` SST entry indirect BlobIndex { generation number, origin, original offset, size, CRCs } | v Version file table: generation number -> live file, or route old file -> P | | live file route found | | v v blob file: read at original offset relocation file P (always one hop): relocation file: look up key look up key (origin, original offset) (origin, original offset) ``` Lifecycle of one value (in this example, F is the blob file where the value is first written, and P is the relocation file that GC writes): ``` write new value goes to blob file F; index: origin = F, generation number = F | census per generation number: the SSTs and references that use it; | per physical file: live bytes, ranked by garbage ratio GC job copy the live values of whole input files (stored bytes, checked | with the stored CRC) into relocation file P, keyed by | (origin, original offset) install one MANIFEST edit: add P, route each input generation number still | in use to P, re-point the routes into the inputs to P, and remove | each input (blob file retirement for a blob file, relocation | file deletion for a relocation file) flatten compaction sets generation number = P in the SSTs it rewrites | route route retirement: route deletion records remove the routes that delete no SST uses ``` Stack map: | Diff | Adds | |---|---| | 1 | Finalize blob metadata outside the DB mutex (prep) | | 2 | Indirect BlobIndex (kIndirectBlob) and exact SST reference metadata <== this diff | | 3 | Relocation file format and stable read path | | 4 | MANIFEST routes and Version ownership | | 5 | Exact reverse reference census | | 6 | Standalone relocation GC (one input file) | | 7 | Pack whole input files; reclaim relocation files | | 8 | Flatten generation numbers; route retirement | | 9 | Remote compaction validation | | 10 | Dynamic options and operations | | 11 | RepairDB rebuilds relocation state; option gate removed | | 12 | db_stress and crash test support | Stack context: terms for later diffs: | Term | Meaning | |---|---| | census | Background count, per generation number, of the SSTs and references that use it, plus live bytes per physical file (diff 5) | | indirect origin | Normal blob file written with the option on; it is the origin of unflattened references | | flatten | Compaction rewrites the generation number to the relocation file that holds the value, so old routes can go. Only flatten makes generation number != origin (diff 8) | | blob file retirement | MANIFEST record (kBlobFileRetirement) that removes a collected blob file from the Version (diff 4) | | route deletion | MANIFEST record that removes a route that no SST uses. Route retirement is the background job that writes these records (diff 8) | Tracking issue: meta#15301 Differential Revision: D124652097
Summary: Pull Request resolved: meta#15540 Why the change is required: The goal of this stack is a blob GC that does not depend on SSTs: it removes garbage from blob files without rewriting any SST. Such a GC needs two things that RocksDB does not have: - A value identity that does not change when the value moves to another file. - A checked list of the blob references that each SST keeps. The list must include older internal versions of a key that the SST still keeps (for example for snapshots), and the blob columns of wide-column entities. This diff adds both. Later diffs build the census (an in-memory reverse index from generation number to the SSTs that use it), GC and repair on top of this list. This diff in one paragraph: With the new column family option enable_blob_indirection on, the blob file writer gives each new value an indirect BlobIndex (type 3, kIndirectBlob). It stores the value's stable identity (origin file, original offset) and its CRCs (an uncompressed value stores only the CRC of the uncompressed value; a compressed value also stores the CRC of its stored bytes). Also with the option on, the block-based table builder collects every blob reference that the SST keeps. It writes them as "exact reference metadata": data blocks, one reference index block, and a 72-byte descriptor table property. In this diff only the table builder writes this metadata, and only tests read it. A temporary option gate (see "option gate" under Terms) keeps the option off outside tests. Separately, wide-column entities with blob columns are now fully checked before their blob indexes are used, also with the option off. Parts of this diff: 1. Indirect BlobIndex (core): types 3 and 4 in db/blob/blob_index.h, and the blob file writer that writes type 3. 2. Exact reference metadata format and decoders (core): db/blob/blob_reference_metadata.{h,cc}. 3. Collector and table builder integration (core): BlobReferenceMetadataCollector, its block cache charge, and BlockBasedTableBuilder. 4. Option enable_blob_indirection and the option gate. Tests need it to turn the feature on. You can skip this part on a first read. 5. Wide-column validation (WideColumnSerialization::ForEachBlobIndex). The table builder uses it to give the blob columns of each entity to the collector. It also changes behavior with the option off (see New behavior). You can skip this part on a first read. 6. Small helpers: canonical varints (util/coding.h) and CacheReservationHandle::Resize(). You can skip these on a first read. How to read this diff: 1. Overview: read "Why" and "This diff in one paragraph" above, and "Terms used in this diff" right below. Then read the top comment of db/blob/blob_reference_metadata.h (terms, layout diagrams, run and entry layout, index layout). 2. Indirect BlobIndex (db/blob/blob_index.h) and canonical varints (util/coding.h GetCanonicalVarint32/64): read the overview comment and the kIndirectBlob / kIndirectFlattenedBlob code. Every later part uses these fields: origin, original offset, generation number, logical CRC and stored CRC. The new formats read varints only in the canonical form. 3. Blob file writer (db/blob/blob_file_builder.cc, blob_log_format.cc, blob_log_writer.cc): see how a new value gets a type-3 index and how its CRC is computed once. 4. Format types and decoders: the structs in db/blob/blob_reference_metadata.h, then in the .cc the constants, GapBase, DecodeEntryBody, DecodeRunEntriesAt, DecodeBlobReferenceIndex and BlobReferenceGroupDecoder. They use only BlobIndex and the canonical varints. Learn the on-disk format here, before you read the writer. 5. Collector and its cache charge: first read the new CacheEntryRole kBlobReferenceMetadataBuildingBuffer (include/rocksdb/cache.h) and CacheReservationHandle::Resize (cache/cache_reservation_manager.h). Then read BlobReferenceMetadataCollector in this order: the class comment, Add, AppendSorted, BuildAndWriteDataBlocks, EncodeIndex, FinishOnce. Then read the unsorted fallback (AppendUnsorted, SortUnsortedGroup, SortAndEncodeGroup). 6. Wide-column change (db/wide/wide_column_serialization.cc, ForEachBlobIndex and WalkV2Columns): it gives every blob index of an entity to a callback after it checks the whole entity. The table builder uses it to feed the collector. UpdateBoundaries, BlobGarbageMeter and the compaction iterator also use it, with the option on or off. 7. Option: enable_blob_indirection (include/rocksdb/advanced_options.h, options/cf_options.cc), and the option checks and gate in db/column_family.cc (ValidateOptions, TEST_SetAllowBlobIndirection), db/db_impl/db_impl_open.cc and db_impl_secondary.cc (the FIFO check for opens that skip ValidateOptions). 8. Table integration: table/block_based/block_based_table_factory.cc (the shared reservation manager), then block_based_table_builder.cc (Rep constructor, Add, WriteBlobReferenceMetadata, EstimatedTailSize). See where the collector is created, fed and finished. block_based_table_reader.cc maps the new block name to a block type. Terms used in this diff: Stack terms: | Term | Meaning | |---|---| | origin | Blob file where the value was first written | | original offset | Value offset in the origin file | | stable identity | (origin, original offset). Never changes, even when the value moves | | generation number | The file number that a reference uses to find its value. A new value has generation number == origin. When GC moves the value, the reference keeps its generation number, and a route sends the read to the relocation file | | indirect BlobIndex | A kIndirectBlob (type 3) or kIndirectFlattenedBlob (type 4) index. It stores the stable identity, so the value can move without an SST rewrite | | unflattened reference | Index with generation number == origin: the value was not moved, or it was moved but no compaction has rewritten its reference yet (see "flatten" in "Stack context: terms for later diffs") | | flattened reference | Index with generation number != origin (type 4). Only flatten (diff 8) writes it; this diff defines the encoding | | route | Maps an old file number (a file that GC has removed from the Version) to the relocation file that now holds its live values. One hop: a target is never itself routed. Stored in the MANIFEST (diff 4) | | relocation file | Block-based table keyed by (origin, original offset) that holds copied values. Each value is the stored bytes plus a 5-byte trailer (kSimpleGen2BlobTrailerSize) | | logical CRC | CRC32C of the uncompressed value (BlobIndex::stable_checksum()) | | stored CRC | CRC32C of the stored bytes plus the compression byte | | physical file | A file that holds value bytes: a normal blob file or a relocation file. A generation number can name a physical file, or it can be only the key of a route | | exact reference metadata | Per-SST sorted list of every blob reference the SST keeps, written when the SST is built. References with the same encoding class, generation number and origin form a group. The list is stored as data blocks, with an index of the groups. An SST that has it is an exact SST | | option gate | Until diff 11, ColumnFamilyData::ValidateOptions returns NotSupported for enable_blob_indirection on a read-write open, CreateColumnFamily and SetOptions, unless a test calls ColumnFamilyData::TEST_SetAllowBlobIndirection(true). It keeps the option off in production while the stack is not complete | Terms of this diff (all defined at the top of db/blob/blob_reference_metadata.h): | Term | Meaning | |---|---| | encoding class (class) | kDirect (a legacy kBlob or kBlobTTL index), unflattened, or flattened | | reference group (group) | All references of one SST with the same (encoding class, generation number, origin) | | run | The part of one group inside one data block | | record bytes | The bytes the value's record takes in the file named by its generation number | | gap | How far a record is from where it would be if the records were back to back. A gap of 0 means back to back | | descriptor | The 72-byte table property that points to the reference index block and holds the totals, the CF ID, the SST file number and the relocation-state fingerprint | | relocation-state fingerprint | 128-bit hash of the routes and relocation files of the Version that an SST was built from (fields route_snapshot_fingerprint_*). Written as 0 in this diff | | legacy | Written the way RocksDB writes it without blob indirection | Previous behavior: - A BlobIndex named only a physical blob file and an offset. A value could not move to another file without rewriting every SST that referenced it. - An SST had no metadata that listed its blob references. The only way to find them was to scan its user keys. - These readers get the blob indexes of a wide-column entity through ForEachBlobFileNumber: UpdateBoundaries, BlobGarbageMeter, the compaction iterator, WideColumnSerialization::RemoveBlobColumns, and the referenced_blob_bytes table property count in BlockBasedTableBuilder::Add. ForEachBlobFileNumber decoded each blob index, but it did not check the rest of the entity framing (name order, section sizes, exact input size), and it could call the callback before it found a later error. New behavior: Separate change (option on or off): - The same readers (UpdateBoundaries, BlobGarbageMeter, the compaction iterator, RemoveBlobColumns, and the referenced_blob_bytes count in BlockBasedTableBuilder::Add) use WideColumnSerialization::ForEachBlobIndex, with the option on or off. With the option on, the table builder also uses it to give the blob columns of each entity to the collector. For an entity with blob columns, it checks the full framing, then decodes and validates every blob index before any callback runs. So an entity with a malformed framing now fails the flush or compaction, even with the option off. The referenced_blob_bytes count itself does not fail the build on a malformed entity. It removes that property from the SST, as in base. Option and option gate: - This diff adds a new column family option, `enable_blob_indirection`. It is off by default, and SetOptions can change it. - The option gate (see Terms) keeps the option off outside tests until the stack is complete. ColumnFamilyData::ValidateOptions rejects the option with NotSupported on a read-write open, CreateColumnFamily and SetOptions. Read-only opens (DBImplReadOnly and CompactedDBImpl) and secondary opens (including DB::OpenAndCompact) do not run ValidateOptions; they check only the FIFO rule (see Implementation). Only tests can turn the option on, through the internal ColumnFamilyData::TEST_SetAllowBlobIndirection(). DBTestBase sets it. - When the option is on, ValidateOptions also requires enable_blob_files, the block-based table format, no blob direct write, and a compaction style other than FIFO. Blob file writer (option on): - The blob file writer encodes each new value with a stable identity (origin file, original offset) and two checksums: - the logical CRC: the CRC32C of the uncompressed value. - the stored CRC: the CRC32C of the exact stored (maybe compressed) bytes, extended with the compression byte. - For an uncompressed value the stored CRC is not written, because the reader can compute it from the logical CRC. - The checksums have no version field of their own. The BlobIndex type byte gives the format version: the two new types, 3 (kIndirectBlob) and 4 (kIndirectFlattenedBlob, see Implementation), are BlobIndex::kIndirectBlobFormatVersion (1). A later format uses new type values. - The read path does not check these checksums yet. The check starts in diff 3. Exact reference metadata (option on): 1. What: a sorted list of every blob reference that the SST keeps, in plain blob values and in the blob columns of wide-column entities. 2. Where: data blocks (target 16 KB each before compression; small blocks waste less I/O, because GC reads a whole block to get a few groups) and one reference index block, written after the SST data blocks and before the SST tail, plus a 72-byte descriptor table property. 3. Who reads it: only tests in this diff. The census, GC and RepairDB read it in later diffs. 4. Details: the top comment of db/blob/blob_reference_metadata.h has the full format with diagrams, the record-bytes rules and the gap rule. - Not collected: SstFileWriter output (it has no DB file number) and any output whose CF is unknown (it has no CF ID). The descriptor needs both values, so these outputs stay in the legacy form. Checks on the metadata: - A repeated or overlapping reference in one group fails the SST build with Corruption. Readers check every count, order, size and total against the index and the descriptor, so corrupt metadata returns Corruption. The writer never makes a data block larger than kTargetDataBlockSize (16 KB), and readers accept up to kMaxDataBlockSize (64 KB). Details: "Readers" and "Integrity" at the top of db/blob/blob_reference_metadata.h, and BlobReferenceGroupDecoder. Collection and memory: - Each reference is encoded when it is added, in the on-disk entry encoding, into its group's chain of 512-byte chunks. Only a group that takes the unsorted fallback is encoded a second time, in Finish(). Inside a group, references normally arrive in rising offset order, so no reference is sorted; at the end only the groups are sorted. A group that gets a lower offset takes the unsorted fallback. The BlobReferenceMetadataCollector class comment explains the chunks, the chunk size and the fallback. - The collector keeps every reference of the SST in memory until the SST is finished, and charges this memory to the block cache under a new CacheEntryRole, kBlobReferenceMetadataBuildingBuffer. The measured cost is about 8.3 to 9 bytes per reference for uncompressed values and 12.8 to 13.5 bytes for compressed values. "Memory usage" and "Memory charge" in the BlobReferenceMetadataCollector class comment give the measurement setup and the charge rule. - There is no fixed limit on references. The collector fails with MemoryLimit above 2^20 groups or 2^20 data blocks in one SST; these are persisted format limits. - There is no charge when the table has no block cache or when charging for this role is disabled. Each block-based table factory holds one cache reservation manager for this role, shared by its builders and the SST summaries. If the charge fails, the SST build fails, so the flush or compaction fails with a background error. The buffers and their charge are released when the SST is finished or abandoned. Implementation: - BlobIndex adds two indirect encodings. The legacy encodings do not change. - Type 3, kIndirectBlob (unflattened, generation number == origin), stores the origin, the original offset, the stored size, the compression type, the logical CRC and, for compressed values, the stored CRC. It does not store the generation number. This saves the size of one varint file number per reference (about 3 to 4 bytes for typical file numbers; an estimate, not measured). - Type 4, kIndirectFlattenedBlob (flattened), stores the generation number first, then the same fields. Its generation number must be above the origin. The decoder rejects type 4 with generation number <= origin, so each reference has one encoding. - The encoders take the fields as one BlobIndex::IndirectBlobFields, set by field name, so values of the same type (file number, offset and size; the two CRCs) cannot be swapped. EncodeIndirectBlob() takes the generation number as a separate argument; EncodeUnflattenedIndirectBlob() uses the origin, so the common path cannot leave the generation number unset. - The new files blob_reference_metadata.{h,cc} define the exact reference metadata. The files also contain: - BlobReferenceMetadataCollector, the collector described above. It takes the SST identity (CF ID, SST file number and the relocation-state fingerprint) as one BlobReferenceSstIdentity, set by field name. - BlobReferenceGroupKey, the key of a group: (encoding class, generation number, origin). It has the order, the hash and the validity check that the collector map, the sort, the decoder and FindBlobReferenceGroup() use. BlobReference and BlobReferenceGroup keep their flat fields and return the key through group_key(). - The index codec and the descriptor table property. - BlobReferenceGroupDecoder, the only decoder of reference runs. It reads a group one data block at a time and holds every corruption check of a group walk. Readers that visit one group (found by a binary search of the index) use it, and so do readers that visit every block in order. - Callers in this diff: only the table builder runs the collector. The decoders (DecodeBlobReferenceIndex, FindBlobReferenceGroup, BlobReferenceGroupDecoder, ForEachBlobReference*) are used only by tests. kMaxDecodedBlobReferenceBytes is used only in compile-time checks of the count limits. The census (diff 5) and GC (diffs 6 and 7) are their first production users. The census is also the first user of BlobReferenceSstSummary (see the last Implementation bullet). - Descriptor: the CF ID and the SST file number are the repair fields. They let RepairDB tell which column family and SST the list belongs to without a MANIFEST. The decoder rejects a descriptor whose repair fields are not set (unknown CF ID or SST file number 0). The descriptor also has a 128-bit relocation-state fingerprint (the route_snapshot_fingerprint fields). Remote compaction uses it in diff 9 to check that an SST built elsewhere used the same routes as the job that asked for it. In this diff the table builder always writes 0, and DecodeFrom() does not check it. The field is in the format now, so the descriptor keeps one format version (1) through the stack. While the option gate is on, no data in this format exists outside tests. - The collector validates each BlobIndex once, when it is added. Its Finish() writes the data blocks and then the reference index block through one block writer, and fills the descriptor, including the index handle. Destroying the collector frees its memory and then its charge, also when the build is abandoned. - BlockBasedTableBuilder creates the collector with the CF ID and SST file number from TableBuilderOptions. It passes the size of each entry's user key to the collector, so the collector can compute the exact size of each blob record. It adds the collector's EstimatedEncodedSize() to its tail size estimate: the blocks are written before the tail, but output file size planning uses this estimate. The reference blocks are written with BlockType::kUserDefinedIndex, which keeps their contents as opaque bytes. Readers read them without the block cache, so prepopulate_block_cache does not warm them into the block cache (WriteMaybeCompressedBlock() with warm_cache false). - With the option on, the blob file builder computes the CRC32C of each stored value once. The blob log record CRC combines it with the key CRC (crc32c::Crc32cCombine). The stored CRC extends it with the compression byte. For an uncompressed value it is also the logical CRC. The bytes written do not change. - WideColumnSerialization::ForEachBlobIndex replaces ForEachBlobFileNumber. It reads the entity header and column types once. For a V1 entity (no column types and no blob columns), it returns right after the header. For a V2 entity without blob columns, it returns right after the type scan. It shares one V2 column walker (WalkV2Columns) with Deserialize(), so the entity is parsed in one pass. - Read-only opens (DBImplReadOnly, CompactedDBImpl) and secondary opens skip ValidateOptions, so they check the compaction style against the option once per column family: in DBImpl::Recover() for read-only opens, and at the start of OpenAsSecondaryImpl() for secondary opens. DB::OpenAndCompact opens through OpenAsSecondaryImpl(), so it gets the same check. - CacheReservationHandle gets Resize(), so each collector or summary can change its own part of the shared charge. - util/coding.h adds GetCanonicalVarint32/64, which reject overlong varint encodings. The new formats use them, so each value has one encoding. - BlobReferenceSstSummary is the in-memory list of an SST's groups and totals that Finish() can return. It has no caller yet: the table builder passes a null summary output. It is checked and does not change after Finish(), so the finished builder, the SST metadata and the census (its first user, in diff 5) can share it. It is in this diff because Finish() hands the group buffer and its cache charge over to the summary. Adding the summary later would mean rewriting the charge code of Finish() in diff 5, so the charge code is written and tested once, here. Safety boundaries and unsupported cases: - Public API: this diff adds the option `enable_blob_indirection` with the C API functions `rocksdb_options_set_enable_blob_indirection` and `rocksdb_options_get_enable_blob_indirection`. It also inserts a new CacheEntryRole value before kMisc, which changes the numeric value of kMisc and kNumCacheEntryRoles. - With the option off, new blob values and SSTs are written in the legacy form. Legacy blob indexes stay readable. - One behavior change applies with the option off too: UpdateBoundaries, BlobGarbageMeter and the compaction iterator fully validate wide-column entities with blob columns and their blob indexes (see "Separate change" above). - Exact reference metadata has one format (version 1) with a fixed 72-byte descriptor. The decoders reject any other version as Corruption. - The metadata blocks are protected by the table's block checksum only. With checksum = kNoChecksum, corrupt metadata is detected only when it breaks the format or order checks. - The index decoder checks each decoded count against the remaining input size before it reserves memory. So a small corrupt index cannot make it allocate a large vector. - Inlined-TTL and same-file blob indexes are not collected and cannot be relocated. Encoding, count and checksum errors return an error; a bad reference is never skipped. - Option gate (see Terms): outside tests no DB writes data in the new format. The gate checks only the option value, never persisted state. Diff 11 removes it, once data written with the option stays readable through the end of the stack and the safety checks for turning the option on are in place. - Downgrading the binary after data was written with the option on is not supported. Validation and known limitations: - Unit tests cover the BlobIndex encodings and their rejection of bad input, the metadata layout and every corruption check, the collector (sorted path, unsorted fallback, memory charge), the flush integration, the option checks and option gate, wide-column validation, and canonical varints. See the Test Plan. - The collection buffer is not spilled to disk. One SST with very many blob references needs a matching amount of charged memory: the per-reference amount given under "Collection and memory", plus up to one partly used 512-byte chunk per group. While a fallback group is sorted, it also needs 16 bytes of sort key per reference and a second copy of its entries. For a test group of 200K references with every other one out of order, the test comment estimates about 44 bytes per reference, and the test checks that the peak charge is at most 56 bytes per reference. - Flush still writes a single output file. Until diff 7, one compaction output collects every reference it gets, charged to the block cache. Diff 7 cuts compaction outputs (including L0) at a reference cap and keeps each user key's history in one output. With the option gate, a large output is reachable only through the test hook. Stack context (same in diffs 2-11, except that diffs 8-11 add the diff 8 route rule to the install line; "<== this diff" marks this diff's part): Goal: remove garbage from blob files without rewriting any SST. A value gets a stable identity when it is first written. GC copies the live values of whole files into a relocation file and installs routes in the MANIFEST. Readers follow the routes. SSTs are not changed. How a read finds a value (P is the relocation file that holds the value now): ``` SST entry indirect BlobIndex { generation number, origin, original offset, size, CRCs } | v Version file table: generation number -> live file, or route old file -> P | | live file route found | | v v blob file: read at original offset relocation file P (always one hop): relocation file: look up key look up key (origin, original offset) (origin, original offset) ``` Lifecycle of one value (in this example, F is the blob file where the value is first written, and P is the relocation file that GC writes): ``` write new value goes to blob file F; index: origin = F, generation number = F | census per generation number: the SSTs and references that use it; | per physical file: live bytes, ranked by garbage ratio GC job copy the live values of whole input files (stored bytes, checked | with the stored CRC) into relocation file P, keyed by | (origin, original offset) install one MANIFEST edit: add P, route each input generation number still | in use to P, re-point the routes into the inputs to P, and remove | each input (blob file retirement for a blob file, relocation | file deletion for a relocation file) flatten compaction sets generation number = P in the SSTs it rewrites | route route retirement: route deletion records remove the routes that delete no SST uses ``` Stack map: | Diff | Adds | |---|---| | 1 | Finalize blob metadata outside the DB mutex (prep) | | 2 | Indirect BlobIndex (kIndirectBlob) and exact SST reference metadata <== this diff | | 3 | Relocation file format and stable read path | | 4 | MANIFEST routes and Version ownership | | 5 | Exact reverse reference census | | 6 | Standalone relocation GC (one input file) | | 7 | Pack whole input files; reclaim relocation files | | 8 | Flatten generation numbers; route retirement | | 9 | Remote compaction validation | | 10 | Dynamic options and operations | | 11 | RepairDB rebuilds relocation state; option gate removed | | 12 | db_stress and crash test support | Stack context: terms for later diffs: | Term | Meaning | |---|---| | census | Background count, per generation number, of the SSTs and references that use it, plus live bytes per physical file (diff 5) | | indirect origin | Normal blob file written with the option on; it is the origin of unflattened references | | flatten | Compaction rewrites the generation number to the relocation file that holds the value, so old routes can go. Only flatten makes generation number != origin (diff 8) | | blob file retirement | MANIFEST record (kBlobFileRetirement) that removes a collected blob file from the Version (diff 4) | | route deletion | MANIFEST record that removes a route that no SST uses. Route retirement is the background job that writes these records (diff 8) | Tracking issue: meta#15301 Differential Revision: D124652097
Summary: Pull Request resolved: meta#15540 Why the change is required: The goal of this stack is a blob GC that does not depend on SSTs: it removes garbage from blob files without rewriting any SST. Such a GC needs two things that RocksDB does not have: - A value identity that does not change when the value moves to another file. - A checked list of the blob references that each SST keeps. The list must include older internal versions of a key that the SST still keeps (for example for snapshots), and the blob columns of wide-column entities. This diff adds both. Later diffs build the census (an in-memory reverse index from generation number to the SSTs that use it), GC and repair on top of this list. This diff in one paragraph: With the new column family option enable_blob_indirection on, the blob file writer gives each new value an indirect BlobIndex (type 3, kIndirectBlob). It stores the value's stable identity (origin file, original offset) and its CRCs (an uncompressed value stores only the CRC of the uncompressed value; a compressed value also stores the CRC of its stored bytes). Also with the option on, the block-based table builder collects every blob reference that the SST keeps. It writes them as "exact reference metadata": data blocks, one reference index block, and a 72-byte descriptor table property. In this diff only the table builder writes this metadata, and only tests read it. A temporary option gate (see "option gate" under Terms) keeps the option off outside tests. Separately, wide-column entities with blob columns are now fully checked before their blob indexes are used, also with the option off. Parts of this diff: 1. Indirect BlobIndex (core): types 3 and 4 in db/blob/blob_index.h, and the blob file writer that writes type 3. 2. Exact reference metadata format and decoders (core): db/blob/blob_reference_metadata.{h,cc}. 3. Collector and table builder integration (core): BlobReferenceMetadataCollector, its block cache charge, and BlockBasedTableBuilder. 4. Option enable_blob_indirection and the option gate. Tests need it to turn the feature on. You can skip this part on a first read. 5. Wide-column validation (WideColumnSerialization::ForEachBlobIndex). The table builder uses it to give the blob columns of each entity to the collector. It also changes behavior with the option off (see New behavior). You can skip this part on a first read. 6. Small helpers: canonical varints (util/coding.h) and CacheReservationHandle::Resize(). You can skip these on a first read. How to read this diff: 1. Overview: read "Why" and "This diff in one paragraph" above, and "Terms used in this diff" right below. Then read the top comment of db/blob/blob_reference_metadata.h (terms, layout diagrams, run and entry layout, index layout). 2. Indirect BlobIndex (db/blob/blob_index.h) and canonical varints (util/coding.h GetCanonicalVarint32/64): read the overview comment and the kIndirectBlob / kIndirectFlattenedBlob code. Every later part uses these fields: origin, original offset, generation number, logical CRC and stored CRC. The new formats read varints only in the canonical form. 3. Blob file writer (db/blob/blob_file_builder.cc, blob_log_format.cc, blob_log_writer.cc): see how a new value gets a type-3 index and how its CRC is computed once. 4. Format types and decoders: the structs in db/blob/blob_reference_metadata.h, then in the .cc the constants, GapBase, DecodeEntryBody, DecodeRunEntriesAt, DecodeBlobReferenceIndex and BlobReferenceGroupDecoder. They use only BlobIndex and the canonical varints. Learn the on-disk format here, before you read the writer. 5. Collector and its cache charge: first read the new CacheEntryRole kBlobReferenceMetadataBuildingBuffer (include/rocksdb/cache.h) and CacheReservationHandle::Resize (cache/cache_reservation_manager.h). Then read BlobReferenceMetadataCollector in this order: the class comment, Add, AppendSorted, BuildAndWriteDataBlocks, EncodeIndex, FinishOnce. Then read the unsorted fallback (AppendUnsorted, SortUnsortedGroup, SortAndEncodeGroup). 6. Wide-column change (db/wide/wide_column_serialization.cc, ForEachBlobIndex and WalkV2Columns): it gives every blob index of an entity to a callback after it checks the whole entity. The table builder uses it to feed the collector. UpdateBoundaries, BlobGarbageMeter and the compaction iterator also use it, with the option on or off. 7. Option: enable_blob_indirection (include/rocksdb/advanced_options.h, options/cf_options.cc), and the option checks and gate in db/column_family.cc (ValidateOptions, TEST_SetAllowBlobIndirection), db/db_impl/db_impl_open.cc and db_impl_secondary.cc (the FIFO check for opens that skip ValidateOptions). 8. Table integration: table/block_based/block_based_table_factory.cc (the shared reservation manager), then block_based_table_builder.cc (Rep constructor, Add, WriteBlobReferenceMetadata, EstimatedTailSize). See where the collector is created, fed and finished. block_based_table_reader.cc maps the new block name to a block type. Terms used in this diff: Stack terms: | Term | Meaning | |---|---| | origin | Blob file where the value was first written | | original offset | Value offset in the origin file | | stable identity | (origin, original offset). Never changes, even when the value moves | | generation number | The file number that a reference uses to find its value. A new value has generation number == origin. When GC moves the value, the reference keeps its generation number, and a route sends the read to the relocation file | | indirect BlobIndex | A kIndirectBlob (type 3) or kIndirectFlattenedBlob (type 4) index. It stores the stable identity, so the value can move without an SST rewrite | | unflattened reference | Index with generation number == origin: the value was not moved, or it was moved but no compaction has rewritten its reference yet (see "flatten" in "Stack context: terms for later diffs") | | flattened reference | Index with generation number != origin (type 4). Only flatten (diff 8) writes it; this diff defines the encoding | | route | Maps an old file number (a file that GC has removed from the Version) to the relocation file that now holds its live values. One hop: a target is never itself routed. Stored in the MANIFEST (diff 4) | | relocation file | Block-based table keyed by (origin, original offset) that holds copied values. Each value is the stored bytes plus a 5-byte trailer (kSimpleGen2BlobTrailerSize) | | logical CRC | CRC32C of the uncompressed value (BlobIndex::stable_checksum()) | | stored CRC | CRC32C of the stored bytes plus the compression byte | | physical file | A file that holds value bytes: a normal blob file or a relocation file. A generation number can name a physical file, or it can be only the key of a route | | exact reference metadata | Per-SST sorted list of every blob reference the SST keeps, written when the SST is built. References with the same encoding class, generation number and origin form a group. The list is stored as data blocks, with an index of the groups. An SST that has it is an exact SST | | option gate | Until diff 11, ColumnFamilyData::ValidateOptions returns NotSupported for enable_blob_indirection on a read-write open, CreateColumnFamily and SetOptions, unless a test calls ColumnFamilyData::TEST_SetAllowBlobIndirection(true). It keeps the option off in production while the stack is not complete | Terms of this diff (all defined at the top of db/blob/blob_reference_metadata.h): | Term | Meaning | |---|---| | encoding class (class) | kDirect (a legacy kBlob or kBlobTTL index), unflattened, or flattened | | reference group (group) | All references of one SST with the same (encoding class, generation number, origin) | | run | The part of one group inside one data block | | record bytes | The bytes the value's record takes in the file named by its generation number | | gap | How far a record is from where it would be if the records were back to back. A gap of 0 means back to back | | descriptor | The 72-byte table property that points to the reference index block and holds the totals, the CF ID, the SST file number and the relocation-state fingerprint | | relocation-state fingerprint | 128-bit hash of the routes and relocation files of the Version that an SST was built from (fields route_snapshot_fingerprint_*). Written as 0 in this diff | | legacy | Written the way RocksDB writes it without blob indirection | Previous behavior: - A BlobIndex named only a physical blob file and an offset. A value could not move to another file without rewriting every SST that referenced it. - An SST had no metadata that listed its blob references. The only way to find them was to scan its user keys. - These readers get the blob indexes of a wide-column entity through ForEachBlobFileNumber: UpdateBoundaries, BlobGarbageMeter, the compaction iterator, WideColumnSerialization::RemoveBlobColumns, and the referenced_blob_bytes table property count in BlockBasedTableBuilder::Add. ForEachBlobFileNumber decoded each blob index, but it did not check the rest of the entity framing (name order, section sizes, exact input size), and it could call the callback before it found a later error. New behavior: Separate change (option on or off): - The same readers (UpdateBoundaries, BlobGarbageMeter, the compaction iterator, RemoveBlobColumns, and the referenced_blob_bytes count in BlockBasedTableBuilder::Add) use WideColumnSerialization::ForEachBlobIndex, with the option on or off. With the option on, the table builder also uses it to give the blob columns of each entity to the collector. For an entity with blob columns, it checks the full framing, then decodes and validates every blob index before any callback runs. So an entity with a malformed framing now fails the flush or compaction, even with the option off. The referenced_blob_bytes count itself does not fail the build on a malformed entity. It removes that property from the SST, as in base. Option and option gate: - This diff adds a new column family option, `enable_blob_indirection`. It is off by default, and SetOptions can change it. - The option gate (see Terms) keeps the option off outside tests until the stack is complete. ColumnFamilyData::ValidateOptions rejects the option with NotSupported on a read-write open, CreateColumnFamily and SetOptions. Read-only opens (DBImplReadOnly and CompactedDBImpl) and secondary opens (including DB::OpenAndCompact) do not run ValidateOptions; they check only the FIFO rule (see Implementation). Only tests can turn the option on, through the internal ColumnFamilyData::TEST_SetAllowBlobIndirection(). DBTestBase sets it. - When the option is on, ValidateOptions also requires enable_blob_files, the block-based table format, no blob direct write, and a compaction style other than FIFO. Blob file writer (option on): - The blob file writer encodes each new value with a stable identity (origin file, original offset) and two checksums: - the logical CRC: the CRC32C of the uncompressed value. - the stored CRC: the CRC32C of the exact stored (maybe compressed) bytes, extended with the compression byte. - For an uncompressed value the stored CRC is not written, because the reader can compute it from the logical CRC. - The checksums have no version field of their own. The BlobIndex type byte gives the format version: the two new types, 3 (kIndirectBlob) and 4 (kIndirectFlattenedBlob, see Implementation), are BlobIndex::kIndirectBlobFormatVersion (1). A later format uses new type values. - The read path does not check these checksums yet. The check starts in diff 3. Exact reference metadata (option on): 1. What: a sorted list of every blob reference that the SST keeps, in plain blob values and in the blob columns of wide-column entities. 2. Where: data blocks (target 16 KB each before compression; small blocks waste less I/O, because GC reads a whole block to get a few groups) and one reference index block, written after the SST data blocks and before the SST tail, plus a 72-byte descriptor table property. 3. Who reads it: only tests in this diff. The census, GC and RepairDB read it in later diffs. 4. Details: the top comment of db/blob/blob_reference_metadata.h has the full format with diagrams, the record-bytes rules and the gap rule. - Not collected: SstFileWriter output (it has no DB file number) and any output whose CF is unknown (it has no CF ID). The descriptor needs both values, so these outputs stay in the legacy form. Checks on the metadata: - A repeated or overlapping reference in one group fails the SST build with Corruption. Readers check every count, order, size and total against the index and the descriptor, so corrupt metadata returns Corruption. The writer never makes a data block larger than kTargetDataBlockSize (16 KB), and readers accept up to kMaxDataBlockSize (64 KB). Details: "Readers" and "Integrity" at the top of db/blob/blob_reference_metadata.h, and BlobReferenceGroupDecoder. Collection and memory: - Each reference is encoded when it is added, in the on-disk entry encoding, into its group's chain of 512-byte chunks. Only a group that takes the unsorted fallback is encoded a second time, in Finish(). Inside a group, references normally arrive in rising offset order, so no reference is sorted; at the end only the groups are sorted. A group that gets a lower offset takes the unsorted fallback. The BlobReferenceMetadataCollector class comment explains the chunks, the chunk size and the fallback. - The collector keeps every reference of the SST in memory until the SST is finished, and charges this memory to the block cache under a new CacheEntryRole, kBlobReferenceMetadataBuildingBuffer. The measured cost is about 8.3 to 9 bytes per reference for uncompressed values and 12.8 to 13.5 bytes for compressed values. "Memory usage" and "Memory charge" in the BlobReferenceMetadataCollector class comment give the measurement setup and the charge rule. - There is no fixed limit on references. The collector fails with MemoryLimit above 2^20 groups or 2^20 data blocks in one SST; these are persisted format limits. - There is no charge when the table has no block cache or when charging for this role is disabled. Each block-based table factory holds one cache reservation manager for this role, shared by its builders and the SST summaries. If the charge fails, the SST build fails, so the flush or compaction fails with a background error. The buffers and their charge are released when the SST is finished or abandoned. Implementation: - BlobIndex adds two indirect encodings. The legacy encodings do not change. - Type 3, kIndirectBlob (unflattened, generation number == origin), stores the origin, the original offset, the stored size, the compression type, the logical CRC and, for compressed values, the stored CRC. It does not store the generation number. This saves the size of one varint file number per reference (about 3 to 4 bytes for typical file numbers; an estimate, not measured). - Type 4, kIndirectFlattenedBlob (flattened), stores the generation number first, then the same fields. Its generation number must be above the origin. The decoder rejects type 4 with generation number <= origin, so each reference has one encoding. - The encoders take the fields as one BlobIndex::IndirectBlobFields, set by field name, so values of the same type (file number, offset and size; the two CRCs) cannot be swapped. EncodeIndirectBlob() takes the generation number as a separate argument; EncodeUnflattenedIndirectBlob() uses the origin, so the common path cannot leave the generation number unset. - The new files blob_reference_metadata.{h,cc} define the exact reference metadata. The files also contain: - BlobReferenceMetadataCollector, the collector described above. It takes the SST identity (CF ID, SST file number and the relocation-state fingerprint) as one BlobReferenceSstIdentity, set by field name. - BlobReferenceGroupKey, the key of a group: (encoding class, generation number, origin). It has the order, the hash and the validity check that the collector map, the sort, the decoder and FindBlobReferenceGroup() use. BlobReference and BlobReferenceGroup keep their flat fields and return the key through group_key(). - The index codec and the descriptor table property. - BlobReferenceGroupDecoder, the only decoder of reference runs. It reads a group one data block at a time and holds every corruption check of a group walk. Readers that visit one group (found by a binary search of the index) use it, and so do readers that visit every block in order. - Callers in this diff: only the table builder runs the collector. The decoders (DecodeBlobReferenceIndex, FindBlobReferenceGroup, BlobReferenceGroupDecoder, ForEachBlobReference*) are used only by tests. kMaxDecodedBlobReferenceBytes is used only in compile-time checks of the count limits. The census (diff 5) and GC (diffs 6 and 7) are their first production users. The census is also the first user of BlobReferenceSstSummary (see the last Implementation bullet). - Descriptor: the CF ID and the SST file number are the repair fields. They let RepairDB tell which column family and SST the list belongs to without a MANIFEST. The decoder rejects a descriptor whose repair fields are not set (unknown CF ID or SST file number 0). The descriptor also has a 128-bit relocation-state fingerprint (the route_snapshot_fingerprint fields). Remote compaction uses it in diff 9 to check that an SST built elsewhere used the same routes as the job that asked for it. In this diff the table builder always writes 0, and DecodeFrom() does not check it. The field is in the format now, so the descriptor keeps one format version (1) through the stack. While the option gate is on, no data in this format exists outside tests. - The collector validates each BlobIndex once, when it is added. Its Finish() writes the data blocks and then the reference index block through one block writer, and fills the descriptor, including the index handle. Destroying the collector frees its memory and then its charge, also when the build is abandoned. - BlockBasedTableBuilder creates the collector with the CF ID and SST file number from TableBuilderOptions. It passes the size of each entry's user key to the collector, so the collector can compute the exact size of each blob record. It adds the collector's EstimatedEncodedSize() to its tail size estimate: the blocks are written before the tail, but output file size planning uses this estimate. The reference blocks are written with BlockType::kUserDefinedIndex, which keeps their contents as opaque bytes. Readers read them without the block cache, so prepopulate_block_cache does not warm them into the block cache (WriteMaybeCompressedBlock() with warm_cache false). - With the option on, the blob file builder computes the CRC32C of each stored value once. The blob log record CRC combines it with the key CRC (crc32c::Crc32cCombine). The stored CRC extends it with the compression byte. For an uncompressed value it is also the logical CRC. The bytes written do not change. - WideColumnSerialization::ForEachBlobIndex replaces ForEachBlobFileNumber. It reads the entity header and column types once. For a V1 entity (no column types and no blob columns), it returns right after the header. For a V2 entity without blob columns, it returns right after the type scan. It shares one V2 column walker (WalkV2Columns) with Deserialize(), so the entity is parsed in one pass. - Read-only opens (DBImplReadOnly, CompactedDBImpl) and secondary opens skip ValidateOptions, so they check the compaction style against the option once per column family: in DBImpl::Recover() for read-only opens, and at the start of OpenAsSecondaryImpl() for secondary opens. DB::OpenAndCompact opens through OpenAsSecondaryImpl(), so it gets the same check. - CacheReservationHandle gets Resize(), so each collector or summary can change its own part of the shared charge. - util/coding.h adds GetCanonicalVarint32/64, which reject overlong varint encodings. The new formats use them, so each value has one encoding. - BlobReferenceSstSummary is the in-memory list of an SST's groups and totals that Finish() can return. It has no caller yet: the table builder passes a null summary output. It is checked and does not change after Finish(), so the finished builder, the SST metadata and the census (its first user, in diff 5) can share it. It is in this diff because Finish() hands the group buffer and its cache charge over to the summary. Adding the summary later would mean rewriting the charge code of Finish() in diff 5, so the charge code is written and tested once, here. Safety boundaries and unsupported cases: - Public API: this diff adds the option `enable_blob_indirection` with the C API functions `rocksdb_options_set_enable_blob_indirection` and `rocksdb_options_get_enable_blob_indirection`. It also inserts a new CacheEntryRole value before kMisc, which changes the numeric value of kMisc and kNumCacheEntryRoles. - With the option off, new blob values and SSTs are written in the legacy form. Legacy blob indexes stay readable. - One behavior change applies with the option off too: UpdateBoundaries, BlobGarbageMeter and the compaction iterator fully validate wide-column entities with blob columns and their blob indexes (see "Separate change" above). - Exact reference metadata has one format (version 1) with a fixed 72-byte descriptor. The decoders reject any other version as Corruption. - The metadata blocks are protected by the table's block checksum only. With checksum = kNoChecksum, corrupt metadata is detected only when it breaks the format or order checks. - The index decoder checks each decoded count against the remaining input size before it reserves memory. So a small corrupt index cannot make it allocate a large vector. - Inlined-TTL and same-file blob indexes are not collected and cannot be relocated. Encoding, count and checksum errors return an error; a bad reference is never skipped. - Option gate (see Terms): outside tests no DB writes data in the new format. The gate checks only the option value, never persisted state. Diff 11 removes it, once data written with the option stays readable through the end of the stack and the safety checks for turning the option on are in place. - Downgrading the binary after data was written with the option on is not supported. Validation and known limitations: - Unit tests cover the BlobIndex encodings and their rejection of bad input, the metadata layout and every corruption check, the collector (sorted path, unsorted fallback, memory charge), the flush integration, the option checks and option gate, wide-column validation, and canonical varints. See the Test Plan. - The collection buffer is not spilled to disk. One SST with very many blob references needs a matching amount of charged memory: the per-reference amount given under "Collection and memory", plus up to one partly used 512-byte chunk per group. While a fallback group is sorted, it also needs 16 bytes of sort key per reference and a second copy of its entries. For a test group of 200K references with every other one out of order, the test comment estimates about 44 bytes per reference, and the test checks that the peak charge is at most 56 bytes per reference. - Flush still writes a single output file. Until diff 7, one compaction output collects every reference it gets, charged to the block cache. Diff 7 cuts compaction outputs (including L0) at a reference cap and keeps each user key's history in one output. With the option gate, a large output is reachable only through the test hook. Stack context (same in diffs 2-11, except that diffs 8-11 add the diff 8 route rule to the install line; "<== this diff" marks this diff's part): Goal: remove garbage from blob files without rewriting any SST. A value gets a stable identity when it is first written. GC copies the live values of whole files into a relocation file and installs routes in the MANIFEST. Readers follow the routes. SSTs are not changed. How a read finds a value (P is the relocation file that holds the value now): ``` SST entry indirect BlobIndex { generation number, origin, original offset, size, CRCs } | v Version file table: generation number -> live file, or route old file -> P | | live file route found | | v v blob file: read at original offset relocation file P (always one hop): relocation file: look up key look up key (origin, original offset) (origin, original offset) ``` Lifecycle of one value (in this example, F is the blob file where the value is first written, and P is the relocation file that GC writes): ``` write new value goes to blob file F; index: origin = F, generation number = F | census per generation number: the SSTs and references that use it; | per physical file: live bytes, ranked by garbage ratio GC job copy the live values of whole input files (stored bytes, checked | with the stored CRC) into relocation file P, keyed by | (origin, original offset) install one MANIFEST edit: add P, route each input generation number still | in use to P, re-point the routes into the inputs to P, and remove | each input (blob file retirement for a blob file, relocation | file deletion for a relocation file) flatten compaction sets generation number = P in the SSTs it rewrites | route route retirement: route deletion records remove the routes that delete no SST uses ``` Stack map: | Diff | Adds | |---|---| | 1 | Finalize blob metadata outside the DB mutex (prep) | | 2 | Indirect BlobIndex (kIndirectBlob) and exact SST reference metadata <== this diff | | 3 | Relocation file format and stable read path | | 4 | MANIFEST routes and Version ownership | | 5 | Exact reverse reference census | | 6 | Standalone relocation GC (one input file) | | 7 | Pack whole input files; reclaim relocation files | | 8 | Flatten generation numbers; route retirement | | 9 | Remote compaction validation | | 10 | Dynamic options and operations | | 11 | RepairDB rebuilds relocation state; option gate removed | | 12 | db_stress and crash test support | Stack context: terms for later diffs: | Term | Meaning | |---|---| | census | Background count, per generation number, of the SSTs and references that use it, plus live bytes per physical file (diff 5) | | indirect origin | Normal blob file written with the option on; it is the origin of unflattened references | | flatten | Compaction rewrites the generation number to the relocation file that holds the value, so old routes can go. Only flatten makes generation number != origin (diff 8) | | blob file retirement | MANIFEST record (kBlobFileRetirement) that removes a collected blob file from the Version (diff 4) | | route deletion | MANIFEST record that removes a route that no SST uses. Route retirement is the background job that writes these records (diff 8) | Tracking issue: meta#15301 Differential Revision: D124652097
Summary: Pull Request resolved: meta#15540 Why the change is required: The goal of this stack is a blob GC that does not depend on SSTs: it removes garbage from blob files without rewriting any SST. Such a GC needs two things that RocksDB does not have: - A value identity that does not change when the value moves to another file. - A checked list of the blob references that each SST keeps. The list must include older internal versions of a key that the SST still keeps (for example for snapshots), and the blob columns of wide-column entities. This diff adds both. Later diffs build the census (an in-memory reverse index from generation number to the SSTs that use it), GC and repair on top of this list. This diff in one paragraph: With the new column family option enable_blob_indirection on, the blob file writer gives each new value an indirect BlobIndex (type 3, kIndirectBlob). It stores the value's stable identity (origin file, original offset) and its CRCs (an uncompressed value stores only the CRC of the uncompressed value; a compressed value also stores the CRC of its stored bytes). Also with the option on, the block-based table builder collects every blob reference that the SST keeps. It writes them as "exact reference metadata": data blocks, one reference index block, and a 72-byte descriptor table property. In this diff only the table builder writes this metadata, and only tests read it. A temporary option gate (see "option gate" under Terms) keeps the option off outside tests. Separately, wide-column entities with blob columns are now fully checked before their blob indexes are used, also with the option off. Parts of this diff: 1. Indirect BlobIndex (core): types 3 and 4 in db/blob/blob_index.h, and the blob file writer that writes type 3. 2. Exact reference metadata format and decoders (core): db/blob/blob_reference_metadata.{h,cc}. 3. Collector and table builder integration (core): BlobReferenceMetadataCollector, its block cache charge, and BlockBasedTableBuilder. 4. Option enable_blob_indirection and the option gate. Tests need it to turn the feature on. You can skip this part on a first read. 5. Wide-column validation (WideColumnSerialization::ForEachBlobIndex). The table builder uses it to give the blob columns of each entity to the collector. It also changes behavior with the option off (see New behavior). You can skip this part on a first read. 6. Small helpers: canonical varints (util/coding.h) and CacheReservationHandle::Resize(). You can skip these on a first read. How to read this diff: 1. Overview: read "Why" and "This diff in one paragraph" above, and "Terms used in this diff" right below. Then read the top comment of db/blob/blob_reference_metadata.h (terms, layout diagrams, run and entry layout, index layout). 2. Indirect BlobIndex (db/blob/blob_index.h) and canonical varints (util/coding.h GetCanonicalVarint32/64): read the overview comment and the kIndirectBlob / kIndirectFlattenedBlob code. Every later part uses these fields: origin, original offset, generation number, logical CRC and stored CRC. The new formats read varints only in the canonical form. 3. Blob file writer (db/blob/blob_file_builder.cc, blob_log_format.cc, blob_log_writer.cc): see how a new value gets a type-3 index and how its CRC is computed once. 4. Format types and decoders: the structs in db/blob/blob_reference_metadata.h, then in the .cc the constants, GapBase, DecodeEntryBody, DecodeRunEntriesAt, DecodeBlobReferenceIndex and BlobReferenceGroupDecoder. They use only BlobIndex and the canonical varints. Learn the on-disk format here, before you read the writer. 5. Collector and its cache charge: first read the new CacheEntryRole kBlobReferenceMetadataBuildingBuffer (include/rocksdb/cache.h) and CacheReservationHandle::Resize (cache/cache_reservation_manager.h). Then read BlobReferenceMetadataCollector in this order: the class comment, Add, AppendSorted, BuildAndWriteDataBlocks, EncodeIndex, FinishOnce. Then read the unsorted fallback (AppendUnsorted, SortUnsortedGroup, SortAndEncodeGroup). 6. Wide-column change (db/wide/wide_column_serialization.cc, ForEachBlobIndex and WalkV2Columns): it gives every blob index of an entity to a callback after it checks the whole entity. The table builder uses it to feed the collector. UpdateBoundaries, BlobGarbageMeter and the compaction iterator also use it, with the option on or off. 7. Option: enable_blob_indirection (include/rocksdb/advanced_options.h, options/cf_options.cc), and the option checks and gate in db/column_family.cc (ValidateOptions, TEST_SetAllowBlobIndirection), db/db_impl/db_impl_open.cc and db_impl_secondary.cc (the FIFO check for opens that skip ValidateOptions). 8. Table integration: table/block_based/block_based_table_factory.cc (the shared reservation manager), then block_based_table_builder.cc (Rep constructor, Add, WriteBlobReferenceMetadata, EstimatedTailSize). See where the collector is created, fed and finished. block_based_table_reader.cc maps the new block name to a block type. Terms used in this diff: Stack terms: | Term | Meaning | |---|---| | origin | Blob file where the value was first written | | original offset | Value offset in the origin file | | stable identity | (origin, original offset). Never changes, even when the value moves | | generation number | The file number that a reference uses to find its value. A new value has generation number == origin. When GC moves the value, the reference keeps its generation number, and a route sends the read to the relocation file | | indirect BlobIndex | A kIndirectBlob (type 3) or kIndirectFlattenedBlob (type 4) index. It stores the stable identity, so the value can move without an SST rewrite | | unflattened reference | Index with generation number == origin: the value was not moved, or it was moved but no compaction has rewritten its reference yet (see "flatten" in "Stack context: terms for later diffs") | | flattened reference | Index with generation number != origin (type 4). Only flatten (diff 8) writes it; this diff defines the encoding | | route | Maps an old file number (a file that GC has removed from the Version) to the relocation file that now holds its live values. One hop: a target is never itself routed. Stored in the MANIFEST (diff 4) | | relocation file | Block-based table keyed by (origin, original offset) that holds copied values. Each value is the stored bytes plus a 5-byte trailer (kSimpleGen2BlobTrailerSize) | | logical CRC | CRC32C of the uncompressed value (BlobIndex::stable_checksum()) | | stored CRC | CRC32C of the stored bytes plus the compression byte | | physical file | A file that holds value bytes: a normal blob file or a relocation file. A generation number can name a physical file, or it can be only the key of a route | | exact reference metadata | Per-SST sorted list of every blob reference the SST keeps, written when the SST is built. References with the same encoding class, generation number and origin form a group. The list is stored as data blocks, with an index of the groups. An SST that has it is an exact SST | | option gate | Until diff 11, ColumnFamilyData::ValidateOptions returns NotSupported for enable_blob_indirection on a read-write open, CreateColumnFamily and SetOptions, unless a test calls ColumnFamilyData::TEST_SetAllowBlobIndirection(true). It keeps the option off in production while the stack is not complete | Terms of this diff (all defined at the top of db/blob/blob_reference_metadata.h): | Term | Meaning | |---|---| | encoding class (class) | kDirect (a legacy kBlob or kBlobTTL index), unflattened, or flattened | | reference group (group) | All references of one SST with the same (encoding class, generation number, origin) | | run | The part of one group inside one data block | | record bytes | The bytes the value's record takes in the file named by its generation number | | gap | How far a record is from where it would be if the records were back to back. A gap of 0 means back to back | | descriptor | The 72-byte table property that points to the reference index block and holds the totals, the CF ID, the SST file number and the relocation-state fingerprint | | relocation-state fingerprint | 128-bit hash of the routes and relocation files of the Version that an SST was built from (fields route_snapshot_fingerprint_*). Written as 0 in this diff | | legacy | Written the way RocksDB writes it without blob indirection | Previous behavior: - A BlobIndex named only a physical blob file and an offset. A value could not move to another file without rewriting every SST that referenced it. - An SST had no metadata that listed its blob references. The only way to find them was to scan its user keys. - These readers get the blob indexes of a wide-column entity through ForEachBlobFileNumber: UpdateBoundaries, BlobGarbageMeter, the compaction iterator, WideColumnSerialization::RemoveBlobColumns, and the referenced_blob_bytes table property count in BlockBasedTableBuilder::Add. ForEachBlobFileNumber decoded each blob index, but it did not check the rest of the entity framing (name order, section sizes, exact input size), and it could call the callback before it found a later error. New behavior: Separate change (option on or off): - The same readers (UpdateBoundaries, BlobGarbageMeter, the compaction iterator, RemoveBlobColumns, and the referenced_blob_bytes count in BlockBasedTableBuilder::Add) use WideColumnSerialization::ForEachBlobIndex, with the option on or off. With the option on, the table builder also uses it to give the blob columns of each entity to the collector. For an entity with blob columns, it checks the full framing, then decodes and validates every blob index before any callback runs. So an entity with a malformed framing now fails the flush or compaction, even with the option off. The referenced_blob_bytes count itself does not fail the build on a malformed entity. It removes that property from the SST, as in base. Option and option gate: - This diff adds a new column family option, `enable_blob_indirection`. It is off by default, and SetOptions can change it. - The option gate (see Terms) keeps the option off outside tests until the stack is complete. ColumnFamilyData::ValidateOptions rejects the option with NotSupported on a read-write open, CreateColumnFamily and SetOptions. Read-only opens (DBImplReadOnly and CompactedDBImpl) and secondary opens (including DB::OpenAndCompact) do not run ValidateOptions; they check only the FIFO rule (see Implementation). Only tests can turn the option on, through the internal ColumnFamilyData::TEST_SetAllowBlobIndirection(). DBTestBase sets it. - When the option is on, ValidateOptions also requires enable_blob_files, the block-based table format, no blob direct write, and a compaction style other than FIFO. Blob file writer (option on): - The blob file writer encodes each new value with a stable identity (origin file, original offset) and two checksums: - the logical CRC: the CRC32C of the uncompressed value. - the stored CRC: the CRC32C of the exact stored (maybe compressed) bytes, extended with the compression byte. - For an uncompressed value the stored CRC is not written, because the reader can compute it from the logical CRC. - The checksums have no version field of their own. The BlobIndex type byte gives the format version: the two new types, 3 (kIndirectBlob) and 4 (kIndirectFlattenedBlob, see Implementation), are BlobIndex::kIndirectBlobFormatVersion (1). A later format uses new type values. - The read path does not check these checksums yet. The check starts in diff 3. Exact reference metadata (option on): 1. What: a sorted list of every blob reference that the SST keeps, in plain blob values and in the blob columns of wide-column entities. 2. Where: data blocks (target 16 KB each before compression; small blocks waste less I/O, because GC reads a whole block to get a few groups) and one reference index block, written after the SST data blocks and before the SST tail, plus a 72-byte descriptor table property. 3. Who reads it: only tests in this diff. The census, GC and RepairDB read it in later diffs. 4. Details: the top comment of db/blob/blob_reference_metadata.h has the full format with diagrams, the record-bytes rules and the gap rule. - Not collected: SstFileWriter output (it has no DB file number) and any output whose CF is unknown (it has no CF ID). The descriptor needs both values, so these outputs stay in the legacy form. Checks on the metadata: - A repeated or overlapping reference in one group fails the SST build with Corruption. Readers check every count, order, size and total against the index and the descriptor, so corrupt metadata returns Corruption. The writer never makes a data block larger than kTargetDataBlockSize (16 KB), and readers accept up to kMaxDataBlockSize (64 KB). Details: "Readers" and "Integrity" at the top of db/blob/blob_reference_metadata.h, and BlobReferenceGroupDecoder. Collection and memory: - Each reference is encoded when it is added, in the on-disk entry encoding, into its group's chain of 512-byte chunks. Only a group that takes the unsorted fallback is encoded a second time, in Finish(). Inside a group, references normally arrive in rising offset order, so no reference is sorted; at the end only the groups are sorted. A group that gets a lower offset takes the unsorted fallback. The BlobReferenceMetadataCollector class comment explains the chunks, the chunk size and the fallback. - The collector keeps every reference of the SST in memory until the SST is finished, and charges this memory to the block cache under a new CacheEntryRole, kBlobReferenceMetadataBuildingBuffer. The measured cost is about 8.3 to 9 bytes per reference for uncompressed values and 12.8 to 13.5 bytes for compressed values. "Memory usage" and "Memory charge" in the BlobReferenceMetadataCollector class comment give the measurement setup and the charge rule. - There is no fixed limit on references. The collector fails with MemoryLimit above 2^20 groups or 2^20 data blocks in one SST; these are persisted format limits. - There is no charge when the table has no block cache or when charging for this role is disabled. Each block-based table factory holds one cache reservation manager for this role, shared by its builders and the SST summaries. If the charge fails, the SST build fails, so the flush or compaction fails with a background error. The buffers and their charge are released when the SST is finished or abandoned. Implementation: - BlobIndex adds two indirect encodings. The legacy encodings do not change. - Type 3, kIndirectBlob (unflattened, generation number == origin), stores the origin, the original offset, the stored size, the compression type, the logical CRC and, for compressed values, the stored CRC. It does not store the generation number. This saves the size of one varint file number per reference (about 3 to 4 bytes for typical file numbers; an estimate, not measured). - Type 4, kIndirectFlattenedBlob (flattened), stores the generation number first, then the same fields. Its generation number must be above the origin. The decoder rejects type 4 with generation number <= origin, so each reference has one encoding. - The encoders take the fields as one BlobIndex::IndirectBlobFields, set by field name, so values of the same type (file number, offset and size; the two CRCs) cannot be swapped. EncodeIndirectBlob() takes the generation number as a separate argument; EncodeUnflattenedIndirectBlob() uses the origin, so the common path cannot leave the generation number unset. - The new files blob_reference_metadata.{h,cc} define the exact reference metadata. The files also contain: - BlobReferenceMetadataCollector, the collector described above. It takes the SST identity (CF ID, SST file number and the relocation-state fingerprint) as one BlobReferenceSstIdentity, set by field name. - BlobReferenceGroupKey, the key of a group: (encoding class, generation number, origin). It has the order, the hash and the validity check that the collector map, the sort, the decoder and FindBlobReferenceGroup() use. BlobReference and BlobReferenceGroup keep their flat fields and return the key through group_key(). - The index codec and the descriptor table property. - BlobReferenceGroupDecoder, the only decoder of reference runs. It reads a group one data block at a time and holds every corruption check of a group walk. Readers that visit one group (found by a binary search of the index) use it, and so do readers that visit every block in order. - Callers in this diff: only the table builder runs the collector. The decoders (DecodeBlobReferenceIndex, FindBlobReferenceGroup, BlobReferenceGroupDecoder, ForEachBlobReference*) are used only by tests. kMaxDecodedBlobReferenceBytes is used only in compile-time checks of the count limits. The census (diff 5) and GC (diffs 6 and 7) are their first production users. The census is also the first user of BlobReferenceSstSummary (see the last Implementation bullet). - Descriptor: the CF ID and the SST file number are the repair fields. They let RepairDB tell which column family and SST the list belongs to without a MANIFEST. The decoder rejects a descriptor whose repair fields are not set (unknown CF ID or SST file number 0). The descriptor also has a 128-bit relocation-state fingerprint (the route_snapshot_fingerprint fields). Remote compaction uses it in diff 9 to check that an SST built elsewhere used the same routes as the job that asked for it. In this diff the table builder always writes 0, and DecodeFrom() does not check it. The field is in the format now, so the descriptor keeps one format version (1) through the stack. While the option gate is on, no data in this format exists outside tests. - The collector validates each BlobIndex once, when it is added. Its Finish() writes the data blocks and then the reference index block through one block writer, and fills the descriptor, including the index handle. Destroying the collector frees its memory and then its charge, also when the build is abandoned. - BlockBasedTableBuilder creates the collector with the CF ID and SST file number from TableBuilderOptions. It passes the size of each entry's user key to the collector, so the collector can compute the exact size of each blob record. It adds the collector's EstimatedEncodedSize() to its tail size estimate: the blocks are written before the tail, but output file size planning uses this estimate. The reference blocks are written with BlockType::kUserDefinedIndex, which keeps their contents as opaque bytes. Readers read them without the block cache, so prepopulate_block_cache does not warm them into the block cache (WriteMaybeCompressedBlock() with warm_cache false). - With the option on, the blob file builder computes the CRC32C of each stored value once. The blob log record CRC combines it with the key CRC (crc32c::Crc32cCombine). The stored CRC extends it with the compression byte. For an uncompressed value it is also the logical CRC. The bytes written do not change. - WideColumnSerialization::ForEachBlobIndex replaces ForEachBlobFileNumber. It reads the entity header and column types once. For a V1 entity (no column types and no blob columns), it returns right after the header. For a V2 entity without blob columns, it returns right after the type scan. It shares one V2 column walker (WalkV2Columns) with Deserialize(), so the entity is parsed in one pass. - Read-only opens (DBImplReadOnly, CompactedDBImpl) and secondary opens skip ValidateOptions, so they check the compaction style against the option once per column family: in DBImpl::Recover() for read-only opens, and at the start of OpenAsSecondaryImpl() for secondary opens. DB::OpenAndCompact opens through OpenAsSecondaryImpl(), so it gets the same check. - CacheReservationHandle gets Resize(), so each collector or summary can change its own part of the shared charge. - util/coding.h adds GetCanonicalVarint32/64, which reject overlong varint encodings. The new formats use them, so each value has one encoding. - BlobReferenceSstSummary is the in-memory list of an SST's groups and totals that Finish() can return. It has no caller yet: the table builder passes a null summary output. It is checked and does not change after Finish(), so the finished builder, the SST metadata and the census (its first user, in diff 5) can share it. It is in this diff because Finish() hands the group buffer and its cache charge over to the summary. Adding the summary later would mean rewriting the charge code of Finish() in diff 5, so the charge code is written and tested once, here. Safety boundaries and unsupported cases: - Public API: this diff adds the option `enable_blob_indirection` with the C API functions `rocksdb_options_set_enable_blob_indirection` and `rocksdb_options_get_enable_blob_indirection`. It also inserts a new CacheEntryRole value before kMisc, which changes the numeric value of kMisc and kNumCacheEntryRoles. - With the option off, new blob values and SSTs are written in the legacy form. Legacy blob indexes stay readable. - One behavior change applies with the option off too: UpdateBoundaries, BlobGarbageMeter and the compaction iterator fully validate wide-column entities with blob columns and their blob indexes (see "Separate change" above). - Exact reference metadata has one format (version 1) with a fixed 72-byte descriptor. The decoders reject any other version as Corruption. - The metadata blocks are protected by the table's block checksum only. With checksum = kNoChecksum, corrupt metadata is detected only when it breaks the format or order checks. - The index decoder checks each decoded count against the remaining input size before it reserves memory. So a small corrupt index cannot make it allocate a large vector. - Inlined-TTL and same-file blob indexes are not collected and cannot be relocated. Encoding, count and checksum errors return an error; a bad reference is never skipped. - Option gate (see Terms): outside tests no DB writes data in the new format. The gate checks only the option value, never persisted state. Diff 11 removes it, once data written with the option stays readable through the end of the stack and the safety checks for turning the option on are in place. - Downgrading the binary after data was written with the option on is not supported. Validation and known limitations: - Unit tests cover the BlobIndex encodings and their rejection of bad input, the metadata layout and every corruption check, the collector (sorted path, unsorted fallback, memory charge), the flush integration, the option checks and option gate, wide-column validation, and canonical varints. See the Test Plan. - The collection buffer is not spilled to disk. One SST with very many blob references needs a matching amount of charged memory: the per-reference amount given under "Collection and memory", plus up to one partly used 512-byte chunk per group. While a fallback group is sorted, it also needs 16 bytes of sort key per reference and a second copy of its entries. For a test group of 200K references with every other one out of order, the test comment estimates about 44 bytes per reference, and the test checks that the peak charge is at most 56 bytes per reference. - Flush still writes a single output file. Until diff 7, one compaction output collects every reference it gets, charged to the block cache. Diff 7 cuts compaction outputs (including L0) at a reference cap and keeps each user key's history in one output. With the option gate, a large output is reachable only through the test hook. Stack context (same in diffs 2-11, except that diffs 8-11 add the diff 8 route rule to the install line; "<== this diff" marks this diff's part): Goal: remove garbage from blob files without rewriting any SST. A value gets a stable identity when it is first written. GC copies the live values of whole files into a relocation file and installs routes in the MANIFEST. Readers follow the routes. SSTs are not changed. How a read finds a value (P is the relocation file that holds the value now): ``` SST entry indirect BlobIndex { generation number, origin, original offset, size, CRCs } | v Version file table: generation number -> live file, or route old file -> P | | live file route found | | v v blob file: read at original offset relocation file P (always one hop): relocation file: look up key look up key (origin, original offset) (origin, original offset) ``` Lifecycle of one value (in this example, F is the blob file where the value is first written, and P is the relocation file that GC writes): ``` write new value goes to blob file F; index: origin = F, generation number = F | census per generation number: the SSTs and references that use it; | per physical file: live bytes, ranked by garbage ratio GC job copy the live values of whole input files (stored bytes, checked | with the stored CRC) into relocation file P, keyed by | (origin, original offset) install one MANIFEST edit: add P, route each input generation number still | in use to P, re-point the routes into the inputs to P, and remove | each input (blob file retirement for a blob file, relocation | file deletion for a relocation file) flatten compaction sets generation number = P in the SSTs it rewrites | route route retirement: route deletion records remove the routes that delete no SST uses ``` Stack map: | Diff | Adds | |---|---| | 1 | Finalize blob metadata outside the DB mutex (prep) | | 2 | Indirect BlobIndex (kIndirectBlob) and exact SST reference metadata <== this diff | | 3 | Relocation file format and stable read path | | 4 | MANIFEST routes and Version ownership | | 5 | Exact reverse reference census | | 6 | Standalone relocation GC (one input file) | | 7 | Pack whole input files; reclaim relocation files | | 8 | Flatten generation numbers; route retirement | | 9 | Remote compaction validation | | 10 | Dynamic options and operations | | 11 | RepairDB rebuilds relocation state; option gate removed | | 12 | db_stress and crash test support | Stack context: terms for later diffs: | Term | Meaning | |---|---| | census | Background count, per generation number, of the SSTs and references that use it, plus live bytes per physical file (diff 5) | | indirect origin | Normal blob file written with the option on; it is the origin of unflattened references | | flatten | Compaction rewrites the generation number to the relocation file that holds the value, so old routes can go. Only flatten makes generation number != origin (diff 8) | | blob file retirement | MANIFEST record (kBlobFileRetirement) that removes a collected blob file from the Version (diff 4) | | route deletion | MANIFEST record that removes a route that no SST uses. Route retirement is the background job that writes these records (diff 8) | Tracking issue: meta#15301 Differential Revision: D124652097
Claude Code Review - OBSOLETESuperseded by a newer AI review. Expand to see the original review.✅ Claude Code ReviewAuto-triggered after CI passed — reviewing commit 084c89f SummaryThis is a well-structured, thoroughly documented diff that adds the foundation for blob GC without SST rewrites. The code quality is high: error handling is comprehensive, overflow checks are pervasive, and the format is well-defended against corrupt inputs. The behavioral change in High-severity findings (2):
Full review (click to expand)Findings🔴 HIGHH1. CacheReservationHandle::Resize() sets incremental_memory_used_ before confirming UpdateCacheReservation succeeds --
|
| Context | Code executes? | Assumptions hold? | Action needed? |
|---|---|---|---|
| WritePreparedTxnDB | YES (flush/compaction) | YES | Safe |
| ReadOnly DB | YES (FIFO check added) | YES | Safe |
| SecondaryInstance | YES (FIFO check added) | YES | Safe |
| User-defined timestamps | YES (orthogonal) | YES | Safe |
| MemPurge | NO (no SST build) | N/A | Safe |
| BlobDB with direct write | NO (rejected by ValidateOptions) | N/A | Safe |
| FIFO compaction | NO (rejected) | N/A | Safe |
| SstFileWriter | NO (collector not created) | N/A | Safe |
| DB mutex | NOT AFFECTED | N/A | Safe |
Positive Observations
- Thorough overflow checking: Every arithmetic operation uses
AddWithoutOverflowor explicit checks. Exemplary defensive coding. - Format self-description: Magic, version, CRC in the descriptor; decoders reject unknown versions.
- Memory accounting: "Charge first, then allocate" pattern prevents uncharged memory growth.
- Excellent documentation: The header comment in
blob_reference_metadata.his a complete format specification with diagrams and measurements. - Named field initialization:
IndirectBlobFieldsandBlobReferenceSstIdentityuse designated initializers, preventing field-swap bugs. - Canonical varints:
GetCanonicalVarint64on decode ensures one encoding per value. - Test coverage: ~2800 lines of new tests covering encoding round-trips, corruption detection, unsorted fallback, memory charging, option validation, and flush integration.
ℹ️ About this response
Generated by Claude Code.
Review methodology: claude_md/ci_review_prompt.md
Limitations:
- Claude may miss context from files not in the diff
- Large PRs may be truncated
- Always apply human judgment to AI suggestions
Commands:
/claude-review [context]— Request a code review/claude-query <question>— Ask about the PR or codebase
Summary: Pull Request resolved: meta#15540 Why the change is required: The goal of this stack is indirect blob GC, a blob GC that does not depend on SSTs: it removes garbage from blob files without rewriting any SST. Such a GC needs two things that RocksDB does not have: - A value identity that does not change when the value moves to another file. - A checked list of the blob references that each SST keeps. The list must include older internal versions of a key that the SST still keeps (for example for snapshots), and the blob columns of wide-column entities. This diff adds both. Later diffs build the census (an in-memory reverse index from generation number to the SSTs that use it), GC and repair on top of this list. This diff in one paragraph: With the new column family option enable_blob_indirection on, the blob file writer gives each new value an indirect BlobIndex (type 3, kIndirectBlob). It stores the value's stable identity (origin file, original offset) and its CRCs (an uncompressed value stores only the CRC of the uncompressed value; a compressed value also stores the CRC of its stored bytes). Also with the option on, the block-based table builder collects every blob reference that the SST keeps. It writes them as "exact reference metadata": data blocks, one reference index block, and a 72-byte descriptor table property. In this diff only the table builder writes this metadata, and only tests read it. A temporary option gate (see "option gate" under Terms) keeps the option off outside tests. Separately, wide-column entities with blob columns are now fully checked before their blob indexes are used, also with the option off. Parts of this diff: 1. Indirect BlobIndex (core): types 3 and 4 in db/blob/blob_index.h, and the blob file writer that writes type 3. 2. Exact reference metadata format and decoders (core): db/blob/blob_reference_metadata.{h,cc}. 3. Collector and table builder integration (core): BlobReferenceMetadataCollector, its block cache charge, and BlockBasedTableBuilder. 4. Option enable_blob_indirection and the option gate. Tests need it to turn the feature on. You can skip this part on a first read. 5. Wide-column validation (WideColumnSerialization::ForEachBlobIndex). The table builder uses it to give the blob columns of each entity to the collector. It also changes behavior with the option off (see New behavior). You can skip this part on a first read. 6. Small helpers: canonical varints (util/coding.h) and CacheReservationHandle::Resize(). You can skip these on a first read. How to read this diff: 1. Overview: read "Why" and "This diff in one paragraph" above, and "Terms used in this diff" right below. Then read the top comment of db/blob/blob_reference_metadata.h (terms, layout diagrams, run and entry layout, index layout). 2. Indirect BlobIndex (db/blob/blob_index.h) and canonical varints (util/coding.h GetCanonicalVarint32/64): read the overview comment and the kIndirectBlob / kIndirectFlattenedBlob code. Every later part uses these fields: origin, original offset, generation number, logical CRC and stored CRC. The new formats read varints only in the canonical form. 3. Blob file writer (db/blob/blob_file_builder.cc, blob_log_format.cc, blob_log_writer.cc): see how a new value gets a type-3 index and how its CRC is computed once. 4. Format types and decoders: the structs in db/blob/blob_reference_metadata.h, then in the .cc the constants, GapBase, DecodeEntryBody, DecodeRunEntriesAt, DecodeBlobReferenceIndex and BlobReferenceGroupDecoder. They use only BlobIndex and the canonical varints. Learn the on-disk format here, before you read the writer. 5. Collector and its cache charge: first read the new CacheEntryRole kBlobReferenceMetadataBuildingBuffer (include/rocksdb/cache.h) and CacheReservationHandle::Resize (cache/cache_reservation_manager.h). Then read BlobReferenceMetadataCollector in this order: the class comment, Add, AppendSorted, BuildAndWriteDataBlocks, EncodeIndex, FinishOnce. Then read the unsorted fallback (AppendUnsorted, SortUnsortedGroup, SortAndEncodeGroup). 6. Wide-column change (db/wide/wide_column_serialization.cc, ForEachBlobIndex and WalkV2Columns): it gives every blob index of an entity to a callback after it checks the whole entity. The table builder uses it to feed the collector. UpdateBoundaries, BlobGarbageMeter and the compaction iterator also use it, with the option on or off. 7. Option: enable_blob_indirection (include/rocksdb/advanced_options.h, options/cf_options.cc), and the option checks and gate in db/column_family.cc (ValidateOptions, TEST_SetAllowBlobIndirection), db/db_impl/db_impl_open.cc and db_impl_secondary.cc (the FIFO check for opens that skip ValidateOptions). 8. Table integration: table/block_based/block_based_table_factory.cc (the shared reservation manager), then block_based_table_builder.cc (Rep constructor, Add, WriteBlobReferenceMetadata, EstimatedTailSize). See where the collector is created, fed and finished. block_based_table_reader.cc maps the new block name to a block type. Terms used in this diff: Stack terms: | Term | Meaning | |---|---| | origin | Blob file where the value was first written | | original offset | Value offset in the origin file | | stable identity | (origin, original offset). Never changes, even when the value moves | | generation number | The file number that a reference uses to find its value. A new value has generation number == origin. When indirect blob GC moves the value, the reference keeps its generation number, and a route sends the read to the relocation file | | indirect BlobIndex | A kIndirectBlob (type 3) or kIndirectFlattenedBlob (type 4) index. It stores the stable identity, so the value can move without an SST rewrite | | unflattened reference | Index with generation number == origin: the value was not moved, or it was moved but no compaction has rewritten its reference yet (see "flatten" in "Stack context: terms for later diffs") | | flattened reference | Index with generation number != origin (type 4). Only flatten (diff 8) writes it; this diff defines the encoding | | route | Maps an old file number (a file that GC has removed from the Version) to the relocation file that now holds its live values. One hop: a target is never itself routed. Stored in the MANIFEST (diff 4) | | relocation file | Block-based table keyed by (origin, original offset) that holds copied values. Each value is the stored bytes plus a 5-byte trailer (kSimpleGen2BlobTrailerSize) | | logical CRC | CRC32C of the uncompressed value (BlobIndex::stable_checksum()) | | stored CRC | CRC32C of the stored bytes plus the compression byte | | physical file | A file that holds value bytes: a normal blob file or a relocation file. A generation number can name a physical file, or it can be only the key of a route | | exact reference metadata | Per-SST sorted list of every blob reference the SST keeps, written when the SST is built. References with the same encoding class, generation number and origin form a group. The list is stored as data blocks, with an index of the groups. An SST that has it is an exact SST | | option gate | Until diff 11, ColumnFamilyData::ValidateOptions returns NotSupported for enable_blob_indirection on a read-write open, CreateColumnFamily and SetOptions, unless a test calls ColumnFamilyData::TEST_SetAllowBlobIndirection(true). It keeps the option off in production while the stack is not complete | Terms of this diff (all defined at the top of db/blob/blob_reference_metadata.h): | Term | Meaning | |---|---| | encoding class (class) | kDirect (a legacy kBlob or kBlobTTL index), unflattened, or flattened | | reference group (group) | All references of one SST with the same (encoding class, generation number, origin) | | run | The part of one group inside one data block | | record bytes | The bytes the value's record takes in the file named by its generation number | | gap | How far a record is from where it would be if the records were back to back. A gap of 0 means back to back | | descriptor | The 72-byte table property that points to the reference index block and holds the totals, the CF ID, the SST file number and the relocation-state fingerprint | | relocation-state fingerprint | 128-bit hash of the routes and relocation files of the Version that an SST was built from (fields route_snapshot_fingerprint_*). Written as 0 in this diff | | legacy | Written the way RocksDB writes it without blob indirection | Previous behavior: - A BlobIndex named only a physical blob file and an offset. A value could not move to another file without rewriting every SST that referenced it. - An SST had no metadata that listed its blob references. The only way to find them was to scan its user keys. - These readers get the blob indexes of a wide-column entity through ForEachBlobFileNumber: UpdateBoundaries, BlobGarbageMeter, the compaction iterator, WideColumnSerialization::RemoveBlobColumns, and the referenced_blob_bytes table property count in BlockBasedTableBuilder::Add. ForEachBlobFileNumber decoded each blob index, but it did not check the rest of the entity framing (name order, section sizes, exact input size), and it could call the callback before it found a later error. New behavior: Separate change (option on or off): - The same readers (UpdateBoundaries, BlobGarbageMeter, the compaction iterator, RemoveBlobColumns, and the referenced_blob_bytes count in BlockBasedTableBuilder::Add) use WideColumnSerialization::ForEachBlobIndex, with the option on or off. With the option on, the table builder also uses it to give the blob columns of each entity to the collector. For an entity with blob columns, it checks the full framing, then decodes and validates every blob index before any callback runs. So an entity with a malformed framing now fails the flush or compaction, even with the option off. The referenced_blob_bytes count itself does not fail the build on a malformed entity. It removes that property from the SST, as in base. Option and option gate: - This diff adds a new column family option, `enable_blob_indirection`. It is off by default, and SetOptions can change it. - The option gate (see Terms) keeps the option off outside tests until the stack is complete. ColumnFamilyData::ValidateOptions rejects the option with NotSupported on a read-write open, CreateColumnFamily and SetOptions. Read-only opens (DBImplReadOnly and CompactedDBImpl) and secondary opens (including DB::OpenAndCompact) do not run ValidateOptions; they check only the FIFO rule (see Implementation). Only tests can turn the option on, through the internal ColumnFamilyData::TEST_SetAllowBlobIndirection(). DBTestBase sets it. - When the option is on, ValidateOptions also requires enable_blob_files, the block-based table format, no blob direct write, and a compaction style other than FIFO. Blob file writer (option on): - The blob file writer encodes each new value with a stable identity (origin file, original offset) and two checksums: - the logical CRC: the CRC32C of the uncompressed value. - the stored CRC: the CRC32C of the exact stored (maybe compressed) bytes, extended with the compression byte. - For an uncompressed value the stored CRC is not written, because the reader can compute it from the logical CRC. - The checksums have no version field of their own. The BlobIndex type byte gives the format version: the two new types, 3 (kIndirectBlob) and 4 (kIndirectFlattenedBlob, see Implementation), are BlobIndex::kIndirectBlobFormatVersion (1). A later format uses new type values. - The read path does not check these checksums yet. The check starts in diff 3. Exact reference metadata (option on): 1. What: a sorted list of every blob reference that the SST keeps, in plain blob values and in the blob columns of wide-column entities. 2. Where: data blocks (target 16 KB each before compression; small blocks waste less I/O, because indirect blob GC reads a whole block to get a few groups) and one reference index block, written after the SST data blocks and before the SST tail, plus a 72-byte descriptor table property. 3. Who reads it: only tests in this diff. The census, GC and RepairDB read it in later diffs. 4. Details: the top comment of db/blob/blob_reference_metadata.h has the full format with diagrams, the record-bytes rules and the gap rule. - Not collected: SstFileWriter output (it has no DB file number) and any output whose CF is unknown (it has no CF ID). The descriptor needs both values, so these outputs stay in the legacy form. Checks on the metadata: - A repeated or overlapping reference in one group fails the SST build with Corruption. Readers check every count, order, size and total against the index and the descriptor, so corrupt metadata returns Corruption. The writer never makes a data block larger than kTargetDataBlockSize (16 KB), and readers accept up to kMaxDataBlockSize (64 KB). Details: "Readers" and "Integrity" at the top of db/blob/blob_reference_metadata.h, and BlobReferenceGroupDecoder. Collection and memory: - Each reference is encoded when it is added, in the on-disk entry encoding, into its group's chain of 512-byte chunks. Only a group that takes the unsorted fallback is encoded a second time, in Finish(). Inside a group, references normally arrive in rising offset order, so no reference is sorted; at the end only the groups are sorted. A group that gets a lower offset takes the unsorted fallback. The BlobReferenceMetadataCollector class comment explains the chunks, the chunk size and the fallback. - The collector keeps every reference of the SST in memory until the SST is finished, and charges this memory to the block cache under a new CacheEntryRole, kBlobReferenceMetadataBuildingBuffer. The measured cost is about 8.3 to 9 bytes per reference for uncompressed values and 12.8 to 13.5 bytes for compressed values. "Memory usage" and "Memory charge" in the BlobReferenceMetadataCollector class comment give the measurement setup and the charge rule. - There is no fixed limit on references. The collector fails with MemoryLimit above 2^20 groups or 2^20 data blocks in one SST; these are persisted format limits. - There is no charge when the table has no block cache or when charging for this role is disabled. Each block-based table factory holds one cache reservation manager for this role, shared by its builders and the SST summaries. If the charge fails, the SST build fails, so the flush or compaction fails with a background error. The buffers and their charge are released when the SST is finished or abandoned. Implementation: - BlobIndex adds two indirect encodings. The legacy encodings do not change. - Type 3, kIndirectBlob (unflattened, generation number == origin), stores the origin, the original offset, the stored size, the compression type, the logical CRC and, for compressed values, the stored CRC. It does not store the generation number. This saves the size of one varint file number per reference (about 3 to 4 bytes for typical file numbers; an estimate, not measured). - Type 4, kIndirectFlattenedBlob (flattened), stores the generation number first, then the same fields. Its generation number must be above the origin. The decoder rejects type 4 with generation number <= origin, so each reference has one encoding. - The encoders take the fields as one BlobIndex::IndirectBlobFields, set by field name, so values of the same type (file number, offset and size; the two CRCs) cannot be swapped. EncodeIndirectBlob() takes the generation number as a separate argument; EncodeUnflattenedIndirectBlob() uses the origin, so the common path cannot leave the generation number unset. - The new files blob_reference_metadata.{h,cc} define the exact reference metadata. The files also contain: - BlobReferenceMetadataCollector, the collector described above. It takes the SST identity (CF ID, SST file number and the relocation-state fingerprint) as one BlobReferenceSstIdentity, set by field name. - BlobReferenceGroupKey, the key of a group: (encoding class, generation number, origin). It has the order, the hash and the validity check that the collector map, the sort, the decoder and FindBlobReferenceGroup() use. BlobReference and BlobReferenceGroup keep their flat fields and return the key through group_key(). - The index codec and the descriptor table property. - BlobReferenceGroupDecoder, the only decoder of reference runs. It reads a group one data block at a time and holds every corruption check of a group walk. Readers that visit one group (found by a binary search of the index) use it, and so do readers that visit every block in order. - Callers in this diff: only the table builder runs the collector. The decoders (DecodeBlobReferenceIndex, FindBlobReferenceGroup, BlobReferenceGroupDecoder, ForEachBlobReference*) are used only by tests. kMaxDecodedBlobReferenceBytes is used only in compile-time checks of the count limits. The census (diff 5) and indirect blob GC (diffs 6 and 7) are their first production users. The census is also the first user of BlobReferenceSstSummary (see the last Implementation bullet). - Descriptor: the CF ID and the SST file number are the repair fields. They let RepairDB tell which column family and SST the list belongs to without a MANIFEST. The decoder rejects a descriptor whose repair fields are not set (unknown CF ID or SST file number 0). The descriptor also has a 128-bit relocation-state fingerprint (the route_snapshot_fingerprint fields). Remote compaction uses it in diff 9 to check that an SST built elsewhere used the same routes as the job that asked for it. In this diff the table builder always writes 0, and DecodeFrom() does not check it. The field is in the format now, so the descriptor keeps one format version (1) through the stack. While the option gate is on, no data in this format exists outside tests. - The collector validates each BlobIndex once, when it is added. Its Finish() writes the data blocks and then the reference index block through one block writer, and fills the descriptor, including the index handle. Destroying the collector frees its memory and then its charge, also when the build is abandoned. - BlockBasedTableBuilder creates the collector with the CF ID and SST file number from TableBuilderOptions. It passes the size of each entry's user key to the collector, so the collector can compute the exact size of each blob record. It adds the collector's EstimatedEncodedSize() to its tail size estimate: the blocks are written before the tail, but output file size planning uses this estimate. The reference blocks are written with BlockType::kUserDefinedIndex, which keeps their contents as opaque bytes. Readers read them without the block cache, so prepopulate_block_cache does not warm them into the block cache (WriteMaybeCompressedBlock() with warm_cache false). - With the option on, the blob file builder computes the CRC32C of each stored value once. The blob log record CRC combines it with the key CRC (crc32c::Crc32cCombine). The stored CRC extends it with the compression byte. For an uncompressed value it is also the logical CRC. The bytes written do not change. - WideColumnSerialization::ForEachBlobIndex replaces ForEachBlobFileNumber. It reads the entity header and column types once. For a V1 entity (no column types and no blob columns), it returns right after the header. For a V2 entity without blob columns, it returns right after the type scan. It shares one V2 column walker (WalkV2Columns) with Deserialize(), so the entity is parsed in one pass. - Read-only opens (DBImplReadOnly, CompactedDBImpl) and secondary opens skip ValidateOptions, so they check the compaction style against the option once per column family: in DBImpl::Recover() for read-only opens, and at the start of OpenAsSecondaryImpl() for secondary opens. DB::OpenAndCompact opens through OpenAsSecondaryImpl(), so it gets the same check. - CacheReservationHandle gets Resize(), so each collector or summary can change its own part of the shared charge. - util/coding.h adds GetCanonicalVarint32/64, which reject overlong varint encodings. The new formats use them, so each value has one encoding. - BlobReferenceSstSummary is the in-memory list of an SST's groups and totals that Finish() can return. It has no caller yet: the table builder passes a null summary output. It is checked and does not change after Finish(), so the finished builder, the SST metadata and the census (its first user, in diff 5) can share it. It is in this diff because Finish() hands the group buffer and its cache charge over to the summary. Adding the summary later would mean rewriting the charge code of Finish() in diff 5, so the charge code is written and tested once, here. Safety boundaries and unsupported cases: - Public API: this diff adds the option `enable_blob_indirection` with the C API functions `rocksdb_options_set_enable_blob_indirection` and `rocksdb_options_get_enable_blob_indirection`. It also inserts a new CacheEntryRole value before kMisc, which changes the numeric value of kMisc and kNumCacheEntryRoles. - With the option off, new blob values and SSTs are written in the legacy form. Legacy blob indexes stay readable. - One behavior change applies with the option off too: UpdateBoundaries, BlobGarbageMeter and the compaction iterator fully validate wide-column entities with blob columns and their blob indexes (see "Separate change" above). - Exact reference metadata has one format (version 1) with a fixed 72-byte descriptor. The decoders reject any other version as Corruption. - The metadata blocks are protected by the table's block checksum only. With checksum = kNoChecksum, corrupt metadata is detected only when it breaks the format or order checks. - The index decoder checks each decoded count against the remaining input size before it reserves memory. So a small corrupt index cannot make it allocate a large vector. - Inlined-TTL and same-file blob indexes are not collected and cannot be relocated. Encoding, count and checksum errors return an error; a bad reference is never skipped. - Option gate (see Terms): outside tests no DB writes data in the new format. The gate checks only the option value, never persisted state. Diff 11 removes it, once data written with the option stays readable through the end of the stack and the safety checks for turning the option on are in place. - Downgrading the binary after data was written with the option on is not supported. Validation and known limitations: - Unit tests cover the BlobIndex encodings and their rejection of bad input, the metadata layout and every corruption check, the collector (sorted path, unsorted fallback, memory charge), the flush integration, the option checks and option gate, wide-column validation, and canonical varints. See the Test Plan. - The collection buffer is not spilled to disk. One SST with very many blob references needs a matching amount of charged memory: the per-reference amount given under "Collection and memory", plus up to one partly used 512-byte chunk per group. While a fallback group is sorted, it also needs 16 bytes of sort key per reference and a second copy of its entries. For a test group of 200K references with every other one out of order, the test comment estimates about 44 bytes per reference, and the test checks that the peak charge is at most 56 bytes per reference. - Flush still writes a single output file. Until diff 7, one compaction output collects every reference it gets, charged to the block cache. Diff 7 cuts compaction outputs (including L0) at a reference cap and keeps each user key's history in one output. With the option gate, a large output is reachable only through the test hook. Stack context (same in diffs 2-11, except that diffs 8-11 add the diff 8 route rule to the install line; "<== this diff" marks this diff's part): Goal: remove garbage from blob files without rewriting any SST. A value gets a stable identity when it is first written. Indirect blob GC copies the live values of whole files into a relocation file and installs routes in the MANIFEST. Readers follow the routes. SSTs are not changed. How a read finds a value (P is the relocation file that holds the value now): ``` SST entry indirect BlobIndex { generation number, origin, original offset, size, CRCs } | v Version file table: generation number -> live file, or route old file -> P | | live file route found | | v v blob file: read at original offset relocation file P (always one hop): relocation file: look up key look up key (origin, original offset) (origin, original offset) ``` Lifecycle of one value (in this example, F is the blob file where the value is first written, and P is the relocation file that GC writes): ``` write new value goes to blob file F; index: origin = F, generation number = F | census per generation number: the SSTs and references that use it; | per physical file: live bytes, ranked by garbage ratio GC job copy the live values of whole input files (stored bytes, checked | with the stored CRC) into relocation file P, keyed by | (origin, original offset) install one MANIFEST edit: add P, route each input generation number still | in use to P, re-point the routes into the inputs to P, and remove | each input (blob file retirement for a blob file, relocation | file deletion for a relocation file) flatten compaction sets generation number = P in the SSTs it rewrites | route route retirement: route deletion records remove the routes that delete no SST uses ``` Stack map: | Diff | Adds | |---|---| | 1 | Finalize blob metadata outside the DB mutex (prep) | | 2 | Indirect BlobIndex (kIndirectBlob) and exact SST reference metadata <== this diff | | 3 | Relocation file format and stable read path | | 4 | MANIFEST routes and Version ownership | | 5 | Exact reverse reference census | | 6 | Indirect blob GC (one input file) | | 7 | Pack whole input files; reclaim relocation files | | 8 | Flatten generation numbers; route retirement | | 9 | Remote compaction validation | | 10 | Dynamic options and operations | | 11 | RepairDB rebuilds relocation state; option gate removed | | 12 | db_stress and crash test support | Stack context: terms for later diffs: | Term | Meaning | |---|---| | census | Background count, per generation number, of the SSTs and references that use it, plus live bytes per physical file (diff 5) | | indirect origin | Normal blob file written with the option on; it is the origin of unflattened references | | flatten | Compaction rewrites the generation number to the relocation file that holds the value, so old routes can go. Only flatten makes generation number != origin (diff 8) | | blob file retirement | MANIFEST record (kBlobFileRetirement) that removes a collected blob file from the Version (diff 4) | | route deletion | MANIFEST record that removes a route that no SST uses. Route retirement is the background job that writes these records (diff 8) | Tracking issue: meta#15301 Differential Revision: D124652097
084c89f to
3fa5380
Compare
Summary: Pull Request resolved: meta#15540 Why the change is required: The goal of this stack is indirect blob GC, a blob GC that does not depend on SSTs: it removes garbage from blob files without rewriting any SST. Such a GC needs two things that RocksDB does not have: - A value identity that does not change when the value moves to another file. - A checked list of the blob references that each SST keeps. The list must include older internal versions of a key that the SST still keeps (for example for snapshots), and the blob columns of wide-column entities. This diff adds both. Later diffs build the census (an in-memory reverse index from generation number to the SSTs that use it), GC and repair on top of this list. This diff in one paragraph: With the new column family option enable_blob_indirection on, the blob file writer gives each new value an indirect BlobIndex (type 3, kIndirectBlob). It stores the value's stable identity (origin file, original offset) and its CRCs (an uncompressed value stores only the CRC of the uncompressed value; a compressed value also stores the CRC of its stored bytes). Also with the option on, the block-based table builder collects every blob reference that the SST keeps. It writes them as "exact reference metadata": data blocks, one reference index block, and a 72-byte descriptor table property. In this diff only the table builder writes this metadata, and only tests read it. A temporary option gate (see "option gate" under Terms) keeps the option off outside tests. Separately, wide-column entities with blob columns are now fully checked before their blob indexes are used, also with the option off. Parts of this diff: 1. Indirect BlobIndex (core): types 3 and 4 in db/blob/blob_index.h, and the blob file writer that writes type 3. 2. Exact reference metadata format and decoders (core): db/blob/blob_reference_metadata.{h,cc}. 3. Collector and table builder integration (core): BlobReferenceMetadataCollector, its block cache charge, and BlockBasedTableBuilder. 4. Option enable_blob_indirection and the option gate. Tests need it to turn the feature on. You can skip this part on a first read. 5. Wide-column validation (WideColumnSerialization::ForEachBlobIndex). The table builder uses it to give the blob columns of each entity to the collector. It also changes behavior with the option off (see New behavior). You can skip this part on a first read. 6. Small helpers: canonical varints (util/coding.h) and CacheReservationHandle::Resize(). You can skip these on a first read. How to read this diff: 1. Overview: read "Why" and "This diff in one paragraph" above, and "Terms used in this diff" right below. Then read the top comment of db/blob/blob_reference_metadata.h (terms, layout diagrams, run and entry layout, index layout). 2. Indirect BlobIndex (db/blob/blob_index.h) and canonical varints (util/coding.h GetCanonicalVarint32/64): read the overview comment and the kIndirectBlob / kIndirectFlattenedBlob code. Every later part uses these fields: origin, original offset, generation number, logical CRC and stored CRC. The new formats read varints only in the canonical form. 3. Blob file writer (db/blob/blob_file_builder.cc, blob_log_format.cc, blob_log_writer.cc): see how a new value gets a type-3 index and how its CRC is computed once. 4. Format types and decoders: the structs in db/blob/blob_reference_metadata.h, then in the .cc the constants, GapBase, DecodeEntryBody, DecodeRunEntriesAt, DecodeBlobReferenceIndex and BlobReferenceGroupDecoder. They use only BlobIndex and the canonical varints. Learn the on-disk format here, before you read the writer. 5. Collector and its cache charge: first read the new CacheEntryRole kBlobReferenceMetadataBuildingBuffer (include/rocksdb/cache.h) and CacheReservationHandle::Resize (cache/cache_reservation_manager.h). Then read BlobReferenceMetadataCollector in this order: the class comment, Add, AppendSorted, BuildAndWriteDataBlocks, EncodeIndex, FinishOnce. Then read the unsorted fallback (AppendUnsorted, SortUnsortedGroup, SortAndEncodeGroup). 6. Wide-column change (db/wide/wide_column_serialization.cc, ForEachBlobIndex and WalkV2Columns): it gives every blob index of an entity to a callback after it checks the whole entity. The table builder uses it to feed the collector. UpdateBoundaries, BlobGarbageMeter and the compaction iterator also use it, with the option on or off. 7. Option: enable_blob_indirection (include/rocksdb/advanced_options.h, options/cf_options.cc), and the option checks and gate in db/column_family.cc (ValidateOptions, TEST_SetAllowBlobIndirection), db/db_impl/db_impl_open.cc and db_impl_secondary.cc (the FIFO check for opens that skip ValidateOptions). 8. Table integration: table/block_based/block_based_table_factory.cc (the shared reservation manager), then block_based_table_builder.cc (Rep constructor, Add, WriteBlobReferenceMetadata, EstimatedTailSize). See where the collector is created, fed and finished. block_based_table_reader.cc maps the new block name to a block type. Terms used in this diff: Stack terms: | Term | Meaning | |---|---| | origin | Blob file where the value was first written | | original offset | Value offset in the origin file | | stable identity | (origin, original offset). Never changes, even when the value moves | | generation number | The file number that a reference uses to find its value. A new value has generation number == origin. When indirect blob GC moves the value, the reference keeps its generation number, and a route sends the read to the relocation file | | indirect BlobIndex | A kIndirectBlob (type 3) or kIndirectFlattenedBlob (type 4) index. It stores the stable identity, so the value can move without an SST rewrite | | unflattened reference | Index with generation number == origin: the value was not moved, or it was moved but no compaction has rewritten its reference yet (see "flatten" in "Stack context: terms for later diffs") | | flattened reference | Index with generation number != origin (type 4). Only flatten (diff 8) writes it; this diff defines the encoding | | route | Maps an old file number (a file that GC has removed from the Version) to the relocation file that now holds its live values. One hop: a target is never itself routed. Stored in the MANIFEST (diff 4) | | relocation file | Block-based table keyed by (origin, original offset) that holds copied values. Each value is the stored bytes plus a 5-byte trailer (kSimpleGen2BlobTrailerSize) | | logical CRC | CRC32C of the uncompressed value (BlobIndex::stable_checksum()) | | stored CRC | CRC32C of the stored bytes plus the compression byte | | physical file | A file that holds value bytes: a normal blob file or a relocation file. A generation number can name a physical file, or it can be only the key of a route | | exact reference metadata | Per-SST sorted list of every blob reference the SST keeps, written when the SST is built. References with the same encoding class, generation number and origin form a group. The list is stored as data blocks, with an index of the groups. An SST that has it is an exact SST | | option gate | Until diff 11, ColumnFamilyData::ValidateOptions returns NotSupported for enable_blob_indirection on a read-write open, CreateColumnFamily and SetOptions, unless a test calls ColumnFamilyData::TEST_SetAllowBlobIndirection(true). It keeps the option off in production while the stack is not complete | Terms of this diff (all defined at the top of db/blob/blob_reference_metadata.h): | Term | Meaning | |---|---| | encoding class (class) | kDirect (a legacy kBlob or kBlobTTL index), unflattened, or flattened | | reference group (group) | All references of one SST with the same (encoding class, generation number, origin) | | run | The part of one group inside one data block | | record bytes | The bytes the value's record takes in the file named by its generation number | | gap | How far a record is from where it would be if the records were back to back. A gap of 0 means back to back | | descriptor | The 72-byte table property that points to the reference index block and holds the totals, the CF ID, the SST file number and the relocation-state fingerprint | | relocation-state fingerprint | 128-bit hash of the routes and relocation files of the Version that an SST was built from (fields route_snapshot_fingerprint_*). Written as 0 in this diff | | legacy | Written the way RocksDB writes it without blob indirection | Previous behavior: - A BlobIndex named only a physical blob file and an offset. A value could not move to another file without rewriting every SST that referenced it. - An SST had no metadata that listed its blob references. The only way to find them was to scan its user keys. - These readers get the blob indexes of a wide-column entity through ForEachBlobFileNumber: UpdateBoundaries, BlobGarbageMeter, the compaction iterator, WideColumnSerialization::RemoveBlobColumns, and the referenced_blob_bytes table property count in BlockBasedTableBuilder::Add. ForEachBlobFileNumber decoded each blob index, but it did not check the rest of the entity framing (name order, section sizes, exact input size), and it could call the callback before it found a later error. New behavior: Separate change (option on or off): - The same readers (UpdateBoundaries, BlobGarbageMeter, the compaction iterator, RemoveBlobColumns, and the referenced_blob_bytes count in BlockBasedTableBuilder::Add) use WideColumnSerialization::ForEachBlobIndex, with the option on or off. With the option on, the table builder also uses it to give the blob columns of each entity to the collector. For an entity with blob columns, it checks the full framing, then decodes and validates every blob index before any callback runs. So an entity with a malformed framing now fails the flush or compaction, even with the option off. The referenced_blob_bytes count itself does not fail the build on a malformed entity. It removes that property from the SST, as in base. Option and option gate: - This diff adds a new column family option, `enable_blob_indirection`. It is off by default, and SetOptions can change it. - The option gate (see Terms) keeps the option off outside tests until the stack is complete. ColumnFamilyData::ValidateOptions rejects the option with NotSupported on a read-write open, CreateColumnFamily and SetOptions. Read-only opens (DBImplReadOnly and CompactedDBImpl) and secondary opens (including DB::OpenAndCompact) do not run ValidateOptions; they check only the FIFO rule (see Implementation). Only tests can turn the option on, through the internal ColumnFamilyData::TEST_SetAllowBlobIndirection(). DBTestBase sets it. - When the option is on, ValidateOptions also requires enable_blob_files, the block-based table format, no blob direct write, and a compaction style other than FIFO. Blob file writer (option on): - The blob file writer encodes each new value with a stable identity (origin file, original offset) and two checksums: - the logical CRC: the CRC32C of the uncompressed value. - the stored CRC: the CRC32C of the exact stored (maybe compressed) bytes, extended with the compression byte. - For an uncompressed value the stored CRC is not written, because the reader can compute it from the logical CRC. - The checksums have no version field of their own. The BlobIndex type byte gives the format version: the two new types, 3 (kIndirectBlob) and 4 (kIndirectFlattenedBlob, see Implementation), are BlobIndex::kIndirectBlobFormatVersion (1). A later format uses new type values. - The read path does not check these checksums yet. The check starts in diff 3. Exact reference metadata (option on): 1. What: a sorted list of every blob reference that the SST keeps, in plain blob values and in the blob columns of wide-column entities. 2. Where: data blocks (target 16 KB each before compression; small blocks waste less I/O, because indirect blob GC reads a whole block to get a few groups) and one reference index block, written after the SST data blocks and before the SST tail, plus a 72-byte descriptor table property. 3. Who reads it: only tests in this diff. The census, GC and RepairDB read it in later diffs. 4. Details: the top comment of db/blob/blob_reference_metadata.h has the full format with diagrams, the record-bytes rules and the gap rule. - Not collected: SstFileWriter output (it has no DB file number) and any output whose CF is unknown (it has no CF ID). The descriptor needs both values, so these outputs stay in the legacy form. Checks on the metadata: - A repeated or overlapping reference in one group fails the SST build with Corruption. Readers check every count, order, size and total against the index and the descriptor, so corrupt metadata returns Corruption. The writer never makes a data block larger than kTargetDataBlockSize (16 KB), and readers accept up to kMaxDataBlockSize (64 KB). Details: "Readers" and "Integrity" at the top of db/blob/blob_reference_metadata.h, and BlobReferenceGroupDecoder. Collection and memory: - Each reference is encoded when it is added, in the on-disk entry encoding, into its group's chain of 512-byte chunks. Only a group that takes the unsorted fallback is encoded a second time, in Finish(). Inside a group, references normally arrive in rising offset order, so no reference is sorted; at the end only the groups are sorted. A group that gets a lower offset takes the unsorted fallback. The BlobReferenceMetadataCollector class comment explains the chunks, the chunk size and the fallback. - The collector keeps every reference of the SST in memory until the SST is finished, and charges this memory to the block cache under a new CacheEntryRole, kBlobReferenceMetadataBuildingBuffer. The measured cost is about 8.3 to 9 bytes per reference for uncompressed values and 12.8 to 13.5 bytes for compressed values. "Memory usage" and "Memory charge" in the BlobReferenceMetadataCollector class comment give the measurement setup and the charge rule. - There is no fixed limit on references. The collector fails with MemoryLimit above 2^20 groups or 2^20 data blocks in one SST; these are persisted format limits. - There is no charge when the table has no block cache or when charging for this role is disabled. Each block-based table factory holds one cache reservation manager for this role, shared by its builders and the SST summaries. If the charge fails, the SST build fails, so the flush or compaction fails with a background error. The buffers and their charge are released when the SST is finished or abandoned. Implementation: - BlobIndex adds two indirect encodings. The legacy encodings do not change. - Type 3, kIndirectBlob (unflattened, generation number == origin), stores the origin, the original offset, the stored size, the compression type, the logical CRC and, for compressed values, the stored CRC. It does not store the generation number. This saves the size of one varint file number per reference (about 3 to 4 bytes for typical file numbers; an estimate, not measured). - Type 4, kIndirectFlattenedBlob (flattened), stores the generation number first, then the same fields. Its generation number must be above the origin. The decoder rejects type 4 with generation number <= origin, so each reference has one encoding. - The encoders take the fields as one BlobIndex::IndirectBlobFields, set by field name, so values of the same type (file number, offset and size; the two CRCs) cannot be swapped. EncodeIndirectBlob() takes the generation number as a separate argument; EncodeUnflattenedIndirectBlob() uses the origin, so the common path cannot leave the generation number unset. - The new files blob_reference_metadata.{h,cc} define the exact reference metadata. The files also contain: - BlobReferenceMetadataCollector, the collector described above. It takes the SST identity (CF ID, SST file number and the relocation-state fingerprint) as one BlobReferenceSstIdentity, set by field name. - BlobReferenceGroupKey, the key of a group: (encoding class, generation number, origin). It has the order, the hash and the validity check that the collector map, the sort, the decoder and FindBlobReferenceGroup() use. BlobReference and BlobReferenceGroup keep their flat fields and return the key through group_key(). - The index codec and the descriptor table property. - BlobReferenceGroupDecoder, the only decoder of reference runs. It reads a group one data block at a time and holds every corruption check of a group walk. Readers that visit one group (found by a binary search of the index) use it, and so do readers that visit every block in order. - Callers in this diff: only the table builder runs the collector. The decoders (DecodeBlobReferenceIndex, FindBlobReferenceGroup, BlobReferenceGroupDecoder, ForEachBlobReference*) are used only by tests. kMaxDecodedBlobReferenceBytes is used only in compile-time checks of the count limits. The census (diff 5) and indirect blob GC (diffs 6 and 7) are their first production users. The census is also the first user of BlobReferenceSstSummary (see the last Implementation bullet). - Descriptor: the CF ID and the SST file number are the repair fields. They let RepairDB tell which column family and SST the list belongs to without a MANIFEST. The decoder rejects a descriptor whose repair fields are not set (unknown CF ID or SST file number 0). The descriptor also has a 128-bit relocation-state fingerprint (the route_snapshot_fingerprint fields). Remote compaction uses it in diff 9 to check that an SST built elsewhere used the same routes as the job that asked for it. In this diff the table builder always writes 0, and DecodeFrom() does not check it. The field is in the format now, so the descriptor keeps one format version (1) through the stack. While the option gate is on, no data in this format exists outside tests. - The collector validates each BlobIndex once, when it is added. Its Finish() writes the data blocks and then the reference index block through one block writer, and fills the descriptor, including the index handle. Destroying the collector frees its memory and then its charge, also when the build is abandoned. - BlockBasedTableBuilder creates the collector with the CF ID and SST file number from TableBuilderOptions. It passes the size of each entry's user key to the collector, so the collector can compute the exact size of each blob record. It adds the collector's EstimatedEncodedSize() to its tail size estimate: the blocks are written before the tail, but output file size planning uses this estimate. The reference blocks are written with BlockType::kUserDefinedIndex, which keeps their contents as opaque bytes. Readers read them without the block cache, so prepopulate_block_cache does not warm them into the block cache (WriteMaybeCompressedBlock() with warm_cache false). - With the option on, the blob file builder computes the CRC32C of each stored value once. The blob log record CRC combines it with the key CRC (crc32c::Crc32cCombine). The stored CRC extends it with the compression byte. For an uncompressed value it is also the logical CRC. The bytes written do not change. - WideColumnSerialization::ForEachBlobIndex replaces ForEachBlobFileNumber. It reads the entity header and column types once. For a V1 entity (no column types and no blob columns), it returns right after the header. For a V2 entity without blob columns, it returns right after the type scan. It shares one V2 column walker (WalkV2Columns) with Deserialize(), so the entity is parsed in one pass. - Read-only opens (DBImplReadOnly, CompactedDBImpl) and secondary opens skip ValidateOptions, so they check the compaction style against the option once per column family: in DBImpl::Recover() for read-only opens, and at the start of OpenAsSecondaryImpl() for secondary opens. DB::OpenAndCompact opens through OpenAsSecondaryImpl(), so it gets the same check. - CacheReservationHandle gets Resize(), so each collector or summary can change its own part of the shared charge. - util/coding.h adds GetCanonicalVarint32/64, which reject overlong varint encodings. The new formats use them, so each value has one encoding. - BlobReferenceSstSummary is the in-memory list of an SST's groups and totals that Finish() can return. It has no caller yet: the table builder passes a null summary output. It is checked and does not change after Finish(), so the finished builder, the SST metadata and the census (its first user, in diff 5) can share it. It is in this diff because Finish() hands the group buffer and its cache charge over to the summary. Adding the summary later would mean rewriting the charge code of Finish() in diff 5, so the charge code is written and tested once, here. Safety boundaries and unsupported cases: - Public API: this diff adds the option `enable_blob_indirection` with the C API functions `rocksdb_options_set_enable_blob_indirection` and `rocksdb_options_get_enable_blob_indirection`. It also inserts a new CacheEntryRole value before kMisc, which changes the numeric value of kMisc and kNumCacheEntryRoles. - With the option off, new blob values and SSTs are written in the legacy form. Legacy blob indexes stay readable. - One behavior change applies with the option off too: UpdateBoundaries, BlobGarbageMeter and the compaction iterator fully validate wide-column entities with blob columns and their blob indexes (see "Separate change" above). - Exact reference metadata has one format (version 1) with a fixed 72-byte descriptor. The decoders reject any other version as Corruption. - The metadata blocks are protected by the table's block checksum only. With checksum = kNoChecksum, corrupt metadata is detected only when it breaks the format or order checks. - The index decoder checks each decoded count against the remaining input size before it reserves memory. So a small corrupt index cannot make it allocate a large vector. - Inlined-TTL and same-file blob indexes are not collected and cannot be relocated. Encoding, count and checksum errors return an error; a bad reference is never skipped. - Option gate (see Terms): outside tests no DB writes data in the new format. The gate checks only the option value, never persisted state. Diff 11 removes it, once data written with the option stays readable through the end of the stack and the safety checks for turning the option on are in place. - Downgrading the binary after data was written with the option on is not supported. Validation and known limitations: - Unit tests cover the BlobIndex encodings and their rejection of bad input, the metadata layout and every corruption check, the collector (sorted path, unsorted fallback, memory charge), the flush integration, the option checks and option gate, wide-column validation, and canonical varints. See the Test Plan. - The collection buffer is not spilled to disk. One SST with very many blob references needs a matching amount of charged memory: the per-reference amount given under "Collection and memory", plus up to one partly used 512-byte chunk per group. While a fallback group is sorted, it also needs 16 bytes of sort key per reference and a second copy of its entries. For a test group of 200K references with every other one out of order, the test comment estimates about 44 bytes per reference, and the test checks that the peak charge is at most 56 bytes per reference. - Flush still writes a single output file. Until diff 7, one compaction output collects every reference it gets, charged to the block cache. Diff 7 cuts compaction outputs (including L0) at a reference cap and keeps each user key's history in one output. With the option gate, a large output is reachable only through the test hook. Stack context (same in diffs 2-11, except that diffs 8-11 add the diff 8 route rule to the install line; "<== this diff" marks this diff's part): Goal: remove garbage from blob files without rewriting any SST. A value gets a stable identity when it is first written. Indirect blob GC copies the live values of whole files into a relocation file and installs routes in the MANIFEST. Readers follow the routes. SSTs are not changed. How a read finds a value (P is the relocation file that holds the value now): ``` SST entry indirect BlobIndex { generation number, origin, original offset, size, CRCs } | v Version file table: generation number -> live file, or route old file -> P | | live file route found | | v v blob file: read at original offset relocation file P (always one hop): relocation file: look up key look up key (origin, original offset) (origin, original offset) ``` Lifecycle of one value (in this example, F is the blob file where the value is first written, and P is the relocation file that GC writes): ``` write new value goes to blob file F; index: origin = F, generation number = F | census per generation number: the SSTs and references that use it; | per physical file: live bytes, ranked by garbage ratio GC job copy the live values of whole input files (stored bytes, checked | with the stored CRC) into relocation file P, keyed by | (origin, original offset) install one MANIFEST edit: add P, route each input generation number still | in use to P, re-point the routes into the inputs to P, and remove | each input (blob file retirement for a blob file, relocation | file deletion for a relocation file) flatten compaction sets generation number = P in the SSTs it rewrites | route route retirement: route deletion records remove the routes that delete no SST uses ``` Stack map: | Diff | Adds | |---|---| | 1 | Finalize blob metadata outside the DB mutex (prep) | | 2 | Indirect BlobIndex (kIndirectBlob) and exact SST reference metadata <== this diff | | 3 | Relocation file format and stable read path | | 4 | MANIFEST routes and Version ownership | | 5 | Exact reverse reference census | | 6 | Indirect blob GC (one input file) | | 7 | Pack whole input files; reclaim relocation files | | 8 | Flatten generation numbers; route retirement | | 9 | Remote compaction validation | | 10 | Dynamic options and operations | | 11 | RepairDB rebuilds relocation state; option gate removed | | 12 | db_stress and crash test support | Stack context: terms for later diffs: | Term | Meaning | |---|---| | census | Background count, per generation number, of the SSTs and references that use it, plus live bytes per physical file (diff 5) | | indirect origin | Normal blob file written with the option on; it is the origin of unflattened references | | flatten | Compaction rewrites the generation number to the relocation file that holds the value, so old routes can go. Only flatten makes generation number != origin (diff 8) | | blob file retirement | MANIFEST record (kBlobFileRetirement) that removes a collected blob file from the Version (diff 4) | | route deletion | MANIFEST record that removes a route that no SST uses. Route retirement is the background job that writes these records (diff 8) | Tracking issue: meta#15301 Differential Revision: D124652097
Summary: Pull Request resolved: meta#15540 Why the change is required: The goal of this stack is indirect blob GC, a blob GC that does not depend on SSTs: it removes garbage from blob files without rewriting any SST. Such a GC needs two things that RocksDB does not have: - A value identity that does not change when the value moves to another file. - A checked list of the blob references that each SST keeps. The list must include older internal versions of a key that the SST still keeps (for example for snapshots), and the blob columns of wide-column entities. This diff adds both. Later diffs build the census (an in-memory reverse index from generation number to the SSTs that use it), GC and repair on top of this list. This diff in one paragraph: With the new column family option enable_blob_indirection on, the blob file writer gives each new value an indirect BlobIndex (type 3, kIndirectBlob). It stores the value's stable identity (origin file, original offset) and its CRCs (an uncompressed value stores only the CRC of the uncompressed value; a compressed value also stores the CRC of its stored bytes). Also with the option on, the block-based table builder collects every blob reference that the SST keeps. It writes them as "exact reference metadata": data blocks, one reference index block, and a 72-byte descriptor table property. In this diff only the table builder writes this metadata, and only tests read it. A temporary option gate (see "option gate" under Terms) keeps the option off outside tests. Separately, wide-column entities with blob columns are now fully checked before their blob indexes are used, also with the option off. Parts of this diff: 1. Indirect BlobIndex (core): types 3 and 4 in db/blob/blob_index.h, and the blob file writer that writes type 3. 2. Exact reference metadata format and decoders (core): db/blob/blob_reference_metadata.{h,cc}. 3. Collector and table builder integration (core): BlobReferenceMetadataCollector, its block cache charge, and BlockBasedTableBuilder. 4. Option enable_blob_indirection and the option gate. Tests need it to turn the feature on. You can skip this part on a first read. 5. Wide-column validation (WideColumnSerialization::ForEachBlobIndex). The table builder uses it to give the blob columns of each entity to the collector. It also changes behavior with the option off (see New behavior). You can skip this part on a first read. 6. Small helpers: canonical varints (util/coding.h) and CacheReservationHandle::Resize(). You can skip these on a first read. How to read this diff: 1. Overview: read "Why" and "This diff in one paragraph" above, and "Terms used in this diff" right below. Then read the top comment of db/blob/blob_reference_metadata.h (terms, layout diagrams, run and entry layout, index layout). 2. Indirect BlobIndex (db/blob/blob_index.h) and canonical varints (util/coding.h GetCanonicalVarint32/64): read the overview comment and the kIndirectBlob / kIndirectFlattenedBlob code. Every later part uses these fields: origin, original offset, generation number, logical CRC and stored CRC. The new formats read varints only in the canonical form. 3. Blob file writer (db/blob/blob_file_builder.cc, blob_log_format.cc, blob_log_writer.cc): see how a new value gets a type-3 index and how its CRC is computed once. 4. Format types and decoders: the structs in db/blob/blob_reference_metadata.h, then in the .cc the constants, GapBase, DecodeEntryBody, DecodeRunEntriesAt, DecodeBlobReferenceIndex and BlobReferenceGroupDecoder. They use only BlobIndex and the canonical varints. Learn the on-disk format here, before you read the writer. 5. Collector and its cache charge: first read the new CacheEntryRole kBlobReferenceMetadataBuildingBuffer (include/rocksdb/cache.h) and CacheReservationHandle::Resize (cache/cache_reservation_manager.h). Then read BlobReferenceMetadataCollector in this order: the class comment, Add, AppendSorted, BuildAndWriteDataBlocks, EncodeIndex, FinishOnce. Then read the unsorted fallback (AppendUnsorted, SortUnsortedGroup, SortAndEncodeGroup). 6. Wide-column change (db/wide/wide_column_serialization.cc, ForEachBlobIndex and WalkV2Columns): it gives every blob index of an entity to a callback after it checks the whole entity. The table builder uses it to feed the collector. UpdateBoundaries, BlobGarbageMeter and the compaction iterator also use it, with the option on or off. 7. Option: enable_blob_indirection (include/rocksdb/advanced_options.h, options/cf_options.cc), and the option checks and gate in db/column_family.cc (ValidateOptions, TEST_SetAllowBlobIndirection), db/db_impl/db_impl_open.cc and db_impl_secondary.cc (the FIFO check for opens that skip ValidateOptions). 8. Table integration: table/block_based/block_based_table_factory.cc (the shared reservation manager), then block_based_table_builder.cc (Rep constructor, Add, WriteBlobReferenceMetadata, EstimatedTailSize). See where the collector is created, fed and finished. block_based_table_reader.cc maps the new block name to a block type. Terms used in this diff: Stack terms: | Term | Meaning | |---|---| | origin | Blob file where the value was first written | | original offset | Value offset in the origin file | | stable identity | (origin, original offset). Never changes, even when the value moves | | generation number | The file number that a reference uses to find its value. A new value has generation number == origin. When indirect blob GC moves the value, the reference keeps its generation number, and a route sends the read to the relocation file | | indirect BlobIndex | A kIndirectBlob (type 3) or kIndirectFlattenedBlob (type 4) index. It stores the stable identity, so the value can move without an SST rewrite | | unflattened reference | Index with generation number == origin: the value was not moved, or it was moved but no compaction has rewritten its reference yet (see "flatten" in "Stack context: terms for later diffs") | | flattened reference | Index with generation number != origin (type 4). Only flatten (diff 8) writes it; this diff defines the encoding | | route | Maps an old file number (a file that GC has removed from the Version) to the relocation file that now holds its live values. One hop: a target is never itself routed. Stored in the MANIFEST (diff 4) | | relocation file | Block-based table keyed by (origin, original offset) that holds copied values. Each value is the stored bytes plus a 5-byte trailer (kSimpleGen2BlobTrailerSize) | | logical CRC | CRC32C of the uncompressed value (BlobIndex::stable_checksum()) | | stored CRC | CRC32C of the stored bytes plus the compression byte | | physical file | A file that holds value bytes: a normal blob file or a relocation file. A generation number can name a physical file, or it can be only the key of a route | | exact reference metadata | Per-SST sorted list of every blob reference the SST keeps, written when the SST is built. References with the same encoding class, generation number and origin form a group. The list is stored as data blocks, with an index of the groups. An SST that has it is an exact SST | | option gate | Until diff 11, ColumnFamilyData::ValidateOptions returns NotSupported for enable_blob_indirection on a read-write open, CreateColumnFamily and SetOptions, unless a test calls ColumnFamilyData::TEST_SetAllowBlobIndirection(true). It keeps the option off in production while the stack is not complete | Terms of this diff (all defined at the top of db/blob/blob_reference_metadata.h): | Term | Meaning | |---|---| | encoding class (class) | kDirect (a legacy kBlob or kBlobTTL index), unflattened, or flattened | | reference group (group) | All references of one SST with the same (encoding class, generation number, origin) | | run | The part of one group inside one data block | | record bytes | The bytes the value's record takes in the file named by its generation number | | gap | How far a record is from where it would be if the records were back to back. A gap of 0 means back to back | | descriptor | The 72-byte table property that points to the reference index block and holds the totals, the CF ID, the SST file number and the relocation-state fingerprint | | relocation-state fingerprint | 128-bit hash of the routes and relocation files of the Version that an SST was built from (fields route_snapshot_fingerprint_*). Written as 0 in this diff | | legacy | Written the way RocksDB writes it without blob indirection | Previous behavior: - A BlobIndex named only a physical blob file and an offset. A value could not move to another file without rewriting every SST that referenced it. - An SST had no metadata that listed its blob references. The only way to find them was to scan its user keys. - These readers get the blob indexes of a wide-column entity through ForEachBlobFileNumber: UpdateBoundaries, BlobGarbageMeter, the compaction iterator, WideColumnSerialization::RemoveBlobColumns, and the referenced_blob_bytes table property count in BlockBasedTableBuilder::Add. ForEachBlobFileNumber decoded each blob index, but it did not check the rest of the entity framing (name order, section sizes, exact input size), and it could call the callback before it found a later error. New behavior: Separate change (option on or off): - The same readers (UpdateBoundaries, BlobGarbageMeter, the compaction iterator, RemoveBlobColumns, and the referenced_blob_bytes count in BlockBasedTableBuilder::Add) use WideColumnSerialization::ForEachBlobIndex, with the option on or off. With the option on, the table builder also uses it to give the blob columns of each entity to the collector. For an entity with blob columns, it checks the full framing, then decodes and validates every blob index before any callback runs. So an entity with a malformed framing now fails the flush or compaction, even with the option off. The referenced_blob_bytes count itself does not fail the build on a malformed entity. It removes that property from the SST, as in base. Option and option gate: - This diff adds a new column family option, `enable_blob_indirection`. It is off by default, and SetOptions can change it. - The option gate (see Terms) keeps the option off outside tests until the stack is complete. ColumnFamilyData::ValidateOptions rejects the option with NotSupported on a read-write open, CreateColumnFamily and SetOptions. Read-only opens (DBImplReadOnly and CompactedDBImpl) and secondary opens (including DB::OpenAndCompact) do not run ValidateOptions; they check only the FIFO rule (see Implementation). Only tests can turn the option on, through the internal ColumnFamilyData::TEST_SetAllowBlobIndirection(). DBTestBase sets it. - When the option is on, ValidateOptions also requires enable_blob_files, the block-based table format, no blob direct write, and a compaction style other than FIFO. Blob file writer (option on): - The blob file writer encodes each new value with a stable identity (origin file, original offset) and two checksums: - the logical CRC: the CRC32C of the uncompressed value. - the stored CRC: the CRC32C of the exact stored (maybe compressed) bytes, extended with the compression byte. - For an uncompressed value the stored CRC is not written, because the reader can compute it from the logical CRC. - The checksums have no version field of their own. The BlobIndex type byte gives the format version: the two new types, 3 (kIndirectBlob) and 4 (kIndirectFlattenedBlob, see Implementation), are BlobIndex::kIndirectBlobFormatVersion (1). A later format uses new type values. - The read path does not check these checksums yet. The check starts in diff 3. Exact reference metadata (option on): 1. What: a sorted list of every blob reference that the SST keeps, in plain blob values and in the blob columns of wide-column entities. 2. Where: data blocks (target 16 KB each before compression; small blocks waste less I/O, because indirect blob GC reads a whole block to get a few groups) and one reference index block, written after the SST data blocks and before the SST tail, plus a 72-byte descriptor table property. 3. Who reads it: only tests in this diff. The census, GC and RepairDB read it in later diffs. 4. Details: the top comment of db/blob/blob_reference_metadata.h has the full format with diagrams, the record-bytes rules and the gap rule. - Not collected: SstFileWriter output (it has no DB file number) and any output whose CF is unknown (it has no CF ID). The descriptor needs both values, so these outputs stay in the legacy form. Checks on the metadata: - A repeated or overlapping reference in one group fails the SST build with Corruption. Readers check every count, order, size and total against the index and the descriptor, so corrupt metadata returns Corruption. The writer never makes a data block larger than kTargetDataBlockSize (16 KB), and readers accept up to kMaxDataBlockSize (64 KB). Details: "Readers" and "Integrity" at the top of db/blob/blob_reference_metadata.h, and BlobReferenceGroupDecoder. Collection and memory: - Each reference is encoded when it is added, in the on-disk entry encoding, into its group's chain of 512-byte chunks. Only a group that takes the unsorted fallback is encoded a second time, in Finish(). Inside a group, references normally arrive in rising offset order, so no reference is sorted; at the end only the groups are sorted. A group that gets a lower offset takes the unsorted fallback. The BlobReferenceMetadataCollector class comment explains the chunks, the chunk size and the fallback. - The collector keeps every reference of the SST in memory until the SST is finished, and charges this memory to the block cache under a new CacheEntryRole, kBlobReferenceMetadataBuildingBuffer. The measured cost is about 8.3 to 9 bytes per reference for uncompressed values and 12.8 to 13.5 bytes for compressed values. "Memory usage" and "Memory charge" in the BlobReferenceMetadataCollector class comment give the measurement setup and the charge rule. - There is no fixed limit on references. The collector fails with MemoryLimit above 2^20 groups or 2^20 data blocks in one SST; these are persisted format limits. - There is no charge when the table has no block cache or when charging for this role is disabled. Each block-based table factory holds one cache reservation manager for this role, shared by its builders and the SST summaries. If the charge fails, the SST build fails, so the flush or compaction fails with a background error. The buffers and their charge are released when the SST is finished or abandoned. Implementation: - BlobIndex adds two indirect encodings. The legacy encodings do not change. - Type 3, kIndirectBlob (unflattened, generation number == origin), stores the origin, the original offset, the stored size, the compression type, the logical CRC and, for compressed values, the stored CRC. It does not store the generation number. This saves the size of one varint file number per reference (about 3 to 4 bytes for typical file numbers; an estimate, not measured). - Type 4, kIndirectFlattenedBlob (flattened), stores the generation number first, then the same fields. Its generation number must be above the origin. The decoder rejects type 4 with generation number <= origin, so each reference has one encoding. - The encoders take the fields as one BlobIndex::IndirectBlobFields, set by field name, so values of the same type (file number, offset and size; the two CRCs) cannot be swapped. EncodeIndirectBlob() takes the generation number as a separate argument; EncodeUnflattenedIndirectBlob() uses the origin, so the common path cannot leave the generation number unset. - The new files blob_reference_metadata.{h,cc} define the exact reference metadata. The files also contain: - BlobReferenceMetadataCollector, the collector described above. It takes the SST identity (CF ID, SST file number and the relocation-state fingerprint) as one BlobReferenceSstIdentity, set by field name. - BlobReferenceGroupKey, the key of a group: (encoding class, generation number, origin). It has the order, the hash and the validity check that the collector map, the sort, the decoder and FindBlobReferenceGroup() use. BlobReference and BlobReferenceGroup keep their flat fields and return the key through group_key(). - The index codec and the descriptor table property. - BlobReferenceGroupDecoder, the only decoder of reference runs. It reads a group one data block at a time and holds every corruption check of a group walk. Readers that visit one group (found by a binary search of the index) use it, and so do readers that visit every block in order. - Callers in this diff: only the table builder runs the collector. The decoders (DecodeBlobReferenceIndex, FindBlobReferenceGroup, BlobReferenceGroupDecoder, ForEachBlobReference*) are used only by tests. kMaxDecodedBlobReferenceBytes is used only in compile-time checks of the count limits. The census (diff 5) and indirect blob GC (diffs 6 and 7) are their first production users. The census is also the first user of BlobReferenceSstSummary (see the last Implementation bullet). - Descriptor: the CF ID and the SST file number are the repair fields. They let RepairDB tell which column family and SST the list belongs to without a MANIFEST. The decoder rejects a descriptor whose repair fields are not set (unknown CF ID or SST file number 0). The descriptor also has a 128-bit relocation-state fingerprint (the route_snapshot_fingerprint fields). Remote compaction uses it in diff 9 to check that an SST built elsewhere used the same routes as the job that asked for it. In this diff the table builder always writes 0, and DecodeFrom() does not check it. The field is in the format now, so the descriptor keeps one format version (1) through the stack. While the option gate is on, no data in this format exists outside tests. - The collector validates each BlobIndex once, when it is added. Its Finish() writes the data blocks and then the reference index block through one block writer, and fills the descriptor, including the index handle. Destroying the collector frees its memory and then its charge, also when the build is abandoned. - BlockBasedTableBuilder creates the collector with the CF ID and SST file number from TableBuilderOptions. It passes the size of each entry's user key to the collector, so the collector can compute the exact size of each blob record. It adds the collector's EstimatedEncodedSize() to its tail size estimate: the blocks are written before the tail, but output file size planning uses this estimate. The reference blocks are written with BlockType::kUserDefinedIndex, which keeps their contents as opaque bytes. Readers read them without the block cache, so prepopulate_block_cache does not warm them into the block cache (WriteMaybeCompressedBlock() with warm_cache false). - With the option on, the blob file builder computes the CRC32C of each stored value once. The blob log record CRC combines it with the key CRC (crc32c::Crc32cCombine). The stored CRC extends it with the compression byte. For an uncompressed value it is also the logical CRC. The bytes written do not change. - WideColumnSerialization::ForEachBlobIndex replaces ForEachBlobFileNumber. It reads the entity header and column types once. For a V1 entity (no column types and no blob columns), it returns right after the header. For a V2 entity without blob columns, it returns right after the type scan. It shares one V2 column walker (WalkV2Columns) with Deserialize(), so the entity is parsed in one pass. - Read-only opens (DBImplReadOnly, CompactedDBImpl) and secondary opens skip ValidateOptions, so they check the compaction style against the option once per column family: in DBImpl::Recover() for read-only opens, and at the start of OpenAsSecondaryImpl() for secondary opens. DB::OpenAndCompact opens through OpenAsSecondaryImpl(), so it gets the same check. - CacheReservationHandle gets Resize(), so each collector or summary can change its own part of the shared charge. - util/coding.h adds GetCanonicalVarint32/64, which reject overlong varint encodings. The new formats use them, so each value has one encoding. - BlobReferenceSstSummary is the in-memory list of an SST's groups and totals that Finish() can return. It has no caller yet: the table builder passes a null summary output. It is checked and does not change after Finish(), so the finished builder, the SST metadata and the census (its first user, in diff 5) can share it. It is in this diff because Finish() hands the group buffer and its cache charge over to the summary. Adding the summary later would mean rewriting the charge code of Finish() in diff 5, so the charge code is written and tested once, here. Safety boundaries and unsupported cases: - Public API: this diff adds the option `enable_blob_indirection` with the C API functions `rocksdb_options_set_enable_blob_indirection` and `rocksdb_options_get_enable_blob_indirection`. It also inserts a new CacheEntryRole value before kMisc, which changes the numeric value of kMisc and kNumCacheEntryRoles. - With the option off, new blob values and SSTs are written in the legacy form. Legacy blob indexes stay readable. - One behavior change applies with the option off too: UpdateBoundaries, BlobGarbageMeter and the compaction iterator fully validate wide-column entities with blob columns and their blob indexes (see "Separate change" above). - Exact reference metadata has one format (version 1) with a fixed 72-byte descriptor. The decoders reject any other version as Corruption. - The metadata blocks are protected by the table's block checksum only. With checksum = kNoChecksum, corrupt metadata is detected only when it breaks the format or order checks. - The index decoder checks each decoded count against the remaining input size before it reserves memory. So a small corrupt index cannot make it allocate a large vector. - Inlined-TTL and same-file blob indexes are not collected and cannot be relocated. Encoding, count and checksum errors return an error; a bad reference is never skipped. - Option gate (see Terms): outside tests no DB writes data in the new format. The gate checks only the option value, never persisted state. Diff 11 removes it, once data written with the option stays readable through the end of the stack and the safety checks for turning the option on are in place. - Downgrading the binary after data was written with the option on is not supported. Validation and known limitations: - Unit tests cover the BlobIndex encodings and their rejection of bad input, the metadata layout and every corruption check, the collector (sorted path, unsorted fallback, memory charge), the flush integration, the option checks and option gate, wide-column validation, and canonical varints. See the Test Plan. - The collection buffer is not spilled to disk. One SST with very many blob references needs a matching amount of charged memory: the per-reference amount given under "Collection and memory", plus up to one partly used 512-byte chunk per group. While a fallback group is sorted, it also needs 16 bytes of sort key per reference and a second copy of its entries. For a test group of 200K references with every other one out of order, the test comment estimates about 44 bytes per reference, and the test checks that the peak charge is at most 56 bytes per reference. - Flush still writes a single output file. Until diff 7, one compaction output collects every reference it gets, charged to the block cache. Diff 7 cuts compaction outputs (including L0) at a reference cap and keeps each user key's history in one output. With the option gate, a large output is reachable only through the test hook. Stack context (same in diffs 2-11, except that diffs 8-11 add the diff 8 route rule to the install line; "<== this diff" marks this diff's part): Goal: remove garbage from blob files without rewriting any SST. A value gets a stable identity when it is first written. Indirect blob GC copies the live values of whole files into a relocation file and installs routes in the MANIFEST. Readers follow the routes. SSTs are not changed. How a read finds a value (P is the relocation file that holds the value now): ``` SST entry indirect BlobIndex { generation number, origin, original offset, size, CRCs } | v Version file table: generation number -> live file, or route old file -> P | | live file route found | | v v blob file: read at original offset relocation file P (always one hop): relocation file: look up key look up key (origin, original offset) (origin, original offset) ``` Lifecycle of one value (in this example, F is the blob file where the value is first written, and P is the relocation file that GC writes): ``` write new value goes to blob file F; index: origin = F, generation number = F | census per generation number: the SSTs and references that use it; | per physical file: live bytes, ranked by garbage ratio GC job copy the live values of whole input files (stored bytes, checked | with the stored CRC) into relocation file P, keyed by | (origin, original offset) install one MANIFEST edit: add P, route each input generation number still | in use to P, re-point the routes into the inputs to P, and remove | each input (blob file retirement for a blob file, relocation | file deletion for a relocation file) flatten compaction sets generation number = P in the SSTs it rewrites | route route retirement: route deletion records remove the routes that delete no SST uses ``` Stack map: | Diff | Adds | |---|---| | 1 | Finalize blob metadata outside the DB mutex (prep) | | 2 | Indirect BlobIndex (kIndirectBlob) and exact SST reference metadata <== this diff | | 3 | Relocation file format and stable read path | | 4 | MANIFEST routes and Version ownership | | 5 | Exact reverse reference census | | 6 | Indirect blob GC (one input file) | | 7 | Pack whole input files; reclaim relocation files | | 8 | Flatten generation numbers; route retirement | | 9 | Remote compaction validation | | 10 | Dynamic options and operations | | 11 | RepairDB rebuilds relocation state; option gate removed | | 12 | db_stress and crash test support | Stack context: terms for later diffs: | Term | Meaning | |---|---| | census | Background count, per generation number, of the SSTs and references that use it, plus live bytes per physical file (diff 5) | | indirect origin | Normal blob file written with the option on; it is the origin of unflattened references | | flatten | Compaction rewrites the generation number to the relocation file that holds the value, so old routes can go. Only flatten makes generation number != origin (diff 8) | | blob file retirement | MANIFEST record (kBlobFileRetirement) that removes a collected blob file from the Version (diff 4) | | route deletion | MANIFEST record that removes a route that no SST uses. Route retirement is the background job that writes these records (diff 8) | Tracking issue: meta#15301 Differential Revision: D124652097
Summary: Pull Request resolved: meta#15540 Why the change is required: The goal of this stack is indirect blob GC, a blob GC that does not depend on SSTs: it removes garbage from blob files without rewriting any SST. Such a GC needs two things that RocksDB does not have: - A value identity that does not change when the value moves to another file. - A checked list of the blob references that each SST keeps. The list must include older internal versions of a key that the SST still keeps (for example for snapshots), and the blob columns of wide-column entities. This diff adds both. Later diffs build the census (an in-memory reverse index from generation number to the SSTs that use it), GC and repair on top of this list. This diff in one paragraph: With the new column family option enable_blob_indirection on, the blob file writer gives each new value an indirect BlobIndex (type 3, kIndirectBlob). It stores the value's stable identity (origin file, original offset) and its CRCs (an uncompressed value stores only the CRC of the uncompressed value; a compressed value also stores the CRC of its stored bytes). Also with the option on, the block-based table builder collects every blob reference that the SST keeps. It writes them as "exact reference metadata": data blocks, one reference index block, and a 72-byte descriptor table property. In this diff only the table builder writes this metadata, and only tests read it. A temporary option gate (see "option gate" under Terms) keeps the option off outside tests. Separately, wide-column entities with blob columns are now fully checked before their blob indexes are used, also with the option off. Parts of this diff: 1. Indirect BlobIndex (core): types 3 and 4 in db/blob/blob_index.h, and the blob file writer that writes type 3. 2. Exact reference metadata format and decoders (core): db/blob/blob_reference_metadata.{h,cc}. 3. Collector and table builder integration (core): BlobReferenceMetadataCollector, its block cache charge, and BlockBasedTableBuilder. 4. Option enable_blob_indirection and the option gate. Tests need it to turn the feature on. You can skip this part on a first read. 5. Wide-column validation (WideColumnSerialization::ForEachBlobIndex). The table builder uses it to give the blob columns of each entity to the collector. It also changes behavior with the option off (see New behavior). You can skip this part on a first read. 6. Small helpers: canonical varints (util/coding.h) and CacheReservationHandle::Resize(). You can skip these on a first read. How to read this diff: 1. Overview: read "Why" and "This diff in one paragraph" above, and "Terms used in this diff" right below. Then read the top comment of db/blob/blob_reference_metadata.h (terms, layout diagrams, run and entry layout, index layout). 2. Indirect BlobIndex (db/blob/blob_index.h) and canonical varints (util/coding.h GetCanonicalVarint32/64): read the overview comment and the kIndirectBlob / kIndirectFlattenedBlob code. Every later part uses these fields: origin, original offset, generation number, logical CRC and stored CRC. The new formats read varints only in the canonical form. 3. Blob file writer (db/blob/blob_file_builder.cc, blob_log_format.cc, blob_log_writer.cc): see how a new value gets a type-3 index and how its CRC is computed once. 4. Format types and decoders: the structs in db/blob/blob_reference_metadata.h, then in the .cc the constants, GapBase, DecodeEntryBody, DecodeRunEntriesAt, DecodeBlobReferenceIndex and BlobReferenceGroupDecoder. They use only BlobIndex and the canonical varints. Learn the on-disk format here, before you read the writer. 5. Collector and its cache charge: first read the new CacheEntryRole kBlobReferenceMetadataBuildingBuffer (include/rocksdb/cache.h) and CacheReservationHandle::Resize (cache/cache_reservation_manager.h). Then read BlobReferenceMetadataCollector in this order: the class comment, Add, AppendSorted, BuildAndWriteDataBlocks, EncodeIndex, FinishOnce. Then read the unsorted fallback (AppendUnsorted, SortUnsortedGroup, SortAndEncodeGroup). 6. Wide-column change (db/wide/wide_column_serialization.cc, ForEachBlobIndex and WalkV2Columns): it gives every blob index of an entity to a callback after it checks the whole entity. The table builder uses it to feed the collector. UpdateBoundaries, BlobGarbageMeter and the compaction iterator also use it, with the option on or off. 7. Option: enable_blob_indirection (include/rocksdb/advanced_options.h, options/cf_options.cc), and the option checks and gate in db/column_family.cc (ValidateOptions, TEST_SetAllowBlobIndirection), db/db_impl/db_impl_open.cc and db_impl_secondary.cc (the FIFO check for opens that skip ValidateOptions). 8. Table integration: table/block_based/block_based_table_factory.cc (the shared reservation manager), then block_based_table_builder.cc (Rep constructor, Add, WriteBlobReferenceMetadata, EstimatedTailSize). See where the collector is created, fed and finished. block_based_table_reader.cc maps the new block name to a block type. Terms used in this diff: Stack terms: | Term | Meaning | |---|---| | origin | Blob file where the value was first written | | original offset | Value offset in the origin file | | stable identity | (origin, original offset). Never changes, even when the value moves | | generation number | The file number that a reference uses to find its value. A new value has generation number == origin. When indirect blob GC moves the value, the reference keeps its generation number, and a route sends the read to the relocation file | | indirect BlobIndex | A kIndirectBlob (type 3) or kIndirectFlattenedBlob (type 4) index. It stores the stable identity, so the value can move without an SST rewrite | | unflattened reference | Index with generation number == origin: the value was not moved, or it was moved but no compaction has rewritten its reference yet (see "flatten" in "Stack context: terms for later diffs") | | flattened reference | Index with generation number != origin (type 4). Only flatten (diff 8) writes it; this diff defines the encoding | | route | Maps an old file number (a file that GC has removed from the Version) to the relocation file that now holds its live values. One hop: a target is never itself routed. Stored in the MANIFEST (diff 4) | | relocation file | Block-based table keyed by (origin, original offset) that holds copied values. Each value is the stored bytes plus a 5-byte trailer (kSimpleGen2BlobTrailerSize) | | logical CRC | CRC32C of the uncompressed value (BlobIndex::stable_checksum()) | | stored CRC | CRC32C of the stored bytes plus the compression byte | | physical file | A file that holds value bytes: a normal blob file or a relocation file. A generation number can name a physical file, or it can be only the key of a route | | exact reference metadata | Per-SST sorted list of every blob reference the SST keeps, written when the SST is built. References with the same encoding class, generation number and origin form a group. The list is stored as data blocks, with an index of the groups. An SST that has it is an exact SST | | option gate | Until diff 11, ColumnFamilyData::ValidateOptions returns NotSupported for enable_blob_indirection on a read-write open, CreateColumnFamily and SetOptions, unless a test calls ColumnFamilyData::TEST_SetAllowBlobIndirection(true). It keeps the option off in production while the stack is not complete | Terms of this diff (all defined at the top of db/blob/blob_reference_metadata.h): | Term | Meaning | |---|---| | encoding class (class) | kDirect (a legacy kBlob or kBlobTTL index), unflattened, or flattened | | reference group (group) | All references of one SST with the same (encoding class, generation number, origin) | | run | The part of one group inside one data block | | record bytes | The bytes the value's record takes in the file named by its generation number | | gap | How far a record is from where it would be if the records were back to back. A gap of 0 means back to back | | descriptor | The 72-byte table property that points to the reference index block and holds the totals, the CF ID, the SST file number and the relocation-state fingerprint | | relocation-state fingerprint | 128-bit hash of the routes and relocation files of the Version that an SST was built from (fields route_snapshot_fingerprint_*). Written as 0 in this diff | | legacy | Written the way RocksDB writes it without blob indirection | Previous behavior: - A BlobIndex named only a physical blob file and an offset. A value could not move to another file without rewriting every SST that referenced it. - An SST had no metadata that listed its blob references. The only way to find them was to scan its user keys. - These readers get the blob indexes of a wide-column entity through ForEachBlobFileNumber: UpdateBoundaries, BlobGarbageMeter, the compaction iterator, WideColumnSerialization::RemoveBlobColumns, and the referenced_blob_bytes table property count in BlockBasedTableBuilder::Add. ForEachBlobFileNumber decoded each blob index, but it did not check the rest of the entity framing (name order, section sizes, exact input size), and it could call the callback before it found a later error. New behavior: Separate change (option on or off): - The same readers (UpdateBoundaries, BlobGarbageMeter, the compaction iterator, RemoveBlobColumns, and the referenced_blob_bytes count in BlockBasedTableBuilder::Add) use WideColumnSerialization::ForEachBlobIndex, with the option on or off. With the option on, the table builder also uses it to give the blob columns of each entity to the collector. For an entity with blob columns, it checks the full framing, then decodes and validates every blob index before any callback runs. So an entity with a malformed framing now fails the flush or compaction, even with the option off. The referenced_blob_bytes count itself does not fail the build on a malformed entity. It removes that property from the SST, as in base. Option and option gate: - This diff adds a new column family option, `enable_blob_indirection`. It is off by default, and SetOptions can change it. - The option gate (see Terms) keeps the option off outside tests until the stack is complete. ColumnFamilyData::ValidateOptions rejects the option with NotSupported on a read-write open, CreateColumnFamily and SetOptions. Read-only opens (DBImplReadOnly and CompactedDBImpl) and secondary opens (including DB::OpenAndCompact) do not run ValidateOptions; they check only the FIFO rule (see Implementation). Only tests can turn the option on, through the internal ColumnFamilyData::TEST_SetAllowBlobIndirection(). DBTestBase sets it. - When the option is on, ValidateOptions also requires enable_blob_files, the block-based table format, no blob direct write, and a compaction style other than FIFO. Blob file writer (option on): - The blob file writer encodes each new value with a stable identity (origin file, original offset) and two checksums: - the logical CRC: the CRC32C of the uncompressed value. - the stored CRC: the CRC32C of the exact stored (maybe compressed) bytes, extended with the compression byte. - For an uncompressed value the stored CRC is not written, because the reader can compute it from the logical CRC. - The checksums have no version field of their own. The BlobIndex type byte gives the format version: the two new types, 3 (kIndirectBlob) and 4 (kIndirectFlattenedBlob, see Implementation), are BlobIndex::kIndirectBlobFormatVersion (1). A later format uses new type values. - The read path does not check these checksums yet. The check starts in diff 3. Exact reference metadata (option on): 1. What: a sorted list of every blob reference that the SST keeps, in plain blob values and in the blob columns of wide-column entities. 2. Where: data blocks (target 16 KB each before compression; small blocks waste less I/O, because indirect blob GC reads a whole block to get a few groups) and one reference index block, written after the SST data blocks and before the SST tail, plus a 72-byte descriptor table property. 3. Who reads it: only tests in this diff. The census, GC and RepairDB read it in later diffs. 4. Details: the top comment of db/blob/blob_reference_metadata.h has the full format with diagrams, the record-bytes rules and the gap rule. - Not collected: SstFileWriter output (it has no DB file number) and any output whose CF is unknown (it has no CF ID). The descriptor needs both values, so these outputs stay in the legacy form. Checks on the metadata: - A repeated or overlapping reference in one group fails the SST build with Corruption. Readers check every count, order, size and total against the index and the descriptor, so corrupt metadata returns Corruption. The writer never makes a data block larger than kTargetDataBlockSize (16 KB), and readers accept up to kMaxDataBlockSize (64 KB). Details: "Readers" and "Integrity" at the top of db/blob/blob_reference_metadata.h, and BlobReferenceGroupDecoder. Collection and memory: - Each reference is encoded when it is added, in the on-disk entry encoding, into its group's chain of 512-byte chunks. Only a group that takes the unsorted fallback is encoded a second time, in Finish(). Inside a group, references normally arrive in rising offset order, so no reference is sorted; at the end only the groups are sorted. A group that gets a lower offset takes the unsorted fallback. The BlobReferenceMetadataCollector class comment explains the chunks, the chunk size and the fallback. - The collector keeps every reference of the SST in memory until the SST is finished, and charges this memory to the block cache under a new CacheEntryRole, kBlobReferenceMetadataBuildingBuffer. The measured cost is about 8.3 to 9 bytes per reference for uncompressed values and 12.8 to 13.5 bytes for compressed values. "Memory usage" and "Memory charge" in the BlobReferenceMetadataCollector class comment give the measurement setup and the charge rule. - There is no fixed limit on references. The collector fails with MemoryLimit above 2^20 groups or 2^20 data blocks in one SST; these are persisted format limits. - There is no charge when the table has no block cache or when charging for this role is disabled. Each block-based table factory holds one cache reservation manager for this role, shared by its builders and the SST summaries. If the charge fails, the SST build fails, so the flush or compaction fails with a background error. The buffers and their charge are released when the SST is finished or abandoned. Implementation: - BlobIndex adds two indirect encodings. The legacy encodings do not change. - Type 3, kIndirectBlob (unflattened, generation number == origin), stores the origin, the original offset, the stored size, the compression type, the logical CRC and, for compressed values, the stored CRC. It does not store the generation number. This saves the size of one varint file number per reference (about 3 to 4 bytes for typical file numbers; an estimate, not measured). - Type 4, kIndirectFlattenedBlob (flattened), stores the generation number first, then the same fields. Its generation number must be above the origin. The decoder rejects type 4 with generation number <= origin, so each reference has one encoding. - The encoders take the fields as one BlobIndex::IndirectBlobFields, set by field name, so values of the same type (file number, offset and size; the two CRCs) cannot be swapped. EncodeIndirectBlob() takes the generation number as a separate argument; EncodeUnflattenedIndirectBlob() uses the origin, so the common path cannot leave the generation number unset. - The new files blob_reference_metadata.{h,cc} define the exact reference metadata. The files also contain: - BlobReferenceMetadataCollector, the collector described above. It takes the SST identity (CF ID, SST file number and the relocation-state fingerprint) as one BlobReferenceSstIdentity, set by field name. - BlobReferenceGroupKey, the key of a group: (encoding class, generation number, origin). It has the order, the hash and the validity check that the collector map, the sort, the decoder and FindBlobReferenceGroup() use. BlobReference and BlobReferenceGroup keep their flat fields and return the key through group_key(). - The index codec and the descriptor table property. - BlobReferenceGroupDecoder, the only decoder of reference runs. It reads a group one data block at a time and holds every corruption check of a group walk. Readers that visit one group (found by a binary search of the index) use it, and so do readers that visit every block in order. - Callers in this diff: only the table builder runs the collector. The decoders (DecodeBlobReferenceIndex, FindBlobReferenceGroup, BlobReferenceGroupDecoder, ForEachBlobReference*) are used only by tests. kMaxDecodedBlobReferenceBytes is used only in compile-time checks of the count limits. The census (diff 5) and indirect blob GC (diffs 6 and 7) are their first production users. The census is also the first user of BlobReferenceSstSummary (see the last Implementation bullet). - Descriptor: the CF ID and the SST file number are the repair fields. They let RepairDB tell which column family and SST the list belongs to without a MANIFEST. The decoder rejects a descriptor whose repair fields are not set (unknown CF ID or SST file number 0). The descriptor also has a 128-bit relocation-state fingerprint (the route_snapshot_fingerprint fields). Remote compaction uses it in diff 9 to check that an SST built elsewhere used the same routes as the job that asked for it. In this diff the table builder always writes 0, and DecodeFrom() does not check it. The field is in the format now, so the descriptor keeps one format version (1) through the stack. While the option gate is on, no data in this format exists outside tests. - The collector validates each BlobIndex once, when it is added. Its Finish() writes the data blocks and then the reference index block through one block writer, and fills the descriptor, including the index handle. Destroying the collector frees its memory and then its charge, also when the build is abandoned. - BlockBasedTableBuilder creates the collector with the CF ID and SST file number from TableBuilderOptions. It passes the size of each entry's user key to the collector, so the collector can compute the exact size of each blob record. It adds the collector's EstimatedEncodedSize() to its tail size estimate: the blocks are written before the tail, but output file size planning uses this estimate. The reference blocks are written with BlockType::kUserDefinedIndex, which keeps their contents as opaque bytes. Readers read them without the block cache, so prepopulate_block_cache does not warm them into the block cache (WriteMaybeCompressedBlock() with warm_cache false). - With the option on, the blob file builder computes the CRC32C of each stored value once. The blob log record CRC combines it with the key CRC (crc32c::Crc32cCombine). The stored CRC extends it with the compression byte. For an uncompressed value it is also the logical CRC. The bytes written do not change. - WideColumnSerialization::ForEachBlobIndex replaces ForEachBlobFileNumber. It reads the entity header and column types once. For a V1 entity (no column types and no blob columns), it returns right after the header. For a V2 entity without blob columns, it returns right after the type scan. It shares one V2 column walker (WalkV2Columns) with Deserialize(), so the entity is parsed in one pass. - Read-only opens (DBImplReadOnly, CompactedDBImpl) and secondary opens skip ValidateOptions, so they check the compaction style against the option once per column family: in DBImpl::Recover() for read-only opens, and at the start of OpenAsSecondaryImpl() for secondary opens. DB::OpenAndCompact opens through OpenAsSecondaryImpl(), so it gets the same check. - CacheReservationHandle gets Resize(), so each collector or summary can change its own part of the shared charge. - util/coding.h adds GetCanonicalVarint32/64, which reject overlong varint encodings. The new formats use them, so each value has one encoding. - BlobReferenceSstSummary is the in-memory list of an SST's groups and totals that Finish() can return. It has no caller yet: the table builder passes a null summary output. It is checked and does not change after Finish(), so the finished builder, the SST metadata and the census (its first user, in diff 5) can share it. It is in this diff because Finish() hands the group buffer and its cache charge over to the summary. Adding the summary later would mean rewriting the charge code of Finish() in diff 5, so the charge code is written and tested once, here. Safety boundaries and unsupported cases: - Public API: this diff adds the option `enable_blob_indirection` with the C API functions `rocksdb_options_set_enable_blob_indirection` and `rocksdb_options_get_enable_blob_indirection`. It also inserts a new CacheEntryRole value before kMisc, which changes the numeric value of kMisc and kNumCacheEntryRoles. - With the option off, new blob values and SSTs are written in the legacy form. Legacy blob indexes stay readable. - One behavior change applies with the option off too: UpdateBoundaries, BlobGarbageMeter and the compaction iterator fully validate wide-column entities with blob columns and their blob indexes (see "Separate change" above). - Exact reference metadata has one format (version 1) with a fixed 72-byte descriptor. The decoders reject any other version as Corruption. - The metadata blocks are protected by the table's block checksum only. With checksum = kNoChecksum, corrupt metadata is detected only when it breaks the format or order checks. - The index decoder checks each decoded count against the remaining input size before it reserves memory. So a small corrupt index cannot make it allocate a large vector. - Inlined-TTL and same-file blob indexes are not collected and cannot be relocated. Encoding, count and checksum errors return an error; a bad reference is never skipped. - Option gate (see Terms): outside tests no DB writes data in the new format. The gate checks only the option value, never persisted state. Diff 11 removes it, once data written with the option stays readable through the end of the stack and the safety checks for turning the option on are in place. - Downgrading the binary after data was written with the option on is not supported. Validation and known limitations: - Unit tests cover the BlobIndex encodings and their rejection of bad input, the metadata layout and every corruption check, the collector (sorted path, unsorted fallback, memory charge), the flush integration, the option checks and option gate, wide-column validation, and canonical varints. See the Test Plan. - The collection buffer is not spilled to disk. One SST with very many blob references needs a matching amount of charged memory: the per-reference amount given under "Collection and memory", plus up to one partly used 512-byte chunk per group. While a fallback group is sorted, it also needs 16 bytes of sort key per reference and a second copy of its entries. For a test group of 200K references with every other one out of order, the test comment estimates about 44 bytes per reference, and the test checks that the peak charge is at most 56 bytes per reference. - Flush still writes a single output file. Until diff 7, one compaction output collects every reference it gets, charged to the block cache. Diff 7 cuts compaction outputs (including L0) at a reference cap and keeps each user key's history in one output. With the option gate, a large output is reachable only through the test hook. Stack context (same in diffs 2-11, except that diffs 8-11 add the diff 8 route rule to the install line; "<== this diff" marks this diff's part): Goal: remove garbage from blob files without rewriting any SST. A value gets a stable identity when it is first written. Indirect blob GC copies the live values of whole files into a relocation file and installs routes in the MANIFEST. Readers follow the routes. SSTs are not changed. How a read finds a value (P is the relocation file that holds the value now): ``` SST entry indirect BlobIndex { generation number, origin, original offset, size, CRCs } | v Version file table: generation number -> live file, or route old file -> P | | live file route found | | v v blob file: read at original offset relocation file P (always one hop): relocation file: look up key look up key (origin, original offset) (origin, original offset) ``` Lifecycle of one value (in this example, F is the blob file where the value is first written, and P is the relocation file that GC writes): ``` write new value goes to blob file F; index: origin = F, generation number = F | census per generation number: the SSTs and references that use it; | per physical file: live bytes, ranked by garbage ratio GC job copy the live values of whole input files (stored bytes, checked | with the stored CRC) into relocation file P, keyed by | (origin, original offset) install one MANIFEST edit: add P, route each input generation number still | in use to P, re-point the routes into the inputs to P, and remove | each input (blob file retirement for a blob file, relocation | file deletion for a relocation file) flatten compaction sets generation number = P in the SSTs it rewrites | route route retirement: route deletion records remove the routes that delete no SST uses ``` Stack map: | Diff | Adds | |---|---| | 1 | Finalize blob metadata outside the DB mutex (prep) | | 2 | Indirect BlobIndex (kIndirectBlob) and exact SST reference metadata <== this diff | | 3 | Relocation file format and stable read path | | 4 | MANIFEST routes and Version ownership | | 5 | Exact reverse reference census | | 6 | Indirect blob GC (one input file) | | 7 | Pack whole input files; reclaim relocation files | | 8 | Flatten generation numbers; route retirement | | 9 | Remote compaction validation | | 10 | Dynamic options and operations | | 11 | RepairDB rebuilds relocation state; option gate removed | | 12 | db_stress and crash test support | Stack context: terms for later diffs: | Term | Meaning | |---|---| | census | Background count, per generation number, of the SSTs and references that use it, plus live bytes per physical file (diff 5) | | indirect origin | Normal blob file written with the option on; it is the origin of unflattened references | | flatten | Compaction rewrites the generation number to the relocation file that holds the value, so old routes can go. Only flatten makes generation number != origin (diff 8) | | blob file retirement | MANIFEST record (kBlobFileRetirement) that removes a collected blob file from the Version (diff 4) | | route deletion | MANIFEST record that removes a route that no SST uses. Route retirement is the background job that writes these records (diff 8) | Tracking issue: meta#15301 Differential Revision: D124652097
Summary: Pull Request resolved: meta#15540 Why the change is required: The goal of this stack is indirect blob GC, a blob GC that does not depend on SSTs: it removes garbage from blob files without rewriting any SST. Such a GC needs two things that RocksDB does not have: - A value identity that does not change when the value moves to another file. - A checked list of the blob references that each SST keeps. The list must include older internal versions of a key that the SST still keeps (for example for snapshots), and the blob columns of wide-column entities. This diff adds both. Later diffs build the census (an in-memory reverse index from generation number to the SSTs that use it), GC and repair on top of this list. This diff in one paragraph: With the new column family option enable_blob_indirection on, the blob file writer gives each new value an indirect BlobIndex (type 3, kIndirectBlob). It stores the value's stable identity (origin file, original offset) and its CRCs (an uncompressed value stores only the CRC of the uncompressed value; a compressed value also stores the CRC of its stored bytes). Also with the option on, the block-based table builder collects every blob reference that the SST keeps. It writes them as "exact reference metadata": data blocks, one reference index block, and a 72-byte descriptor table property. In this diff only the table builder writes this metadata, and only tests read it. A temporary option gate (see "option gate" under Terms) keeps the option off outside tests. Separately, wide-column entities with blob columns are now fully checked before their blob indexes are used, also with the option off. Parts of this diff: 1. Indirect BlobIndex (core): types 3 and 4 in db/blob/blob_index.h, and the blob file writer that writes type 3. 2. Exact reference metadata format and decoders (core): db/blob/blob_reference_metadata.{h,cc}. 3. Collector and table builder integration (core): BlobReferenceMetadataCollector, its block cache charge, and BlockBasedTableBuilder. 4. Option enable_blob_indirection and the option gate. Tests need it to turn the feature on. You can skip this part on a first read. 5. Wide-column validation (WideColumnSerialization::ForEachBlobIndex). The table builder uses it to give the blob columns of each entity to the collector. It also changes behavior with the option off (see New behavior). You can skip this part on a first read. 6. Small helpers: canonical varints (util/coding.h) and CacheReservationHandle::Resize(). You can skip these on a first read. How to read this diff: 1. Overview: read "Why" and "This diff in one paragraph" above, and "Terms used in this diff" right below. Then read the top comment of db/blob/blob_reference_metadata.h (terms, layout diagrams, run and entry layout, index layout). 2. Indirect BlobIndex (db/blob/blob_index.h) and canonical varints (util/coding.h GetCanonicalVarint32/64): read the overview comment and the kIndirectBlob / kIndirectFlattenedBlob code. Every later part uses these fields: origin, original offset, generation number, logical CRC and stored CRC. The new formats read varints only in the canonical form. 3. Blob file writer (db/blob/blob_file_builder.cc, blob_log_format.cc, blob_log_writer.cc): see how a new value gets a type-3 index and how its CRC is computed once. 4. Format types and decoders: the structs in db/blob/blob_reference_metadata.h, then in the .cc the constants, GapBase, DecodeEntryBody, DecodeRunEntriesAt, DecodeBlobReferenceIndex and BlobReferenceGroupDecoder. They use only BlobIndex and the canonical varints. Learn the on-disk format here, before you read the writer. 5. Collector and its cache charge: first read the new CacheEntryRole kBlobReferenceMetadataBuildingBuffer (include/rocksdb/cache.h) and CacheReservationHandle::Resize (cache/cache_reservation_manager.h). Then read BlobReferenceMetadataCollector in this order: the class comment, Add, AppendSorted, BuildAndWriteDataBlocks, EncodeIndex, FinishOnce. Then read the unsorted fallback (AppendUnsorted, SortUnsortedGroup, SortAndEncodeGroup). 6. Wide-column change (db/wide/wide_column_serialization.cc, ForEachBlobIndex and WalkV2Columns): it gives every blob index of an entity to a callback after it checks the whole entity. The table builder uses it to feed the collector. UpdateBoundaries, BlobGarbageMeter and the compaction iterator also use it, with the option on or off. 7. Option: enable_blob_indirection (include/rocksdb/advanced_options.h, options/cf_options.cc), and the option checks and gate in db/column_family.cc (ValidateOptions, TEST_SetAllowBlobIndirection), db/db_impl/db_impl_open.cc and db_impl_secondary.cc (the FIFO check for opens that skip ValidateOptions). 8. Table integration: table/block_based/block_based_table_factory.cc (the shared reservation manager), then block_based_table_builder.cc (Rep constructor, Add, WriteBlobReferenceMetadata, EstimatedTailSize). See where the collector is created, fed and finished. block_based_table_reader.cc maps the new block name to a block type. Terms used in this diff: Stack terms: | Term | Meaning | |---|---| | origin | Blob file where the value was first written | | original offset | Value offset in the origin file | | stable identity | (origin, original offset). Never changes, even when the value moves | | generation number | The file number that a reference uses to find its value. A new value has generation number == origin. When indirect blob GC moves the value, the reference keeps its generation number, and a route sends the read to the relocation file | | indirect BlobIndex | A kIndirectBlob (type 3) or kIndirectFlattenedBlob (type 4) index. It stores the stable identity, so the value can move without an SST rewrite | | unflattened reference | Index with generation number == origin: the value was not moved, or it was moved but no compaction has rewritten its reference yet (see "flatten" in "Stack context: terms for later diffs") | | flattened reference | Index with generation number != origin (type 4). Only flatten (diff 8) writes it; this diff defines the encoding | | route | Maps an old file number (a file that GC has removed from the Version) to the relocation file that now holds its live values. One hop: a target is never itself routed. Stored in the MANIFEST (diff 4) | | relocation file | Block-based table keyed by (origin, original offset) that holds copied values. Each value is the stored bytes plus a 5-byte trailer (kSimpleGen2BlobTrailerSize) | | logical CRC | CRC32C of the uncompressed value (BlobIndex::stable_checksum()) | | stored CRC | CRC32C of the stored bytes plus the compression byte | | physical file | A file that holds value bytes: a normal blob file or a relocation file. A generation number can name a physical file, or it can be only the key of a route | | exact reference metadata | Per-SST sorted list of every blob reference the SST keeps, written when the SST is built. References with the same encoding class, generation number and origin form a group. The list is stored as data blocks, with an index of the groups. An SST that has it is an exact SST | | option gate | Until diff 11, ColumnFamilyData::ValidateOptions returns NotSupported for enable_blob_indirection on a read-write open, CreateColumnFamily and SetOptions, unless a test calls ColumnFamilyData::TEST_SetAllowBlobIndirection(true). It keeps the option off in production while the stack is not complete | Terms of this diff (all defined at the top of db/blob/blob_reference_metadata.h): | Term | Meaning | |---|---| | encoding class (class) | kDirect (a legacy kBlob or kBlobTTL index), unflattened, or flattened | | reference group (group) | All references of one SST with the same (encoding class, generation number, origin) | | run | The part of one group inside one data block | | record bytes | The bytes the value's record takes in the file named by its generation number | | gap | How far a record is from where it would be if the records were back to back. A gap of 0 means back to back | | descriptor | The 72-byte table property that points to the reference index block and holds the totals, the CF ID, the SST file number and the relocation-state fingerprint | | relocation-state fingerprint | 128-bit hash of the routes and relocation files of the Version that an SST was built from (fields route_snapshot_fingerprint_*). Written as 0 in this diff | | legacy | Written the way RocksDB writes it without blob indirection | Previous behavior: - A BlobIndex named only a physical blob file and an offset. A value could not move to another file without rewriting every SST that referenced it. - An SST had no metadata that listed its blob references. The only way to find them was to scan its user keys. - These readers get the blob indexes of a wide-column entity through ForEachBlobFileNumber: UpdateBoundaries, BlobGarbageMeter, the compaction iterator, WideColumnSerialization::RemoveBlobColumns, and the referenced_blob_bytes table property count in BlockBasedTableBuilder::Add. ForEachBlobFileNumber decoded each blob index, but it did not check the rest of the entity framing (name order, section sizes, exact input size), and it could call the callback before it found a later error. New behavior: Separate change (option on or off): - The same readers (UpdateBoundaries, BlobGarbageMeter, the compaction iterator, RemoveBlobColumns, and the referenced_blob_bytes count in BlockBasedTableBuilder::Add) use WideColumnSerialization::ForEachBlobIndex, with the option on or off. With the option on, the table builder also uses it to give the blob columns of each entity to the collector. For an entity with blob columns, it checks the full framing, then decodes and validates every blob index before any callback runs. So an entity with a malformed framing now fails the flush or compaction, even with the option off. The referenced_blob_bytes count itself does not fail the build on a malformed entity. It removes that property from the SST, as in base. Option and option gate: - This diff adds a new column family option, `enable_blob_indirection`. It is off by default, and SetOptions can change it. - The option gate (see Terms) keeps the option off outside tests until the stack is complete. ColumnFamilyData::ValidateOptions rejects the option with NotSupported on a read-write open, CreateColumnFamily and SetOptions. Read-only opens (DBImplReadOnly and CompactedDBImpl) and secondary opens (including DB::OpenAndCompact) do not run ValidateOptions; they check only the FIFO rule (see Implementation). Only tests can turn the option on, through the internal ColumnFamilyData::TEST_SetAllowBlobIndirection(). DBTestBase sets it. - When the option is on, ValidateOptions also requires enable_blob_files, the block-based table format, no blob direct write, and a compaction style other than FIFO. Blob file writer (option on): - The blob file writer encodes each new value with a stable identity (origin file, original offset) and two checksums: - the logical CRC: the CRC32C of the uncompressed value. - the stored CRC: the CRC32C of the exact stored (maybe compressed) bytes, extended with the compression byte. - For an uncompressed value the stored CRC is not written, because the reader can compute it from the logical CRC. - The checksums have no version field of their own. The BlobIndex type byte gives the format version: the two new types, 3 (kIndirectBlob) and 4 (kIndirectFlattenedBlob, see Implementation), are BlobIndex::kIndirectBlobFormatVersion (1). A later format uses new type values. - The read path does not check these checksums yet. The check starts in diff 3. Exact reference metadata (option on): 1. What: a sorted list of every blob reference that the SST keeps, in plain blob values and in the blob columns of wide-column entities. 2. Where: data blocks (target 16 KB each before compression; small blocks waste less I/O, because indirect blob GC reads a whole block to get a few groups) and one reference index block, written after the SST data blocks and before the SST tail, plus a 72-byte descriptor table property. 3. Who reads it: only tests in this diff. The census, GC and RepairDB read it in later diffs. 4. Details: the top comment of db/blob/blob_reference_metadata.h has the full format with diagrams, the record-bytes rules and the gap rule. - Not collected: SstFileWriter output (it has no DB file number) and any output whose CF is unknown (it has no CF ID). The descriptor needs both values, so these outputs stay in the legacy form. Checks on the metadata: - A repeated or overlapping reference in one group fails the SST build with Corruption. Readers check every count, order, size and total against the index and the descriptor, so corrupt metadata returns Corruption. The writer never makes a data block larger than kTargetDataBlockSize (16 KB), and readers accept up to kMaxDataBlockSize (64 KB). Details: "Readers" and "Integrity" at the top of db/blob/blob_reference_metadata.h, and BlobReferenceGroupDecoder. Collection and memory: - Each reference is encoded when it is added, in the on-disk entry encoding, into its group's chain of 512-byte chunks. Only a group that takes the unsorted fallback is encoded a second time, in Finish(). Inside a group, references normally arrive in rising offset order, so no reference is sorted; at the end only the groups are sorted. A group that gets a lower offset takes the unsorted fallback. The BlobReferenceMetadataCollector class comment explains the chunks, the chunk size and the fallback. - The collector keeps every reference of the SST in memory until the SST is finished, and charges this memory to the block cache under a new CacheEntryRole, kBlobReferenceMetadataBuildingBuffer. The measured cost is about 8.3 to 9 bytes per reference for uncompressed values and 12.8 to 13.5 bytes for compressed values. "Memory usage" and "Memory charge" in the BlobReferenceMetadataCollector class comment give the measurement setup and the charge rule. - There is no fixed limit on references. The collector fails with MemoryLimit above 2^20 groups or 2^20 data blocks in one SST; these are persisted format limits. - There is no charge when the table has no block cache or when charging for this role is disabled. Each block-based table factory holds one cache reservation manager for this role, shared by its builders and the SST summaries. If the charge fails, the SST build fails, so the flush or compaction fails with a background error. The buffers and their charge are released when the SST is finished or abandoned. Implementation: - BlobIndex adds two indirect encodings. The legacy encodings do not change. - Type 3, kIndirectBlob (unflattened, generation number == origin), stores the origin, the original offset, the stored size, the compression type, the logical CRC and, for compressed values, the stored CRC. It does not store the generation number. This saves the size of one varint file number per reference (about 3 to 4 bytes for typical file numbers; an estimate, not measured). - Type 4, kIndirectFlattenedBlob (flattened), stores the generation number first, then the same fields. Its generation number must be above the origin. The decoder rejects type 4 with generation number <= origin, so each reference has one encoding. - The encoders take the fields as one BlobIndex::IndirectBlobFields, set by field name, so values of the same type (file number, offset and size; the two CRCs) cannot be swapped. EncodeIndirectBlob() takes the generation number as a separate argument; EncodeUnflattenedIndirectBlob() uses the origin, so the common path cannot leave the generation number unset. - The new files blob_reference_metadata.{h,cc} define the exact reference metadata. The files also contain: - BlobReferenceMetadataCollector, the collector described above. It takes the SST identity (CF ID, SST file number and the relocation-state fingerprint) as one BlobReferenceSstIdentity, set by field name. - BlobReferenceGroupKey, the key of a group: (encoding class, generation number, origin). It has the order, the hash and the validity check that the collector map, the sort, the decoder and FindBlobReferenceGroup() use. BlobReference and BlobReferenceGroup keep their flat fields and return the key through group_key(). - The index codec and the descriptor table property. - BlobReferenceGroupDecoder, the only decoder of reference runs. It reads a group one data block at a time and holds every corruption check of a group walk. Readers that visit one group (found by a binary search of the index) use it, and so do readers that visit every block in order. - Callers in this diff: only the table builder runs the collector. The decoders (DecodeBlobReferenceIndex, FindBlobReferenceGroup, BlobReferenceGroupDecoder, ForEachBlobReference*) are used only by tests. kMaxDecodedBlobReferenceBytes is used only in compile-time checks of the count limits. The census (diff 5) and indirect blob GC (diffs 6 and 7) are their first production users. The census is also the first user of BlobReferenceSstSummary (see the last Implementation bullet). - Descriptor: the CF ID and the SST file number are the repair fields. They let RepairDB tell which column family and SST the list belongs to without a MANIFEST. The decoder rejects a descriptor whose repair fields are not set (unknown CF ID or SST file number 0). The descriptor also has a 128-bit relocation-state fingerprint (the route_snapshot_fingerprint fields). Remote compaction uses it in diff 9 to check that an SST built elsewhere used the same routes as the job that asked for it. In this diff the table builder always writes 0, and DecodeFrom() does not check it. The field is in the format now, so the descriptor keeps one format version (1) through the stack. While the option gate is on, no data in this format exists outside tests. - The collector validates each BlobIndex once, when it is added. Its Finish() writes the data blocks and then the reference index block through one block writer, and fills the descriptor, including the index handle. Destroying the collector frees its memory and then its charge, also when the build is abandoned. - BlockBasedTableBuilder creates the collector with the CF ID and SST file number from TableBuilderOptions. It passes the size of each entry's user key to the collector, so the collector can compute the exact size of each blob record. It adds the collector's EstimatedEncodedSize() to its tail size estimate: the blocks are written before the tail, but output file size planning uses this estimate. The reference blocks are written with BlockType::kUserDefinedIndex, which keeps their contents as opaque bytes. Readers read them without the block cache, so prepopulate_block_cache does not warm them into the block cache (WriteMaybeCompressedBlock() with warm_cache false). - With the option on, the blob file builder computes the CRC32C of each stored value once. The blob log record CRC combines it with the key CRC (crc32c::Crc32cCombine). The stored CRC extends it with the compression byte. For an uncompressed value it is also the logical CRC. The bytes written do not change. - WideColumnSerialization::ForEachBlobIndex replaces ForEachBlobFileNumber. It reads the entity header and column types once. For a V1 entity (no column types and no blob columns), it returns right after the header. For a V2 entity without blob columns, it returns right after the type scan. It shares one V2 column walker (WalkV2Columns) with Deserialize(), so the entity is parsed in one pass. - Read-only opens (DBImplReadOnly, CompactedDBImpl) and secondary opens skip ValidateOptions, so they check the compaction style against the option once per column family: in DBImpl::Recover() for read-only opens, and at the start of OpenAsSecondaryImpl() for secondary opens. DB::OpenAndCompact opens through OpenAsSecondaryImpl(), so it gets the same check. - CacheReservationHandle gets Resize(), so each collector or summary can change its own part of the shared charge. - util/coding.h adds GetCanonicalVarint32/64, which reject overlong varint encodings. The new formats use them, so each value has one encoding. - BlobReferenceSstSummary is the in-memory list of an SST's groups and totals that Finish() can return. It has no caller yet: the table builder passes a null summary output. It is checked and does not change after Finish(), so the finished builder, the SST metadata and the census (its first user, in diff 5) can share it. It is in this diff because Finish() hands the group buffer and its cache charge over to the summary. Adding the summary later would mean rewriting the charge code of Finish() in diff 5, so the charge code is written and tested once, here. Safety boundaries and unsupported cases: - Public API: this diff adds the option `enable_blob_indirection` with the C API functions `rocksdb_options_set_enable_blob_indirection` and `rocksdb_options_get_enable_blob_indirection`. It also inserts a new CacheEntryRole value before kMisc, which changes the numeric value of kMisc and kNumCacheEntryRoles. - With the option off, new blob values and SSTs are written in the legacy form. Legacy blob indexes stay readable. - One behavior change applies with the option off too: UpdateBoundaries, BlobGarbageMeter and the compaction iterator fully validate wide-column entities with blob columns and their blob indexes (see "Separate change" above). - Exact reference metadata has one format (version 1) with a fixed 72-byte descriptor. The decoders reject any other version as Corruption. - The metadata blocks are protected by the table's block checksum only. With checksum = kNoChecksum, corrupt metadata is detected only when it breaks the format or order checks. - The index decoder checks each decoded count against the remaining input size before it reserves memory. So a small corrupt index cannot make it allocate a large vector. - Inlined-TTL and same-file blob indexes are not collected and cannot be relocated. Encoding, count and checksum errors return an error; a bad reference is never skipped. - Option gate (see Terms): outside tests no DB writes data in the new format. The gate checks only the option value, never persisted state. Diff 11 removes it, once data written with the option stays readable through the end of the stack and the safety checks for turning the option on are in place. - Downgrading the binary after data was written with the option on is not supported. Validation and known limitations: - Unit tests cover the BlobIndex encodings and their rejection of bad input, the metadata layout and every corruption check, the collector (sorted path, unsorted fallback, memory charge), the flush integration, the option checks and option gate, wide-column validation, and canonical varints. See the Test Plan. - The collection buffer is not spilled to disk. One SST with very many blob references needs a matching amount of charged memory: the per-reference amount given under "Collection and memory", plus up to one partly used 512-byte chunk per group. While a fallback group is sorted, it also needs 16 bytes of sort key per reference and a second copy of its entries. For a test group of 200K references with every other one out of order, the test comment estimates about 44 bytes per reference, and the test checks that the peak charge is at most 56 bytes per reference. - Flush still writes a single output file. Until diff 7, one compaction output collects every reference it gets, charged to the block cache. Diff 7 cuts compaction outputs (including L0) at a reference cap and keeps each user key's history in one output. With the option gate, a large output is reachable only through the test hook. Stack context (same in diffs 2-11, except that diffs 8-11 add the diff 8 route rule to the install line; "<== this diff" marks this diff's part): Goal: remove garbage from blob files without rewriting any SST. A value gets a stable identity when it is first written. Indirect blob GC copies the live values of whole files into a relocation file and installs routes in the MANIFEST. Readers follow the routes. SSTs are not changed. How a read finds a value (P is the relocation file that holds the value now): ``` SST entry indirect BlobIndex { generation number, origin, original offset, size, CRCs } | v Version file table: generation number -> live file, or route old file -> P | | live file route found | | v v blob file: read at original offset relocation file P (always one hop): relocation file: look up key look up key (origin, original offset) (origin, original offset) ``` Lifecycle of one value (in this example, F is the blob file where the value is first written, and P is the relocation file that GC writes): ``` write new value goes to blob file F; index: origin = F, generation number = F | census per generation number: the SSTs and references that use it; | per physical file: live bytes, ranked by garbage ratio GC job copy the live values of whole input files (stored bytes, checked | with the stored CRC) into relocation file P, keyed by | (origin, original offset) install one MANIFEST edit: add P, route each input generation number still | in use to P, re-point the routes into the inputs to P, and remove | each input (blob file retirement for a blob file, relocation | file deletion for a relocation file) flatten compaction sets generation number = P in the SSTs it rewrites | route route retirement: route deletion records remove the routes that delete no SST uses ``` Stack map: | Diff | Adds | |---|---| | 1 | Finalize blob metadata outside the DB mutex (prep) | | 2 | Indirect BlobIndex (kIndirectBlob) and exact SST reference metadata <== this diff | | 3 | Relocation file format and stable read path | | 4 | MANIFEST routes and Version ownership | | 5 | Exact reverse reference census | | 6 | Indirect blob GC (one input file) | | 7 | Pack whole input files; reclaim relocation files | | 8 | Flatten generation numbers; route retirement | | 9 | Remote compaction validation | | 10 | Dynamic options and operations | | 11 | RepairDB rebuilds relocation state; option gate removed | | 12 | db_stress and crash test support | Stack context: terms for later diffs: | Term | Meaning | |---|---| | census | Background count, per generation number, of the SSTs and references that use it, plus live bytes per physical file (diff 5) | | indirect origin | Normal blob file written with the option on; it is the origin of unflattened references | | flatten | Compaction rewrites the generation number to the relocation file that holds the value, so old routes can go. Only flatten makes generation number != origin (diff 8) | | blob file retirement | MANIFEST record (kBlobFileRetirement) that removes a collected blob file from the Version (diff 4) | | route deletion | MANIFEST record that removes a route that no SST uses. Route retirement is the background job that writes these records (diff 8) | Tracking issue: meta#15301 Differential Revision: D124652097
Summary: Pull Request resolved: meta#15540 Why the change is required: The goal of this stack is indirect blob GC, a blob GC that does not depend on SSTs: it removes garbage from blob files without rewriting any SST. Such a GC needs two things that RocksDB does not have: - A value identity that does not change when the value moves to another file. - A checked list of the blob references that each SST keeps. The list must include older internal versions of a key that the SST still keeps (for example for snapshots), and the blob columns of wide-column entities. This diff adds both. Later diffs build the census (an in-memory reverse index from generation number to the SSTs that use it), GC and repair on top of this list. This diff in one paragraph: With the new column family option enable_blob_indirection on, the blob file writer gives each new value an indirect BlobIndex (type 3, kIndirectBlob). It stores the value's stable identity (origin file, original offset) and its CRCs (an uncompressed value stores only the CRC of the uncompressed value; a compressed value also stores the CRC of its stored bytes). Also with the option on, the block-based table builder collects every blob reference that the SST keeps. It writes them as "exact reference metadata": data blocks, one reference index block, and a 72-byte descriptor table property. In this diff only the table builder writes this metadata, and only tests read it. A temporary option gate (see "option gate" under Terms) keeps the option off outside tests. Separately, wide-column entities with blob columns are now fully checked before their blob indexes are used, also with the option off. Parts of this diff: 1. Indirect BlobIndex (core): types 3 and 4 in db/blob/blob_index.h, and the blob file writer that writes type 3. 2. Exact reference metadata format and decoders (core): db/blob/blob_reference_metadata.{h,cc}. 3. Collector and table builder integration (core): BlobReferenceMetadataCollector, its block cache charge, and BlockBasedTableBuilder. 4. Option enable_blob_indirection and the option gate. Tests need it to turn the feature on. You can skip this part on a first read. 5. Wide-column validation (WideColumnSerialization::ForEachBlobIndex). The table builder uses it to give the blob columns of each entity to the collector. It also changes behavior with the option off (see New behavior). You can skip this part on a first read. 6. Small helpers: canonical varints (util/coding.h) and CacheReservationHandle::Resize(). You can skip these on a first read. How to read this diff: 1. Overview: read "Why" and "This diff in one paragraph" above, and "Terms used in this diff" right below. Then read the top comment of db/blob/blob_reference_metadata.h (terms, layout diagrams, run and entry layout, index layout). 2. Indirect BlobIndex (db/blob/blob_index.h) and canonical varints (util/coding.h GetCanonicalVarint32/64): read the overview comment and the kIndirectBlob / kIndirectFlattenedBlob code. Every later part uses these fields: origin, original offset, generation number, logical CRC and stored CRC. The new formats read varints only in the canonical form. 3. Blob file writer (db/blob/blob_file_builder.cc, blob_log_format.cc, blob_log_writer.cc): see how a new value gets a type-3 index and how its CRC is computed once. 4. Format types and decoders: the structs in db/blob/blob_reference_metadata.h, then in the .cc the constants, GapBase, DecodeEntryBody, DecodeRunEntriesAt, DecodeBlobReferenceIndex and BlobReferenceGroupDecoder. They use only BlobIndex and the canonical varints. Learn the on-disk format here, before you read the writer. 5. Collector and its cache charge: first read the new CacheEntryRole kBlobReferenceMetadataBuildingBuffer (include/rocksdb/cache.h) and CacheReservationHandle::Resize (cache/cache_reservation_manager.h). Then read BlobReferenceMetadataCollector in this order: the class comment, Add, AppendSorted, BuildAndWriteDataBlocks, EncodeIndex, FinishOnce. Then read the unsorted fallback (AppendUnsorted, SortUnsortedGroup, SortAndEncodeGroup). 6. Wide-column change (db/wide/wide_column_serialization.cc, ForEachBlobIndex and WalkV2Columns): it gives every blob index of an entity to a callback after it checks the whole entity. The table builder uses it to feed the collector. UpdateBoundaries, BlobGarbageMeter and the compaction iterator also use it, with the option on or off. 7. Option: enable_blob_indirection (include/rocksdb/advanced_options.h, options/cf_options.cc), and the option checks and gate in db/column_family.cc (ValidateOptions, TEST_SetAllowBlobIndirection), db/db_impl/db_impl_open.cc and db_impl_secondary.cc (the FIFO check for opens that skip ValidateOptions). 8. Table integration: table/block_based/block_based_table_factory.cc (the shared reservation manager), then block_based_table_builder.cc (Rep constructor, Add, WriteBlobReferenceMetadata, EstimatedTailSize). See where the collector is created, fed and finished. block_based_table_reader.cc maps the new block name to a block type. Terms used in this diff: Stack terms: | Term | Meaning | |---|---| | origin | Blob file where the value was first written | | original offset | Value offset in the origin file | | stable identity | (origin, original offset). Never changes, even when the value moves | | generation number | The file number that a reference uses to find its value. A new value has generation number == origin. When indirect blob GC moves the value, the reference keeps its generation number, and a route sends the read to the relocation file | | indirect BlobIndex | A kIndirectBlob (type 3) or kIndirectFlattenedBlob (type 4) index. It stores the stable identity, so the value can move without an SST rewrite | | unflattened reference | Index with generation number == origin: the value was not moved, or it was moved but no compaction has rewritten its reference yet (see "flatten" in "Stack context: terms for later diffs") | | flattened reference | Index with generation number != origin (type 4). Only flatten (diff 8) writes it; this diff defines the encoding | | route | Maps an old file number (a file that GC has removed from the Version) to the relocation file that now holds its live values. One hop: a target is never itself routed. Stored in the MANIFEST (diff 4) | | relocation file | Block-based table keyed by (origin, original offset) that holds copied values. Each value is the stored bytes plus a 5-byte trailer (kSimpleGen2BlobTrailerSize) | | logical CRC | CRC32C of the uncompressed value (BlobIndex::stable_checksum()) | | stored CRC | CRC32C of the stored bytes plus the compression byte | | physical file | A file that holds value bytes: a normal blob file or a relocation file. A generation number can name a physical file, or it can be only the key of a route | | exact reference metadata | Per-SST sorted list of every blob reference the SST keeps, written when the SST is built. References with the same encoding class, generation number and origin form a group. The list is stored as data blocks, with an index of the groups. An SST that has it is an exact SST | | option gate | Until diff 11, ColumnFamilyData::ValidateOptions returns NotSupported for enable_blob_indirection on a read-write open, CreateColumnFamily and SetOptions, unless a test calls ColumnFamilyData::TEST_SetAllowBlobIndirection(true). It keeps the option off in production while the stack is not complete | Terms of this diff (all defined at the top of db/blob/blob_reference_metadata.h): | Term | Meaning | |---|---| | encoding class (class) | kDirect (a legacy kBlob or kBlobTTL index), unflattened, or flattened | | reference group (group) | All references of one SST with the same (encoding class, generation number, origin) | | run | The part of one group inside one data block | | record bytes | The bytes the value's record takes in the file named by its generation number | | gap | How far a record is from where it would be if the records were back to back. A gap of 0 means back to back | | descriptor | The 72-byte table property that points to the reference index block and holds the totals, the CF ID, the SST file number and the relocation-state fingerprint | | relocation-state fingerprint | 128-bit hash of the routes and relocation files of the Version that an SST was built from (fields route_snapshot_fingerprint_*). Written as 0 in this diff | | legacy | Written the way RocksDB writes it without blob indirection | Previous behavior: - A BlobIndex named only a physical blob file and an offset. A value could not move to another file without rewriting every SST that referenced it. - An SST had no metadata that listed its blob references. The only way to find them was to scan its user keys. - These readers get the blob indexes of a wide-column entity through ForEachBlobFileNumber: UpdateBoundaries, BlobGarbageMeter, the compaction iterator, WideColumnSerialization::RemoveBlobColumns, and the referenced_blob_bytes table property count in BlockBasedTableBuilder::Add. ForEachBlobFileNumber decoded each blob index, but it did not check the rest of the entity framing (name order, section sizes, exact input size), and it could call the callback before it found a later error. New behavior: Separate change (option on or off): - The same readers (UpdateBoundaries, BlobGarbageMeter, the compaction iterator, RemoveBlobColumns, and the referenced_blob_bytes count in BlockBasedTableBuilder::Add) use WideColumnSerialization::ForEachBlobIndex, with the option on or off. With the option on, the table builder also uses it to give the blob columns of each entity to the collector. For an entity with blob columns, it checks the full framing, then decodes and validates every blob index before any callback runs. So an entity with a malformed framing now fails the flush or compaction, even with the option off. The referenced_blob_bytes count itself does not fail the build on a malformed entity. It removes that property from the SST, as in base. Option and option gate: - This diff adds a new column family option, `enable_blob_indirection`. It is off by default, and SetOptions can change it. - The option gate (see Terms) keeps the option off outside tests until the stack is complete. ColumnFamilyData::ValidateOptions rejects the option with NotSupported on a read-write open, CreateColumnFamily and SetOptions. Read-only opens (DBImplReadOnly and CompactedDBImpl) and secondary opens (including DB::OpenAndCompact) do not run ValidateOptions; they check only the FIFO rule (see Implementation). Only tests can turn the option on, through the internal ColumnFamilyData::TEST_SetAllowBlobIndirection(). DBTestBase sets it. - When the option is on, ValidateOptions also requires enable_blob_files, the block-based table format, no blob direct write, and a compaction style other than FIFO. Blob file writer (option on): - The blob file writer encodes each new value with a stable identity (origin file, original offset) and two checksums: - the logical CRC: the CRC32C of the uncompressed value. - the stored CRC: the CRC32C of the exact stored (maybe compressed) bytes, extended with the compression byte. - For an uncompressed value the stored CRC is not written, because the reader can compute it from the logical CRC. - The checksums have no version field of their own. The BlobIndex type byte gives the format version: the two new types, 3 (kIndirectBlob) and 4 (kIndirectFlattenedBlob, see Implementation), are BlobIndex::kIndirectBlobFormatVersion (1). A later format uses new type values. - The read path does not check these checksums yet. The check starts in diff 3. Exact reference metadata (option on): 1. What: a sorted list of every blob reference that the SST keeps, in plain blob values and in the blob columns of wide-column entities. 2. Where: data blocks (target 16 KB each before compression; small blocks waste less I/O, because indirect blob GC reads a whole block to get a few groups) and one reference index block, written after the SST data blocks and before the SST tail, plus a 72-byte descriptor table property. 3. Who reads it: only tests in this diff. The census, GC and RepairDB read it in later diffs. 4. Details: the top comment of db/blob/blob_reference_metadata.h has the full format with diagrams, the record-bytes rules and the gap rule. - Not collected: SstFileWriter output (it has no DB file number) and any output whose CF is unknown (it has no CF ID). The descriptor needs both values, so these outputs stay in the legacy form. Checks on the metadata: - A repeated or overlapping reference in one group fails the SST build with Corruption. Readers check every count, order, size and total against the index and the descriptor, so corrupt metadata returns Corruption. The writer never makes a data block larger than kTargetDataBlockSize (16 KB), and readers accept up to kMaxDataBlockSize (64 KB). Details: "Readers" and "Integrity" at the top of db/blob/blob_reference_metadata.h, and BlobReferenceGroupDecoder. Collection and memory: - Each reference is encoded when it is added, in the on-disk entry encoding, into its group's chain of 512-byte chunks. Only a group that takes the unsorted fallback is encoded a second time, in Finish(). Inside a group, references normally arrive in rising offset order, so no reference is sorted; at the end only the groups are sorted. A group that gets a lower offset takes the unsorted fallback. The BlobReferenceMetadataCollector class comment explains the chunks, the chunk size and the fallback. - The collector keeps every reference of the SST in memory until the SST is finished, and charges this memory to the block cache under a new CacheEntryRole, kBlobReferenceMetadataBuildingBuffer. The measured cost is about 8.3 to 9 bytes per reference for uncompressed values and 12.8 to 13.5 bytes for compressed values. "Memory usage" and "Memory charge" in the BlobReferenceMetadataCollector class comment give the measurement setup and the charge rule. - There is no fixed limit on references. The collector fails with MemoryLimit above 2^20 groups or 2^20 data blocks in one SST; these are persisted format limits. - There is no charge when the table has no block cache or when charging for this role is disabled. Each block-based table factory holds one cache reservation manager for this role, shared by its builders and the SST summaries. If the charge fails, the SST build fails, so the flush or compaction fails with a background error. The buffers and their charge are released when the SST is finished or abandoned. Implementation: - BlobIndex adds two indirect encodings. The legacy encodings do not change. - Type 3, kIndirectBlob (unflattened, generation number == origin), stores the origin, the original offset, the stored size, the compression type, the logical CRC and, for compressed values, the stored CRC. It does not store the generation number. This saves the size of one varint file number per reference (about 3 to 4 bytes for typical file numbers; an estimate, not measured). - Type 4, kIndirectFlattenedBlob (flattened), stores the generation number first, then the same fields. Its generation number must be above the origin. The decoder rejects type 4 with generation number <= origin, so each reference has one encoding. - The encoders take the fields as one BlobIndex::IndirectBlobFields, set by field name, so values of the same type (file number, offset and size; the two CRCs) cannot be swapped. EncodeIndirectBlob() takes the generation number as a separate argument; EncodeUnflattenedIndirectBlob() uses the origin, so the common path cannot leave the generation number unset. - The new files blob_reference_metadata.{h,cc} define the exact reference metadata. The files also contain: - BlobReferenceMetadataCollector, the collector described above. It takes the SST identity (CF ID, SST file number and the relocation-state fingerprint) as one BlobReferenceSstIdentity, set by field name. - BlobReferenceGroupKey, the key of a group: (encoding class, generation number, origin). It has the order, the hash and the validity check that the collector map, the sort, the decoder and FindBlobReferenceGroup() use. BlobReference and BlobReferenceGroup keep their flat fields and return the key through group_key(). - The index codec and the descriptor table property. - BlobReferenceGroupDecoder, the only decoder of reference runs. It reads a group one data block at a time and holds every corruption check of a group walk. Readers that visit one group (found by a binary search of the index) use it, and so do readers that visit every block in order. - Callers in this diff: only the table builder runs the collector. The decoders (DecodeBlobReferenceIndex, FindBlobReferenceGroup, BlobReferenceGroupDecoder, ForEachBlobReference*) are used only by tests. kMaxDecodedBlobReferenceBytes is used only in compile-time checks of the count limits. The census (diff 5) and indirect blob GC (diffs 6 and 7) are their first production users. The census is also the first user of BlobReferenceSstSummary (see the last Implementation bullet). - Descriptor: the CF ID and the SST file number are the repair fields. They let RepairDB tell which column family and SST the list belongs to without a MANIFEST. The decoder rejects a descriptor whose repair fields are not set (unknown CF ID or SST file number 0). The descriptor also has a 128-bit relocation-state fingerprint (the route_snapshot_fingerprint fields). Remote compaction uses it in diff 9 to check that an SST built elsewhere used the same routes as the job that asked for it. In this diff the table builder always writes 0, and DecodeFrom() does not check it. The field is in the format now, so the descriptor keeps one format version (1) through the stack. While the option gate is on, no data in this format exists outside tests. - The collector validates each BlobIndex once, when it is added. Its Finish() writes the data blocks and then the reference index block through one block writer, and fills the descriptor, including the index handle. Destroying the collector frees its memory and then its charge, also when the build is abandoned. - BlockBasedTableBuilder creates the collector with the CF ID and SST file number from TableBuilderOptions. It passes the size of each entry's user key to the collector, so the collector can compute the exact size of each blob record. It adds the collector's EstimatedEncodedSize() to its tail size estimate: the blocks are written before the tail, but output file size planning uses this estimate. The reference blocks are written with BlockType::kUserDefinedIndex, which keeps their contents as opaque bytes. Readers read them without the block cache, so prepopulate_block_cache does not warm them into the block cache (WriteMaybeCompressedBlock() with warm_cache false). - With the option on, the blob file builder computes the CRC32C of each stored value once. The blob log record CRC combines it with the key CRC (crc32c::Crc32cCombine). The stored CRC extends it with the compression byte. For an uncompressed value it is also the logical CRC. The bytes written do not change. - WideColumnSerialization::ForEachBlobIndex replaces ForEachBlobFileNumber. It reads the entity header and column types once. For a V1 entity (no column types and no blob columns), it returns right after the header. For a V2 entity without blob columns, it returns right after the type scan. It shares one V2 column walker (WalkV2Columns) with Deserialize(), so the entity is parsed in one pass. - Read-only opens (DBImplReadOnly, CompactedDBImpl) and secondary opens skip ValidateOptions, so they check the compaction style against the option once per column family: in DBImpl::Recover() for read-only opens, and at the start of OpenAsSecondaryImpl() for secondary opens. DB::OpenAndCompact opens through OpenAsSecondaryImpl(), so it gets the same check. - CacheReservationHandle gets Resize(), so each collector or summary can change its own part of the shared charge. - util/coding.h adds GetCanonicalVarint32/64, which reject overlong varint encodings. The new formats use them, so each value has one encoding. - BlobReferenceSstSummary is the in-memory list of an SST's groups and totals that Finish() can return. It has no caller yet: the table builder passes a null summary output. It is checked and does not change after Finish(), so the finished builder, the SST metadata and the census (its first user, in diff 5) can share it. It is in this diff because Finish() hands the group buffer and its cache charge over to the summary. Adding the summary later would mean rewriting the charge code of Finish() in diff 5, so the charge code is written and tested once, here. Safety boundaries and unsupported cases: - Public API: this diff adds the option `enable_blob_indirection` with the C API functions `rocksdb_options_set_enable_blob_indirection` and `rocksdb_options_get_enable_blob_indirection`. It also inserts a new CacheEntryRole value before kMisc, which changes the numeric value of kMisc and kNumCacheEntryRoles. - With the option off, new blob values and SSTs are written in the legacy form. Legacy blob indexes stay readable. - One behavior change applies with the option off too: UpdateBoundaries, BlobGarbageMeter and the compaction iterator fully validate wide-column entities with blob columns and their blob indexes (see "Separate change" above). - Exact reference metadata has one format (version 1) with a fixed 72-byte descriptor. The decoders reject any other version as Corruption. - The metadata blocks are protected by the table's block checksum only. With checksum = kNoChecksum, corrupt metadata is detected only when it breaks the format or order checks. - The index decoder checks each decoded count against the remaining input size before it reserves memory. So a small corrupt index cannot make it allocate a large vector. - Inlined-TTL and same-file blob indexes are not collected and cannot be relocated. Encoding, count and checksum errors return an error; a bad reference is never skipped. - Option gate (see Terms): outside tests no DB writes data in the new format. The gate checks only the option value, never persisted state. Diff 11 removes it, once data written with the option stays readable through the end of the stack and the safety checks for turning the option on are in place. - Downgrading the binary after data was written with the option on is not supported. Validation and known limitations: - Unit tests cover the BlobIndex encodings and their rejection of bad input, the metadata layout and every corruption check, the collector (sorted path, unsorted fallback, memory charge), the flush integration, the option checks and option gate, wide-column validation, and canonical varints. See the Test Plan. - The collection buffer is not spilled to disk. One SST with very many blob references needs a matching amount of charged memory: the per-reference amount given under "Collection and memory", plus up to one partly used 512-byte chunk per group. While a fallback group is sorted, it also needs 16 bytes of sort key per reference and a second copy of its entries. For a test group of 200K references with every other one out of order, the test comment estimates about 44 bytes per reference, and the test checks that the peak charge is at most 56 bytes per reference. - Flush still writes a single output file. Until diff 7, one compaction output collects every reference it gets, charged to the block cache. Diff 7 cuts compaction outputs (including L0) at a reference cap and keeps each user key's history in one output. With the option gate, a large output is reachable only through the test hook. Stack context (same in diffs 2-11, except that diffs 8-11 add the diff 8 route rule to the install line; "<== this diff" marks this diff's part): Goal: remove garbage from blob files without rewriting any SST. A value gets a stable identity when it is first written. Indirect blob GC copies the live values of whole files into a relocation file and installs routes in the MANIFEST. Readers follow the routes. SSTs are not changed. How a read finds a value (P is the relocation file that holds the value now): ``` SST entry indirect BlobIndex { generation number, origin, original offset, size, CRCs } | v Version file table: generation number -> live file, or route old file -> P | | live file route found | | v v blob file: read at original offset relocation file P (always one hop): relocation file: look up key look up key (origin, original offset) (origin, original offset) ``` Lifecycle of one value (in this example, F is the blob file where the value is first written, and P is the relocation file that GC writes): ``` write new value goes to blob file F; index: origin = F, generation number = F | census per generation number: the SSTs and references that use it; | per physical file: live bytes, ranked by garbage ratio GC job copy the live values of whole input files (stored bytes, checked | with the stored CRC) into relocation file P, keyed by | (origin, original offset) install one MANIFEST edit: add P, route each input generation number still | in use to P, re-point the routes into the inputs to P, and remove | each input (blob file retirement for a blob file, relocation | file deletion for a relocation file) flatten compaction sets generation number = P in the SSTs it rewrites | route route retirement: route deletion records remove the routes that delete no SST uses ``` Stack map: | Diff | Adds | |---|---| | 1 | Finalize blob metadata outside the DB mutex (prep) | | 2 | Indirect BlobIndex (kIndirectBlob) and exact SST reference metadata <== this diff | | 3 | Relocation file format and stable read path | | 4 | MANIFEST routes and Version ownership | | 5 | Exact reverse reference census | | 6 | Indirect blob GC (one input file) | | 7 | Pack whole input files; reclaim relocation files | | 8 | Flatten generation numbers; route retirement | | 9 | Remote compaction validation | | 10 | Dynamic options and operations | | 11 | RepairDB rebuilds relocation state; option gate removed | | 12 | db_stress and crash test support | Stack context: terms for later diffs: | Term | Meaning | |---|---| | census | Background count, per generation number, of the SSTs and references that use it, plus live bytes per physical file (diff 5) | | indirect origin | Normal blob file written with the option on; it is the origin of unflattened references | | flatten | Compaction rewrites the generation number to the relocation file that holds the value, so old routes can go. Only flatten makes generation number != origin (diff 8) | | blob file retirement | MANIFEST record (kBlobFileRetirement) that removes a collected blob file from the Version (diff 4) | | route deletion | MANIFEST record that removes a route that no SST uses. Route retirement is the background job that writes these records (diff 8) | Tracking issue: meta#15301 Differential Revision: D124652097
Summary: Pull Request resolved: meta#15540 Why the change is required: The goal of this stack is indirect blob GC, a blob GC that does not depend on SSTs: it removes garbage from blob files without rewriting any SST. Such a GC needs two things that RocksDB does not have: - A value identity that does not change when the value moves to another file. - A checked list of the blob references that each SST keeps. The list must include older internal versions of a key that the SST still keeps (for example for snapshots), and the blob columns of wide-column entities. This diff adds both. Later diffs build the census (an in-memory reverse index from generation number to the SSTs that use it), GC and repair on top of this list. This diff in one paragraph: With the new column family option enable_blob_indirection on, the blob file writer gives each new value an indirect BlobIndex (type 3, kIndirectBlob). It stores the value's stable identity (origin file, original offset) and its CRCs (an uncompressed value stores only the CRC of the uncompressed value; a compressed value also stores the CRC of its stored bytes). Also with the option on, the block-based table builder collects every blob reference that the SST keeps. It writes them as "exact reference metadata": data blocks, one reference index block, and a 72-byte descriptor table property. In this diff only the table builder writes this metadata, and only tests read it. A temporary option gate (see "option gate" under Terms) keeps the option off outside tests. Separately, wide-column entities with blob columns are now fully checked before their blob indexes are used, also with the option off. Parts of this diff: 1. Indirect BlobIndex (core): types 3 and 4 in db/blob/blob_index.h, and the blob file writer that writes type 3. 2. Exact reference metadata format and decoders (core): db/blob/blob_reference_metadata.{h,cc}. 3. Collector and table builder integration (core): BlobReferenceMetadataCollector, its block cache charge, and BlockBasedTableBuilder. 4. Option enable_blob_indirection and the option gate. Tests need it to turn the feature on. You can skip this part on a first read. 5. Wide-column validation (WideColumnSerialization::ForEachBlobIndex). The table builder uses it to give the blob columns of each entity to the collector. It also changes behavior with the option off (see New behavior). You can skip this part on a first read. 6. Small helpers: canonical varints (util/coding.h) and CacheReservationHandle::Resize(). You can skip these on a first read. How to read this diff: 1. Overview: read "Why" and "This diff in one paragraph" above, and "Terms used in this diff" right below. Then read the top comment of db/blob/blob_reference_metadata.h (terms, layout diagrams, run and entry layout, index layout). 2. Indirect BlobIndex (db/blob/blob_index.h) and canonical varints (util/coding.h GetCanonicalVarint32/64): read the overview comment and the kIndirectBlob / kIndirectFlattenedBlob code. Every later part uses these fields: origin, original offset, generation number, logical CRC and stored CRC. The new formats read varints only in the canonical form. 3. Blob file writer (db/blob/blob_file_builder.cc, blob_log_format.cc, blob_log_writer.cc): see how a new value gets a type-3 index and how its CRC is computed once. 4. Format types and decoders: the structs in db/blob/blob_reference_metadata.h, then in the .cc the constants, GapBase, DecodeEntryBody, DecodeRunEntriesAt, DecodeBlobReferenceIndex and BlobReferenceGroupDecoder. They use only BlobIndex and the canonical varints. Learn the on-disk format here, before you read the writer. 5. Collector and its cache charge: first read the new CacheEntryRole kBlobReferenceMetadataBuildingBuffer (include/rocksdb/cache.h) and CacheReservationHandle::Resize (cache/cache_reservation_manager.h). Then read BlobReferenceMetadataCollector in this order: the class comment, Add, AppendSorted, BuildAndWriteDataBlocks, EncodeIndex, FinishOnce. Then read the unsorted fallback (AppendUnsorted, SortUnsortedGroup, SortAndEncodeGroup). 6. Wide-column change (db/wide/wide_column_serialization.cc, ForEachBlobIndex and WalkV2Columns): it gives every blob index of an entity to a callback after it checks the whole entity. The table builder uses it to feed the collector. UpdateBoundaries, BlobGarbageMeter and the compaction iterator also use it, with the option on or off. 7. Option: enable_blob_indirection (include/rocksdb/advanced_options.h, options/cf_options.cc), and the option checks and gate in db/column_family.cc (ValidateOptions, TEST_SetAllowBlobIndirection), db/db_impl/db_impl_open.cc and db_impl_secondary.cc (the FIFO check for opens that skip ValidateOptions). 8. Table integration: table/block_based/block_based_table_factory.cc (the shared reservation manager), then block_based_table_builder.cc (Rep constructor, Add, WriteBlobReferenceMetadata, EstimatedTailSize). See where the collector is created, fed and finished. block_based_table_reader.cc maps the new block name to a block type. Terms used in this diff: Stack terms: | Term | Meaning | |---|---| | origin | Blob file where the value was first written | | original offset | Value offset in the origin file | | stable identity | (origin, original offset). Never changes, even when the value moves | | generation number | The file number that a reference uses to find its value. A new value has generation number == origin. When indirect blob GC moves the value, the reference keeps its generation number, and a route sends the read to the relocation file | | indirect BlobIndex | A kIndirectBlob (type 3) or kIndirectFlattenedBlob (type 4) index. It stores the stable identity, so the value can move without an SST rewrite | | unflattened reference | Index with generation number == origin: the value was not moved, or it was moved but no compaction has rewritten its reference yet (see "flatten" in "Stack context: terms for later diffs") | | flattened reference | Index with generation number != origin (type 4). Only flatten (diff 8) writes it; this diff defines the encoding | | route | Maps an old file number (a file that GC has removed from the Version) to the relocation file that now holds its live values. One hop: a target is never itself routed. Stored in the MANIFEST (diff 4) | | relocation file | Block-based table keyed by (origin, original offset) that holds copied values. Each value is the stored bytes plus a 5-byte trailer (kSimpleGen2BlobTrailerSize) | | logical CRC | CRC32C of the uncompressed value (BlobIndex::stable_checksum()) | | stored CRC | CRC32C of the stored bytes plus the compression byte | | physical file | A file that holds value bytes: a normal blob file or a relocation file. A generation number can name a physical file, or it can be only the key of a route | | exact reference metadata | Per-SST sorted list of every blob reference the SST keeps, written when the SST is built. References with the same encoding class, generation number and origin form a group. The list is stored as data blocks, with an index of the groups. An SST that has it is an exact SST | | option gate | Until diff 11, ColumnFamilyData::ValidateOptions returns NotSupported for enable_blob_indirection on a read-write open, CreateColumnFamily and SetOptions, unless a test calls ColumnFamilyData::TEST_SetAllowBlobIndirection(true). It keeps the option off in production while the stack is not complete | Terms of this diff (all defined at the top of db/blob/blob_reference_metadata.h): | Term | Meaning | |---|---| | encoding class (class) | kDirect (a legacy kBlob or kBlobTTL index), unflattened, or flattened | | reference group (group) | All references of one SST with the same (encoding class, generation number, origin) | | run | The part of one group inside one data block | | record bytes | The bytes the value's record takes in the file named by its generation number | | gap | How far a record is from where it would be if the records were back to back. A gap of 0 means back to back | | descriptor | The 72-byte table property that points to the reference index block and holds the totals, the CF ID, the SST file number and the relocation-state fingerprint | | relocation-state fingerprint | 128-bit hash of the routes and relocation files of the Version that an SST was built from (fields route_snapshot_fingerprint_*). Written as 0 in this diff | | legacy | Written the way RocksDB writes it without blob indirection | Previous behavior: - A BlobIndex named only a physical blob file and an offset. A value could not move to another file without rewriting every SST that referenced it. - An SST had no metadata that listed its blob references. The only way to find them was to scan its user keys. - These readers get the blob indexes of a wide-column entity through ForEachBlobFileNumber: UpdateBoundaries, BlobGarbageMeter, the compaction iterator, WideColumnSerialization::RemoveBlobColumns, and the referenced_blob_bytes table property count in BlockBasedTableBuilder::Add. ForEachBlobFileNumber decoded each blob index, but it did not check the rest of the entity framing (name order, section sizes, exact input size), and it could call the callback before it found a later error. New behavior: Separate change (option on or off): - The same readers (UpdateBoundaries, BlobGarbageMeter, the compaction iterator, RemoveBlobColumns, and the referenced_blob_bytes count in BlockBasedTableBuilder::Add) use WideColumnSerialization::ForEachBlobIndex, with the option on or off. With the option on, the table builder also uses it to give the blob columns of each entity to the collector. For an entity with blob columns, it checks the full framing, then decodes and validates every blob index before any callback runs. So an entity with a malformed framing now fails the flush or compaction, even with the option off. The referenced_blob_bytes count itself does not fail the build on a malformed entity. It removes that property from the SST, as in base. Option and option gate: - This diff adds a new column family option, `enable_blob_indirection`. It is off by default, and SetOptions can change it. - The option gate (see Terms) keeps the option off outside tests until the stack is complete. ColumnFamilyData::ValidateOptions rejects the option with NotSupported on a read-write open, CreateColumnFamily and SetOptions. Read-only opens (DBImplReadOnly and CompactedDBImpl) and secondary opens (including DB::OpenAndCompact) do not run ValidateOptions; they check only the FIFO rule (see Implementation). Only tests can turn the option on, through the internal ColumnFamilyData::TEST_SetAllowBlobIndirection(). DBTestBase sets it. - When the option is on, ValidateOptions also requires enable_blob_files, the block-based table format, no blob direct write, and a compaction style other than FIFO. Blob file writer (option on): - The blob file writer encodes each new value with a stable identity (origin file, original offset) and two checksums: - the logical CRC: the CRC32C of the uncompressed value. - the stored CRC: the CRC32C of the exact stored (maybe compressed) bytes, extended with the compression byte. - For an uncompressed value the stored CRC is not written, because the reader can compute it from the logical CRC. - The checksums have no version field of their own. The BlobIndex type byte gives the format version: the two new types, 3 (kIndirectBlob) and 4 (kIndirectFlattenedBlob, see Implementation), are BlobIndex::kIndirectBlobFormatVersion (1). A later format uses new type values. - The read path does not check these checksums yet. The check starts in diff 3. Exact reference metadata (option on): 1. What: a sorted list of every blob reference that the SST keeps, in plain blob values and in the blob columns of wide-column entities. 2. Where: data blocks (target 16 KB each before compression; small blocks waste less I/O, because indirect blob GC reads a whole block to get a few groups) and one reference index block, written after the SST data blocks and before the SST tail, plus a 72-byte descriptor table property. 3. Who reads it: only tests in this diff. The census, GC and RepairDB read it in later diffs. 4. Details: the top comment of db/blob/blob_reference_metadata.h has the full format with diagrams, the record-bytes rules and the gap rule. - Not collected: SstFileWriter output (it has no DB file number) and any output whose CF is unknown (it has no CF ID). The descriptor needs both values, so these outputs stay in the legacy form. Checks on the metadata: - A repeated or overlapping reference in one group fails the SST build with Corruption. Readers check every count, order, size and total against the index and the descriptor, so corrupt metadata returns Corruption. The writer never makes a data block larger than kTargetDataBlockSize (16 KB), and readers accept up to kMaxDataBlockSize (64 KB). Details: "Readers" and "Integrity" at the top of db/blob/blob_reference_metadata.h, and BlobReferenceGroupDecoder. Collection and memory: - Each reference is encoded when it is added, in the on-disk entry encoding, into its group's chain of 512-byte chunks. Only a group that takes the unsorted fallback is encoded a second time, in Finish(). Inside a group, references normally arrive in rising offset order, so no reference is sorted; at the end only the groups are sorted. A group that gets a lower offset takes the unsorted fallback. The BlobReferenceMetadataCollector class comment explains the chunks, the chunk size and the fallback. - The collector keeps every reference of the SST in memory until the SST is finished, and charges this memory to the block cache under a new CacheEntryRole, kBlobReferenceMetadataBuildingBuffer. The measured cost is about 8.3 to 9 bytes per reference for uncompressed values and 12.8 to 13.5 bytes for compressed values. "Memory usage" and "Memory charge" in the BlobReferenceMetadataCollector class comment give the measurement setup and the charge rule. - There is no fixed limit on references. The collector fails with MemoryLimit above 2^20 groups or 2^20 data blocks in one SST; these are persisted format limits. - There is no charge when the table has no block cache or when charging for this role is disabled. Each block-based table factory holds one cache reservation manager for this role, shared by its builders and the SST summaries. If the charge fails, the SST build fails, so the flush or compaction fails with a background error. The buffers and their charge are released when the SST is finished or abandoned. Implementation: - BlobIndex adds two indirect encodings. The legacy encodings do not change. - Type 3, kIndirectBlob (unflattened, generation number == origin), stores the origin, the original offset, the stored size, the compression type, the logical CRC and, for compressed values, the stored CRC. It does not store the generation number. This saves the size of one varint file number per reference (about 3 to 4 bytes for typical file numbers; an estimate, not measured). - Type 4, kIndirectFlattenedBlob (flattened), stores the generation number first, then the same fields. Its generation number must be above the origin. The decoder rejects type 4 with generation number <= origin, so each reference has one encoding. - The encoders take the fields as one BlobIndex::IndirectBlobFields, set by field name, so values of the same type (file number, offset and size; the two CRCs) cannot be swapped. EncodeIndirectBlob() takes the generation number as a separate argument; EncodeUnflattenedIndirectBlob() uses the origin, so the common path cannot leave the generation number unset. - The new files blob_reference_metadata.{h,cc} define the exact reference metadata. The files also contain: - BlobReferenceMetadataCollector, the collector described above. It takes the SST identity (CF ID, SST file number and the relocation-state fingerprint) as one BlobReferenceSstIdentity, set by field name. - BlobReferenceGroupKey, the key of a group: (encoding class, generation number, origin). It has the order, the hash and the validity check that the collector map, the sort, the decoder and FindBlobReferenceGroup() use. BlobReference and BlobReferenceGroup keep their flat fields and return the key through group_key(). - The index codec and the descriptor table property. - BlobReferenceGroupDecoder, the only decoder of reference runs. It reads a group one data block at a time and holds every corruption check of a group walk. Readers that visit one group (found by a binary search of the index) use it, and so do readers that visit every block in order. - Callers in this diff: only the table builder runs the collector. The decoders (DecodeBlobReferenceIndex, FindBlobReferenceGroup, BlobReferenceGroupDecoder, ForEachBlobReference*) are used only by tests. kMaxDecodedBlobReferenceBytes is used only in compile-time checks of the count limits. The census (diff 5) and indirect blob GC (diffs 6 and 7) are their first production users. The census is also the first user of BlobReferenceSstSummary (see the last Implementation bullet). - Descriptor: the CF ID and the SST file number are the repair fields. They let RepairDB tell which column family and SST the list belongs to without a MANIFEST. The decoder rejects a descriptor whose repair fields are not set (unknown CF ID or SST file number 0). The descriptor also has a 128-bit relocation-state fingerprint (the route_snapshot_fingerprint fields). Remote compaction uses it in diff 9 to check that an SST built elsewhere used the same routes as the job that asked for it. In this diff the table builder always writes 0, and DecodeFrom() does not check it. The field is in the format now, so the descriptor keeps one format version (1) through the stack. While the option gate is on, no data in this format exists outside tests. - The collector validates each BlobIndex once, when it is added. Its Finish() writes the data blocks and then the reference index block through one block writer, and fills the descriptor, including the index handle. Destroying the collector frees its memory and then its charge, also when the build is abandoned. - BlockBasedTableBuilder creates the collector with the CF ID and SST file number from TableBuilderOptions. It passes the size of each entry's user key to the collector, so the collector can compute the exact size of each blob record. It adds the collector's EstimatedEncodedSize() to its tail size estimate: the blocks are written before the tail, but output file size planning uses this estimate. The reference blocks are written with BlockType::kUserDefinedIndex, which keeps their contents as opaque bytes. Readers read them without the block cache, so prepopulate_block_cache does not warm them into the block cache (WriteMaybeCompressedBlock() with warm_cache false). - With the option on, the blob file builder computes the CRC32C of each stored value once. The blob log record CRC combines it with the key CRC (crc32c::Crc32cCombine). The stored CRC extends it with the compression byte. For an uncompressed value it is also the logical CRC. The bytes written do not change. - WideColumnSerialization::ForEachBlobIndex replaces ForEachBlobFileNumber. It reads the entity header and column types once. For a V1 entity (no column types and no blob columns), it returns right after the header. For a V2 entity without blob columns, it returns right after the type scan. It shares one V2 column walker (WalkV2Columns) with Deserialize(), so the entity is parsed in one pass. - Read-only opens (DBImplReadOnly, CompactedDBImpl) and secondary opens skip ValidateOptions, so they check the compaction style against the option once per column family: in DBImpl::Recover() for read-only opens, and at the start of OpenAsSecondaryImpl() for secondary opens. DB::OpenAndCompact opens through OpenAsSecondaryImpl(), so it gets the same check. - CacheReservationHandle gets Resize(), so each collector or summary can change its own part of the shared charge. - util/coding.h adds GetCanonicalVarint32/64, which reject overlong varint encodings. The new formats use them, so each value has one encoding. - BlobReferenceSstSummary is the in-memory list of an SST's groups and totals that Finish() can return. It has no caller yet: the table builder passes a null summary output. It is checked and does not change after Finish(), so the finished builder, the SST metadata and the census (its first user, in diff 5) can share it. It is in this diff because Finish() hands the group buffer and its cache charge over to the summary. Adding the summary later would mean rewriting the charge code of Finish() in diff 5, so the charge code is written and tested once, here. Safety boundaries and unsupported cases: - Public API: this diff adds the option `enable_blob_indirection` with the C API functions `rocksdb_options_set_enable_blob_indirection` and `rocksdb_options_get_enable_blob_indirection`. It also inserts a new CacheEntryRole value before kMisc, which changes the numeric value of kMisc and kNumCacheEntryRoles. - With the option off, new blob values and SSTs are written in the legacy form. Legacy blob indexes stay readable. - One behavior change applies with the option off too: UpdateBoundaries, BlobGarbageMeter and the compaction iterator fully validate wide-column entities with blob columns and their blob indexes (see "Separate change" above). - Exact reference metadata has one format (version 1) with a fixed 72-byte descriptor. The decoders reject any other version as Corruption. - The metadata blocks are protected by the table's block checksum only. With checksum = kNoChecksum, corrupt metadata is detected only when it breaks the format or order checks. - The index decoder checks each decoded count against the remaining input size before it reserves memory. So a small corrupt index cannot make it allocate a large vector. - Inlined-TTL and same-file blob indexes are not collected and cannot be relocated. Encoding, count and checksum errors return an error; a bad reference is never skipped. - Option gate (see Terms): outside tests no DB writes data in the new format. The gate checks only the option value, never persisted state. Diff 11 removes it, once data written with the option stays readable through the end of the stack and the safety checks for turning the option on are in place. - Downgrading the binary after data was written with the option on is not supported. Validation and known limitations: - Unit tests cover the BlobIndex encodings and their rejection of bad input, the metadata layout and every corruption check, the collector (sorted path, unsorted fallback, memory charge), the flush integration, the option checks and option gate, wide-column validation, and canonical varints. See the Test Plan. - The collection buffer is not spilled to disk. One SST with very many blob references needs a matching amount of charged memory: the per-reference amount given under "Collection and memory", plus up to one partly used 512-byte chunk per group. While a fallback group is sorted, it also needs 16 bytes of sort key per reference and a second copy of its entries. For a test group of 200K references with every other one out of order, the test comment estimates about 44 bytes per reference, and the test checks that the peak charge is at most 56 bytes per reference. - Flush still writes a single output file. Until diff 7, one compaction output collects every reference it gets, charged to the block cache. Diff 7 cuts compaction outputs (including L0) at a reference cap and keeps each user key's history in one output. With the option gate, a large output is reachable only through the test hook. Stack context (same in diffs 2-11, except that diffs 8-11 add the diff 8 route rule to the install line; "<== this diff" marks this diff's part): Goal: remove garbage from blob files without rewriting any SST. A value gets a stable identity when it is first written. Indirect blob GC copies the live values of whole files into a relocation file and installs routes in the MANIFEST. Readers follow the routes. SSTs are not changed. How a read finds a value (P is the relocation file that holds the value now): ``` SST entry indirect BlobIndex { generation number, origin, original offset, size, CRCs } | v Version file table: generation number -> live file, or route old file -> P | | live file route found | | v v blob file: read at original offset relocation file P (always one hop): relocation file: look up key look up key (origin, original offset) (origin, original offset) ``` Lifecycle of one value (in this example, F is the blob file where the value is first written, and P is the relocation file that GC writes): ``` write new value goes to blob file F; index: origin = F, generation number = F | census per generation number: the SSTs and references that use it; | per physical file: live bytes, ranked by garbage ratio GC job copy the live values of whole input files (stored bytes, checked | with the stored CRC) into relocation file P, keyed by | (origin, original offset) install one MANIFEST edit: add P, route each input generation number still | in use to P, re-point the routes into the inputs to P, and remove | each input (blob file retirement for a blob file, relocation | file deletion for a relocation file) flatten compaction sets generation number = P in the SSTs it rewrites | route route retirement: route deletion records remove the routes that delete no SST uses ``` Stack map: | Diff | Adds | |---|---| | 1 | Finalize blob metadata outside the DB mutex (prep) | | 2 | Indirect BlobIndex (kIndirectBlob) and exact SST reference metadata <== this diff | | 3 | Relocation file format and stable read path | | 4 | MANIFEST routes and Version ownership | | 5 | Exact reverse reference census | | 6 | Indirect blob GC (one input file) | | 7 | Pack whole input files; reclaim relocation files | | 8 | Flatten generation numbers; route retirement | | 9 | Remote compaction validation | | 10 | Dynamic options and operations | | 11 | RepairDB rebuilds relocation state; option gate removed | | 12 | db_stress and crash test support | Stack context: terms for later diffs: | Term | Meaning | |---|---| | census | Background count, per generation number, of the SSTs and references that use it, plus live bytes per physical file (diff 5) | | indirect origin | Normal blob file written with the option on; it is the origin of unflattened references | | flatten | Compaction rewrites the generation number to the relocation file that holds the value, so old routes can go. Only flatten makes generation number != origin (diff 8) | | blob file retirement | MANIFEST record (kBlobFileRetirement) that removes a collected blob file from the Version (diff 4) | | route deletion | MANIFEST record that removes a route that no SST uses. Route retirement is the background job that writes these records (diff 8) | Tracking issue: meta#15301 Differential Revision: D124652097
Claude Code Review - OBSOLETESuperseded by a newer AI review. Expand to see the original review.✅ Claude Code ReviewAuto-triggered after CI passed — reviewing commit 3fa5380 SummaryThis is a large, well-documented foundational diff (diff 2/12) adding indirect BlobIndex types and exact SST reference metadata for indirect blob GC. The design is thorough, with detailed format specifications, strong validation, and careful memory charging. The option gate keeps this off in production during the stack build-out. Key concerns are: (1) a behavioral compatibility change that affects the option-off code path, (2) new defaulted parameters that CLAUDE.md explicitly warns against, (3) potential build system gaps, and (4) the CacheEntryRole enum insertion changing kMisc's numeric value. High-severity findings (3):
Full review (click to expand)Findings🔴 HIGHH1. Defaulted parameters on
|
| Context | Affected? | Assessment |
|---|---|---|
| WritePreparedTxnDB | Yes (blob writes) | Safe: option gated; changes behind enable_blob_indirection_ |
| ReadOnly DB | Yes (skips ValidateOptions) | Safe: only reads; FIFO check added |
| SecondaryInstance | Yes (skips ValidateOptions) | Safe: FIFO check added; no writes |
| CompactionService | Yes | Safe for this diff: option gate prevents production use |
| User-defined timestamps | Yes | Safe: collector uses key size, not key contents |
| FIFO compaction | Blocked | Safe: ValidateOptions rejects the combination |
| MemPurge | Yes (flush path) | Safe: table builder integration is standard |
Positive Observations
- Exceptional documentation: The PR description with format diagrams, reading guides, and terminology tables is best-in-class for a diff of this complexity.
- Defensive encoding: Designated initializers for
IndirectBlobFieldsprevent field-swap bugs. Canonical varints ensure one-encoding-per-value. - Memory discipline: Block cache charging for collector buffers, with proper cleanup on abandon, follows RocksDB best practices.
- CRC optimization: Computing stored-bytes CRC once and deriving record CRC via
crc32c::Crc32cCombineavoids a redundant full-value pass. - Option gate pattern: Keeping the feature off in production while building the stack is sound engineering.
- Test coverage: Tests cover sorted path, unsorted fallback, memory charging, flush integration, option validation, canonical varint rejection, CRC correctness, and failure modes.
ℹ️ About this response
Generated by Claude Code.
Review methodology: claude_md/ci_review_prompt.md
Limitations:
- Claude may miss context from files not in the diff
- Large PRs may be truncated
- Always apply human judgment to AI suggestions
Commands:
/claude-review [context]— Request a code review/claude-query <question>— Ask about the PR or codebase
Summary: Pull Request resolved: meta#15540 Why the change is required: The goal of this stack is indirect blob GC, a blob GC that does not depend on SSTs: it removes garbage from blob files without rewriting any SST. Such a GC needs two things that RocksDB does not have: - A value identity that does not change when the value moves to another file. - A checked list of the blob references that each SST keeps. The list must include older internal versions of a key that the SST still keeps (for example for snapshots), and the blob columns of wide-column entities. This diff adds both. Later diffs build the census (an in-memory reverse index from generation number to the SSTs that use it), GC and repair on top of this list. This diff in one paragraph: With the new column family option enable_blob_indirection on, the blob file writer gives each new value an indirect BlobIndex (type 3, kIndirectBlob). It stores the value's stable identity (origin file, original offset) and its CRCs (an uncompressed value stores only the CRC of the uncompressed value; a compressed value also stores the CRC of its stored bytes). Also with the option on, the block-based table builder collects every blob reference that the SST keeps. It writes them as "exact reference metadata": data blocks, one reference index block, and a 72-byte descriptor table property. In this diff only the table builder writes this metadata, and only tests read it. A temporary option gate (see "option gate" under Terms) keeps the option off outside tests. Separately, wide-column entities with blob columns are now fully checked before their blob indexes are used, also with the option off. Parts of this diff: 1. Indirect BlobIndex (core): types 3 and 4 in db/blob/blob_index.h, and the blob file writer that writes type 3. 2. Exact reference metadata format and decoders (core): db/blob/blob_reference_metadata.{h,cc}. 3. Collector and table builder integration (core): BlobReferenceMetadataCollector, its block cache charge, and BlockBasedTableBuilder. 4. Option enable_blob_indirection and the option gate. Tests need it to turn the feature on. You can skip this part on a first read. 5. Wide-column validation (WideColumnSerialization::ForEachBlobIndex). The table builder uses it to give the blob columns of each entity to the collector. It also changes behavior with the option off (see New behavior). You can skip this part on a first read. 6. Small helpers: canonical varints (util/coding.h) and CacheReservationHandle::Resize(). You can skip these on a first read. How to read this diff: 1. Overview: read "Why" and "This diff in one paragraph" above, and "Terms used in this diff" right below. Then read the top comment of db/blob/blob_reference_metadata.h (terms, layout diagrams, run and entry layout, index layout). 2. Indirect BlobIndex (db/blob/blob_index.h) and canonical varints (util/coding.h GetCanonicalVarint32/64): read the overview comment and the kIndirectBlob / kIndirectFlattenedBlob code. Every later part uses these fields: origin, original offset, generation number, logical CRC and stored CRC. The new formats read varints only in the canonical form. 3. Blob file writer (db/blob/blob_file_builder.cc, blob_log_format.cc, blob_log_writer.cc): see how a new value gets a type-3 index and how its CRC is computed once. 4. Format types and decoders: the structs in db/blob/blob_reference_metadata.h, then in the .cc the constants, GapBase, DecodeEntryBody, DecodeRunEntriesAt, DecodeBlobReferenceIndex and BlobReferenceGroupDecoder. They use only BlobIndex and the canonical varints. Learn the on-disk format here, before you read the writer. 5. Collector and its cache charge: first read the new CacheEntryRole kBlobReferenceMetadataBuildingBuffer (include/rocksdb/cache.h) and CacheReservationHandle::Resize (cache/cache_reservation_manager.h). Then read BlobReferenceMetadataCollector in this order: the class comment, Add, AppendSorted, BuildAndWriteDataBlocks, EncodeIndex, FinishOnce. Then read the unsorted fallback (AppendUnsorted, SortUnsortedGroup, SortAndEncodeGroup). 6. Wide-column change (db/wide/wide_column_serialization.cc, ForEachBlobIndex and WalkV2Columns): it gives every blob index of an entity to a callback after it checks the whole entity. The table builder uses it to feed the collector. UpdateBoundaries, BlobGarbageMeter and the compaction iterator also use it, with the option on or off. 7. Option: enable_blob_indirection (include/rocksdb/advanced_options.h, options/cf_options.cc), and the option checks and gate in db/column_family.cc (ValidateOptions, TEST_SetAllowBlobIndirection), db/db_impl/db_impl_open.cc and db_impl_secondary.cc (the FIFO check for opens that skip ValidateOptions). 8. Table integration: table/block_based/block_based_table_factory.cc (the shared reservation manager), then block_based_table_builder.cc (Rep constructor, Add, WriteBlobReferenceMetadata, EstimatedTailSize). See where the collector is created, fed and finished. block_based_table_reader.cc maps the new block name to a block type. Terms used in this diff: Stack terms: | Term | Meaning | |---|---| | origin | Blob file where the value was first written | | original offset | Value offset in the origin file | | stable identity | (origin, original offset). Never changes, even when the value moves | | generation number | The file number that a reference uses to find its value. A new value has generation number == origin. When indirect blob GC moves the value, the reference keeps its generation number, and a route sends the read to the relocation file | | indirect BlobIndex | A kIndirectBlob (type 3) or kIndirectFlattenedBlob (type 4) index. It stores the stable identity, so the value can move without an SST rewrite | | unflattened reference | Index with generation number == origin: the value was not moved, or it was moved but no compaction has rewritten its reference yet (see "flatten" in "Stack context: terms for later diffs") | | flattened reference | Index with generation number != origin (type 4). Only flatten (diff 8) writes it; this diff defines the encoding | | route | Maps an old file number (a file that GC has removed from the Version) to the relocation file that now holds its live values. One hop: a target is never itself routed. Stored in the MANIFEST (diff 4) | | relocation file | Block-based table keyed by (origin, original offset) that holds copied values. Each value is the stored bytes plus a 5-byte trailer (kSimpleGen2BlobTrailerSize) | | logical CRC | CRC32C of the uncompressed value (BlobIndex::stable_checksum()) | | stored CRC | CRC32C of the stored bytes plus the compression byte | | physical file | A file that holds value bytes: a normal blob file or a relocation file. A generation number can name a physical file, or it can be only the key of a route | | exact reference metadata | Per-SST sorted list of every blob reference the SST keeps, written when the SST is built. References with the same encoding class, generation number and origin form a group. The list is stored as data blocks, with an index of the groups. An SST that has it is an exact SST | | option gate | Until diff 11, ColumnFamilyData::ValidateOptions returns NotSupported for enable_blob_indirection on a read-write open, CreateColumnFamily and SetOptions, unless a test calls ColumnFamilyData::TEST_SetAllowBlobIndirection(true). It keeps the option off in production while the stack is not complete | Terms of this diff (all defined at the top of db/blob/blob_reference_metadata.h): | Term | Meaning | |---|---| | encoding class (class) | kDirect (a legacy kBlob or kBlobTTL index), unflattened, or flattened | | reference group (group) | All references of one SST with the same (encoding class, generation number, origin) | | run | The part of one group inside one data block | | record bytes | The bytes the value's record takes in the file named by its generation number | | gap | How far a record is from where it would be if the records were back to back. A gap of 0 means back to back | | descriptor | The 72-byte table property that points to the reference index block and holds the totals, the CF ID, the SST file number and the relocation-state fingerprint | | relocation-state fingerprint | 128-bit hash of the routes and relocation files of the Version that an SST was built from (fields route_snapshot_fingerprint_*). Written as 0 in this diff | | legacy | Written the way RocksDB writes it without blob indirection | Previous behavior: - A BlobIndex named only a physical blob file and an offset. A value could not move to another file without rewriting every SST that referenced it. - An SST had no metadata that listed its blob references. The only way to find them was to scan its user keys. - These readers get the blob indexes of a wide-column entity through ForEachBlobFileNumber: UpdateBoundaries, BlobGarbageMeter, the compaction iterator, WideColumnSerialization::RemoveBlobColumns, and the referenced_blob_bytes table property count in BlockBasedTableBuilder::Add. ForEachBlobFileNumber decoded each blob index, but it did not check the rest of the entity framing (name order, section sizes, exact input size), and it could call the callback before it found a later error. New behavior: Separate change (option on or off): - The same readers (UpdateBoundaries, BlobGarbageMeter, the compaction iterator, RemoveBlobColumns, and the referenced_blob_bytes count in BlockBasedTableBuilder::Add) use WideColumnSerialization::ForEachBlobIndex, with the option on or off. With the option on, the table builder also uses it to give the blob columns of each entity to the collector. For an entity with blob columns, it checks the full framing, then decodes and validates every blob index before any callback runs. So an entity with a malformed framing now fails the flush or compaction, even with the option off. The referenced_blob_bytes count itself does not fail the build on a malformed entity. It removes that property from the SST, as in base. Option and option gate: - This diff adds a new column family option, `enable_blob_indirection`. It is off by default, and SetOptions can change it. - The option gate (see Terms) keeps the option off outside tests until the stack is complete. ColumnFamilyData::ValidateOptions rejects the option with NotSupported on a read-write open, CreateColumnFamily and SetOptions. Read-only opens (DBImplReadOnly and CompactedDBImpl) and secondary opens (including DB::OpenAndCompact) do not run ValidateOptions; they check only the FIFO rule (see Implementation). Only tests can turn the option on, through the internal ColumnFamilyData::TEST_SetAllowBlobIndirection(). DBTestBase sets it. - When the option is on, ValidateOptions also requires enable_blob_files, the block-based table format, no blob direct write, and a compaction style other than FIFO. Blob file writer (option on): - The blob file writer encodes each new value with a stable identity (origin file, original offset) and two checksums: - the logical CRC: the CRC32C of the uncompressed value. - the stored CRC: the CRC32C of the exact stored (maybe compressed) bytes, extended with the compression byte. - For an uncompressed value the stored CRC is not written, because the reader can compute it from the logical CRC. - The checksums have no version field of their own. The BlobIndex type byte gives the format version: the two new types, 3 (kIndirectBlob) and 4 (kIndirectFlattenedBlob, see Implementation), are BlobIndex::kIndirectBlobFormatVersion (1). A later format uses new type values. - The read path does not check these checksums yet. The check starts in diff 3. Exact reference metadata (option on): 1. What: a sorted list of every blob reference that the SST keeps, in plain blob values and in the blob columns of wide-column entities. 2. Where: data blocks (target 16 KB each before compression; small blocks waste less I/O, because indirect blob GC reads a whole block to get a few groups) and one reference index block, written after the SST data blocks and before the SST tail, plus a 72-byte descriptor table property. 3. Who reads it: only tests in this diff. The census, GC and RepairDB read it in later diffs. 4. Details: the top comment of db/blob/blob_reference_metadata.h has the full format with diagrams, the record-bytes rules and the gap rule. - Not collected: SstFileWriter output (it has no DB file number) and any output whose CF is unknown (it has no CF ID). The descriptor needs both values, so these outputs stay in the legacy form. Checks on the metadata: - A repeated or overlapping reference in one group fails the SST build with Corruption. Readers check every count, order, size and total against the index and the descriptor, so corrupt metadata returns Corruption. The writer never makes a data block larger than kTargetDataBlockSize (16 KB), and readers accept up to kMaxDataBlockSize (64 KB). Details: "Readers" and "Integrity" at the top of db/blob/blob_reference_metadata.h, and BlobReferenceGroupDecoder. Collection and memory: - Each reference is encoded when it is added, in the on-disk entry encoding, into its group's chain of 512-byte chunks. Only a group that takes the unsorted fallback is encoded a second time, in Finish(). Inside a group, references normally arrive in rising offset order, so no reference is sorted; at the end only the groups are sorted. A group that gets a lower offset takes the unsorted fallback. The BlobReferenceMetadataCollector class comment explains the chunks, the chunk size and the fallback. - The collector keeps every reference of the SST in memory until the SST is finished, and charges this memory to the block cache under a new CacheEntryRole, kBlobReferenceMetadataBuildingBuffer. The measured cost is about 8.3 to 9 bytes per reference for uncompressed values and 12.8 to 13.5 bytes for compressed values. "Memory usage" and "Memory charge" in the BlobReferenceMetadataCollector class comment give the measurement setup and the charge rule. - There is no fixed limit on references. The collector fails with MemoryLimit above 2^20 groups or 2^20 data blocks in one SST; these are persisted format limits. - There is no charge when the table has no block cache or when charging for this role is disabled. Each block-based table factory holds one cache reservation manager for this role, shared by its builders and the SST summaries. If the charge fails, the SST build fails, so the flush or compaction fails with a background error. The buffers and their charge are released when the SST is finished or abandoned. Implementation: - BlobIndex adds two indirect encodings. The legacy encodings do not change. - Type 3, kIndirectBlob (unflattened, generation number == origin), stores the origin, the original offset, the stored size, the compression type, the logical CRC and, for compressed values, the stored CRC. It does not store the generation number. This saves the size of one varint file number per reference (about 3 to 4 bytes for typical file numbers; an estimate, not measured). - Type 4, kIndirectFlattenedBlob (flattened), stores the generation number first, then the same fields. Its generation number must be above the origin. The decoder rejects type 4 with generation number <= origin, so each reference has one encoding. - The encoders take the fields as one BlobIndex::IndirectBlobFields, set by field name, so values of the same type (file number, offset and size; the two CRCs) cannot be swapped. EncodeIndirectBlob() takes the generation number as a separate argument; EncodeUnflattenedIndirectBlob() uses the origin, so the common path cannot leave the generation number unset. - The new files blob_reference_metadata.{h,cc} define the exact reference metadata. The files also contain: - BlobReferenceMetadataCollector, the collector described above. It takes the SST identity (CF ID, SST file number and the relocation-state fingerprint) as one BlobReferenceSstIdentity, set by field name. - BlobReferenceGroupKey, the key of a group: (encoding class, generation number, origin). It has the order, the hash and the validity check that the collector map, the sort, the decoder and FindBlobReferenceGroup() use. BlobReference and BlobReferenceGroup keep their flat fields and return the key through group_key(). - The index codec and the descriptor table property. - BlobReferenceGroupDecoder, the only decoder of reference runs. It reads a group one data block at a time and holds every corruption check of a group walk. Readers that visit one group (found by a binary search of the index) use it, and so do readers that visit every block in order. - Callers in this diff: only the table builder runs the collector. The decoders (DecodeBlobReferenceIndex, FindBlobReferenceGroup, BlobReferenceGroupDecoder, ForEachBlobReference*) are used only by tests. kMaxDecodedBlobReferenceBytes is used only in compile-time checks of the count limits. The census (diff 5) and indirect blob GC (diffs 6 and 7) are their first production users. The census is also the first user of BlobReferenceSstSummary (see the last Implementation bullet). - Descriptor: the CF ID and the SST file number are the repair fields. They let RepairDB tell which column family and SST the list belongs to without a MANIFEST. The decoder rejects a descriptor whose repair fields are not set (unknown CF ID or SST file number 0). The descriptor also has a 128-bit relocation-state fingerprint (the route_snapshot_fingerprint fields). Remote compaction uses it in diff 9 to check that an SST built elsewhere used the same routes as the job that asked for it. In this diff the table builder always writes 0, and DecodeFrom() does not check it. The field is in the format now, so the descriptor keeps one format version (1) through the stack. While the option gate is on, no data in this format exists outside tests. - The collector validates each BlobIndex once, when it is added. Its Finish() writes the data blocks and then the reference index block through one block writer, and fills the descriptor, including the index handle. Destroying the collector frees its memory and then its charge, also when the build is abandoned. - BlockBasedTableBuilder creates the collector with the CF ID and SST file number from TableBuilderOptions. It passes the size of each entry's user key to the collector, so the collector can compute the exact size of each blob record. It adds the collector's EstimatedEncodedSize() to its tail size estimate: the blocks are written before the tail, but output file size planning uses this estimate. The reference blocks are written with BlockType::kUserDefinedIndex, which keeps their contents as opaque bytes. Readers read them without the block cache, so prepopulate_block_cache does not warm them into the block cache (WriteMaybeCompressedBlock() with warm_cache false). - With the option on, the blob file builder computes the CRC32C of each stored value once. The blob log record CRC combines it with the key CRC (crc32c::Crc32cCombine). The stored CRC extends it with the compression byte. For an uncompressed value it is also the logical CRC. The bytes written do not change. - WideColumnSerialization::ForEachBlobIndex replaces ForEachBlobFileNumber. It reads the entity header and column types once. For a V1 entity (no column types and no blob columns), it returns right after the header. For a V2 entity without blob columns, it returns right after the type scan. It shares one V2 column walker (WalkV2Columns) with Deserialize(), so the entity is parsed in one pass. - Read-only opens (DBImplReadOnly, CompactedDBImpl) and secondary opens skip ValidateOptions, so they check the compaction style against the option once per column family: in DBImpl::Recover() for read-only opens, and at the start of OpenAsSecondaryImpl() for secondary opens. DB::OpenAndCompact opens through OpenAsSecondaryImpl(), so it gets the same check. - CacheReservationHandle gets Resize(), so each collector or summary can change its own part of the shared charge. - util/coding.h adds GetCanonicalVarint32/64, which reject overlong varint encodings. The new formats use them, so each value has one encoding. - BlobReferenceSstSummary is the in-memory list of an SST's groups and totals that Finish() can return. It has no caller yet: the table builder passes a null summary output. It is checked and does not change after Finish(), so the finished builder, the SST metadata and the census (its first user, in diff 5) can share it. It is in this diff because Finish() hands the group buffer and its cache charge over to the summary. Adding the summary later would mean rewriting the charge code of Finish() in diff 5, so the charge code is written and tested once, here. Safety boundaries and unsupported cases: - Public API: this diff adds the option `enable_blob_indirection` with the C API functions `rocksdb_options_set_enable_blob_indirection` and `rocksdb_options_get_enable_blob_indirection`. It also inserts a new CacheEntryRole value before kMisc, which changes the numeric value of kMisc and kNumCacheEntryRoles. - With the option off, new blob values and SSTs are written in the legacy form. Legacy blob indexes stay readable. - One behavior change applies with the option off too: UpdateBoundaries, BlobGarbageMeter and the compaction iterator fully validate wide-column entities with blob columns and their blob indexes (see "Separate change" above). - Exact reference metadata has one format (version 1) with a fixed 72-byte descriptor. The decoders reject any other version as Corruption. - The metadata blocks are protected by the table's block checksum only. With checksum = kNoChecksum, corrupt metadata is detected only when it breaks the format or order checks. - The index decoder checks each decoded count against the remaining input size before it reserves memory. So a small corrupt index cannot make it allocate a large vector. - Inlined-TTL and same-file blob indexes are not collected and cannot be relocated. Encoding, count and checksum errors return an error; a bad reference is never skipped. - Option gate (see Terms): outside tests no DB writes data in the new format. The gate checks only the option value, never persisted state. Diff 11 removes it, once data written with the option stays readable through the end of the stack and the safety checks for turning the option on are in place. - Downgrading the binary after data was written with the option on is not supported. Validation and known limitations: - Unit tests cover the BlobIndex encodings and their rejection of bad input, the metadata layout and every corruption check, the collector (sorted path, unsorted fallback, memory charge), the flush integration, the option checks and option gate, wide-column validation, and canonical varints. See the Test Plan. - The collection buffer is not spilled to disk. One SST with very many blob references needs a matching amount of charged memory: the per-reference amount given under "Collection and memory", plus up to one partly used 512-byte chunk per group. While a fallback group is sorted, it also needs 16 bytes of sort key per reference and a second copy of its entries. For a test group of 200K references with every other one out of order, the test comment estimates about 44 bytes per reference, and the test checks that the peak charge is at most 56 bytes per reference. - Flush still writes a single output file. Until diff 7, one compaction output collects every reference it gets, charged to the block cache. Diff 7 cuts compaction outputs (including L0) at a reference cap and keeps each user key's history in one output. With the option gate, a large output is reachable only through the test hook. Stack context (same in diffs 2-11, except that diffs 8-11 add the diff 8 route rule to the install line; "<== this diff" marks this diff's part): Goal: remove garbage from blob files without rewriting any SST. A value gets a stable identity when it is first written. Indirect blob GC copies the live values of whole files into a relocation file and installs routes in the MANIFEST. Readers follow the routes. SSTs are not changed. How a read finds a value (P is the relocation file that holds the value now): ``` SST entry indirect BlobIndex { generation number, origin, original offset, size, CRCs } | v Version file table: generation number -> live file, or route old file -> P | | live file route found | | v v blob file: read at original offset relocation file P (always one hop): relocation file: look up key look up key (origin, original offset) (origin, original offset) ``` Lifecycle of one value (in this example, F is the blob file where the value is first written, and P is the relocation file that GC writes): ``` write new value goes to blob file F; index: origin = F, generation number = F | census per generation number: the SSTs and references that use it; | per physical file: live bytes, ranked by garbage ratio GC job copy the live values of whole input files (stored bytes, checked | with the stored CRC) into relocation file P, keyed by | (origin, original offset) install one MANIFEST edit: add P, route each input generation number still | in use to P, re-point the routes into the inputs to P, and remove | each input (blob file retirement for a blob file, relocation | file deletion for a relocation file) flatten compaction sets generation number = P in the SSTs it rewrites | route route retirement: route deletion records remove the routes that delete no SST uses ``` Stack map: | Diff | Adds | |---|---| | 1 | Finalize blob metadata outside the DB mutex (prep) | | 2 | Indirect BlobIndex (kIndirectBlob) and exact SST reference metadata <== this diff | | 3 | Relocation file format and stable read path | | 4 | MANIFEST routes and Version ownership | | 5 | Exact reverse reference census | | 6 | Indirect blob GC (one input file) | | 7 | Pack whole input files; reclaim relocation files | | 8 | Flatten generation numbers; route retirement | | 9 | Remote compaction validation | | 10 | Dynamic options and operations | | 11 | RepairDB rebuilds relocation state; option gate removed | | 12 | db_stress and crash test support | Stack context: terms for later diffs: | Term | Meaning | |---|---| | census | Background count, per generation number, of the SSTs and references that use it, plus live bytes per physical file (diff 5) | | indirect origin | Normal blob file written with the option on; it is the origin of unflattened references | | flatten | Compaction rewrites the generation number to the relocation file that holds the value, so old routes can go. Only flatten makes generation number != origin (diff 8) | | blob file retirement | MANIFEST record (kBlobFileRetirement) that removes a collected blob file from the Version (diff 4) | | route deletion | MANIFEST record that removes a route that no SST uses. Route retirement is the background job that writes these records (diff 8) | Tracking issue: meta#15301 Differential Revision: D124652097
3fa5380 to
1777ad6
Compare
✅ Claude Code ReviewAuto-triggered after CI passed — reviewing commit 1777ad6 SummaryThis is a large, well-structured foundational diff (2/15 in the indirect blob GC stack) that adds indirect BlobIndex types, exact SST reference metadata with a collector/encoder/decoder, and supporting infrastructure. The design is thorough with strong integrity checks and memory accounting. The option gate appropriately prevents production use. The diff has a few medium-severity issues around CLAUDE.md guideline violations (defaulted parameters), a public API compatibility concern with the CacheEntryRole enum change, and some behavioral change documentation gaps. High-severity findings (0): No high-severity findings. Full review (click to expand)Findings🔴 HIGHNo high-severity findings. 🟡 MEDIUMM1. Defaulted parameters added to
|
| Context | Affected? | Assessment |
|---|---|---|
| WritePreparedTxnDB | No - option gate prevents production use | Safe |
| ReadOnly DB | Yes - skips ValidateOptions, checks FIFO only | Correctly handled per diff |
| Secondary Instance | Yes - same as ReadOnly | Correctly handled |
| User-defined timestamps | No direct interaction | Safe |
| MemPurge | No interaction with blob reference metadata | Safe |
| BlobDB (column family) | Yes - prerequisite enable_blob_files required | Correctly validated |
| FIFO compaction | Rejected by ValidateOptions and open-time checks | Correctly handled |
| Concurrent writers | Collector is single-threaded per table build | Safe |
| SstFileWriter | Not collected (no DB file number) | Correctly excluded |
Positive Observations
- CRC computation efficiency: Computing the CRC once and reusing it via
Crc32cCombinefor the blob record CRC avoids reading the value bytes twice. - Canonical varints: Rejecting overlong varint encodings ensures each value has exactly one encoding, preventing format ambiguity.
- Memory accounting: The collector charges all memory to the block cache with careful tracking of slabs, chunks, map nodes, and sort scratch. The
Resize()method enables efficient in-place charge updates. - Option gate pattern: Keeping the feature off in production while the stack is incomplete is disciplined incremental development.
- Designated initializers for
IndirectBlobFields: Using{.origin_file_number = ..., .original_value_offset = ...}prevents field-swap bugs. - Format validation: Decoders check counts against remaining input size before reserving memory, preventing amplification from corrupt metadata.
BlobReference::FromBlobIndexvalidates before collection: A malformed index fails the SST build rather than being silently written into metadata.
ℹ️ About this response
Generated by Claude Code.
Review methodology: claude_md/ci_review_prompt.md
Limitations:
- Claude may miss context from files not in the diff
- Large PRs may be truncated
- Always apply human judgment to AI suggestions
Commands:
/claude-review [context]— Request a code review/claude-query <question>— Ask about the PR or codebase
Summary: Pull Request resolved: meta#15540 Why the change is required: The goal of this stack is indirect blob GC, a blob GC that does not depend on SSTs: it removes garbage from blob files without rewriting any SST. Such a GC needs two things that RocksDB does not have: - A value identity that does not change when the value moves to another file. - A checked list of the blob references that each SST keeps. The list must include older internal versions of a key that the SST still keeps (for example for snapshots), and the blob columns of wide-column entities. This diff adds both. Later diffs build the census (an in-memory reverse index from generation number to the SSTs that use it), GC and repair on top of this list. This diff in one paragraph: With the new column family option enable_blob_indirection on, the blob file writer gives each new value an indirect BlobIndex (type 3, kIndirectBlob). It stores the value's stable identity (origin file, original offset) and its CRCs (an uncompressed value stores only the CRC of the uncompressed value; a compressed value also stores the CRC of its stored bytes). Also with the option on, the block-based table builder collects every blob reference that the SST keeps. It writes them as "exact reference metadata": data blocks, one reference index block, and a 72-byte descriptor table property. In this diff only the table builder writes this metadata, and only tests read it. A temporary option gate (see "option gate" under Terms) keeps the option off outside tests. Separately, wide-column entities with blob columns are now fully checked before their blob indexes are used, also with the option off. Parts of this diff: 1. Indirect BlobIndex (core): types 3 and 4 in db/blob/blob_index.h, and the blob file writer that writes type 3. 2. Exact reference metadata format and decoders (core): db/blob/blob_reference_metadata.{h,cc}. 3. Collector and table builder integration (core): BlobReferenceMetadataCollector, its block cache charge, and BlockBasedTableBuilder. 4. Option enable_blob_indirection and the option gate. Tests need it to turn the feature on. You can skip this part on a first read. 5. Wide-column validation (WideColumnSerialization::ForEachBlobIndex). The table builder uses it to give the blob columns of each entity to the collector. It also changes behavior with the option off (see New behavior). You can skip this part on a first read. 6. Small helpers: canonical varints (util/coding.h) and CacheReservationHandle::Resize(). You can skip these on a first read. How to read this diff: 1. Overview: read "Why" and "This diff in one paragraph" above, and "Terms used in this diff" right below. Then read the top comment of db/blob/blob_reference_metadata.h (terms, layout diagrams, run and entry layout, index layout). 2. Indirect BlobIndex (db/blob/blob_index.h) and canonical varints (util/coding.h GetCanonicalVarint32/64): read the overview comment and the kIndirectBlob / kIndirectFlattenedBlob code. Every later part uses these fields: origin, original offset, generation number, logical CRC and stored CRC. The new formats read varints only in the canonical form. 3. Blob file writer (db/blob/blob_file_builder.cc, blob_log_format.cc, blob_log_writer.cc): see how a new value gets a type-3 index and how its CRC is computed once. 4. Format types and decoders: the structs in db/blob/blob_reference_metadata.h, then in the .cc the constants, GapBase, DecodeEntryBody, DecodeRunEntriesAt, DecodeBlobReferenceIndex and BlobReferenceGroupDecoder. They use only BlobIndex and the canonical varints. Learn the on-disk format here, before you read the writer. 5. Collector and its cache charge: first read the new CacheEntryRole kBlobReferenceMetadataBuildingBuffer (include/rocksdb/cache.h) and CacheReservationHandle::Resize (cache/cache_reservation_manager.h). Then read BlobReferenceMetadataCollector in this order: the class comment, Add, AppendSorted, BuildAndWriteDataBlocks, EncodeIndex, FinishOnce. Then read the unsorted fallback (AppendUnsorted, SortUnsortedGroup, SortAndEncodeGroup). 6. Wide-column change (db/wide/wide_column_serialization.cc, ForEachBlobIndex and WalkV2Columns): it gives every blob index of an entity to a callback after it checks the whole entity. The table builder uses it to feed the collector. UpdateBoundaries, BlobGarbageMeter and the compaction iterator also use it, with the option on or off. 7. Option: enable_blob_indirection (include/rocksdb/advanced_options.h, options/cf_options.cc), and the option checks and gate in db/column_family.cc (ValidateOptions, TEST_SetAllowBlobIndirection), db/db_impl/db_impl_open.cc and db_impl_secondary.cc (the FIFO check for opens that skip ValidateOptions). 8. Table integration: table/block_based/block_based_table_factory.cc (the shared reservation manager), then block_based_table_builder.cc (Rep constructor, AddImpl, WriteBlobReferenceMetadata, EstimatedTailSize). See where the collector is created, fed (AddImpl(), which both Add() and TryAdd() call) and finished. block_based_table_reader.cc maps the new block name to a block type. Terms used in this diff: Stack terms: | Term | Meaning | |---|---| | origin | Blob file where the value was first written | | original offset | Value offset in the origin file | | stable identity | (origin, original offset). Never changes, even when the value moves | | generation number | The file number that a reference uses to find its value. A new value has generation number == origin. When indirect blob GC moves the value, the reference keeps its generation number, and a route sends the read to the relocation file | | indirect BlobIndex | A kIndirectBlob (type 3) or kIndirectFlattenedBlob (type 4) index. It stores the stable identity, so the value can move without an SST rewrite | | unflattened reference | Index with generation number == origin: the value was not moved, or it was moved but no compaction has rewritten its reference yet (see "flatten" in "Stack context: terms for later diffs") | | flattened reference | Index with generation number != origin (type 4). Only flatten (diff 10) writes it; this diff defines the encoding | | route | Maps an old file number (a file that GC has removed from the Version) to the relocation file that now holds its live values. One hop: a target is never itself routed. Stored in the MANIFEST (diff 4) | | relocation file | Block-based table keyed by (origin, original offset) that holds copied values. Each value is the stored bytes plus a 5-byte trailer (kSimpleGen2BlobTrailerSize) | | logical CRC | CRC32C of the uncompressed value (BlobIndex::stable_checksum()) | | stored CRC | CRC32C of the stored bytes plus the compression byte | | physical file | A file that holds value bytes: a normal blob file or a relocation file. A generation number can name a physical file, or it can be only the key of a route | | exact reference metadata | Per-SST sorted list of every blob reference the SST keeps, written when the SST is built. References with the same encoding class, generation number and origin form a group. The list is stored as data blocks, with an index of the groups. An SST that has it is an exact SST | | option gate | Until diff 14, ColumnFamilyData::ValidateOptions returns NotSupported for enable_blob_indirection on a read-write open, CreateColumnFamily and SetOptions, unless a test calls ColumnFamilyData::TEST_SetAllowBlobIndirection(true). It keeps the option off in production while the stack is not complete | Terms of this diff (all defined at the top of db/blob/blob_reference_metadata.h): | Term | Meaning | |---|---| | encoding class (class) | kDirect (a legacy kBlob or kBlobTTL index), unflattened, or flattened | | reference group (group) | All references of one SST with the same (encoding class, generation number, origin) | | run | The part of one group inside one data block | | record bytes | The bytes the value's record takes in the file named by its generation number | | gap | How far a record is from where it would be if the records were back to back. A gap of 0 means back to back | | descriptor | The 72-byte table property that points to the reference index block and holds the totals, the CF ID, the SST file number and the relocation-state fingerprint | | relocation-state fingerprint | 128-bit hash of the routes and relocation files of the Version that an SST was built from (fields route_snapshot_fingerprint_*). Written as 0 in this diff | | legacy | Written the way RocksDB writes it without blob indirection | Previous behavior: - A BlobIndex named only a physical blob file and an offset. A value could not move to another file without rewriting every SST that referenced it. - An SST had no metadata that listed its blob references. The only way to find them was to scan its user keys. - These readers get the blob indexes of a wide-column entity through ForEachBlobFileNumber: UpdateBoundaries, BlobGarbageMeter, the compaction iterator, WideColumnSerialization::RemoveBlobColumns, and the referenced_blob_bytes table property count in BlockBasedTableBuilder::Add. ForEachBlobFileNumber decoded each blob index, but it did not check the rest of the entity framing (name order, section sizes, exact input size), and it could call the callback before it found a later error. New behavior: Separate change (option on or off): - The same readers (UpdateBoundaries, BlobGarbageMeter, the compaction iterator, RemoveBlobColumns, and the referenced_blob_bytes count in BlockBasedTableBuilder::Add) use WideColumnSerialization::ForEachBlobIndex, with the option on or off. With the option on, the table builder also uses it to give the blob columns of each entity to the collector. For an entity with blob columns, it checks the full framing, then decodes and validates every blob index before any callback runs. So an entity with a malformed framing now fails the flush or compaction, even with the option off. The referenced_blob_bytes count itself does not fail the build on a malformed entity. It removes that property from the SST, as in base. Option and option gate: - This diff adds a new column family option, `enable_blob_indirection`. It is off by default, and SetOptions can change it. - The option gate (see Terms) keeps the option off outside tests until the stack is complete. ColumnFamilyData::ValidateOptions rejects the option with NotSupported on a read-write open, CreateColumnFamily and SetOptions. Read-only opens (DBImplReadOnly and CompactedDBImpl) and secondary opens (including DB::OpenAndCompact) do not run ValidateOptions; they check only the FIFO rule (see Implementation). Only tests can turn the option on, through the internal ColumnFamilyData::TEST_SetAllowBlobIndirection(). DBTestBase sets it. - When the option is on, ValidateOptions also requires enable_blob_files, the block-based table format, no blob direct write, and a compaction style other than FIFO. Blob file writer (option on): - The blob file writer encodes each new value with a stable identity (origin file, original offset) and two checksums: - the logical CRC: the CRC32C of the uncompressed value. - the stored CRC: the CRC32C of the exact stored (maybe compressed) bytes, extended with the compression byte. - For an uncompressed value the stored CRC is not written, because the reader can compute it from the logical CRC. - The checksums have no version field of their own. The BlobIndex type byte gives the format version: the two new types, 3 (kIndirectBlob) and 4 (kIndirectFlattenedBlob, see Implementation), are BlobIndex::kIndirectBlobFormatVersion (1). A later format uses new type values. - The read path does not check these checksums yet. The check starts in diff 3. Exact reference metadata (option on): 1. What: a sorted list of every blob reference that the SST keeps, in plain blob values and in the blob columns of wide-column entities. 2. Where: data blocks (target 16 KB each before compression; small blocks waste less I/O, because indirect blob GC reads a whole block to get a few groups) and one reference index block, written after the SST data blocks and before the SST tail, plus a 72-byte descriptor table property. 3. Who reads it: only tests in this diff. The census, GC and RepairDB read it in later diffs. 4. Details: the top comment of db/blob/blob_reference_metadata.h has the full format with diagrams, the record-bytes rules and the gap rule. - Not collected: SstFileWriter output (it has no DB file number) and any output whose CF is unknown (it has no CF ID). The descriptor needs both values, so these outputs stay in the legacy form. Checks on the metadata: - A repeated or overlapping reference in one group fails the SST build with Corruption. Readers check every count, order, size and total against the index and the descriptor, so corrupt metadata returns Corruption. The writer never makes a data block larger than kTargetDataBlockSize (16 KB), and readers accept up to kMaxDataBlockSize (64 KB). Details: "Readers" and "Integrity" at the top of db/blob/blob_reference_metadata.h, and BlobReferenceGroupDecoder. Collection and memory: - Each reference is encoded when it is added, in the on-disk entry encoding, into its group's chain of 512-byte chunks. Only a group that takes the unsorted fallback is encoded a second time, in Finish(). Inside a group, references normally arrive in rising offset order, so no reference is sorted; at the end only the groups are sorted. A group that gets a lower offset takes the unsorted fallback. The BlobReferenceMetadataCollector class comment explains the chunks, the chunk size and the fallback. - The collector keeps every reference of the SST in memory until the SST is finished, and charges this memory to the block cache under a new CacheEntryRole, kBlobReferenceMetadataBuildingBuffer. "Memory usage" and "Memory charge" in the BlobReferenceMetadataCollector class comment explain the parts of this memory and the charge rule. The measured amounts are under "Validation and known limitations". - There is no fixed limit on references. The collector fails with MemoryLimit above 2^20 groups or 2^20 data blocks in one SST; these are persisted format limits. It also fails with MemoryLimit when the reference index block could be larger than 64 MB (kMaxTopLevelMetadataBlockSize), which can happen from about 945K groups (64 MB / kMaxIndexEntrySize, 71 bytes per group), or from fewer groups when the SST also has many data blocks. - There is no charge when the table has no block cache or when charging for this role is disabled. Each block-based table factory holds one cache reservation manager for this role, shared by its builders and the SST summaries. If the charge fails, the SST build fails, so the flush or compaction fails with a background error. The buffers and their charge are released when the SST is finished or abandoned. Implementation: - BlobIndex adds two indirect encodings. The legacy encodings do not change. - Type 3, kIndirectBlob (unflattened, generation number == origin), stores the origin, the original offset, the stored size, the compression type, the logical CRC and, for compressed values, the stored CRC. It does not store the generation number. This saves the size of one varint file number per reference (about 3 to 4 bytes for typical file numbers; an estimate, not measured). - Type 4, kIndirectFlattenedBlob (flattened), stores the generation number first, then the same fields. Its generation number must be above the origin. The decoder rejects type 4 with generation number <= origin, so each reference has one encoding. - stored_checksum() computes the stored CRC of an uncompressed value when a caller asks for it: the logical CRC extended with the compression byte. A decode does not compute it, because most decodes (for example in compaction) never read it. - The encoders take the fields as one BlobIndex::IndirectBlobFields, set by field name, so values of the same type (file number, offset and size; the two CRCs) cannot be swapped. EncodeIndirectBlob() takes the generation number as a separate argument; EncodeUnflattenedIndirectBlob() uses the origin, so the common path cannot leave the generation number unset. - The new files blob_reference_metadata.{h,cc} define the exact reference metadata. The files also contain: - BlobReferenceMetadataCollector, the collector described above. It takes the SST identity (CF ID, SST file number and the relocation-state fingerprint) as one BlobReferenceSstIdentity, set by field name. - BlobReferenceGroupKey, the key of a group: (encoding class, generation number, origin). It has the order, the hash and the validity check that the collector map, the sort, the decoder and FindBlobReferenceGroup() use. BlobReference and BlobReferenceGroup keep their flat fields and return the key through group_key(). - The index codec and the descriptor table property. - BlobReferenceGroupDecoder, the only decoder of reference runs. It reads a group one data block at a time and holds every corruption check of a group walk. Readers that visit one group (found by a binary search of the index) use it, and so do readers that visit every block in order. - Callers in this diff: only the table builder runs the collector. The decoders (DecodeBlobReferenceIndex, FindBlobReferenceGroup, BlobReferenceGroupDecoder, ForEachBlobReference*) are used only by tests. kMaxDecodedBlobReferenceBytes is used only in compile-time checks of the count limits. The census (diff 5) and indirect blob GC (diffs 8 and 9) are their first production users. The census is also the first user of BlobReferenceSstSummary (see the last Implementation bullet). - Descriptor: the CF ID and the SST file number are the repair fields. They let RepairDB tell which column family and SST the list belongs to without a MANIFEST. The decoder rejects a descriptor whose repair fields are not set (unknown CF ID or SST file number 0). The descriptor also has a 128-bit relocation-state fingerprint (the route_snapshot_fingerprint fields). Remote compaction uses it in diff 11 to check that an SST built elsewhere used the same routes as the job that asked for it. In this diff the table builder always writes 0, and DecodeFrom() does not check it. The field is in the format now, so the descriptor keeps one format version (1) through the stack. While the option gate is on, no data in this format exists outside tests. - The collector validates each BlobIndex once, when it is added. Its Finish() writes the data blocks and then the reference index block through one block writer, and fills the descriptor, including the index handle. Destroying the collector frees its memory and then its charge, also when the build is abandoned. - BlockBasedTableBuilder creates the collector with the CF ID and SST file number from TableBuilderOptions. It passes the size of each entry's user key to the collector, so the collector can compute the exact size of each blob record. It adds the collector's EstimatedEncodedSize() to its tail size estimate: the blocks are written before the tail, but output file size planning uses this estimate. The reference blocks are written with BlockType::kUserDefinedIndex, which keeps their contents as opaque bytes. Readers read them without the block cache, so prepopulate_block_cache does not warm them into the block cache (WriteMaybeCompressedBlock() with warm_cache false). - To save CPU, compaction decodes each kTypeBlobIndex output value once, in CompactionOutputs::AddToOutput(). It passes the decode to TableBuilder::TryAdd() in the new field TableBuilderAddContext::decoded_blob_index, and to new overloads of BlobGarbageMeter::ProcessOutFlow() and FileMetaData::UpdateBoundaries(). BlockBasedTableBuilder decodes a kTypeBlobIndex value once for both the referenced_blob_bytes count and the collector, and uses the caller's decode when it has one. If the decode fails, no decode is passed, and each reader decodes the value itself and returns its usual error. This applies with the option on or off. The SST bytes are the same as when each reader decodes the value itself. - The collector converts each BlobIndex into a reference without deriving the stored CRC of an uncompressed value (ConvertBlobIndex() in db/blob/blob_reference_metadata.cc). The collector does not write that CRC, because a reader derives it again from the logical CRC. Deriving it would cost CPU for every reference of every output SST. BlobReference::FromBlobIndex() still derives it. - With the option on, the blob file builder computes the CRC32C of each stored value once. The blob log record CRC combines it with the key CRC (crc32c::Crc32cCombine). The stored CRC extends it with the compression byte. For an uncompressed value it is also the logical CRC. The bytes written do not change. - WideColumnSerialization::ForEachBlobIndex replaces ForEachBlobFileNumber. It reads the entity header and column types once. For a V1 entity (no column types and no blob columns), it returns right after the header. For a V2 entity without blob columns, it returns right after the type scan. It shares one V2 column walker (WalkV2Columns) with Deserialize(), so the entity is parsed in one pass. - Read-only opens (DBImplReadOnly, CompactedDBImpl) and secondary opens skip ValidateOptions, so they check the compaction style against the option once per column family: in DBImpl::Recover() for read-only opens, and at the start of OpenAsSecondaryImpl() for secondary opens. DB::OpenAndCompact opens through OpenAsSecondaryImpl(), so it gets the same check. - CacheReservationHandle gets Resize(), so each collector or summary can change its own part of the shared charge. - util/coding.h adds GetCanonicalVarint32/64, which reject overlong varint encodings. The new formats use them, so each value has one encoding. - BlobReferenceSstSummary is the in-memory list of an SST's groups and totals that Finish() can return. It has no caller yet: the table builder passes a null summary output. It is checked and does not change after Finish(), so the finished builder, the SST metadata and the census (its first user, in diff 5) can share it. It is in this diff because Finish() hands the group buffer and its cache charge over to the summary. Adding the summary later would mean rewriting the charge code of Finish() in diff 5, so the charge code is written and tested once, here. - sst_dump prints a "Blob reference metadata" section after the raw user collected properties of an SST that has exact metadata: the reference count, the stored value bytes, the column family ID, the SST file number when built and the index block handle. An SST without the descriptor prints nothing more. Safety boundaries and unsupported cases: - Public API: this diff adds the option `enable_blob_indirection` with the C API functions `rocksdb_options_set_enable_blob_indirection` and `rocksdb_options_get_enable_blob_indirection`. It also inserts a new CacheEntryRole value before kMisc, which changes the numeric value of kMisc and kNumCacheEntryRoles. - With the option off, new blob values and SSTs are written in the legacy form. Legacy blob indexes stay readable. - One behavior change applies with the option off too: the five readers listed under "Separate change" fully validate wide-column entities with blob columns and their blob indexes (see "Separate change" above). - Exact reference metadata has one format (version 1) with a fixed 72-byte descriptor. The decoders reject any other version as Corruption. - The metadata blocks are protected by the table's block checksum only. With checksum = kNoChecksum, corrupt metadata is detected only when it breaks the format or order checks. - The index decoder checks each decoded count against the remaining input size before it reserves memory. So a small corrupt index cannot make it allocate a large vector. - Inlined-TTL and same-file blob indexes are not collected and cannot be relocated. Encoding, count and checksum errors return an error; a bad reference is never skipped. - Option gate (see Terms): outside tests no DB writes data in the new format. The gate checks only the option value, never persisted state. Diff 14 removes it, once data written with the option stays readable through the end of the stack and the safety checks for turning the option on are in place. - Downgrading the binary after data was written with the option on is not supported. Validation and known limitations: - Collector memory, measured in a release build that drives the collector directly (bench/collector-chunk-20261111/README.md; host not recorded; 16-byte keys, 1 KB values, 62K to 1M references per SST in 1 to 16 groups): the buffers used about 8.3 to 9 bytes per reference for uncompressed values and 12.8 to 13.5 bytes for compressed values, including the chunk headers. With 1000 groups in a 62K-reference SST, they used about 14 bytes per uncompressed reference. - Chunk size, from the same measurement: for an SST with 1M references, the chunk size changed memory by less than 5% between 256 and 4096 bytes. For an SST with 62K references in 1000 groups, the partly filled chunks dominated: about 14 bytes per reference at 512 bytes, 22 at 1024 and 72 at 4096. 256 bytes used 3-4% more memory than 512 and made Finish() up to 1.6 times slower with 100 or more groups. - Reference index block size, measured in census db_bench runs (source report and host not recorded; 16-byte keys, 200-byte values, 1 MB blob files, no compression, fillrandom and then a full compaction): about 1.7 KB per SST with 32 MB write buffers and a target file size of 8 MB or 2 MB, and about 10 KB per SST with 1 MB write buffers and a target of 2 MB or 200 KB. The size is the bytes read for the index blocks divided by the number of SSTs. - Unit tests cover the BlobIndex encodings and their rejection of bad input, the metadata layout and every corruption check, the collector (sorted path, unsorted fallback, memory charge), the flush integration, the option checks and option gate, wide-column validation, and canonical varints. See the Test Plan. - The collection buffer is not spilled to disk. One SST with very many blob references needs a matching amount of charged memory: the per-reference amount given under "Collection and memory", plus up to one partly used 512-byte chunk per group. While a fallback group is sorted, it also needs 16 bytes of sort key per reference and a second copy of its entries. For a test group of 200K references with every other one out of order, the test comment estimates about 44 bytes per reference, and the test checks that the peak charge is at most 56 bytes per reference. - Flush still writes a single output file. Until diff 7, one compaction output collects every reference it gets, charged to the block cache. Diff 7 cuts compaction outputs (including L0) at a reference cap and keeps each user key's history in one output. With the option gate, a large output is reachable only through the test hook. Stack context (same in diffs 2-14, except that diffs 10-14 add the diff 10 route rule to the install line; "<== this diff" marks this diff's part): Goal: remove garbage from blob files without rewriting any SST. A value gets a stable identity when it is first written. Indirect blob GC copies the live values of whole files into a relocation file and installs routes in the MANIFEST. Readers follow the routes. SSTs are not changed. How a read finds a value (P is the relocation file that holds the value now): ``` SST entry indirect BlobIndex { generation number, origin, original offset, size, CRCs } | v Version file table: generation number -> live file, or route old file -> P | | live file route found | | v v blob file: read at original offset relocation file P (always one hop): relocation file: look up key look up key (origin, original offset) (origin, original offset) ``` Lifecycle of one value (in this example, F is the blob file where the value is first written, and P is the relocation file that GC writes): ``` write new value goes to blob file F; index: origin = F, generation number = F | census per generation number: the SSTs and references that use it; | per physical file: live bytes, ranked by garbage ratio GC job copy the live values of whole input files (stored bytes, checked | with the stored CRC) into relocation file P, keyed by | (origin, original offset) install one MANIFEST edit: add P, route each input generation number still | in use to P, re-point the routes into the inputs to P, and remove | each input (blob file retirement for a blob file, relocation | file deletion for a relocation file) flatten compaction sets generation number = P in the SSTs it rewrites | route route retirement: route deletion records remove the routes that delete no SST uses ``` Stack map: | Diff | Adds | |---|---| | 1 | Finalize blob metadata outside the DB mutex (prep) | | 2 | Indirect BlobIndex (kIndirectBlob) and exact SST reference metadata <== this diff | | 3 | Relocation file format and stable read path | | 4 | MANIFEST routes and relocation state in the Version | | 5 | Exact reverse reference census | | 6 | MANIFEST apply outside the DB mutex, with a precondition re-check | | 7 | Coarse flush metadata, per-SST reference cap, epoch runs | | 8 | Indirect blob GC building blocks | | 9 | Indirect blob GC job | | 10 | Flatten generation numbers; route retirement | | 11 | Remote compaction validation | | 12 | Dynamic options and operations | | 13 | Lost relocation files | | 14 | RepairDB rebuilds relocation state; option gate removed | | 15 | db_stress and crash test support | Stack context: terms for later diffs: | Term | Meaning | |---|---| | census | Background count, per generation number, of the SSTs and references that use it, plus live bytes per physical file (diff 5) | | indirect origin | Normal blob file written with the option on; it is the origin of unflattened references | | flatten | Compaction rewrites the generation number to the relocation file that holds the value, so old routes can go. Only flatten makes generation number != origin (diff 10) | | blob file retirement | MANIFEST record (kBlobFileRetirement) that removes a collected blob file from the Version (diff 4) | | route deletion | MANIFEST record that removes a route that no SST uses. Route retirement is the background job that writes these records (diff 10) | Tracking issue: meta#15301 Differential Revision: D124652097
Summary: Pull Request resolved: meta#15540 Why the change is required: The goal of this stack is indirect blob GC, a blob GC that does not depend on SSTs: it removes garbage from blob files without rewriting any SST. Such a GC needs two things that RocksDB does not have: - A value identity that does not change when the value moves to another file. - A checked list of the blob references that each SST keeps. The list must include older internal versions of a key that the SST still keeps (for example for snapshots), and the blob columns of wide-column entities. This diff adds both. Later diffs build the census (an in-memory reverse index from generation number to the SSTs that use it), GC and repair on top of this list. This diff in one paragraph: With the new column family option enable_blob_indirection on, the blob file writer gives each new value an indirect BlobIndex (type 3, kIndirectBlob). It stores the value's stable identity (origin file, original offset) and its CRCs (an uncompressed value stores only the CRC of the uncompressed value; a compressed value also stores the CRC of its stored bytes). Also with the option on, the block-based table builder collects every blob reference that the SST keeps. It writes them as "exact reference metadata": data blocks, one reference index block, and a 72-byte descriptor table property. In this diff only the table builder writes this metadata, and only tests read it. A temporary option gate (see "option gate" under Terms) keeps the option off outside tests. Separately, wide-column entities with blob columns are now fully checked before their blob indexes are used, also with the option off. Parts of this diff: 1. Indirect BlobIndex (core): types 3 and 4 in db/blob/blob_index.h, and the blob file writer that writes type 3. 2. Exact reference metadata format and decoders (core): db/blob/blob_reference_metadata.{h,cc}. 3. Collector and table builder integration (core): BlobReferenceMetadataCollector, its block cache charge, and BlockBasedTableBuilder. 4. Option enable_blob_indirection and the option gate. Tests need it to turn the feature on. You can skip this part on a first read. 5. Wide-column validation (WideColumnSerialization::ForEachBlobIndex). The table builder uses it to give the blob columns of each entity to the collector. It also changes behavior with the option off (see New behavior). You can skip this part on a first read. 6. Small helpers: canonical varints (util/coding.h) and CacheReservationHandle::Resize(). You can skip these on a first read. How to read this diff: 1. Overview: read "Why" and "This diff in one paragraph" above, and "Terms used in this diff" right below. Then read the top comment of db/blob/blob_reference_metadata.h (terms, layout diagrams, run and entry layout, index layout). 2. Indirect BlobIndex (db/blob/blob_index.h) and canonical varints (util/coding.h GetCanonicalVarint32/64): read the overview comment and the kIndirectBlob / kIndirectFlattenedBlob code. Every later part uses these fields: origin, original offset, generation number, logical CRC and stored CRC. The new formats read varints only in the canonical form. 3. Blob file writer (db/blob/blob_file_builder.cc, blob_log_format.cc, blob_log_writer.cc): see how a new value gets a type-3 index and how its CRC is computed once. 4. Format types and decoders: the structs in db/blob/blob_reference_metadata.h, then in the .cc the constants, GapBase, DecodeEntryBody, DecodeRunEntriesAt, DecodeBlobReferenceIndex and BlobReferenceGroupDecoder. They use only BlobIndex and the canonical varints. Learn the on-disk format here, before you read the writer. 5. Collector and its cache charge: first read the new CacheEntryRole kBlobReferenceMetadataBuildingBuffer (include/rocksdb/cache.h) and CacheReservationHandle::Resize (cache/cache_reservation_manager.h). Then read BlobReferenceMetadataCollector in this order: the class comment, Add, AppendSorted, BuildAndWriteDataBlocks, EncodeIndex, FinishOnce. Then read the unsorted fallback (AppendUnsorted, SortUnsortedGroup, SortAndEncodeGroup). 6. Wide-column change (db/wide/wide_column_serialization.cc, ForEachBlobIndex and WalkV2Columns): it gives every blob index of an entity to a callback after it checks the whole entity. The table builder uses it to feed the collector. UpdateBoundaries, BlobGarbageMeter and the compaction iterator also use it, with the option on or off. 7. Option: enable_blob_indirection (include/rocksdb/advanced_options.h, options/cf_options.cc), and the option checks and gate in db/column_family.cc (ValidateOptions, TEST_SetAllowBlobIndirection), db/db_impl/db_impl_open.cc and db_impl_secondary.cc (the FIFO check for opens that skip ValidateOptions). 8. Table integration: table/block_based/block_based_table_factory.cc (the shared reservation manager), then block_based_table_builder.cc (Rep constructor, AddImpl, WriteBlobReferenceMetadata, EstimatedTailSize). See where the collector is created, fed (AddImpl(), which both Add() and TryAdd() call) and finished. block_based_table_reader.cc maps the new block name to a block type. Terms used in this diff: Stack terms: | Term | Meaning | |---|---| | origin | Blob file where the value was first written | | original offset | Value offset in the origin file | | stable identity | (origin, original offset). Never changes, even when the value moves | | generation number | The file number that a reference uses to find its value. A new value has generation number == origin. When indirect blob GC moves the value, the reference keeps its generation number, and a route sends the read to the relocation file | | indirect BlobIndex | A kIndirectBlob (type 3) or kIndirectFlattenedBlob (type 4) index. It stores the stable identity, so the value can move without an SST rewrite | | unflattened reference | Index with generation number == origin: the value was not moved, or it was moved but no compaction has rewritten its reference yet (see "flatten" in "Stack context: terms for later diffs") | | flattened reference | Index with generation number != origin (type 4). Only flatten (diff 10) writes it; this diff defines the encoding | | route | Maps an old file number (a file that GC has removed from the Version) to the relocation file that now holds its live values. One hop: a target is never itself routed. Stored in the MANIFEST (diff 4) | | relocation file | Block-based table keyed by (origin, original offset) that holds copied values. Each value is the stored bytes plus a 5-byte trailer (kSimpleGen2BlobTrailerSize) | | logical CRC | CRC32C of the uncompressed value (BlobIndex::stable_checksum()) | | stored CRC | CRC32C of the stored bytes plus the compression byte | | physical file | A file that holds value bytes: a normal blob file or a relocation file. A generation number can name a physical file, or it can be only the key of a route | | exact reference metadata | Per-SST sorted list of every blob reference the SST keeps, written when the SST is built. References with the same encoding class, generation number and origin form a group. The list is stored as data blocks, with an index of the groups. An SST that has it is an exact SST | | option gate | Until diff 14, ColumnFamilyData::ValidateOptions returns NotSupported for enable_blob_indirection on a read-write open, CreateColumnFamily and SetOptions, unless a test calls ColumnFamilyData::TEST_SetAllowBlobIndirection(true). It keeps the option off in production while the stack is not complete | Terms of this diff (all defined at the top of db/blob/blob_reference_metadata.h): | Term | Meaning | |---|---| | encoding class (class) | kDirect (a legacy kBlob or kBlobTTL index), unflattened, or flattened | | reference group (group) | All references of one SST with the same (encoding class, generation number, origin) | | run | The part of one group inside one data block | | record bytes | The bytes the value's record takes in the file named by its generation number | | gap | How far a record is from where it would be if the records were back to back. A gap of 0 means back to back | | descriptor | The 72-byte table property that points to the reference index block and holds the totals, the CF ID, the SST file number and the relocation-state fingerprint | | relocation-state fingerprint | 128-bit hash of the routes and relocation files of the Version that an SST was built from (fields route_snapshot_fingerprint_*). Written as 0 in this diff | | legacy | Written the way RocksDB writes it without blob indirection | Previous behavior: - A BlobIndex named only a physical blob file and an offset. A value could not move to another file without rewriting every SST that referenced it. - An SST had no metadata that listed its blob references. The only way to find them was to scan its user keys. - These readers get the blob indexes of a wide-column entity through ForEachBlobFileNumber: UpdateBoundaries, BlobGarbageMeter, the compaction iterator, WideColumnSerialization::RemoveBlobColumns, and the referenced_blob_bytes table property count in BlockBasedTableBuilder::Add. ForEachBlobFileNumber decoded each blob index, but it did not check the rest of the entity framing (name order, section sizes, exact input size), and it could call the callback before it found a later error. New behavior: Separate change (option on or off): - The same readers (UpdateBoundaries, BlobGarbageMeter, the compaction iterator, RemoveBlobColumns, and the referenced_blob_bytes count in BlockBasedTableBuilder::Add) use WideColumnSerialization::ForEachBlobIndex, with the option on or off. With the option on, the table builder also uses it to give the blob columns of each entity to the collector. For an entity with blob columns, it checks the full framing, then decodes and validates every blob index before any callback runs. So an entity with a malformed framing now fails the flush or compaction, even with the option off. The referenced_blob_bytes count itself does not fail the build on a malformed entity. It removes that property from the SST, as in base. Option and option gate: - This diff adds a new column family option, `enable_blob_indirection`. It is off by default, and SetOptions can change it. - The option gate (see Terms) keeps the option off outside tests until the stack is complete. ColumnFamilyData::ValidateOptions rejects the option with NotSupported on a read-write open, CreateColumnFamily and SetOptions. Read-only opens (DBImplReadOnly and CompactedDBImpl) and secondary opens (including DB::OpenAndCompact) do not run ValidateOptions; they check only the FIFO rule (see Implementation). Only tests can turn the option on, through the internal ColumnFamilyData::TEST_SetAllowBlobIndirection(). DBTestBase sets it. - When the option is on, ValidateOptions also requires enable_blob_files, the block-based table format, no blob direct write, and a compaction style other than FIFO. Blob file writer (option on): - The blob file writer encodes each new value with a stable identity (origin file, original offset) and two checksums: - the logical CRC: the CRC32C of the uncompressed value. - the stored CRC: the CRC32C of the exact stored (maybe compressed) bytes, extended with the compression byte. - For an uncompressed value the stored CRC is not written, because the reader can compute it from the logical CRC. - The checksums have no version field of their own. The BlobIndex type byte gives the format version: the two new types, 3 (kIndirectBlob) and 4 (kIndirectFlattenedBlob, see Implementation), are BlobIndex::kIndirectBlobFormatVersion (1). A later format uses new type values. - The read path does not check these checksums yet. The check starts in diff 3. Exact reference metadata (option on): 1. What: a sorted list of every blob reference that the SST keeps, in plain blob values and in the blob columns of wide-column entities. 2. Where: data blocks (target 16 KB each before compression; small blocks waste less I/O, because indirect blob GC reads a whole block to get a few groups) and one reference index block, written after the SST data blocks and before the SST tail, plus a 72-byte descriptor table property. 3. Who reads it: only tests in this diff. The census, GC and RepairDB read it in later diffs. 4. Details: the top comment of db/blob/blob_reference_metadata.h has the full format with diagrams, the record-bytes rules and the gap rule. - Not collected: SstFileWriter output (it has no DB file number) and any output whose CF is unknown (it has no CF ID). The descriptor needs both values, so these outputs stay in the legacy form. Checks on the metadata: - A repeated or overlapping reference in one group fails the SST build with Corruption. Readers check every count, order, size and total against the index and the descriptor, so corrupt metadata returns Corruption. The writer never makes a data block larger than kTargetDataBlockSize (16 KB), and readers accept up to kMaxDataBlockSize (64 KB). Details: "Readers" and "Integrity" at the top of db/blob/blob_reference_metadata.h, and BlobReferenceGroupDecoder. Collection and memory: - Each reference is encoded when it is added, in the on-disk entry encoding, into its group's chain of 512-byte chunks. Only a group that takes the unsorted fallback is encoded a second time, in Finish(). Inside a group, references normally arrive in rising offset order, so no reference is sorted; at the end only the groups are sorted. A group that gets a lower offset takes the unsorted fallback. The BlobReferenceMetadataCollector class comment explains the chunks, the chunk size and the fallback. - The collector keeps every reference of the SST in memory until the SST is finished, and charges this memory to the block cache under a new CacheEntryRole, kBlobReferenceMetadataBuildingBuffer. "Memory usage" and "Memory charge" in the BlobReferenceMetadataCollector class comment explain the parts of this memory and the charge rule. The measured amounts are under "Validation and known limitations". - There is no fixed limit on references. The collector fails with MemoryLimit above 2^20 groups or 2^20 data blocks in one SST; these are persisted format limits. It also fails with MemoryLimit when the reference index block could be larger than 64 MB (kMaxTopLevelMetadataBlockSize), which can happen from about 945K groups (64 MB / kMaxIndexEntrySize, 71 bytes per group), or from fewer groups when the SST also has many data blocks. - There is no charge when the table has no block cache or when charging for this role is disabled. Each block-based table factory holds one cache reservation manager for this role, shared by its builders and the SST summaries. If the charge fails, the SST build fails, so the flush or compaction fails with a background error. The buffers and their charge are released when the SST is finished or abandoned. Implementation: - BlobIndex adds two indirect encodings. The legacy encodings do not change. - Type 3, kIndirectBlob (unflattened, generation number == origin), stores the origin, the original offset, the stored size, the compression type, the logical CRC and, for compressed values, the stored CRC. It does not store the generation number. This saves the size of one varint file number per reference (about 3 to 4 bytes for typical file numbers; an estimate, not measured). - Type 4, kIndirectFlattenedBlob (flattened), stores the generation number first, then the same fields. Its generation number must be above the origin. The decoder rejects type 4 with generation number <= origin, so each reference has one encoding. - stored_checksum() computes the stored CRC of an uncompressed value when a caller asks for it: the logical CRC extended with the compression byte. A decode does not compute it, because most decodes (for example in compaction) never read it. - The encoders take the fields as one BlobIndex::IndirectBlobFields, set by field name, so values of the same type (file number, offset and size; the two CRCs) cannot be swapped. EncodeIndirectBlob() takes the generation number as a separate argument; EncodeUnflattenedIndirectBlob() uses the origin, so the common path cannot leave the generation number unset. - The new files blob_reference_metadata.{h,cc} define the exact reference metadata. The files also contain: - BlobReferenceMetadataCollector, the collector described above. It takes the SST identity (CF ID, SST file number and the relocation-state fingerprint) as one BlobReferenceSstIdentity, set by field name. - BlobReferenceGroupKey, the key of a group: (encoding class, generation number, origin). It has the order, the hash and the validity check that the collector map, the sort, the decoder and FindBlobReferenceGroup() use. BlobReference and BlobReferenceGroup keep their flat fields and return the key through group_key(). - The index codec and the descriptor table property. - BlobReferenceGroupDecoder, the only decoder of reference runs. It reads a group one data block at a time and holds every corruption check of a group walk. Readers that visit one group (found by a binary search of the index) use it, and so do readers that visit every block in order. - Callers in this diff: only the table builder runs the collector. The decoders (DecodeBlobReferenceIndex, FindBlobReferenceGroup, BlobReferenceGroupDecoder, ForEachBlobReference*) are used only by tests. kMaxDecodedBlobReferenceBytes is used only in compile-time checks of the count limits. The census (diff 5) and indirect blob GC (diffs 8 and 9) are their first production users. The census is also the first user of BlobReferenceSstSummary (see the last Implementation bullet). - Descriptor: the CF ID and the SST file number are the repair fields. They let RepairDB tell which column family and SST the list belongs to without a MANIFEST. The decoder rejects a descriptor whose repair fields are not set (unknown CF ID or SST file number 0). The descriptor also has a 128-bit relocation-state fingerprint (the route_snapshot_fingerprint fields). Remote compaction uses it in diff 11 to check that an SST built elsewhere used the same routes as the job that asked for it. In this diff the table builder always writes 0, and DecodeFrom() does not check it. The field is in the format now, so the descriptor keeps one format version (1) through the stack. While the option gate is on, no data in this format exists outside tests. - The collector validates each BlobIndex once, when it is added. Its Finish() writes the data blocks and then the reference index block through one block writer, and fills the descriptor, including the index handle. Destroying the collector frees its memory and then its charge, also when the build is abandoned. - BlockBasedTableBuilder creates the collector with the CF ID and SST file number from TableBuilderOptions. It passes the size of each entry's user key to the collector, so the collector can compute the exact size of each blob record. It adds the collector's EstimatedEncodedSize() to its tail size estimate: the blocks are written before the tail, but output file size planning uses this estimate. The reference blocks are written with BlockType::kUserDefinedIndex, which keeps their contents as opaque bytes. Readers read them without the block cache, so prepopulate_block_cache does not warm them into the block cache (WriteMaybeCompressedBlock() with warm_cache false). - To save CPU, compaction decodes each kTypeBlobIndex output value once, in CompactionOutputs::AddToOutput(). It passes the decode to TableBuilder::TryAdd() in the new field TableBuilderAddContext::decoded_blob_index, and to new overloads of BlobGarbageMeter::ProcessOutFlow() and FileMetaData::UpdateBoundaries(). BlockBasedTableBuilder decodes a kTypeBlobIndex value once for both the referenced_blob_bytes count and the collector, and uses the caller's decode when it has one. If the decode fails, no decode is passed, and each reader decodes the value itself and returns its usual error. This applies with the option on or off. The SST bytes are the same as when each reader decodes the value itself. - The collector converts each BlobIndex into a reference without deriving the stored CRC of an uncompressed value (ConvertBlobIndex() in db/blob/blob_reference_metadata.cc). The collector does not write that CRC, because a reader derives it again from the logical CRC. Deriving it would cost CPU for every reference of every output SST. BlobReference::FromBlobIndex() still derives it. - With the option on, the blob file builder computes the CRC32C of each stored value once. The blob log record CRC combines it with the key CRC (crc32c::Crc32cCombine). The stored CRC extends it with the compression byte. For an uncompressed value it is also the logical CRC. The bytes written do not change. - WideColumnSerialization::ForEachBlobIndex replaces ForEachBlobFileNumber. It reads the entity header and column types once. For a V1 entity (no column types and no blob columns), it returns right after the header. For a V2 entity without blob columns, it returns right after the type scan. It shares one V2 column walker (WalkV2Columns) with Deserialize(), so the entity is parsed in one pass. - Read-only opens (DBImplReadOnly, CompactedDBImpl) and secondary opens skip ValidateOptions, so they check the compaction style against the option once per column family: in DBImpl::Recover() for read-only opens, and at the start of OpenAsSecondaryImpl() for secondary opens. DB::OpenAndCompact opens through OpenAsSecondaryImpl(), so it gets the same check. - CacheReservationHandle gets Resize(), so each collector or summary can change its own part of the shared charge. - util/coding.h adds GetCanonicalVarint32/64, which reject overlong varint encodings. The new formats use them, so each value has one encoding. - BlobReferenceSstSummary is the in-memory list of an SST's groups and totals that Finish() can return. It has no caller yet: the table builder passes a null summary output. It is checked and does not change after Finish(), so the finished builder, the SST metadata and the census (its first user, in diff 5) can share it. It is in this diff because Finish() hands the group buffer and its cache charge over to the summary. Adding the summary later would mean rewriting the charge code of Finish() in diff 5, so the charge code is written and tested once, here. - sst_dump prints a "Blob reference metadata" section after the raw user collected properties of an SST that has exact metadata: the reference count, the stored value bytes, the column family ID, the SST file number when built and the index block handle. An SST without the descriptor prints nothing more. Safety boundaries and unsupported cases: - Public API: this diff adds the option `enable_blob_indirection` with the C API functions `rocksdb_options_set_enable_blob_indirection` and `rocksdb_options_get_enable_blob_indirection`. It also inserts a new CacheEntryRole value before kMisc, which changes the numeric value of kMisc and kNumCacheEntryRoles. - With the option off, new blob values and SSTs are written in the legacy form. Legacy blob indexes stay readable. - One behavior change applies with the option off too: the five readers listed under "Separate change" fully validate wide-column entities with blob columns and their blob indexes (see "Separate change" above). - Exact reference metadata has one format (version 1) with a fixed 72-byte descriptor. The decoders reject any other version as Corruption. - The metadata blocks are protected by the table's block checksum only. With checksum = kNoChecksum, corrupt metadata is detected only when it breaks the format or order checks. - The index decoder checks each decoded count against the remaining input size before it reserves memory. So a small corrupt index cannot make it allocate a large vector. - Inlined-TTL and same-file blob indexes are not collected and cannot be relocated. Encoding, count and checksum errors return an error; a bad reference is never skipped. - Option gate (see Terms): outside tests no DB writes data in the new format. The gate checks only the option value, never persisted state. Diff 14 removes it, once data written with the option stays readable through the end of the stack and the safety checks for turning the option on are in place. - Downgrading the binary after data was written with the option on is not supported. Validation and known limitations: - Collector memory, measured in a release build that drives the collector directly (bench/collector-chunk-20261111/README.md; host not recorded; 16-byte keys, 1 KB values, 62K to 1M references per SST in 1 to 16 groups): the buffers used about 8.3 to 9 bytes per reference for uncompressed values and 12.8 to 13.5 bytes for compressed values, including the chunk headers. With 1000 groups in a 62K-reference SST, they used about 14 bytes per uncompressed reference. - Chunk size, from the same measurement: for an SST with 1M references, the chunk size changed memory by less than 5% between 256 and 4096 bytes. For an SST with 62K references in 1000 groups, the partly filled chunks dominated: about 14 bytes per reference at 512 bytes, 22 at 1024 and 72 at 4096. 256 bytes used 3-4% more memory than 512 and made Finish() up to 1.6 times slower with 100 or more groups. - Reference index block size, measured in census db_bench runs (source report and host not recorded; 16-byte keys, 200-byte values, 1 MB blob files, no compression, fillrandom and then a full compaction): about 1.7 KB per SST with 32 MB write buffers and a target file size of 8 MB or 2 MB, and about 10 KB per SST with 1 MB write buffers and a target of 2 MB or 200 KB. The size is the bytes read for the index blocks divided by the number of SSTs. - Unit tests cover the BlobIndex encodings and their rejection of bad input, the metadata layout and every corruption check, the collector (sorted path, unsorted fallback, memory charge), the flush integration, the option checks and option gate, wide-column validation, and canonical varints. See the Test Plan. - The collection buffer is not spilled to disk. One SST with very many blob references needs a matching amount of charged memory: the per-reference amount given under "Collection and memory", plus up to one partly used 512-byte chunk per group. While a fallback group is sorted, it also needs 16 bytes of sort key per reference and a second copy of its entries. For a test group of 200K references with every other one out of order, the test comment estimates about 44 bytes per reference, and the test checks that the peak charge is at most 56 bytes per reference. - Flush still writes a single output file. Until diff 7, one compaction output collects every reference it gets, charged to the block cache. Diff 7 cuts compaction outputs (including L0) at a reference cap and keeps each user key's history in one output. With the option gate, a large output is reachable only through the test hook. Stack context (same in diffs 2-14, except that diffs 10-14 add the diff 10 route rule to the install line; "<== this diff" marks this diff's part): Goal: remove garbage from blob files without rewriting any SST. A value gets a stable identity when it is first written. Indirect blob GC copies the live values of whole files into a relocation file and installs routes in the MANIFEST. Readers follow the routes. SSTs are not changed. How a read finds a value (P is the relocation file that holds the value now): ``` SST entry indirect BlobIndex { generation number, origin, original offset, size, CRCs } | v Version file table: generation number -> live file, or route old file -> P | | live file route found | | v v blob file: read at original offset relocation file P (always one hop): relocation file: look up key look up key (origin, original offset) (origin, original offset) ``` Lifecycle of one value (in this example, F is the blob file where the value is first written, and P is the relocation file that GC writes): ``` write new value goes to blob file F; index: origin = F, generation number = F | census per generation number: the SSTs and references that use it; | per physical file: live bytes, ranked by garbage ratio GC job copy the live values of whole input files (stored bytes, checked | with the stored CRC) into relocation file P, keyed by | (origin, original offset) install one MANIFEST edit: add P, route each input generation number still | in use to P, re-point the routes into the inputs to P, and remove | each input (blob file retirement for a blob file, relocation | file deletion for a relocation file) flatten compaction sets generation number = P in the SSTs it rewrites | route route retirement: route deletion records remove the routes that delete no SST uses ``` Stack map: | Diff | Adds | |---|---| | 1 | Finalize blob metadata outside the DB mutex (prep) | | 2 | Indirect BlobIndex (kIndirectBlob) and exact SST reference metadata <== this diff | | 3 | Relocation file format and stable read path | | 4 | MANIFEST routes and relocation state in the Version | | 5 | Exact reverse reference census | | 6 | MANIFEST apply outside the DB mutex, with a precondition re-check | | 7 | Coarse flush metadata, per-SST reference cap, epoch runs | | 8 | Indirect blob GC building blocks | | 9 | Indirect blob GC job | | 10 | Flatten generation numbers; route retirement | | 11 | Remote compaction validation | | 12 | Dynamic options and operations | | 13 | Lost relocation files | | 14 | RepairDB rebuilds relocation state; option gate removed | | 15 | db_stress and crash test support | Stack context: terms for later diffs: | Term | Meaning | |---|---| | census | Background count, per generation number, of the SSTs and references that use it, plus live bytes per physical file (diff 5) | | indirect origin | Normal blob file written with the option on; it is the origin of unflattened references | | flatten | Compaction rewrites the generation number to the relocation file that holds the value, so old routes can go. Only flatten makes generation number != origin (diff 10) | | blob file retirement | MANIFEST record (kBlobFileRetirement) that removes a collected blob file from the Version (diff 4) | | route deletion | MANIFEST record that removes a route that no SST uses. Route retirement is the background job that writes these records (diff 10) | Tracking issue: meta#15301 Differential Revision: D124652097
1777ad6 to
4da30c3
Compare
Summary: Pull Request resolved: meta#15540 Why the change is required: The goal of this stack is indirect blob GC, a blob GC that does not depend on SSTs: it removes garbage from blob files without rewriting any SST. Such a GC needs two things that RocksDB does not have: - A value identity that does not change when the value moves to another file. - A checked list of the blob references that each SST keeps. The list must include older internal versions of a key that the SST still keeps (for example for snapshots), and the blob columns of wide-column entities. This diff adds both. Later diffs build the census (an in-memory reverse index from generation number to the SSTs that use it), GC and repair on top of this list. This diff in one paragraph: With the new column family option enable_blob_indirection on, the blob file writer gives each new value an indirect BlobIndex (type 3, kIndirectBlob). It stores the value's stable identity (origin file, original offset) and its CRCs (an uncompressed value stores only the CRC of the uncompressed value; a compressed value also stores the CRC of its stored bytes). Also with the option on, the block-based table builder collects every blob reference that the SST keeps. It writes them as "exact reference metadata": data blocks, one reference index block, and a 72-byte descriptor table property. In this diff only the table builder writes this metadata, and only tests read it. A temporary option gate (see "option gate" under Terms) keeps the option off outside tests. Separately, wide-column entities with blob columns are now fully checked before their blob indexes are used, also with the option off. Parts of this diff: 1. Indirect BlobIndex (core): types 3 and 4 in db/blob/blob_index.h, and the blob file writer that writes type 3. 2. Exact reference metadata format and decoders (core): db/blob/blob_reference_metadata.{h,cc}. 3. Collector and table builder integration (core): BlobReferenceMetadataCollector, its block cache charge, and BlockBasedTableBuilder. 4. Option enable_blob_indirection and the option gate. Tests need it to turn the feature on. You can skip this part on a first read. 5. Wide-column validation (WideColumnSerialization::ForEachBlobIndex). The table builder uses it to give the blob columns of each entity to the collector. It also changes behavior with the option off (see New behavior). You can skip this part on a first read. 6. Small helpers: canonical varints (util/coding.h) and CacheReservationHandle::Resize(). You can skip these on a first read. How to read this diff: 1. Overview: read "Why" and "This diff in one paragraph" above, and "Terms used in this diff" right below. Then read the top comment of db/blob/blob_reference_metadata.h (terms, layout diagrams, run and entry layout, index layout). 2. Indirect BlobIndex (db/blob/blob_index.h) and canonical varints (util/coding.h GetCanonicalVarint32/64): read the overview comment and the kIndirectBlob / kIndirectFlattenedBlob code. Every later part uses these fields: origin, original offset, generation number, logical CRC and stored CRC. The new formats read varints only in the canonical form. 3. Blob file writer (db/blob/blob_file_builder.cc, blob_log_format.cc, blob_log_writer.cc): see how a new value gets a type-3 index and how its CRC is computed once. 4. Format types and decoders: the structs in db/blob/blob_reference_metadata.h, then in the .cc the constants, GapBase, DecodeEntryBody, DecodeRunEntriesAt, DecodeBlobReferenceIndex and BlobReferenceGroupDecoder. They use only BlobIndex and the canonical varints. Learn the on-disk format here, before you read the writer. 5. Collector and its cache charge: first read the new CacheEntryRole kBlobReferenceMetadataBuildingBuffer (include/rocksdb/cache.h) and CacheReservationHandle::Resize (cache/cache_reservation_manager.h). Then read BlobReferenceMetadataCollector in this order: the class comment, Add, AppendSorted, BuildAndWriteDataBlocks, EncodeIndex, FinishOnce. Then read the unsorted fallback (AppendUnsorted, SortUnsortedGroup, SortAndEncodeGroup). 6. Wide-column change (db/wide/wide_column_serialization.cc, ForEachBlobIndex and WalkV2Columns): it gives every blob index of an entity to a callback after it checks the whole entity. The table builder uses it to feed the collector. UpdateBoundaries, BlobGarbageMeter and the compaction iterator also use it, with the option on or off. 7. Option: enable_blob_indirection (include/rocksdb/advanced_options.h, options/cf_options.cc), and the option checks and gate in db/column_family.cc (ValidateOptions, TEST_SetAllowBlobIndirection), db/db_impl/db_impl_open.cc and db_impl_secondary.cc (the FIFO check for opens that skip ValidateOptions). 8. Table integration: table/block_based/block_based_table_factory.cc (the shared reservation manager), then block_based_table_builder.cc (Rep constructor, AddImpl, WriteBlobReferenceMetadata, EstimatedTailSize). See where the collector is created, fed (AddImpl(), which both Add() and TryAdd() call) and finished. block_based_table_reader.cc maps the new block name to a block type. Terms used in this diff: Stack terms: | Term | Meaning | |---|---| | origin | Blob file where the value was first written | | original offset | Value offset in the origin file | | stable identity | (origin, original offset). Never changes, even when the value moves | | generation number | The file number that a reference uses to find its value. A new value has generation number == origin. When indirect blob GC moves the value, the reference keeps its generation number, and a route sends the read to the relocation file | | indirect BlobIndex | A kIndirectBlob (type 3) or kIndirectFlattenedBlob (type 4) index. It stores the stable identity, so the value can move without an SST rewrite | | unflattened reference | Index with generation number == origin: the value was not moved, or it was moved but no compaction has rewritten its reference yet (see "flatten" in "Stack context: terms for later diffs") | | flattened reference | Index with generation number != origin (type 4). Only flatten (diff 10) writes it; this diff defines the encoding | | route | Maps an old file number (a file that GC has removed from the Version) to the relocation file that now holds its live values. One hop: a target is never itself routed. Stored in the MANIFEST (diff 4) | | relocation file | Block-based table keyed by (origin, original offset) that holds copied values. Each value is the stored bytes plus a 5-byte trailer (kSimpleGen2BlobTrailerSize) | | logical CRC | CRC32C of the uncompressed value (BlobIndex::stable_checksum()) | | stored CRC | CRC32C of the stored bytes plus the compression byte | | physical file | A file that holds value bytes: a normal blob file or a relocation file. A generation number can name a physical file, or it can be only the key of a route | | exact reference metadata | Per-SST sorted list of every blob reference the SST keeps, written when the SST is built. References with the same encoding class, generation number and origin form a group. The list is stored as data blocks, with an index of the groups. An SST that has it is an exact SST | | option gate | Until diff 14, ColumnFamilyData::ValidateOptions returns NotSupported for enable_blob_indirection on a read-write open, CreateColumnFamily and SetOptions, unless a test calls ColumnFamilyData::TEST_SetAllowBlobIndirection(true). It keeps the option off in production while the stack is not complete | Terms of this diff (all defined at the top of db/blob/blob_reference_metadata.h): | Term | Meaning | |---|---| | encoding class (class) | kDirect (a legacy kBlob or kBlobTTL index), unflattened, or flattened | | reference group (group) | All references of one SST with the same (encoding class, generation number, origin) | | run | The part of one group inside one data block | | record bytes | The bytes the value's record takes in the file named by its generation number | | gap | How far a record is from where it would be if the records were back to back. A gap of 0 means back to back | | descriptor | The 72-byte table property that points to the reference index block and holds the totals, the CF ID, the SST file number and the relocation-state fingerprint | | relocation-state fingerprint | 128-bit hash of the routes and relocation files of the Version that an SST was built from (fields route_snapshot_fingerprint_*). Written as 0 in this diff | | legacy | Written the way RocksDB writes it without blob indirection | Previous behavior: - A BlobIndex named only a physical blob file and an offset. A value could not move to another file without rewriting every SST that referenced it. - An SST had no metadata that listed its blob references. The only way to find them was to scan its user keys. - These readers get the blob indexes of a wide-column entity through ForEachBlobFileNumber: UpdateBoundaries, BlobGarbageMeter, the compaction iterator, WideColumnSerialization::RemoveBlobColumns, and the referenced_blob_bytes table property count in BlockBasedTableBuilder::Add. ForEachBlobFileNumber decoded each blob index, but it did not check the rest of the entity framing (name order, section sizes, exact input size), and it could call the callback before it found a later error. New behavior: Separate change (option on or off): - The same readers (UpdateBoundaries, BlobGarbageMeter, the compaction iterator, RemoveBlobColumns, and the referenced_blob_bytes count in BlockBasedTableBuilder::Add) use WideColumnSerialization::ForEachBlobIndex, with the option on or off. With the option on, the table builder also uses it to give the blob columns of each entity to the collector. For an entity with blob columns, it checks the full framing, then decodes and validates every blob index before any callback runs. So an entity with a malformed framing now fails the flush or compaction, even with the option off. The referenced_blob_bytes count itself does not fail the build on a malformed entity. It removes that property from the SST, as in base. Option and option gate: - This diff adds a new column family option, `enable_blob_indirection`. It is off by default, and SetOptions can change it. - The option gate (see Terms) keeps the option off outside tests until the stack is complete. ColumnFamilyData::ValidateOptions rejects the option with NotSupported on a read-write open, CreateColumnFamily and SetOptions. Read-only opens (DBImplReadOnly and CompactedDBImpl) and secondary opens (including DB::OpenAndCompact) do not run ValidateOptions; they check only the FIFO rule (see Implementation). Only tests can turn the option on, through the internal ColumnFamilyData::TEST_SetAllowBlobIndirection(). DBTestBase sets it. - When the option is on, ValidateOptions also requires enable_blob_files, the block-based table format, no blob direct write, and a compaction style other than FIFO. Blob file writer (option on): - The blob file writer encodes each new value with a stable identity (origin file, original offset) and two checksums: - the logical CRC: the CRC32C of the uncompressed value. - the stored CRC: the CRC32C of the exact stored (maybe compressed) bytes, extended with the compression byte. - For an uncompressed value the stored CRC is not written, because the reader can compute it from the logical CRC. - The checksums have no version field of their own. The BlobIndex type byte gives the format version: the two new types, 3 (kIndirectBlob) and 4 (kIndirectFlattenedBlob, see Implementation), are BlobIndex::kIndirectBlobFormatVersion (1). A later format uses new type values. - The read path does not check these checksums yet. The check starts in diff 3. Exact reference metadata (option on): 1. What: a sorted list of every blob reference that the SST keeps, in plain blob values and in the blob columns of wide-column entities. 2. Where: data blocks (target 16 KB each before compression; small blocks waste less I/O, because indirect blob GC reads a whole block to get a few groups) and one reference index block, written after the SST data blocks and before the SST tail, plus a 72-byte descriptor table property. 3. Who reads it: only tests in this diff. The census, GC and RepairDB read it in later diffs. 4. Details: the top comment of db/blob/blob_reference_metadata.h has the full format with diagrams, the record-bytes rules and the gap rule. - Not collected: SstFileWriter output (it has no DB file number) and any output whose CF is unknown (it has no CF ID). The descriptor needs both values, so these outputs stay in the legacy form. Checks on the metadata: - A repeated or overlapping reference in one group fails the SST build with Corruption. Readers check every count, order, size and total against the index and the descriptor, so corrupt metadata returns Corruption. The writer never makes a data block larger than kTargetDataBlockSize (16 KB), and readers accept up to kMaxDataBlockSize (64 KB). Details: "Readers" and "Integrity" at the top of db/blob/blob_reference_metadata.h, and BlobReferenceGroupDecoder. Collection and memory: - Each reference is encoded when it is added, in the on-disk entry encoding, into its group's chain of 512-byte chunks. Only a group that takes the unsorted fallback is encoded a second time, in Finish(). Inside a group, references normally arrive in rising offset order, so no reference is sorted; at the end only the groups are sorted. A group that gets a lower offset takes the unsorted fallback. The BlobReferenceMetadataCollector class comment explains the chunks, the chunk size and the fallback. - The collector keeps every reference of the SST in memory until the SST is finished, and charges this memory to the block cache under a new CacheEntryRole, kBlobReferenceMetadataBuildingBuffer. "Memory usage" and "Memory charge" in the BlobReferenceMetadataCollector class comment explain the parts of this memory and the charge rule. The measured amounts are under "Validation and known limitations". - There is no fixed limit on references. The collector fails with MemoryLimit above 2^20 groups or 2^20 data blocks in one SST; these are persisted format limits. It also fails with MemoryLimit when the reference index block could be larger than 64 MB (kMaxTopLevelMetadataBlockSize), which can happen from about 945K groups (64 MB / kMaxIndexEntrySize, 71 bytes per group), or from fewer groups when the SST also has many data blocks. - There is no charge when the table has no block cache or when charging for this role is disabled. Each block-based table factory holds one cache reservation manager for this role, shared by its builders and the SST summaries. If the charge fails, the SST build fails, so the flush or compaction fails with a background error. The buffers and their charge are released when the SST is finished or abandoned. Implementation: - BlobIndex adds two indirect encodings. The legacy encodings do not change. - Type 3, kIndirectBlob (unflattened, generation number == origin), stores the origin, the original offset, the stored size, the compression type, the logical CRC and, for compressed values, the stored CRC. It does not store the generation number. This saves the size of one varint file number per reference (about 3 to 4 bytes for typical file numbers; an estimate, not measured). - Type 4, kIndirectFlattenedBlob (flattened), stores the generation number first, then the same fields. Its generation number must be above the origin. The decoder rejects type 4 with generation number <= origin, so each reference has one encoding. - stored_checksum() computes the stored CRC of an uncompressed value when a caller asks for it: the logical CRC extended with the compression byte. A decode does not compute it, because most decodes (for example in compaction) never read it. - The encoders take the fields as one BlobIndex::IndirectBlobFields, set by field name, so values of the same type (file number, offset and size; the two CRCs) cannot be swapped. EncodeIndirectBlob() takes the generation number as a separate argument; EncodeUnflattenedIndirectBlob() uses the origin, so the common path cannot leave the generation number unset. - The new files blob_reference_metadata.{h,cc} define the exact reference metadata. The files also contain: - BlobReferenceMetadataCollector, the collector described above. It takes the SST identity (CF ID, SST file number and the relocation-state fingerprint) as one BlobReferenceSstIdentity, set by field name. - BlobReferenceGroupKey, the key of a group: (encoding class, generation number, origin). It has the order, the hash and the validity check that the collector map, the sort, the decoder and FindBlobReferenceGroup() use. BlobReference and BlobReferenceGroup keep their flat fields and return the key through group_key(). - The index codec and the descriptor table property. - BlobReferenceGroupDecoder, the only decoder of reference runs. It reads a group one data block at a time and holds every corruption check of a group walk. Readers that visit one group (found by a binary search of the index) use it, and so do readers that visit every block in order. - Callers in this diff: only the table builder runs the collector. The decoders (DecodeBlobReferenceIndex, FindBlobReferenceGroup, BlobReferenceGroupDecoder, ForEachBlobReference*) are used only by tests. kMaxDecodedBlobReferenceBytes is used only in compile-time checks of the count limits. The census (diff 5) and indirect blob GC (diffs 8 and 9) are their first production users. The census is also the first user of BlobReferenceSstSummary (see the last Implementation bullet). - Descriptor: the CF ID and the SST file number are the repair fields. They let RepairDB tell which column family and SST the list belongs to without a MANIFEST. The decoder rejects a descriptor whose repair fields are not set (unknown CF ID or SST file number 0). The descriptor also has a 128-bit relocation-state fingerprint (the route_snapshot_fingerprint fields). Remote compaction uses it in diff 11 to check that an SST built elsewhere used the same routes as the job that asked for it. In this diff the table builder always writes 0, and DecodeFrom() does not check it. The field is in the format now, so the descriptor keeps one format version (1) through the stack. While the option gate is on, no data in this format exists outside tests. - The collector validates each BlobIndex once, when it is added. Its Finish() writes the data blocks and then the reference index block through one block writer, and fills the descriptor, including the index handle. Destroying the collector frees its memory and then its charge, also when the build is abandoned. - BlockBasedTableBuilder creates the collector with the CF ID and SST file number from TableBuilderOptions. It passes the size of each entry's user key to the collector, so the collector can compute the exact size of each blob record. It adds the collector's EstimatedEncodedSize() to its tail size estimate: the blocks are written before the tail, but output file size planning uses this estimate. The reference blocks are written with BlockType::kUserDefinedIndex, which keeps their contents as opaque bytes. Readers read them without the block cache, so prepopulate_block_cache does not warm them into the block cache (WriteMaybeCompressedBlock() with warm_cache false). - To save CPU, compaction decodes each kTypeBlobIndex output value once, in CompactionOutputs::AddToOutput(). It passes the decode to TableBuilder::TryAdd() in the new field TableBuilderAddContext::decoded_blob_index, and to new overloads of BlobGarbageMeter::ProcessOutFlow() and FileMetaData::UpdateBoundaries(). BlockBasedTableBuilder decodes a kTypeBlobIndex value once for both the referenced_blob_bytes count and the collector, and uses the caller's decode when it has one. If the decode fails, no decode is passed, and each reader decodes the value itself and returns its usual error. This applies with the option on or off. The SST bytes are the same as when each reader decodes the value itself. - The collector converts each BlobIndex into a reference without deriving the stored CRC of an uncompressed value (ConvertBlobIndex() in db/blob/blob_reference_metadata.cc). The collector does not write that CRC, because a reader derives it again from the logical CRC. Deriving it would cost CPU for every reference of every output SST. BlobReference::FromBlobIndex() still derives it. - With the option on, the blob file builder computes the CRC32C of each stored value once. The blob log record CRC combines it with the key CRC (crc32c::Crc32cCombine). The stored CRC extends it with the compression byte. For an uncompressed value it is also the logical CRC. The bytes written do not change. - WideColumnSerialization::ForEachBlobIndex replaces ForEachBlobFileNumber. It reads the entity header and column types once. For a V1 entity (no column types and no blob columns), it returns right after the header. For a V2 entity without blob columns, it returns right after the type scan. It shares one V2 column walker (WalkV2Columns) with Deserialize(), so the entity is parsed in one pass. - Read-only opens (DBImplReadOnly, CompactedDBImpl) and secondary opens skip ValidateOptions, so they check the compaction style against the option once per column family: in DBImpl::Recover() for read-only opens, and at the start of OpenAsSecondaryImpl() for secondary opens. DB::OpenAndCompact opens through OpenAsSecondaryImpl(), so it gets the same check. - CacheReservationHandle gets Resize(), so each collector or summary can change its own part of the shared charge. - util/coding.h adds GetCanonicalVarint32/64, which reject overlong varint encodings. The new formats use them, so each value has one encoding. - BlobReferenceSstSummary is the in-memory list of an SST's groups and totals that Finish() can return. It has no caller yet: the table builder passes a null summary output. It is checked and does not change after Finish(), so the finished builder, the SST metadata and the census (its first user, in diff 5) can share it. It is in this diff because Finish() hands the group buffer and its cache charge over to the summary. Adding the summary later would mean rewriting the charge code of Finish() in diff 5, so the charge code is written and tested once, here. - sst_dump prints a "Blob reference metadata" section after the raw user collected properties of an SST that has exact metadata: the reference count, the stored value bytes, the column family ID, the SST file number when built and the index block handle. An SST without the descriptor prints nothing more. Safety boundaries and unsupported cases: - Public API: this diff adds the option `enable_blob_indirection` with the C API functions `rocksdb_options_set_enable_blob_indirection` and `rocksdb_options_get_enable_blob_indirection`. It also inserts a new CacheEntryRole value before kMisc, which changes the numeric value of kMisc and kNumCacheEntryRoles. - With the option off, new blob values and SSTs are written in the legacy form. Legacy blob indexes stay readable. - One behavior change applies with the option off too: the five readers listed under "Separate change" fully validate wide-column entities with blob columns and their blob indexes (see "Separate change" above). - Exact reference metadata has one format (version 1) with a fixed 72-byte descriptor. The decoders reject any other version as Corruption. - The metadata blocks are protected by the table's block checksum only. With checksum = kNoChecksum, corrupt metadata is detected only when it breaks the format or order checks. - The index decoder checks each decoded count against the remaining input size before it reserves memory. So a small corrupt index cannot make it allocate a large vector. - Inlined-TTL and same-file blob indexes are not collected and cannot be relocated. Encoding, count and checksum errors return an error; a bad reference is never skipped. - Option gate (see Terms): outside tests no DB writes data in the new format. The gate checks only the option value, never persisted state. Diff 14 removes it, once data written with the option stays readable through the end of the stack and the safety checks for turning the option on are in place. - Downgrading the binary after data was written with the option on is not supported. Validation and known limitations: - Collector memory, measured in a release build that drives the collector directly (bench/collector-chunk-20261111/README.md; host not recorded; 16-byte keys, 1 KB values, 62K to 1M references per SST in 1 to 16 groups): the buffers used about 8.3 to 9 bytes per reference for uncompressed values and 12.8 to 13.5 bytes for compressed values, including the chunk headers. With 1000 groups in a 62K-reference SST, they used about 14 bytes per uncompressed reference. - Chunk size, from the same measurement: for an SST with 1M references, the chunk size changed memory by less than 5% between 256 and 4096 bytes. For an SST with 62K references in 1000 groups, the partly filled chunks dominated: about 14 bytes per reference at 512 bytes, 22 at 1024 and 72 at 4096. 256 bytes used 3-4% more memory than 512 and made Finish() up to 1.6 times slower with 100 or more groups. - Reference index block size, measured in census db_bench runs (source report and host not recorded; 16-byte keys, 200-byte values, 1 MB blob files, no compression, fillrandom and then a full compaction): about 1.7 KB per SST with 32 MB write buffers and a target file size of 8 MB or 2 MB, and about 10 KB per SST with 1 MB write buffers and a target of 2 MB or 200 KB. The size is the bytes read for the index blocks divided by the number of SSTs. - Unit tests cover the BlobIndex encodings and their rejection of bad input, the metadata layout and every corruption check, the collector (sorted path, unsorted fallback, memory charge), the flush integration, the option checks and option gate, wide-column validation, and canonical varints. See the Test Plan. - The collection buffer is not spilled to disk. One SST with very many blob references needs a matching amount of charged memory: the per-reference amount given under "Collection and memory", plus up to one partly used 512-byte chunk per group. While a fallback group is sorted, it also needs 16 bytes of sort key per reference and a second copy of its entries. For a test group of 200K references with every other one out of order, the test comment estimates about 44 bytes per reference, and the test checks that the peak charge is at most 56 bytes per reference. - Flush still writes a single output file. Until diff 7, one compaction output collects every reference it gets, charged to the block cache. Diff 7 cuts compaction outputs (including L0) at a reference cap and keeps each user key's history in one output. With the option gate, a large output is reachable only through the test hook. Stack context (same in diffs 2-14, except that diffs 10-14 add the diff 10 route rule to the install line; "<== this diff" marks this diff's part): Goal: remove garbage from blob files without rewriting any SST. A value gets a stable identity when it is first written. Indirect blob GC copies the live values of whole files into a relocation file and installs routes in the MANIFEST. Readers follow the routes. SSTs are not changed. How a read finds a value (P is the relocation file that holds the value now): ``` SST entry indirect BlobIndex { generation number, origin, original offset, size, CRCs } | v Version file table: generation number -> live file, or route old file -> P | | live file route found | | v v blob file: read at original offset relocation file P (always one hop): relocation file: look up key look up key (origin, original offset) (origin, original offset) ``` Lifecycle of one value (in this example, F is the blob file where the value is first written, and P is the relocation file that GC writes): ``` write new value goes to blob file F; index: origin = F, generation number = F | census per generation number: the SSTs and references that use it; | per physical file: live bytes, ranked by garbage ratio GC job copy the live values of whole input files (stored bytes, checked | with the stored CRC) into relocation file P, keyed by | (origin, original offset) install one MANIFEST edit: add P, route each input generation number still | in use to P, re-point the routes into the inputs to P, and remove | each input (blob file retirement for a blob file, relocation | file deletion for a relocation file) flatten compaction sets generation number = P in the SSTs it rewrites | route route retirement: route deletion records remove the routes that delete no SST uses ``` Stack map: | Diff | Adds | |---|---| | 1 | Finalize blob metadata outside the DB mutex (prep) | | 2 | Indirect BlobIndex (kIndirectBlob) and exact SST reference metadata <== this diff | | 3 | Relocation file format and stable read path | | 4 | MANIFEST routes and relocation state in the Version | | 5 | Exact reverse reference census | | 6 | MANIFEST apply outside the DB mutex, with a precondition re-check | | 7 | Coarse flush metadata, per-SST reference cap, epoch runs | | 8 | Indirect blob GC building blocks | | 9 | Indirect blob GC job | | 10 | Flatten generation numbers; route retirement | | 11 | Remote compaction validation | | 12 | Dynamic options and operations | | 13 | Lost relocation files | | 14 | RepairDB rebuilds relocation state; option gate removed | | 15 | db_stress and crash test support | Stack context: terms for later diffs: | Term | Meaning | |---|---| | census | Background count, per generation number, of the SSTs and references that use it, plus live bytes per physical file (diff 5) | | indirect origin | Normal blob file written with the option on; it is the origin of unflattened references | | flatten | Compaction rewrites the generation number to the relocation file that holds the value, so old routes can go. Only flatten makes generation number != origin (diff 10) | | blob file retirement | MANIFEST record (kBlobFileRetirement) that removes a collected blob file from the Version (diff 4) | | route deletion | MANIFEST record that removes a route that no SST uses. Route retirement is the background job that writes these records (diff 10) | Tracking issue: meta#15301 Differential Revision: D124652097
Summary: Pull Request resolved: meta#15540 Why the change is required: The goal of this stack is indirect blob GC, a blob GC that does not depend on SSTs: it removes garbage from blob files without rewriting any SST. Such a GC needs two things that RocksDB does not have: - A value identity that does not change when the value moves to another file. - A checked list of the blob references that each SST keeps. The list must include older internal versions of a key that the SST still keeps (for example for snapshots), and the blob columns of wide-column entities. This diff adds both. Later diffs build the census (an in-memory reverse index from generation number to the SSTs that use it), GC and repair on top of this list. This diff in one paragraph: With the new column family option enable_blob_indirection on, the blob file writer gives each new value an indirect BlobIndex (type 3, kIndirectBlob). It stores the value's stable identity (origin file, original offset) and its CRCs (an uncompressed value stores only the CRC of the uncompressed value; a compressed value also stores the CRC of its stored bytes). Also with the option on, the block-based table builder collects every blob reference that the SST keeps. It writes them as "exact reference metadata": data blocks, one reference index block, and a 72-byte descriptor table property. In this diff only the table builder writes this metadata, and only tests read it. A temporary option gate (see "option gate" under Terms) keeps the option off outside tests. Separately, wide-column entities with blob columns are now fully checked before their blob indexes are used, also with the option off. Parts of this diff: 1. Indirect BlobIndex (core): types 3 and 4 in db/blob/blob_index.h, and the blob file writer that writes type 3. 2. Exact reference metadata format and decoders (core): db/blob/blob_reference_metadata.{h,cc}. 3. Collector and table builder integration (core): BlobReferenceMetadataCollector, its block cache charge, and BlockBasedTableBuilder. 4. Option enable_blob_indirection and the option gate. Tests need it to turn the feature on. You can skip this part on a first read. 5. Wide-column validation (WideColumnSerialization::ForEachBlobIndex). The table builder uses it to give the blob columns of each entity to the collector. It also changes behavior with the option off (see New behavior). You can skip this part on a first read. 6. Small helpers: canonical varints (util/coding.h) and CacheReservationHandle::Resize(). You can skip these on a first read. How to read this diff: 1. Overview: read "Why" and "This diff in one paragraph" above, and "Terms used in this diff" right below. Then read the top comment of db/blob/blob_reference_metadata.h (terms, layout diagrams, run and entry layout, index layout). 2. Indirect BlobIndex (db/blob/blob_index.h) and canonical varints (util/coding.h GetCanonicalVarint32/64): read the overview comment and the kIndirectBlob / kIndirectFlattenedBlob code. Every later part uses these fields: origin, original offset, generation number, logical CRC and stored CRC. The new formats read varints only in the canonical form. 3. Blob file writer (db/blob/blob_file_builder.cc, blob_log_format.cc, blob_log_writer.cc): see how a new value gets a type-3 index and how its CRC is computed once. 4. Format types and decoders: the structs in db/blob/blob_reference_metadata.h, then in the .cc the constants, GapBase, DecodeEntryBody, DecodeRunEntriesAt, DecodeBlobReferenceIndex and BlobReferenceGroupDecoder. They use only BlobIndex and the canonical varints. Learn the on-disk format here, before you read the writer. 5. Collector and its cache charge: first read the new CacheEntryRole kBlobReferenceMetadataBuildingBuffer (include/rocksdb/cache.h) and CacheReservationHandle::Resize (cache/cache_reservation_manager.h). Then read BlobReferenceMetadataCollector in this order: the class comment, Add, AppendSorted, BuildAndWriteDataBlocks, EncodeIndex, FinishOnce. Then read the unsorted fallback (AppendUnsorted, SortUnsortedGroup, SortAndEncodeGroup). 6. Wide-column change (db/wide/wide_column_serialization.cc, ForEachBlobIndex and WalkV2Columns): it gives every blob index of an entity to a callback after it checks the whole entity. The table builder uses it to feed the collector. UpdateBoundaries, BlobGarbageMeter and the compaction iterator also use it, with the option on or off. 7. Option: enable_blob_indirection (include/rocksdb/advanced_options.h, options/cf_options.cc), and the option checks and gate in db/column_family.cc (ValidateOptions, TEST_SetAllowBlobIndirection), db/db_impl/db_impl_open.cc and db_impl_secondary.cc (the FIFO check for opens that skip ValidateOptions). 8. Table integration: table/block_based/block_based_table_factory.cc (the shared reservation manager), then block_based_table_builder.cc (Rep constructor, AddImpl, WriteBlobReferenceMetadata, EstimatedTailSize). See where the collector is created, fed (AddImpl(), which both Add() and TryAdd() call) and finished. block_based_table_reader.cc maps the new block name to a block type. Terms used in this diff: Stack terms: | Term | Meaning | |---|---| | origin | Blob file where the value was first written | | original offset | Value offset in the origin file | | stable identity | (origin, original offset). Never changes, even when the value moves | | generation number | The file number that a reference uses to find its value. A new value has generation number == origin. When indirect blob GC moves the value, the reference keeps its generation number, and a route sends the read to the relocation file | | indirect BlobIndex | A kIndirectBlob (type 3) or kIndirectFlattenedBlob (type 4) index. It stores the stable identity, so the value can move without an SST rewrite | | unflattened reference | Index with generation number == origin: the value was not moved, or it was moved but no compaction has rewritten its reference yet (see "flatten" in "Stack context: terms for later diffs") | | flattened reference | Index with generation number != origin (type 4). Only flatten (diff 10) writes it; this diff defines the encoding | | route | Maps an old file number (a file that GC has removed from the Version) to the relocation file that now holds its live values. One hop: a target is never itself routed. Stored in the MANIFEST (diff 4) | | relocation file | Block-based table keyed by (origin, original offset) that holds copied values. Each value is the stored bytes plus a 5-byte trailer (kSimpleGen2BlobTrailerSize) | | logical CRC | CRC32C of the uncompressed value (BlobIndex::stable_checksum()) | | stored CRC | CRC32C of the stored bytes plus the compression byte | | physical file | A file that holds value bytes: a normal blob file or a relocation file. A generation number can name a physical file, or it can be only the key of a route | | exact reference metadata | Per-SST sorted list of every blob reference the SST keeps, written when the SST is built. References with the same encoding class, generation number and origin form a group. The list is stored as data blocks, with an index of the groups. An SST that has it is an exact SST | | option gate | Until diff 14, ColumnFamilyData::ValidateOptions returns NotSupported for enable_blob_indirection on a read-write open, CreateColumnFamily and SetOptions, unless a test calls ColumnFamilyData::TEST_SetAllowBlobIndirection(true). It keeps the option off in production while the stack is not complete | Terms of this diff (all defined at the top of db/blob/blob_reference_metadata.h): | Term | Meaning | |---|---| | encoding class (class) | kDirect (a legacy kBlob or kBlobTTL index), unflattened, or flattened | | reference group (group) | All references of one SST with the same (encoding class, generation number, origin) | | run | The part of one group inside one data block | | record bytes | The bytes the value's record takes in the file named by its generation number | | gap | How far a record is from where it would be if the records were back to back. A gap of 0 means back to back | | descriptor | The 72-byte table property that points to the reference index block and holds the totals, the CF ID, the SST file number and the relocation-state fingerprint | | relocation-state fingerprint | 128-bit hash of the routes and relocation files of the Version that an SST was built from (fields route_snapshot_fingerprint_*). Written as 0 in this diff | | legacy | Written the way RocksDB writes it without blob indirection | Previous behavior: - A BlobIndex named only a physical blob file and an offset. A value could not move to another file without rewriting every SST that referenced it. - An SST had no metadata that listed its blob references. The only way to find them was to scan its user keys. - These readers get the blob indexes of a wide-column entity through ForEachBlobFileNumber: UpdateBoundaries, BlobGarbageMeter, the compaction iterator, WideColumnSerialization::RemoveBlobColumns, and the referenced_blob_bytes table property count in BlockBasedTableBuilder::Add. ForEachBlobFileNumber decoded each blob index, but it did not check the rest of the entity framing (name order, section sizes, exact input size), and it could call the callback before it found a later error. New behavior: Separate change (option on or off): - The same readers (UpdateBoundaries, BlobGarbageMeter, the compaction iterator, RemoveBlobColumns, and the referenced_blob_bytes count in BlockBasedTableBuilder::Add) use WideColumnSerialization::ForEachBlobIndex, with the option on or off. With the option on, the table builder also uses it to give the blob columns of each entity to the collector. For an entity with blob columns, it checks the full framing, then decodes and validates every blob index before any callback runs. So an entity with a malformed framing now fails the flush or compaction, even with the option off. The referenced_blob_bytes count itself does not fail the build on a malformed entity. It removes that property from the SST, as in base. Option and option gate: - This diff adds a new column family option, `enable_blob_indirection`. It is off by default, and SetOptions can change it. - The option gate (see Terms) keeps the option off outside tests until the stack is complete. ColumnFamilyData::ValidateOptions rejects the option with NotSupported on a read-write open, CreateColumnFamily and SetOptions. Read-only opens (DBImplReadOnly and CompactedDBImpl) and secondary opens (including DB::OpenAndCompact) do not run ValidateOptions; they check only the FIFO rule (see Implementation). Only tests can turn the option on, through the internal ColumnFamilyData::TEST_SetAllowBlobIndirection(). DBTestBase sets it. - When the option is on, ValidateOptions also requires enable_blob_files, the block-based table format, no blob direct write, and a compaction style other than FIFO. Blob file writer (option on): - The blob file writer encodes each new value with a stable identity (origin file, original offset) and two checksums: - the logical CRC: the CRC32C of the uncompressed value. - the stored CRC: the CRC32C of the exact stored (maybe compressed) bytes, extended with the compression byte. - For an uncompressed value the stored CRC is not written, because the reader can compute it from the logical CRC. - The checksums have no version field of their own. The BlobIndex type byte gives the format version: the two new types, 3 (kIndirectBlob) and 4 (kIndirectFlattenedBlob, see Implementation), are BlobIndex::kIndirectBlobFormatVersion (1). A later format uses new type values. - The read path does not check these checksums yet. The check starts in diff 3. Exact reference metadata (option on): 1. What: a sorted list of every blob reference that the SST keeps, in plain blob values and in the blob columns of wide-column entities. 2. Where: data blocks (target 16 KB each before compression; small blocks waste less I/O, because indirect blob GC reads a whole block to get a few groups) and one reference index block, written after the SST data blocks and before the SST tail, plus a 72-byte descriptor table property. 3. Who reads it: only tests in this diff. The census, GC and RepairDB read it in later diffs. 4. Details: the top comment of db/blob/blob_reference_metadata.h has the full format with diagrams, the record-bytes rules and the gap rule. - Not collected: SstFileWriter output (it has no DB file number) and any output whose CF is unknown (it has no CF ID). The descriptor needs both values, so these outputs stay in the legacy form. Checks on the metadata: - A repeated or overlapping reference in one group fails the SST build with Corruption. Readers check every count, order, size and total against the index and the descriptor, so corrupt metadata returns Corruption. The writer never makes a data block larger than kTargetDataBlockSize (16 KB), and readers accept up to kMaxDataBlockSize (64 KB). Details: "Readers" and "Integrity" at the top of db/blob/blob_reference_metadata.h, and BlobReferenceGroupDecoder. Collection and memory: - Each reference is encoded when it is added, in the on-disk entry encoding, into its group's chain of 512-byte chunks. Only a group that takes the unsorted fallback is encoded a second time, in Finish(). Inside a group, references normally arrive in rising offset order, so no reference is sorted; at the end only the groups are sorted. A group that gets a lower offset takes the unsorted fallback. The BlobReferenceMetadataCollector class comment explains the chunks, the chunk size and the fallback. - The collector keeps every reference of the SST in memory until the SST is finished, and charges this memory to the block cache under a new CacheEntryRole, kBlobReferenceMetadataBuildingBuffer. "Memory usage" and "Memory charge" in the BlobReferenceMetadataCollector class comment explain the parts of this memory and the charge rule. The measured amounts are under "Validation and known limitations". - There is no fixed limit on references. The collector fails with MemoryLimit above 2^20 groups or 2^20 data blocks in one SST; these are persisted format limits. It also fails with MemoryLimit when the reference index block could be larger than 64 MB (kMaxTopLevelMetadataBlockSize), which can happen from about 945K groups (64 MB / kMaxIndexEntrySize, 71 bytes per group), or from fewer groups when the SST also has many data blocks. - There is no charge when the table has no block cache or when charging for this role is disabled. Each block-based table factory holds one cache reservation manager for this role, shared by its builders and the SST summaries. If the charge fails, the SST build fails, so the flush or compaction fails with a background error. The buffers and their charge are released when the SST is finished or abandoned. Implementation: - BlobIndex adds two indirect encodings. The legacy encodings do not change. - Type 3, kIndirectBlob (unflattened, generation number == origin), stores the origin, the original offset, the stored size, the compression type, the logical CRC and, for compressed values, the stored CRC. It does not store the generation number. This saves the size of one varint file number per reference (about 3 to 4 bytes for typical file numbers; an estimate, not measured). - Type 4, kIndirectFlattenedBlob (flattened), stores the generation number first, then the same fields. Its generation number must be above the origin. The decoder rejects type 4 with generation number <= origin, so each reference has one encoding. - stored_checksum() computes the stored CRC of an uncompressed value when a caller asks for it: the logical CRC extended with the compression byte. A decode does not compute it, because most decodes (for example in compaction) never read it. - The encoders take the fields as one BlobIndex::IndirectBlobFields, set by field name, so values of the same type (file number, offset and size; the two CRCs) cannot be swapped. EncodeIndirectBlob() takes the generation number as a separate argument; EncodeUnflattenedIndirectBlob() uses the origin, so the common path cannot leave the generation number unset. - The new files blob_reference_metadata.{h,cc} define the exact reference metadata. The files also contain: - BlobReferenceMetadataCollector, the collector described above. It takes the SST identity (CF ID, SST file number and the relocation-state fingerprint) as one BlobReferenceSstIdentity, set by field name. - BlobReferenceGroupKey, the key of a group: (encoding class, generation number, origin). It has the order, the hash and the validity check that the collector map, the sort, the decoder and FindBlobReferenceGroup() use. BlobReference and BlobReferenceGroup keep their flat fields and return the key through group_key(). - The index codec and the descriptor table property. - BlobReferenceGroupDecoder, the only decoder of reference runs. It reads a group one data block at a time and holds every corruption check of a group walk. Readers that visit one group (found by a binary search of the index) use it, and so do readers that visit every block in order. - Callers in this diff: only the table builder runs the collector. The decoders (DecodeBlobReferenceIndex, FindBlobReferenceGroup, BlobReferenceGroupDecoder, ForEachBlobReference*) are used only by tests. kMaxDecodedBlobReferenceBytes is used only in compile-time checks of the count limits. The census (diff 5) and indirect blob GC (diffs 8 and 9) are their first production users. The census is also the first user of BlobReferenceSstSummary (see the last Implementation bullet). - Descriptor: the CF ID and the SST file number are the repair fields. They let RepairDB tell which column family and SST the list belongs to without a MANIFEST. The decoder rejects a descriptor whose repair fields are not set (unknown CF ID or SST file number 0). The descriptor also has a 128-bit relocation-state fingerprint (the route_snapshot_fingerprint fields). Remote compaction uses it in diff 11 to check that an SST built elsewhere used the same routes as the job that asked for it. In this diff the table builder always writes 0, and DecodeFrom() does not check it. The field is in the format now, so the descriptor keeps one format version (1) through the stack. While the option gate is on, no data in this format exists outside tests. - The collector validates each BlobIndex once, when it is added. Its Finish() writes the data blocks and then the reference index block through one block writer, and fills the descriptor, including the index handle. Destroying the collector frees its memory and then its charge, also when the build is abandoned. - BlockBasedTableBuilder creates the collector with the CF ID and SST file number from TableBuilderOptions. It passes the size of each entry's user key to the collector, so the collector can compute the exact size of each blob record. It adds the collector's EstimatedEncodedSize() to its tail size estimate: the blocks are written before the tail, but output file size planning uses this estimate. The reference blocks are written with BlockType::kUserDefinedIndex, which keeps their contents as opaque bytes. Readers read them without the block cache, so prepopulate_block_cache does not warm them into the block cache (WriteMaybeCompressedBlock() with warm_cache false). - To save CPU, compaction decodes each kTypeBlobIndex output value once, in CompactionOutputs::AddToOutput(). It passes the decode to TableBuilder::TryAdd() in the new field TableBuilderAddContext::decoded_blob_index, and to new overloads of BlobGarbageMeter::ProcessOutFlow() and FileMetaData::UpdateBoundaries(). BlockBasedTableBuilder decodes a kTypeBlobIndex value once for both the referenced_blob_bytes count and the collector, and uses the caller's decode when it has one. If the decode fails, no decode is passed, and each reader decodes the value itself and returns its usual error. This applies with the option on or off. The SST bytes are the same as when each reader decodes the value itself. - The collector converts each BlobIndex into a reference without deriving the stored CRC of an uncompressed value (ConvertBlobIndex() in db/blob/blob_reference_metadata.cc). The collector does not write that CRC, because a reader derives it again from the logical CRC. Deriving it would cost CPU for every reference of every output SST. BlobReference::FromBlobIndex() still derives it. - With the option on, the blob file builder computes the CRC32C of each stored value once. The blob log record CRC combines it with the key CRC (crc32c::Crc32cCombine). The stored CRC extends it with the compression byte. For an uncompressed value it is also the logical CRC. The bytes written do not change. - WideColumnSerialization::ForEachBlobIndex replaces ForEachBlobFileNumber. It reads the entity header and column types once. For a V1 entity (no column types and no blob columns), it returns right after the header. For a V2 entity without blob columns, it returns right after the type scan. It shares one V2 column walker (WalkV2Columns) with Deserialize(), so the entity is parsed in one pass. - Read-only opens (DBImplReadOnly, CompactedDBImpl) and secondary opens skip ValidateOptions, so they check the compaction style against the option once per column family: in DBImpl::Recover() for read-only opens, and at the start of OpenAsSecondaryImpl() for secondary opens. DB::OpenAndCompact opens through OpenAsSecondaryImpl(), so it gets the same check. - CacheReservationHandle gets Resize(), so each collector or summary can change its own part of the shared charge. - util/coding.h adds GetCanonicalVarint32/64, which reject overlong varint encodings. The new formats use them, so each value has one encoding. - BlobReferenceSstSummary is the in-memory list of an SST's groups and totals that Finish() can return. It has no caller yet: the table builder passes a null summary output. It is checked and does not change after Finish(), so the finished builder, the SST metadata and the census (its first user, in diff 5) can share it. It is in this diff because Finish() hands the group buffer and its cache charge over to the summary. Adding the summary later would mean rewriting the charge code of Finish() in diff 5, so the charge code is written and tested once, here. - sst_dump prints a "Blob reference metadata" section after the raw user collected properties of an SST that has exact metadata: the reference count, the stored value bytes, the column family ID, the SST file number when built and the index block handle. An SST without the descriptor prints nothing more. Safety boundaries and unsupported cases: - Public API: this diff adds the option `enable_blob_indirection` with the C API functions `rocksdb_options_set_enable_blob_indirection` and `rocksdb_options_get_enable_blob_indirection`. It also inserts a new CacheEntryRole value before kMisc, which changes the numeric value of kMisc and kNumCacheEntryRoles. - With the option off, new blob values and SSTs are written in the legacy form. Legacy blob indexes stay readable. - One behavior change applies with the option off too: the five readers listed under "Separate change" fully validate wide-column entities with blob columns and their blob indexes (see "Separate change" above). - Exact reference metadata has one format (version 1) with a fixed 72-byte descriptor. The decoders reject any other version as Corruption. - The metadata blocks are protected by the table's block checksum only. With checksum = kNoChecksum, corrupt metadata is detected only when it breaks the format or order checks. - The index decoder checks each decoded count against the remaining input size before it reserves memory. So a small corrupt index cannot make it allocate a large vector. - Inlined-TTL and same-file blob indexes are not collected and cannot be relocated. Encoding, count and checksum errors return an error; a bad reference is never skipped. - Option gate (see Terms): outside tests no DB writes data in the new format. The gate checks only the option value, never persisted state. Diff 14 removes it, once data written with the option stays readable through the end of the stack and the safety checks for turning the option on are in place. - Downgrading the binary after data was written with the option on is not supported. Validation and known limitations: - Collector memory, measured in a release build that drives the collector directly (bench/collector-chunk-20261111/README.md; host not recorded; 16-byte keys, 1 KB values, 62K to 1M references per SST in 1 to 16 groups): the buffers used about 8.3 to 9 bytes per reference for uncompressed values and 12.8 to 13.5 bytes for compressed values, including the chunk headers. With 1000 groups in a 62K-reference SST, they used about 14 bytes per uncompressed reference. - Chunk size, from the same measurement: for an SST with 1M references, the chunk size changed memory by less than 5% between 256 and 4096 bytes. For an SST with 62K references in 1000 groups, the partly filled chunks dominated: about 14 bytes per reference at 512 bytes, 22 at 1024 and 72 at 4096. 256 bytes used 3-4% more memory than 512 and made Finish() up to 1.6 times slower with 100 or more groups. - Reference index block size, measured in census db_bench runs (source report and host not recorded; 16-byte keys, 200-byte values, 1 MB blob files, no compression, fillrandom and then a full compaction): about 1.7 KB per SST with 32 MB write buffers and a target file size of 8 MB or 2 MB, and about 10 KB per SST with 1 MB write buffers and a target of 2 MB or 200 KB. The size is the bytes read for the index blocks divided by the number of SSTs. - Unit tests cover the BlobIndex encodings and their rejection of bad input, the metadata layout and every corruption check, the collector (sorted path, unsorted fallback, memory charge), the flush integration, the option checks and option gate, wide-column validation, and canonical varints. See the Test Plan. - The collection buffer is not spilled to disk. One SST with very many blob references needs a matching amount of charged memory: the per-reference amount given under "Collection and memory", plus up to one partly used 512-byte chunk per group. While a fallback group is sorted, it also needs 16 bytes of sort key per reference and a second copy of its entries. For a test group of 200K references with every other one out of order, the test comment estimates about 44 bytes per reference, and the test checks that the peak charge is at most 56 bytes per reference. - Flush still writes a single output file. Until diff 7, one compaction output collects every reference it gets, charged to the block cache. Diff 7 cuts compaction outputs (including L0) at a reference cap and keeps each user key's history in one output. With the option gate, a large output is reachable only through the test hook. Stack context (same in diffs 2-14, except that diffs 10-14 add the diff 10 route rule to the install line; "<== this diff" marks this diff's part): Goal: remove garbage from blob files without rewriting any SST. A value gets a stable identity when it is first written. Indirect blob GC copies the live values of whole files into a relocation file and installs routes in the MANIFEST. Readers follow the routes. SSTs are not changed. How a read finds a value (P is the relocation file that holds the value now): ``` SST entry indirect BlobIndex { generation number, origin, original offset, size, CRCs } | v Version file table: generation number -> live file, or route old file -> P | | live file route found | | v v blob file: read at original offset relocation file P (always one hop): relocation file: look up key look up key (origin, original offset) (origin, original offset) ``` Lifecycle of one value (in this example, F is the blob file where the value is first written, and P is the relocation file that GC writes): ``` write new value goes to blob file F; index: origin = F, generation number = F | census per generation number: the SSTs and references that use it; | per physical file: live bytes, ranked by garbage ratio GC job copy the live values of whole input files (stored bytes, checked | with the stored CRC) into relocation file P, keyed by | (origin, original offset) install one MANIFEST edit: add P, route each input generation number still | in use to P, re-point the routes into the inputs to P, and remove | each input (blob file retirement for a blob file, relocation | file deletion for a relocation file) flatten compaction sets generation number = P in the SSTs it rewrites | route route retirement: route deletion records remove the routes that delete no SST uses ``` Stack map: | Diff | Adds | |---|---| | 1 | Finalize blob metadata outside the DB mutex (prep) | | 2 | Indirect BlobIndex (kIndirectBlob) and exact SST reference metadata <== this diff | | 3 | Relocation file format and stable read path | | 4 | MANIFEST routes and relocation state in the Version | | 5 | Exact reverse reference census | | 6 | MANIFEST apply outside the DB mutex, with a precondition re-check | | 7 | Coarse flush metadata, per-SST reference cap, epoch runs | | 8 | Indirect blob GC building blocks | | 9 | Indirect blob GC job | | 10 | Flatten generation numbers; route retirement | | 11 | Remote compaction validation | | 12 | Dynamic options and operations | | 13 | Lost relocation files | | 14 | RepairDB rebuilds relocation state; option gate removed | | 15 | db_stress and crash test support | Stack context: terms for later diffs: | Term | Meaning | |---|---| | census | Background count, per generation number, of the SSTs and references that use it, plus live bytes per physical file (diff 5) | | indirect origin | Normal blob file written with the option on; it is the origin of unflattened references | | flatten | Compaction rewrites the generation number to the relocation file that holds the value, so old routes can go. Only flatten makes generation number != origin (diff 10) | | blob file retirement | MANIFEST record (kBlobFileRetirement) that removes a collected blob file from the Version (diff 4) | | route deletion | MANIFEST record that removes a route that no SST uses. Route retirement is the background job that writes these records (diff 10) | Tracking issue: meta#15301 Differential Revision: D124652097
Summary: Pull Request resolved: meta#15540 Why the change is required: The goal of this stack is indirect blob GC, a blob GC that does not depend on SSTs: it removes garbage from blob files without rewriting any SST. Such a GC needs two things that RocksDB does not have: - A value identity that does not change when the value moves to another file. - A checked list of the blob references that each SST keeps. The list must include older internal versions of a key that the SST still keeps (for example for snapshots), and the blob columns of wide-column entities. This diff adds both. Later diffs build the census (an in-memory reverse index from generation number to the SSTs that use it), GC and repair on top of this list. This diff in one paragraph: With the new column family option enable_blob_indirection on, the blob file writer gives each new value an indirect BlobIndex (type 3, kIndirectBlob). It stores the value's stable identity (origin file, original offset) and its CRCs (an uncompressed value stores only the CRC of the uncompressed value; a compressed value also stores the CRC of its stored bytes). Also with the option on, the block-based table builder collects every blob reference that the SST keeps. It writes them as "exact reference metadata": data blocks, one reference index block, and a 72-byte descriptor table property. In this diff only the table builder writes this metadata, and only tests read it. A temporary option gate (see "option gate" under Terms) keeps the option off outside tests. Separately, wide-column entities with blob columns are now fully checked before their blob indexes are used, also with the option off. Parts of this diff: 1. Indirect BlobIndex (core): types 3 and 4 in db/blob/blob_index.h, and the blob file writer that writes type 3. 2. Exact reference metadata format and decoders (core): db/blob/blob_reference_metadata.{h,cc}. 3. Collector and table builder integration (core): BlobReferenceMetadataCollector, its block cache charge, and BlockBasedTableBuilder. 4. Option enable_blob_indirection and the option gate. Tests need it to turn the feature on. You can skip this part on a first read. 5. Wide-column validation (WideColumnSerialization::ForEachBlobIndex). The table builder uses it to give the blob columns of each entity to the collector. It also changes behavior with the option off (see New behavior). You can skip this part on a first read. 6. Small helpers: canonical varints (util/coding.h) and CacheReservationHandle::Resize(). You can skip these on a first read. How to read this diff: 1. Overview: read "Why" and "This diff in one paragraph" above, and "Terms used in this diff" right below. Then read the top comment of db/blob/blob_reference_metadata.h (terms, layout diagrams, run and entry layout, index layout). 2. Indirect BlobIndex (db/blob/blob_index.h) and canonical varints (util/coding.h GetCanonicalVarint32/64): read the overview comment and the kIndirectBlob / kIndirectFlattenedBlob code. Every later part uses these fields: origin, original offset, generation number, logical CRC and stored CRC. The new formats read varints only in the canonical form. 3. Blob file writer (db/blob/blob_file_builder.cc, blob_log_format.cc, blob_log_writer.cc): see how a new value gets a type-3 index and how its CRC is computed once. 4. Format types and decoders: the structs in db/blob/blob_reference_metadata.h, then in the .cc the constants, GapBase, DecodeEntryBody, DecodeRunEntriesAt, DecodeBlobReferenceIndex and BlobReferenceGroupDecoder. They use only BlobIndex and the canonical varints. Learn the on-disk format here, before you read the writer. 5. Collector and its cache charge: first read the new CacheEntryRole kBlobReferenceMetadataBuildingBuffer (include/rocksdb/cache.h) and CacheReservationHandle::Resize (cache/cache_reservation_manager.h). Then read BlobReferenceMetadataCollector in this order: the class comment, Add, AppendSorted, BuildAndWriteDataBlocks, EncodeIndex, FinishOnce. Then read the unsorted fallback (AppendUnsorted, SortUnsortedGroup, SortAndEncodeGroup). 6. Wide-column change (db/wide/wide_column_serialization.cc, ForEachBlobIndex and WalkV2Columns): it gives every blob index of an entity to a callback after it checks the whole entity. The table builder uses it to feed the collector. UpdateBoundaries, BlobGarbageMeter and the compaction iterator also use it, with the option on or off. 7. Option: enable_blob_indirection (include/rocksdb/advanced_options.h, options/cf_options.cc), and the option checks and gate in db/column_family.cc (ValidateOptions, TEST_SetAllowBlobIndirection), db/db_impl/db_impl_open.cc and db_impl_secondary.cc (the FIFO check for opens that skip ValidateOptions). 8. Table integration: table/block_based/block_based_table_factory.cc (the shared reservation manager), then block_based_table_builder.cc (Rep constructor, AddImpl, WriteBlobReferenceMetadata, EstimatedTailSize). See where the collector is created, fed (AddImpl(), which both Add() and TryAdd() call) and finished. block_based_table_reader.cc maps the new block name to a block type. Terms used in this diff: Stack terms: | Term | Meaning | |---|---| | origin | Blob file where the value was first written | | original offset | Value offset in the origin file | | stable identity | (origin, original offset). Never changes, even when the value moves | | generation number | The file number that a reference uses to find its value. A new value has generation number == origin. When indirect blob GC moves the value, the reference keeps its generation number, and a route sends the read to the relocation file | | indirect BlobIndex | A kIndirectBlob (type 3) or kIndirectFlattenedBlob (type 4) index. It stores the stable identity, so the value can move without an SST rewrite | | unflattened reference | Index with generation number == origin: the value was not moved, or it was moved but no compaction has rewritten its reference yet (see "flatten" in "Stack context: terms for later diffs") | | flattened reference | Index with generation number != origin (type 4). Only flatten (diff 10) writes it; this diff defines the encoding | | route | Maps an old file number (a file that GC has removed from the Version) to the relocation file that now holds its live values. One hop: a target is never itself routed. Stored in the MANIFEST (diff 4) | | relocation file | Block-based table keyed by (origin, original offset) that holds copied values. Each value is the stored bytes plus a 5-byte trailer (kSimpleGen2BlobTrailerSize) | | logical CRC | CRC32C of the uncompressed value (BlobIndex::stable_checksum()) | | stored CRC | CRC32C of the stored bytes plus the compression byte | | physical file | A file that holds value bytes: a normal blob file or a relocation file. A generation number can name a physical file, or it can be only the key of a route | | exact reference metadata | Per-SST sorted list of every blob reference the SST keeps, written when the SST is built. References with the same encoding class, generation number and origin form a group. The list is stored as data blocks, with an index of the groups. An SST that has it is an exact SST | | option gate | Until diff 14, ColumnFamilyData::ValidateOptions returns NotSupported for enable_blob_indirection on a read-write open, CreateColumnFamily and SetOptions, unless a test calls ColumnFamilyData::TEST_SetAllowBlobIndirection(true). It keeps the option off in production while the stack is not complete | Terms of this diff (all defined at the top of db/blob/blob_reference_metadata.h): | Term | Meaning | |---|---| | encoding class (class) | kDirect (a legacy kBlob or kBlobTTL index), unflattened, or flattened | | reference group (group) | All references of one SST with the same (encoding class, generation number, origin) | | run | The part of one group inside one data block | | record bytes | The bytes the value's record takes in the file named by its generation number | | gap | How far a record is from where it would be if the records were back to back. A gap of 0 means back to back | | descriptor | The 72-byte table property that points to the reference index block and holds the totals, the CF ID, the SST file number and the relocation-state fingerprint | | relocation-state fingerprint | 128-bit hash of the routes and relocation files of the Version that an SST was built from (fields route_snapshot_fingerprint_*). Written as 0 in this diff | | legacy | Written the way RocksDB writes it without blob indirection | Previous behavior: - A BlobIndex named only a physical blob file and an offset. A value could not move to another file without rewriting every SST that referenced it. - An SST had no metadata that listed its blob references. The only way to find them was to scan its user keys. - These readers get the blob indexes of a wide-column entity through ForEachBlobFileNumber: UpdateBoundaries, BlobGarbageMeter, the compaction iterator, WideColumnSerialization::RemoveBlobColumns, and the referenced_blob_bytes table property count in BlockBasedTableBuilder::Add. ForEachBlobFileNumber decoded each blob index, but it did not check the rest of the entity framing (name order, section sizes, exact input size), and it could call the callback before it found a later error. New behavior: Separate change (option on or off): - The same readers (UpdateBoundaries, BlobGarbageMeter, the compaction iterator, RemoveBlobColumns, and the referenced_blob_bytes count in BlockBasedTableBuilder::Add) use WideColumnSerialization::ForEachBlobIndex, with the option on or off. With the option on, the table builder also uses it to give the blob columns of each entity to the collector. For an entity with blob columns, it checks the full framing, then decodes and validates every blob index before any callback runs. So an entity with a malformed framing now fails the flush or compaction, even with the option off. The referenced_blob_bytes count itself does not fail the build on a malformed entity. It removes that property from the SST, as in base. Option and option gate: - This diff adds a new column family option, `enable_blob_indirection`. It is off by default, and SetOptions can change it. - The option gate (see Terms) keeps the option off outside tests until the stack is complete. ColumnFamilyData::ValidateOptions rejects the option with NotSupported on a read-write open, CreateColumnFamily and SetOptions. Read-only opens (DBImplReadOnly and CompactedDBImpl) and secondary opens (including DB::OpenAndCompact) do not run ValidateOptions; they check only the FIFO rule (see Implementation). Only tests can turn the option on, through the internal ColumnFamilyData::TEST_SetAllowBlobIndirection(). DBTestBase sets it. - When the option is on, ValidateOptions also requires enable_blob_files, the block-based table format, no blob direct write, and a compaction style other than FIFO. Blob file writer (option on): - The blob file writer encodes each new value with a stable identity (origin file, original offset) and two checksums: - the logical CRC: the CRC32C of the uncompressed value. - the stored CRC: the CRC32C of the exact stored (maybe compressed) bytes, extended with the compression byte. - For an uncompressed value the stored CRC is not written, because the reader can compute it from the logical CRC. - The checksums have no version field of their own. The BlobIndex type byte gives the format version: the two new types, 3 (kIndirectBlob) and 4 (kIndirectFlattenedBlob, see Implementation), are BlobIndex::kIndirectBlobFormatVersion (1). A later format uses new type values. - The read path does not check these checksums yet. The check starts in diff 3. Exact reference metadata (option on): 1. What: a sorted list of every blob reference that the SST keeps, in plain blob values and in the blob columns of wide-column entities. 2. Where: data blocks (target 16 KB each before compression; small blocks waste less I/O, because indirect blob GC reads a whole block to get a few groups) and one reference index block, written after the SST data blocks and before the SST tail, plus a 72-byte descriptor table property. 3. Who reads it: only tests in this diff. The census, GC and RepairDB read it in later diffs. 4. Details: the top comment of db/blob/blob_reference_metadata.h has the full format with diagrams, the record-bytes rules and the gap rule. - Not collected: SstFileWriter output (it has no DB file number) and any output whose CF is unknown (it has no CF ID). The descriptor needs both values, so these outputs stay in the legacy form. Checks on the metadata: - A repeated or overlapping reference in one group fails the SST build with Corruption. Readers check every count, order, size and total against the index and the descriptor, so corrupt metadata returns Corruption. The writer never makes a data block larger than kTargetDataBlockSize (16 KB), and readers accept up to kMaxDataBlockSize (64 KB). Details: "Readers" and "Integrity" at the top of db/blob/blob_reference_metadata.h, and BlobReferenceGroupDecoder. Collection and memory: - Each reference is encoded when it is added, in the on-disk entry encoding, into its group's chain of 512-byte chunks. Only a group that takes the unsorted fallback is encoded a second time, in Finish(). Inside a group, references normally arrive in rising offset order, so no reference is sorted; at the end only the groups are sorted. A group that gets a lower offset takes the unsorted fallback. The BlobReferenceMetadataCollector class comment explains the chunks, the chunk size and the fallback. - The collector keeps every reference of the SST in memory until the SST is finished, and charges this memory to the block cache under a new CacheEntryRole, kBlobReferenceMetadataBuildingBuffer. "Memory usage" and "Memory charge" in the BlobReferenceMetadataCollector class comment explain the parts of this memory and the charge rule. The measured amounts are under "Validation and known limitations". - There is no fixed limit on references. The collector fails with MemoryLimit above 2^20 groups or 2^20 data blocks in one SST; these are persisted format limits. It also fails with MemoryLimit when the reference index block could be larger than 64 MB (kMaxTopLevelMetadataBlockSize), which can happen from about 945K groups (64 MB / kMaxIndexEntrySize, 71 bytes per group), or from fewer groups when the SST also has many data blocks. - There is no charge when the table has no block cache or when charging for this role is disabled. Each block-based table factory holds one cache reservation manager for this role, shared by its builders and the SST summaries. If the charge fails, the SST build fails, so the flush or compaction fails with a background error. The buffers and their charge are released when the SST is finished or abandoned. Implementation: - BlobIndex adds two indirect encodings. The legacy encodings do not change. - Type 3, kIndirectBlob (unflattened, generation number == origin), stores the origin, the original offset, the stored size, the compression type, the logical CRC and, for compressed values, the stored CRC. It does not store the generation number. This saves the size of one varint file number per reference (about 3 to 4 bytes for typical file numbers; an estimate, not measured). - Type 4, kIndirectFlattenedBlob (flattened), stores the generation number first, then the same fields. Its generation number must be above the origin. The decoder rejects type 4 with generation number <= origin, so each reference has one encoding. - stored_checksum() computes the stored CRC of an uncompressed value when a caller asks for it: the logical CRC extended with the compression byte. A decode does not compute it, because most decodes (for example in compaction) never read it. - The encoders take the fields as one BlobIndex::IndirectBlobFields, set by field name, so values of the same type (file number, offset and size; the two CRCs) cannot be swapped. EncodeIndirectBlob() takes the generation number as a separate argument; EncodeUnflattenedIndirectBlob() uses the origin, so the common path cannot leave the generation number unset. - The new files blob_reference_metadata.{h,cc} define the exact reference metadata. The files also contain: - BlobReferenceMetadataCollector, the collector described above. It takes the SST identity (CF ID, SST file number and the relocation-state fingerprint) as one BlobReferenceSstIdentity, set by field name. - BlobReferenceGroupKey, the key of a group: (encoding class, generation number, origin). It has the order, the hash and the validity check that the collector map, the sort, the decoder and FindBlobReferenceGroup() use. BlobReference and BlobReferenceGroup keep their flat fields and return the key through group_key(). - The index codec and the descriptor table property. - BlobReferenceGroupDecoder, the only decoder of reference runs. It reads a group one data block at a time and holds every corruption check of a group walk. Readers that visit one group (found by a binary search of the index) use it, and so do readers that visit every block in order. - Callers in this diff: only the table builder runs the collector. The decoders (DecodeBlobReferenceIndex, FindBlobReferenceGroup, BlobReferenceGroupDecoder, ForEachBlobReference*) are used only by tests. kMaxDecodedBlobReferenceBytes is used only in compile-time checks of the count limits. The census (diff 5) and indirect blob GC (diffs 8 and 9) are their first production users. The census is also the first user of BlobReferenceSstSummary (see the last Implementation bullet). - Descriptor: the CF ID and the SST file number are the repair fields. They let RepairDB tell which column family and SST the list belongs to without a MANIFEST. The decoder rejects a descriptor whose repair fields are not set (unknown CF ID or SST file number 0). The descriptor also has a 128-bit relocation-state fingerprint (the route_snapshot_fingerprint fields). Remote compaction uses it in diff 11 to check that an SST built elsewhere used the same routes as the job that asked for it. In this diff the table builder always writes 0, and DecodeFrom() does not check it. The field is in the format now, so the descriptor keeps one format version (1) through the stack. While the option gate is on, no data in this format exists outside tests. - The collector validates each BlobIndex once, when it is added. Its Finish() writes the data blocks and then the reference index block through one block writer, and fills the descriptor, including the index handle. Destroying the collector frees its memory and then its charge, also when the build is abandoned. - BlockBasedTableBuilder creates the collector with the CF ID and SST file number from TableBuilderOptions. It passes the size of each entry's user key to the collector, so the collector can compute the exact size of each blob record. It adds the collector's EstimatedEncodedSize() to its tail size estimate: the blocks are written before the tail, but output file size planning uses this estimate. The reference blocks are written with BlockType::kUserDefinedIndex, which keeps their contents as opaque bytes. Readers read them without the block cache, so prepopulate_block_cache does not warm them into the block cache (WriteMaybeCompressedBlock() with warm_cache false). - To save CPU, compaction decodes each kTypeBlobIndex output value once, in CompactionOutputs::AddToOutput(). It passes the decode to TableBuilder::TryAdd() in the new field TableBuilderAddContext::decoded_blob_index, and to new overloads of BlobGarbageMeter::ProcessOutFlow() and FileMetaData::UpdateBoundaries(). BlockBasedTableBuilder decodes a kTypeBlobIndex value once for both the referenced_blob_bytes count and the collector, and uses the caller's decode when it has one. If the decode fails, no decode is passed, and each reader decodes the value itself and returns its usual error. This applies with the option on or off. The SST bytes are the same as when each reader decodes the value itself. - The collector converts each BlobIndex into a reference without deriving the stored CRC of an uncompressed value (ConvertBlobIndex() in db/blob/blob_reference_metadata.cc). The collector does not write that CRC, because a reader derives it again from the logical CRC. Deriving it would cost CPU for every reference of every output SST. BlobReference::FromBlobIndex() still derives it. - With the option on, the blob file builder computes the CRC32C of each stored value once. The blob log record CRC combines it with the key CRC (crc32c::Crc32cCombine). The stored CRC extends it with the compression byte. For an uncompressed value it is also the logical CRC. The bytes written do not change. - WideColumnSerialization::ForEachBlobIndex replaces ForEachBlobFileNumber. It reads the entity header and column types once. For a V1 entity (no column types and no blob columns), it returns right after the header. For a V2 entity without blob columns, it returns right after the type scan. It shares one V2 column walker (WalkV2Columns) with Deserialize(), so the entity is parsed in one pass. - Read-only opens (DBImplReadOnly, CompactedDBImpl) and secondary opens skip ValidateOptions, so they check the compaction style against the option once per column family: in DBImpl::Recover() for read-only opens, and at the start of OpenAsSecondaryImpl() for secondary opens. DB::OpenAndCompact opens through OpenAsSecondaryImpl(), so it gets the same check. - CacheReservationHandle gets Resize(), so each collector or summary can change its own part of the shared charge. - util/coding.h adds GetCanonicalVarint32/64, which reject overlong varint encodings. The new formats use them, so each value has one encoding. - BlobReferenceSstSummary is the in-memory list of an SST's groups and totals that Finish() can return. It has no caller yet: the table builder passes a null summary output. It is checked and does not change after Finish(), so the finished builder, the SST metadata and the census (its first user, in diff 5) can share it. It is in this diff because Finish() hands the group buffer and its cache charge over to the summary. Adding the summary later would mean rewriting the charge code of Finish() in diff 5, so the charge code is written and tested once, here. - sst_dump prints a "Blob reference metadata" section after the raw user collected properties of an SST that has exact metadata: the reference count, the stored value bytes, the column family ID, the SST file number when built and the index block handle. An SST without the descriptor prints nothing more. Safety boundaries and unsupported cases: - Public API: this diff adds the option `enable_blob_indirection` with the C API functions `rocksdb_options_set_enable_blob_indirection` and `rocksdb_options_get_enable_blob_indirection`. It also inserts a new CacheEntryRole value before kMisc, which changes the numeric value of kMisc and kNumCacheEntryRoles. - With the option off, new blob values and SSTs are written in the legacy form. Legacy blob indexes stay readable. - One behavior change applies with the option off too: the five readers listed under "Separate change" fully validate wide-column entities with blob columns and their blob indexes (see "Separate change" above). - Exact reference metadata has one format (version 1) with a fixed 72-byte descriptor. The decoders reject any other version as Corruption. - The metadata blocks are protected by the table's block checksum only. With checksum = kNoChecksum, corrupt metadata is detected only when it breaks the format or order checks. - The index decoder checks each decoded count against the remaining input size before it reserves memory. So a small corrupt index cannot make it allocate a large vector. - Inlined-TTL and same-file blob indexes are not collected and cannot be relocated. Encoding, count and checksum errors return an error; a bad reference is never skipped. - Option gate (see Terms): outside tests no DB writes data in the new format. The gate checks only the option value, never persisted state. Diff 14 removes it, once data written with the option stays readable through the end of the stack and the safety checks for turning the option on are in place. - Downgrading the binary after data was written with the option on is not supported. Validation and known limitations: - Collector memory, measured in a release build that drives the collector directly (bench/collector-chunk-20261111/README.md; host not recorded; 16-byte keys, 1 KB values, 62K to 1M references per SST in 1 to 16 groups): the buffers used about 8.3 to 9 bytes per reference for uncompressed values and 12.8 to 13.5 bytes for compressed values, including the chunk headers. With 1000 groups in a 62K-reference SST, they used about 14 bytes per uncompressed reference. - Chunk size, from the same measurement: for an SST with 1M references, the chunk size changed memory by less than 5% between 256 and 4096 bytes. For an SST with 62K references in 1000 groups, the partly filled chunks dominated: about 14 bytes per reference at 512 bytes, 22 at 1024 and 72 at 4096. 256 bytes used 3-4% more memory than 512 and made Finish() up to 1.6 times slower with 100 or more groups. - Reference index block size, measured in census db_bench runs (source report and host not recorded; 16-byte keys, 200-byte values, 1 MB blob files, no compression, fillrandom and then a full compaction): about 1.7 KB per SST with 32 MB write buffers and a target file size of 8 MB or 2 MB, and about 10 KB per SST with 1 MB write buffers and a target of 2 MB or 200 KB. The size is the bytes read for the index blocks divided by the number of SSTs. - Unit tests cover the BlobIndex encodings and their rejection of bad input, the metadata layout and every corruption check, the collector (sorted path, unsorted fallback, memory charge), the flush integration, the option checks and option gate, wide-column validation, and canonical varints. See the Test Plan. - The collection buffer is not spilled to disk. One SST with very many blob references needs a matching amount of charged memory: the per-reference amount given under "Collection and memory", plus up to one partly used 512-byte chunk per group. While a fallback group is sorted, it also needs 16 bytes of sort key per reference and a second copy of its entries. For a test group of 200K references with every other one out of order, the test comment estimates about 44 bytes per reference, and the test checks that the peak charge is at most 56 bytes per reference. - Flush still writes a single output file. Until diff 7, one compaction output collects every reference it gets, charged to the block cache. Diff 7 cuts compaction outputs (including L0) at a reference cap and keeps each user key's history in one output. With the option gate, a large output is reachable only through the test hook. Stack context (same in diffs 2-14, except that diffs 10-14 add the diff 10 route rule to the install line; "<== this diff" marks this diff's part): Goal: remove garbage from blob files without rewriting any SST. A value gets a stable identity when it is first written. Indirect blob GC copies the live values of whole files into a relocation file and installs routes in the MANIFEST. Readers follow the routes. SSTs are not changed. How a read finds a value (P is the relocation file that holds the value now): ``` SST entry indirect BlobIndex { generation number, origin, original offset, size, CRCs } | v Version file table: generation number -> live file, or route old file -> P | | live file route found | | v v blob file: read at original offset relocation file P (always one hop): relocation file: look up key look up key (origin, original offset) (origin, original offset) ``` Lifecycle of one value (in this example, F is the blob file where the value is first written, and P is the relocation file that GC writes): ``` write new value goes to blob file F; index: origin = F, generation number = F | census per generation number: the SSTs and references that use it; | per physical file: live bytes, ranked by garbage ratio GC job copy the live values of whole input files (stored bytes, checked | with the stored CRC) into relocation file P, keyed by | (origin, original offset) install one MANIFEST edit: add P, route each input generation number still | in use to P, re-point the routes into the inputs to P, and remove | each input (blob file retirement for a blob file, relocation | file deletion for a relocation file) flatten compaction sets generation number = P in the SSTs it rewrites | route route retirement: route deletion records remove the routes that delete no SST uses ``` Stack map: | Diff | Adds | |---|---| | 1 | Finalize blob metadata outside the DB mutex (prep) | | 2 | Indirect BlobIndex (kIndirectBlob) and exact SST reference metadata <== this diff | | 3 | Relocation file format and stable read path | | 4 | MANIFEST routes and relocation state in the Version | | 5 | Exact reverse reference census | | 6 | MANIFEST apply outside the DB mutex, with a precondition re-check | | 7 | Coarse flush metadata, per-SST reference cap, epoch runs | | 8 | Indirect blob GC building blocks | | 9 | Indirect blob GC job | | 10 | Flatten generation numbers; route retirement | | 11 | Remote compaction validation | | 12 | Dynamic options and operations | | 13 | Lost relocation files | | 14 | RepairDB rebuilds relocation state; option gate removed | | 15 | db_stress and crash test support | Stack context: terms for later diffs: | Term | Meaning | |---|---| | census | Background count, per generation number, of the SSTs and references that use it, plus live bytes per physical file (diff 5) | | indirect origin | Normal blob file written with the option on; it is the origin of unflattened references | | flatten | Compaction rewrites the generation number to the relocation file that holds the value, so old routes can go. Only flatten makes generation number != origin (diff 10) | | blob file retirement | MANIFEST record (kBlobFileRetirement) that removes a collected blob file from the Version (diff 4) | | route deletion | MANIFEST record that removes a route that no SST uses. Route retirement is the background job that writes these records (diff 10) | Tracking issue: meta#15301 Differential Revision: D124652097
Summary: Pull Request resolved: meta#15540 Why the change is required: The goal of this stack is indirect blob GC, a blob GC that does not depend on SSTs: it removes garbage from blob files without rewriting any SST. Such a GC needs two things that RocksDB does not have: - A value identity that does not change when the value moves to another file. - A checked list of the blob references that each SST keeps. The list must include older internal versions of a key that the SST still keeps (for example for snapshots), and the blob columns of wide-column entities. This diff adds both. Later diffs build the census (an in-memory reverse index from generation number to the SSTs that use it), GC and repair on top of this list. This diff in one paragraph: With the new column family option enable_blob_indirection on, the blob file writer gives each new value an indirect BlobIndex (type 3, kIndirectBlob). It stores the value's stable identity (origin file, original offset) and its CRCs (an uncompressed value stores only the CRC of the uncompressed value; a compressed value also stores the CRC of its stored bytes). Also with the option on, the block-based table builder collects every blob reference that the SST keeps. It writes them as "exact reference metadata": data blocks, one reference index block, and a 72-byte descriptor table property. In this diff only the table builder writes this metadata, and only tests read it. A temporary option gate (see "option gate" under Terms) keeps the option off outside tests. Separately, wide-column entities with blob columns are now fully checked before their blob indexes are used, also with the option off. Parts of this diff: 1. Indirect BlobIndex (core): types 3 and 4 in db/blob/blob_index.h, and the blob file writer that writes type 3. 2. Exact reference metadata format and decoders (core): db/blob/blob_reference_metadata.{h,cc}. 3. Collector and table builder integration (core): BlobReferenceMetadataCollector, its block cache charge, and BlockBasedTableBuilder. 4. Option enable_blob_indirection and the option gate. Tests need it to turn the feature on. You can skip this part on a first read. 5. Wide-column validation (WideColumnSerialization::ForEachBlobIndex). The table builder uses it to give the blob columns of each entity to the collector. It also changes behavior with the option off (see New behavior). You can skip this part on a first read. 6. Small helpers: canonical varints (util/coding.h) and CacheReservationHandle::Resize(). You can skip these on a first read. How to read this diff: 1. Overview: read "Why" and "This diff in one paragraph" above, and "Terms used in this diff" right below. Then read the top comment of db/blob/blob_reference_metadata.h (terms, layout diagrams, run and entry layout, index layout). 2. Indirect BlobIndex (db/blob/blob_index.h) and canonical varints (util/coding.h GetCanonicalVarint32/64): read the overview comment and the kIndirectBlob / kIndirectFlattenedBlob code. Every later part uses these fields: origin, original offset, generation number, logical CRC and stored CRC. The new formats read varints only in the canonical form. 3. Blob file writer (db/blob/blob_file_builder.cc, blob_log_format.cc, blob_log_writer.cc): see how a new value gets a type-3 index and how its CRC is computed once. 4. Format types and decoders: the structs in db/blob/blob_reference_metadata.h, then in the .cc the constants, GapBase, DecodeEntryBody, DecodeRunEntriesAt, DecodeBlobReferenceIndex and BlobReferenceGroupDecoder. They use only BlobIndex and the canonical varints. Learn the on-disk format here, before you read the writer. 5. Collector and its cache charge: first read the new CacheEntryRole kBlobReferenceMetadataBuildingBuffer (include/rocksdb/cache.h) and CacheReservationHandle::Resize (cache/cache_reservation_manager.h). Then read BlobReferenceMetadataCollector in this order: the class comment, Add, AppendSorted, BuildAndWriteDataBlocks, EncodeIndex, FinishOnce. Then read the unsorted fallback (AppendUnsorted, SortUnsortedGroup, SortAndEncodeGroup). 6. Wide-column change (db/wide/wide_column_serialization.cc, ForEachBlobIndex and WalkV2Columns): it gives every blob index of an entity to a callback after it checks the whole entity. The table builder uses it to feed the collector. UpdateBoundaries, BlobGarbageMeter and the compaction iterator also use it, with the option on or off. 7. Option: enable_blob_indirection (include/rocksdb/advanced_options.h, options/cf_options.cc), and the option checks and gate in db/column_family.cc (ValidateOptions, TEST_SetAllowBlobIndirection), db/db_impl/db_impl_open.cc and db_impl_secondary.cc (the FIFO check for opens that skip ValidateOptions). 8. Table integration: table/block_based/block_based_table_factory.cc (the shared reservation manager), then block_based_table_builder.cc (Rep constructor, AddImpl, WriteBlobReferenceMetadata, EstimatedTailSize). See where the collector is created, fed (AddImpl(), which both Add() and TryAdd() call) and finished. block_based_table_reader.cc maps the new block name to a block type. Terms used in this diff: Stack terms: | Term | Meaning | |---|---| | origin | Blob file where the value was first written | | original offset | Value offset in the origin file | | stable identity | (origin, original offset). Never changes, even when the value moves | | generation number | The file number that a reference uses to find its value. A new value has generation number == origin. When indirect blob GC moves the value, the reference keeps its generation number, and a route sends the read to the relocation file | | indirect BlobIndex | A kIndirectBlob (type 3) or kIndirectFlattenedBlob (type 4) index. It stores the stable identity, so the value can move without an SST rewrite | | unflattened reference | Index with generation number == origin: the value was not moved, or it was moved but no compaction has rewritten its reference yet (see "flatten" in "Stack context: terms for later diffs") | | flattened reference | Index with generation number != origin (type 4). Only flatten (diff 10) writes it; this diff defines the encoding | | route | Maps an old file number (a file that GC has removed from the Version) to the relocation file that now holds its live values. One hop: a target is never itself routed. Stored in the MANIFEST (diff 4) | | relocation file | Block-based table keyed by (origin, original offset) that holds copied values. Each value is the stored bytes plus a 5-byte trailer (kSimpleGen2BlobTrailerSize) | | logical CRC | CRC32C of the uncompressed value (BlobIndex::stable_checksum()) | | stored CRC | CRC32C of the stored bytes plus the compression byte | | physical file | A file that holds value bytes: a normal blob file or a relocation file. A generation number can name a physical file, or it can be only the key of a route | | exact reference metadata | Per-SST sorted list of every blob reference the SST keeps, written when the SST is built. References with the same encoding class, generation number and origin form a group. The list is stored as data blocks, with an index of the groups. An SST that has it is an exact SST | | option gate | Until diff 14, ColumnFamilyData::ValidateOptions returns NotSupported for enable_blob_indirection on a read-write open, CreateColumnFamily and SetOptions, unless a test calls ColumnFamilyData::TEST_SetAllowBlobIndirection(true). It keeps the option off in production while the stack is not complete | Terms of this diff (all defined at the top of db/blob/blob_reference_metadata.h): | Term | Meaning | |---|---| | encoding class (class) | kDirect (a legacy kBlob or kBlobTTL index), unflattened, or flattened | | reference group (group) | All references of one SST with the same (encoding class, generation number, origin) | | run | The part of one group inside one data block | | record bytes | The bytes the value's record takes in the file named by its generation number | | gap | How far a record is from where it would be if the records were back to back. A gap of 0 means back to back | | descriptor | The 72-byte table property that points to the reference index block and holds the totals, the CF ID, the SST file number and the relocation-state fingerprint | | relocation-state fingerprint | 128-bit hash of the routes and relocation files of the Version that an SST was built from (fields route_snapshot_fingerprint_*). Written as 0 in this diff | | legacy | Written the way RocksDB writes it without blob indirection | Previous behavior: - A BlobIndex named only a physical blob file and an offset. A value could not move to another file without rewriting every SST that referenced it. - An SST had no metadata that listed its blob references. The only way to find them was to scan its user keys. - These readers get the blob indexes of a wide-column entity through ForEachBlobFileNumber: UpdateBoundaries, BlobGarbageMeter, the compaction iterator, WideColumnSerialization::RemoveBlobColumns, and the referenced_blob_bytes table property count in BlockBasedTableBuilder::Add. ForEachBlobFileNumber decoded each blob index, but it did not check the rest of the entity framing (name order, section sizes, exact input size), and it could call the callback before it found a later error. New behavior: Separate change (option on or off): - The same readers (UpdateBoundaries, BlobGarbageMeter, the compaction iterator, RemoveBlobColumns, and the referenced_blob_bytes count in BlockBasedTableBuilder::Add) use WideColumnSerialization::ForEachBlobIndex, with the option on or off. With the option on, the table builder also uses it to give the blob columns of each entity to the collector. For an entity with blob columns, it checks the full framing, then decodes and validates every blob index before any callback runs. So an entity with a malformed framing now fails the flush or compaction, even with the option off. The referenced_blob_bytes count itself does not fail the build on a malformed entity. It removes that property from the SST, as in base. Option and option gate: - This diff adds a new column family option, `enable_blob_indirection`. It is off by default, and SetOptions can change it. - The option gate (see Terms) keeps the option off outside tests until the stack is complete. ColumnFamilyData::ValidateOptions rejects the option with NotSupported on a read-write open, CreateColumnFamily and SetOptions. Read-only opens (DBImplReadOnly and CompactedDBImpl) and secondary opens (including DB::OpenAndCompact) do not run ValidateOptions; they check only the FIFO rule (see Implementation). Only tests can turn the option on, through the internal ColumnFamilyData::TEST_SetAllowBlobIndirection(). DBTestBase sets it. - When the option is on, ValidateOptions also requires enable_blob_files, the block-based table format, no blob direct write, and a compaction style other than FIFO. Blob file writer (option on): - The blob file writer encodes each new value with a stable identity (origin file, original offset) and two checksums: - the logical CRC: the CRC32C of the uncompressed value. - the stored CRC: the CRC32C of the exact stored (maybe compressed) bytes, extended with the compression byte. - For an uncompressed value the stored CRC is not written, because the reader can compute it from the logical CRC. - The checksums have no version field of their own. The BlobIndex type byte gives the format version: the two new types, 3 (kIndirectBlob) and 4 (kIndirectFlattenedBlob, see Implementation), are BlobIndex::kIndirectBlobFormatVersion (1). A later format uses new type values. - The read path does not check these checksums yet. The check starts in diff 3. Exact reference metadata (option on): 1. What: a sorted list of every blob reference that the SST keeps, in plain blob values and in the blob columns of wide-column entities. 2. Where: data blocks (target 16 KB each before compression; small blocks waste less I/O, because indirect blob GC reads a whole block to get a few groups) and one reference index block, written after the SST data blocks and before the SST tail, plus a 72-byte descriptor table property. 3. Who reads it: only tests in this diff. The census, GC and RepairDB read it in later diffs. 4. Details: the top comment of db/blob/blob_reference_metadata.h has the full format with diagrams, the record-bytes rules and the gap rule. - Not collected: SstFileWriter output (it has no DB file number) and any output whose CF is unknown (it has no CF ID). The descriptor needs both values, so these outputs stay in the legacy form. Checks on the metadata: - A repeated or overlapping reference in one group fails the SST build with Corruption. Readers check every count, order, size and total against the index and the descriptor, so corrupt metadata returns Corruption. The writer never makes a data block larger than kTargetDataBlockSize (16 KB), and readers accept up to kMaxDataBlockSize (64 KB). Details: "Readers" and "Integrity" at the top of db/blob/blob_reference_metadata.h, and BlobReferenceGroupDecoder. Collection and memory: - Each reference is encoded when it is added, in the on-disk entry encoding, into its group's chain of 512-byte chunks. Only a group that takes the unsorted fallback is encoded a second time, in Finish(). Inside a group, references normally arrive in rising offset order, so no reference is sorted; at the end only the groups are sorted. A group that gets a lower offset takes the unsorted fallback. The BlobReferenceMetadataCollector class comment explains the chunks, the chunk size and the fallback. - The collector keeps every reference of the SST in memory until the SST is finished, and charges this memory to the block cache under a new CacheEntryRole, kBlobReferenceMetadataBuildingBuffer. "Memory usage" and "Memory charge" in the BlobReferenceMetadataCollector class comment explain the parts of this memory and the charge rule. The measured amounts are under "Validation and known limitations". - There is no fixed limit on references. The collector fails with MemoryLimit above 2^20 groups or 2^20 data blocks in one SST; these are persisted format limits. It also fails with MemoryLimit when the reference index block could be larger than 64 MB (kMaxTopLevelMetadataBlockSize), which can happen from about 945K groups (64 MB / kMaxIndexEntrySize, 71 bytes per group), or from fewer groups when the SST also has many data blocks. - There is no charge when the table has no block cache or when charging for this role is disabled. Each block-based table factory holds one cache reservation manager for this role, shared by its builders and the SST summaries. If the charge fails, the SST build fails, so the flush or compaction fails with a background error. The buffers and their charge are released when the SST is finished or abandoned. Implementation: - BlobIndex adds two indirect encodings. The legacy encodings do not change. - Type 3, kIndirectBlob (unflattened, generation number == origin), stores the origin, the original offset, the stored size, the compression type, the logical CRC and, for compressed values, the stored CRC. It does not store the generation number. This saves the size of one varint file number per reference (about 3 to 4 bytes for typical file numbers; an estimate, not measured). - Type 4, kIndirectFlattenedBlob (flattened), stores the generation number first, then the same fields. Its generation number must be above the origin. The decoder rejects type 4 with generation number <= origin, so each reference has one encoding. - stored_checksum() computes the stored CRC of an uncompressed value when a caller asks for it: the logical CRC extended with the compression byte. A decode does not compute it, because most decodes (for example in compaction) never read it. - The encoders take the fields as one BlobIndex::IndirectBlobFields, set by field name, so values of the same type (file number, offset and size; the two CRCs) cannot be swapped. EncodeIndirectBlob() takes the generation number as a separate argument; EncodeUnflattenedIndirectBlob() uses the origin, so the common path cannot leave the generation number unset. - The new files blob_reference_metadata.{h,cc} define the exact reference metadata. The files also contain: - BlobReferenceMetadataCollector, the collector described above. It takes the SST identity (CF ID, SST file number and the relocation-state fingerprint) as one BlobReferenceSstIdentity, set by field name. - BlobReferenceGroupKey, the key of a group: (encoding class, generation number, origin). It has the order, the hash and the validity check that the collector map, the sort, the decoder and FindBlobReferenceGroup() use. BlobReference and BlobReferenceGroup keep their flat fields and return the key through group_key(). - The index codec and the descriptor table property. - BlobReferenceGroupDecoder, the only decoder of reference runs. It reads a group one data block at a time and holds every corruption check of a group walk. Readers that visit one group (found by a binary search of the index) use it, and so do readers that visit every block in order. - Callers in this diff: only the table builder runs the collector. The decoders (DecodeBlobReferenceIndex, FindBlobReferenceGroup, BlobReferenceGroupDecoder, ForEachBlobReference*) are used only by tests. kMaxDecodedBlobReferenceBytes is used only in compile-time checks of the count limits. The census (diff 5) and indirect blob GC (diffs 8 and 9) are their first production users. The census is also the first user of BlobReferenceSstSummary (see the last Implementation bullet). - Descriptor: the CF ID and the SST file number are the repair fields. They let RepairDB tell which column family and SST the list belongs to without a MANIFEST. The decoder rejects a descriptor whose repair fields are not set (unknown CF ID or SST file number 0). The descriptor also has a 128-bit relocation-state fingerprint (the route_snapshot_fingerprint fields). Remote compaction uses it in diff 11 to check that an SST built elsewhere used the same routes as the job that asked for it. In this diff the table builder always writes 0, and DecodeFrom() does not check it. The field is in the format now, so the descriptor keeps one format version (1) through the stack. While the option gate is on, no data in this format exists outside tests. - The collector validates each BlobIndex once, when it is added. Its Finish() writes the data blocks and then the reference index block through one block writer, and fills the descriptor, including the index handle. Destroying the collector frees its memory and then its charge, also when the build is abandoned. - BlockBasedTableBuilder creates the collector with the CF ID and SST file number from TableBuilderOptions. It passes the size of each entry's user key to the collector, so the collector can compute the exact size of each blob record. It adds the collector's EstimatedEncodedSize() to its tail size estimate: the blocks are written before the tail, but output file size planning uses this estimate. The reference blocks are written with BlockType::kUserDefinedIndex, which keeps their contents as opaque bytes. Readers read them without the block cache, so prepopulate_block_cache does not warm them into the block cache (WriteMaybeCompressedBlock() with warm_cache false). - To save CPU, compaction decodes each kTypeBlobIndex output value once, in CompactionOutputs::AddToOutput(). It passes the decode to TableBuilder::TryAdd() in the new field TableBuilderAddContext::decoded_blob_index, and to new overloads of BlobGarbageMeter::ProcessOutFlow() and FileMetaData::UpdateBoundaries(). BlockBasedTableBuilder decodes a kTypeBlobIndex value once for both the referenced_blob_bytes count and the collector, and uses the caller's decode when it has one. If the decode fails, no decode is passed, and each reader decodes the value itself and returns its usual error. This applies with the option on or off. The SST bytes are the same as when each reader decodes the value itself. - The collector converts each BlobIndex into a reference without deriving the stored CRC of an uncompressed value (ConvertBlobIndex() in db/blob/blob_reference_metadata.cc). The collector does not write that CRC, because a reader derives it again from the logical CRC. Deriving it would cost CPU for every reference of every output SST. BlobReference::FromBlobIndex() still derives it. - With the option on, the blob file builder computes the CRC32C of each stored value once. The blob log record CRC combines it with the key CRC (crc32c::Crc32cCombine). The stored CRC extends it with the compression byte. For an uncompressed value it is also the logical CRC. The bytes written do not change. - WideColumnSerialization::ForEachBlobIndex replaces ForEachBlobFileNumber. It reads the entity header and column types once. For a V1 entity (no column types and no blob columns), it returns right after the header. For a V2 entity without blob columns, it returns right after the type scan. It shares one V2 column walker (WalkV2Columns) with Deserialize(), so the entity is parsed in one pass. - Read-only opens (DBImplReadOnly, CompactedDBImpl) and secondary opens skip ValidateOptions, so they check the compaction style against the option once per column family: in DBImpl::Recover() for read-only opens, and at the start of OpenAsSecondaryImpl() for secondary opens. DB::OpenAndCompact opens through OpenAsSecondaryImpl(), so it gets the same check. - CacheReservationHandle gets Resize(), so each collector or summary can change its own part of the shared charge. - util/coding.h adds GetCanonicalVarint32/64, which reject overlong varint encodings. The new formats use them, so each value has one encoding. - BlobReferenceSstSummary is the in-memory list of an SST's groups and totals that Finish() can return. It has no caller yet: the table builder passes a null summary output. It is checked and does not change after Finish(), so the finished builder, the SST metadata and the census (its first user, in diff 5) can share it. It is in this diff because Finish() hands the group buffer and its cache charge over to the summary. Adding the summary later would mean rewriting the charge code of Finish() in diff 5, so the charge code is written and tested once, here. - sst_dump prints a "Blob reference metadata" section after the raw user collected properties of an SST that has exact metadata: the reference count, the stored value bytes, the column family ID, the SST file number when built and the index block handle. An SST without the descriptor prints nothing more. Safety boundaries and unsupported cases: - Public API: this diff adds the option `enable_blob_indirection` with the C API functions `rocksdb_options_set_enable_blob_indirection` and `rocksdb_options_get_enable_blob_indirection`. It also inserts a new CacheEntryRole value before kMisc, which changes the numeric value of kMisc and kNumCacheEntryRoles. - With the option off, new blob values and SSTs are written in the legacy form. Legacy blob indexes stay readable. - One behavior change applies with the option off too: the five readers listed under "Separate change" fully validate wide-column entities with blob columns and their blob indexes (see "Separate change" above). - Exact reference metadata has one format (version 1) with a fixed 72-byte descriptor. The decoders reject any other version as Corruption. - The metadata blocks are protected by the table's block checksum only. With checksum = kNoChecksum, corrupt metadata is detected only when it breaks the format or order checks. - The index decoder checks each decoded count against the remaining input size before it reserves memory. So a small corrupt index cannot make it allocate a large vector. - Inlined-TTL and same-file blob indexes are not collected and cannot be relocated. Encoding, count and checksum errors return an error; a bad reference is never skipped. - Option gate (see Terms): outside tests no DB writes data in the new format. The gate checks only the option value, never persisted state. Diff 14 removes it, once data written with the option stays readable through the end of the stack and the safety checks for turning the option on are in place. - Downgrading the binary after data was written with the option on is not supported. Validation and known limitations: - Collector memory, measured in a release build that drives the collector directly (bench/collector-chunk-20261111/README.md; host not recorded; 16-byte keys, 1 KB values, 62K to 1M references per SST in 1 to 16 groups): the buffers used about 8.3 to 9 bytes per reference for uncompressed values and 12.8 to 13.5 bytes for compressed values, including the chunk headers. With 1000 groups in a 62K-reference SST, they used about 14 bytes per uncompressed reference. - Chunk size, from the same measurement: for an SST with 1M references, the chunk size changed memory by less than 5% between 256 and 4096 bytes. For an SST with 62K references in 1000 groups, the partly filled chunks dominated: about 14 bytes per reference at 512 bytes, 22 at 1024 and 72 at 4096. 256 bytes used 3-4% more memory than 512 and made Finish() up to 1.6 times slower with 100 or more groups. - Reference index block size, measured in census db_bench runs (source report and host not recorded; 16-byte keys, 200-byte values, 1 MB blob files, no compression, fillrandom and then a full compaction): about 1.7 KB per SST with 32 MB write buffers and a target file size of 8 MB or 2 MB, and about 10 KB per SST with 1 MB write buffers and a target of 2 MB or 200 KB. The size is the bytes read for the index blocks divided by the number of SSTs. - Unit tests cover the BlobIndex encodings and their rejection of bad input, the metadata layout and every corruption check, the collector (sorted path, unsorted fallback, memory charge), the flush integration, the option checks and option gate, wide-column validation, and canonical varints. See the Test Plan. - The collection buffer is not spilled to disk. One SST with very many blob references needs a matching amount of charged memory: the per-reference amount given under "Collection and memory", plus up to one partly used 512-byte chunk per group. While a fallback group is sorted, it also needs 16 bytes of sort key per reference and a second copy of its entries. For a test group of 200K references with every other one out of order, the test comment estimates about 44 bytes per reference, and the test checks that the peak charge is at most 56 bytes per reference. - Flush still writes a single output file. Until diff 7, one compaction output collects every reference it gets, charged to the block cache. Diff 7 cuts compaction outputs (including L0) at a reference cap and keeps each user key's history in one output. With the option gate, a large output is reachable only through the test hook. Stack context (same in diffs 2-14, except that diffs 10-14 add the diff 10 route rule to the install line; "<== this diff" marks this diff's part): Goal: remove garbage from blob files without rewriting any SST. A value gets a stable identity when it is first written. Indirect blob GC copies the live values of whole files into a relocation file and installs routes in the MANIFEST. Readers follow the routes. SSTs are not changed. How a read finds a value (P is the relocation file that holds the value now): ``` SST entry indirect BlobIndex { generation number, origin, original offset, size, CRCs } | v Version file table: generation number -> live file, or route old file -> P | | live file route found | | v v blob file: read at original offset relocation file P (always one hop): relocation file: look up key look up key (origin, original offset) (origin, original offset) ``` Lifecycle of one value (in this example, F is the blob file where the value is first written, and P is the relocation file that GC writes): ``` write new value goes to blob file F; index: origin = F, generation number = F | census per generation number: the SSTs and references that use it; | per physical file: live bytes, ranked by garbage ratio GC job copy the live values of whole input files (stored bytes, checked | with the stored CRC) into relocation file P, keyed by | (origin, original offset) install one MANIFEST edit: add P, route each input generation number still | in use to P, re-point the routes into the inputs to P, and remove | each input (blob file retirement for a blob file, relocation | file deletion for a relocation file) flatten compaction sets generation number = P in the SSTs it rewrites | route route retirement: route deletion records remove the routes that delete no SST uses ``` Stack map: | Diff | Adds | |---|---| | 1 | Finalize blob metadata outside the DB mutex (prep) | | 2 | Indirect BlobIndex (kIndirectBlob) and exact SST reference metadata <== this diff | | 3 | Relocation file format and stable read path | | 4 | MANIFEST routes and relocation state in the Version | | 5 | Exact reverse reference census | | 6 | MANIFEST apply outside the DB mutex, with a precondition re-check | | 7 | Coarse flush metadata, per-SST reference cap, epoch runs | | 8 | Indirect blob GC building blocks | | 9 | Indirect blob GC job | | 10 | Flatten generation numbers; route retirement | | 11 | Remote compaction validation | | 12 | Dynamic options and operations | | 13 | Lost relocation files | | 14 | RepairDB rebuilds relocation state; option gate removed | | 15 | db_stress and crash test support | Stack context: terms for later diffs: | Term | Meaning | |---|---| | census | Background count, per generation number, of the SSTs and references that use it, plus live bytes per physical file (diff 5) | | indirect origin | Normal blob file written with the option on; it is the origin of unflattened references | | flatten | Compaction rewrites the generation number to the relocation file that holds the value, so old routes can go. Only flatten makes generation number != origin (diff 10) | | blob file retirement | MANIFEST record (kBlobFileRetirement) that removes a collected blob file from the Version (diff 4) | | route deletion | MANIFEST record that removes a route that no SST uses. Route retirement is the background job that writes these records (diff 10) | Tracking issue: meta#15301 Differential Revision: D124652097
Summary: Pull Request resolved: meta#15540 Why the change is required: The goal of this stack is indirect blob GC, a blob GC that does not depend on SSTs: it removes garbage from blob files without rewriting any SST. Such a GC needs two things that RocksDB does not have: - A value identity that does not change when the value moves to another file. - A checked list of the blob references that each SST keeps. The list must include older internal versions of a key that the SST still keeps (for example for snapshots), and the blob columns of wide-column entities. This diff adds both. Later diffs build the census (an in-memory reverse index from generation number to the SSTs that use it), GC and repair on top of this list. This diff in one paragraph: With the new column family option enable_blob_indirection on, the blob file writer gives each new value an indirect BlobIndex (type 3, kIndirectBlob). It stores the value's stable identity (origin file, original offset) and its CRCs (an uncompressed value stores only the CRC of the uncompressed value; a compressed value also stores the CRC of its stored bytes). Also with the option on, the block-based table builder collects every blob reference that the SST keeps. It writes them as "exact reference metadata": data blocks, one reference index block, and a 72-byte descriptor table property. In this diff only the table builder writes this metadata, and only tests read it. A temporary option gate (see "option gate" under Terms) keeps the option off outside tests. Separately, wide-column entities with blob columns are now fully checked before their blob indexes are used, also with the option off. Parts of this diff: 1. Indirect BlobIndex (core): types 3 and 4 in db/blob/blob_index.h, and the blob file writer that writes type 3. 2. Exact reference metadata format and decoders (core): db/blob/blob_reference_metadata.{h,cc}. 3. Collector and table builder integration (core): BlobReferenceMetadataCollector, its block cache charge, and BlockBasedTableBuilder. 4. Option enable_blob_indirection and the option gate. Tests need it to turn the feature on. You can skip this part on a first read. 5. Wide-column validation (WideColumnSerialization::ForEachBlobIndex). The table builder uses it to give the blob columns of each entity to the collector. It also changes behavior with the option off (see New behavior). You can skip this part on a first read. 6. Small helpers: canonical varints (util/coding.h) and CacheReservationHandle::Resize(). You can skip these on a first read. How to read this diff: 1. Overview: read "Why" and "This diff in one paragraph" above, and "Terms used in this diff" right below. Then read the top comment of db/blob/blob_reference_metadata.h (terms, layout diagrams, run and entry layout, index layout). 2. Indirect BlobIndex (db/blob/blob_index.h) and canonical varints (util/coding.h GetCanonicalVarint32/64): read the overview comment and the kIndirectBlob / kIndirectFlattenedBlob code. Every later part uses these fields: origin, original offset, generation number, logical CRC and stored CRC. The new formats read varints only in the canonical form. 3. Blob file writer (db/blob/blob_file_builder.cc, blob_log_format.cc, blob_log_writer.cc): see how a new value gets a type-3 index and how its CRC is computed once. 4. Format types and decoders: the structs in db/blob/blob_reference_metadata.h, then in the .cc the constants, GapBase, DecodeEntryBody, DecodeRunEntriesAt, DecodeBlobReferenceIndex and BlobReferenceGroupDecoder. They use only BlobIndex and the canonical varints. Learn the on-disk format here, before you read the writer. 5. Collector and its cache charge: first read the new CacheEntryRole kBlobReferenceMetadataBuildingBuffer (include/rocksdb/cache.h) and CacheReservationHandle::Resize (cache/cache_reservation_manager.h). Then read BlobReferenceMetadataCollector in this order: the class comment, Add, AppendSorted, BuildAndWriteDataBlocks, EncodeIndex, FinishOnce. Then read the unsorted fallback (AppendUnsorted, SortUnsortedGroup, SortAndEncodeGroup). 6. Wide-column change (db/wide/wide_column_serialization.cc, ForEachBlobIndex and WalkV2Columns): it gives every blob index of an entity to a callback after it checks the whole entity. The table builder uses it to feed the collector. UpdateBoundaries, BlobGarbageMeter and the compaction iterator also use it, with the option on or off. 7. Option: enable_blob_indirection (include/rocksdb/advanced_options.h, options/cf_options.cc), and the option checks and gate in db/column_family.cc (ValidateOptions, TEST_SetAllowBlobIndirection), db/db_impl/db_impl_open.cc and db_impl_secondary.cc (the FIFO check for opens that skip ValidateOptions). 8. Table integration: table/block_based/block_based_table_factory.cc (the shared reservation manager), then block_based_table_builder.cc (Rep constructor, AddImpl, WriteBlobReferenceMetadata, EstimatedTailSize). See where the collector is created, fed (AddImpl(), which both Add() and TryAdd() call) and finished. block_based_table_reader.cc maps the new block name to a block type. Terms used in this diff: Stack terms: | Term | Meaning | |---|---| | origin | Blob file where the value was first written | | original offset | Value offset in the origin file | | stable identity | (origin, original offset). Never changes, even when the value moves | | generation number | The file number that a reference uses to find its value. A new value has generation number == origin. When indirect blob GC moves the value, the reference keeps its generation number, and a route sends the read to the relocation file | | indirect BlobIndex | A kIndirectBlob (type 3) or kIndirectFlattenedBlob (type 4) index. It stores the stable identity, so the value can move without an SST rewrite | | unflattened reference | Index with generation number == origin: the value was not moved, or it was moved but no compaction has rewritten its reference yet (see "flatten" in "Stack context: terms for later diffs") | | flattened reference | Index with generation number != origin (type 4). Only flatten (diff 10) writes it; this diff defines the encoding | | route | Maps an old file number (a file that GC has removed from the Version) to the relocation file that now holds its live values. One hop: a target is never itself routed. Stored in the MANIFEST (diff 4) | | relocation file | Block-based table keyed by (origin, original offset) that holds copied values. Each value is the stored bytes plus a 5-byte trailer (kSimpleGen2BlobTrailerSize) | | logical CRC | CRC32C of the uncompressed value (BlobIndex::stable_checksum()) | | stored CRC | CRC32C of the stored bytes plus the compression byte | | physical file | A file that holds value bytes: a normal blob file or a relocation file. A generation number can name a physical file, or it can be only the key of a route | | exact reference metadata | Per-SST sorted list of every blob reference the SST keeps, written when the SST is built. References with the same encoding class, generation number and origin form a group. The list is stored as data blocks, with an index of the groups. An SST that has it is an exact SST | | option gate | Until diff 14, ColumnFamilyData::ValidateOptions returns NotSupported for enable_blob_indirection on a read-write open, CreateColumnFamily and SetOptions, unless a test calls ColumnFamilyData::TEST_SetAllowBlobIndirection(true). It keeps the option off in production while the stack is not complete | Terms of this diff (all defined at the top of db/blob/blob_reference_metadata.h): | Term | Meaning | |---|---| | encoding class (class) | kDirect (a legacy kBlob or kBlobTTL index), unflattened, or flattened | | reference group (group) | All references of one SST with the same (encoding class, generation number, origin) | | run | The part of one group inside one data block | | record bytes | The bytes the value's record takes in the file named by its generation number | | gap | How far a record is from where it would be if the records were back to back. A gap of 0 means back to back | | descriptor | The 72-byte table property that points to the reference index block and holds the totals, the CF ID, the SST file number and the relocation-state fingerprint | | relocation-state fingerprint | 128-bit hash of the routes and relocation files of the Version that an SST was built from (fields route_snapshot_fingerprint_*). Written as 0 in this diff | | legacy | Written the way RocksDB writes it without blob indirection | Previous behavior: - A BlobIndex named only a physical blob file and an offset. A value could not move to another file without rewriting every SST that referenced it. - An SST had no metadata that listed its blob references. The only way to find them was to scan its user keys. - These readers get the blob indexes of a wide-column entity through ForEachBlobFileNumber: UpdateBoundaries, BlobGarbageMeter, the compaction iterator, WideColumnSerialization::RemoveBlobColumns, and the referenced_blob_bytes table property count in BlockBasedTableBuilder::Add. ForEachBlobFileNumber decoded each blob index, but it did not check the rest of the entity framing (name order, section sizes, exact input size), and it could call the callback before it found a later error. New behavior: Separate change (option on or off): - The same readers (UpdateBoundaries, BlobGarbageMeter, the compaction iterator, RemoveBlobColumns, and the referenced_blob_bytes count in BlockBasedTableBuilder::Add) use WideColumnSerialization::ForEachBlobIndex, with the option on or off. With the option on, the table builder also uses it to give the blob columns of each entity to the collector. For an entity with blob columns, it checks the full framing, then decodes and validates every blob index before any callback runs. So an entity with a malformed framing now fails the flush or compaction, even with the option off. The referenced_blob_bytes count itself does not fail the build on a malformed entity. It removes that property from the SST, as in base. Option and option gate: - This diff adds a new column family option, `enable_blob_indirection`. It is off by default, and SetOptions can change it. - The option gate (see Terms) keeps the option off outside tests until the stack is complete. ColumnFamilyData::ValidateOptions rejects the option with NotSupported on a read-write open, CreateColumnFamily and SetOptions. Read-only opens (DBImplReadOnly and CompactedDBImpl) and secondary opens (including DB::OpenAndCompact) do not run ValidateOptions; they check only the FIFO rule (see Implementation). Only tests can turn the option on, through the internal ColumnFamilyData::TEST_SetAllowBlobIndirection(). DBTestBase sets it. - When the option is on, ValidateOptions also requires enable_blob_files, the block-based table format, no blob direct write, and a compaction style other than FIFO. Blob file writer (option on): - The blob file writer encodes each new value with a stable identity (origin file, original offset) and two checksums: - the logical CRC: the CRC32C of the uncompressed value. - the stored CRC: the CRC32C of the exact stored (maybe compressed) bytes, extended with the compression byte. - For an uncompressed value the stored CRC is not written, because the reader can compute it from the logical CRC. - The checksums have no version field of their own. The BlobIndex type byte gives the format version: the two new types, 3 (kIndirectBlob) and 4 (kIndirectFlattenedBlob, see Implementation), are BlobIndex::kIndirectBlobFormatVersion (1). A later format uses new type values. - The read path does not check these checksums yet. The check starts in diff 3. Exact reference metadata (option on): 1. What: a sorted list of every blob reference that the SST keeps, in plain blob values and in the blob columns of wide-column entities. 2. Where: data blocks (target 16 KB each before compression; small blocks waste less I/O, because indirect blob GC reads a whole block to get a few groups) and one reference index block, written after the SST data blocks and before the SST tail, plus a 72-byte descriptor table property. 3. Who reads it: only tests in this diff. The census, GC and RepairDB read it in later diffs. 4. Details: the top comment of db/blob/blob_reference_metadata.h has the full format with diagrams, the record-bytes rules and the gap rule. - Not collected: SstFileWriter output (it has no DB file number) and any output whose CF is unknown (it has no CF ID). The descriptor needs both values, so these outputs stay in the legacy form. Checks on the metadata: - A repeated or overlapping reference in one group fails the SST build with Corruption. Readers check every count, order, size and total against the index and the descriptor, so corrupt metadata returns Corruption. The writer never makes a data block larger than kTargetDataBlockSize (16 KB), and readers accept up to kMaxDataBlockSize (64 KB). Details: "Readers" and "Integrity" at the top of db/blob/blob_reference_metadata.h, and BlobReferenceGroupDecoder. Collection and memory: - Each reference is encoded when it is added, in the on-disk entry encoding, into its group's chain of 512-byte chunks. Only a group that takes the unsorted fallback is encoded a second time, in Finish(). Inside a group, references normally arrive in rising offset order, so no reference is sorted; at the end only the groups are sorted. A group that gets a lower offset takes the unsorted fallback. The BlobReferenceMetadataCollector class comment explains the chunks, the chunk size and the fallback. - The collector keeps every reference of the SST in memory until the SST is finished, and charges this memory to the block cache under a new CacheEntryRole, kBlobReferenceMetadataBuildingBuffer. "Memory usage" and "Memory charge" in the BlobReferenceMetadataCollector class comment explain the parts of this memory and the charge rule. The measured amounts are under "Validation and known limitations". - There is no fixed limit on references. The collector fails with MemoryLimit above 2^20 groups or 2^20 data blocks in one SST; these are persisted format limits. It also fails with MemoryLimit when the reference index block could be larger than 64 MB (kMaxTopLevelMetadataBlockSize), which can happen from about 945K groups (64 MB / kMaxIndexEntrySize, 71 bytes per group), or from fewer groups when the SST also has many data blocks. - There is no charge when the table has no block cache or when charging for this role is disabled. Each block-based table factory holds one cache reservation manager for this role, shared by its builders and the SST summaries. If the charge fails, the SST build fails, so the flush or compaction fails with a background error. The buffers and their charge are released when the SST is finished or abandoned. Implementation: - BlobIndex adds two indirect encodings. The legacy encodings do not change. - Type 3, kIndirectBlob (unflattened, generation number == origin), stores the origin, the original offset, the stored size, the compression type, the logical CRC and, for compressed values, the stored CRC. It does not store the generation number. This saves the size of one varint file number per reference (about 3 to 4 bytes for typical file numbers; an estimate, not measured). - Type 4, kIndirectFlattenedBlob (flattened), stores the generation number first, then the same fields. Its generation number must be above the origin. The decoder rejects type 4 with generation number <= origin, so each reference has one encoding. - stored_checksum() computes the stored CRC of an uncompressed value when a caller asks for it: the logical CRC extended with the compression byte. A decode does not compute it, because most decodes (for example in compaction) never read it. - The encoders take the fields as one BlobIndex::IndirectBlobFields, set by field name, so values of the same type (file number, offset and size; the two CRCs) cannot be swapped. EncodeIndirectBlob() takes the generation number as a separate argument; EncodeUnflattenedIndirectBlob() uses the origin, so the common path cannot leave the generation number unset. - The new files blob_reference_metadata.{h,cc} define the exact reference metadata. The files also contain: - BlobReferenceMetadataCollector, the collector described above. It takes the SST identity (CF ID, SST file number and the relocation-state fingerprint) as one BlobReferenceSstIdentity, set by field name. - BlobReferenceGroupKey, the key of a group: (encoding class, generation number, origin). It has the order, the hash and the validity check that the collector map, the sort, the decoder and FindBlobReferenceGroup() use. BlobReference and BlobReferenceGroup keep their flat fields and return the key through group_key(). - The index codec and the descriptor table property. - BlobReferenceGroupDecoder, the only decoder of reference runs. It reads a group one data block at a time and holds every corruption check of a group walk. Readers that visit one group (found by a binary search of the index) use it, and so do readers that visit every block in order. - Callers in this diff: only the table builder runs the collector. The decoders (DecodeBlobReferenceIndex, FindBlobReferenceGroup, BlobReferenceGroupDecoder, ForEachBlobReference*) are used only by tests. kMaxDecodedBlobReferenceBytes is used only in compile-time checks of the count limits. The census (diff 5) and indirect blob GC (diffs 8 and 9) are their first production users. The census is also the first user of BlobReferenceSstSummary (see the last Implementation bullet). - Descriptor: the CF ID and the SST file number are the repair fields. They let RepairDB tell which column family and SST the list belongs to without a MANIFEST. The decoder rejects a descriptor whose repair fields are not set (unknown CF ID or SST file number 0). The descriptor also has a 128-bit relocation-state fingerprint (the route_snapshot_fingerprint fields). Remote compaction uses it in diff 11 to check that an SST built elsewhere used the same routes as the job that asked for it. In this diff the table builder always writes 0, and DecodeFrom() does not check it. The field is in the format now, so the descriptor keeps one format version (1) through the stack. While the option gate is on, no data in this format exists outside tests. - The collector validates each BlobIndex once, when it is added. Its Finish() writes the data blocks and then the reference index block through one block writer, and fills the descriptor, including the index handle. Destroying the collector frees its memory and then its charge, also when the build is abandoned. - BlockBasedTableBuilder creates the collector with the CF ID and SST file number from TableBuilderOptions. It passes the size of each entry's user key to the collector, so the collector can compute the exact size of each blob record. It adds the collector's EstimatedEncodedSize() to its tail size estimate: the blocks are written before the tail, but output file size planning uses this estimate. The reference blocks are written with BlockType::kUserDefinedIndex, which keeps their contents as opaque bytes. Readers read them without the block cache, so prepopulate_block_cache does not warm them into the block cache (WriteMaybeCompressedBlock() with warm_cache false). - To save CPU, compaction decodes each kTypeBlobIndex output value once, in CompactionOutputs::AddToOutput(). It passes the decode to TableBuilder::TryAdd() in the new field TableBuilderAddContext::decoded_blob_index, and to new overloads of BlobGarbageMeter::ProcessOutFlow() and FileMetaData::UpdateBoundaries(). BlockBasedTableBuilder decodes a kTypeBlobIndex value once for both the referenced_blob_bytes count and the collector, and uses the caller's decode when it has one. If the decode fails, no decode is passed, and each reader decodes the value itself and returns its usual error. This applies with the option on or off. The SST bytes are the same as when each reader decodes the value itself. - The collector converts each BlobIndex into a reference without deriving the stored CRC of an uncompressed value (ConvertBlobIndex() in db/blob/blob_reference_metadata.cc). The collector does not write that CRC, because a reader derives it again from the logical CRC. Deriving it would cost CPU for every reference of every output SST. BlobReference::FromBlobIndex() still derives it. - With the option on, the blob file builder computes the CRC32C of each stored value once. The blob log record CRC combines it with the key CRC (crc32c::Crc32cCombine). The stored CRC extends it with the compression byte. For an uncompressed value it is also the logical CRC. The bytes written do not change. - WideColumnSerialization::ForEachBlobIndex replaces ForEachBlobFileNumber. It reads the entity header and column types once. For a V1 entity (no column types and no blob columns), it returns right after the header. For a V2 entity without blob columns, it returns right after the type scan. It shares one V2 column walker (WalkV2Columns) with Deserialize(), so the entity is parsed in one pass. - Read-only opens (DBImplReadOnly, CompactedDBImpl) and secondary opens skip ValidateOptions, so they check the compaction style against the option once per column family: in DBImpl::Recover() for read-only opens, and at the start of OpenAsSecondaryImpl() for secondary opens. DB::OpenAndCompact opens through OpenAsSecondaryImpl(), so it gets the same check. - CacheReservationHandle gets Resize(), so each collector or summary can change its own part of the shared charge. - util/coding.h adds GetCanonicalVarint32/64, which reject overlong varint encodings. The new formats use them, so each value has one encoding. - BlobReferenceSstSummary is the in-memory list of an SST's groups and totals that Finish() can return. It has no caller yet: the table builder passes a null summary output. It is checked and does not change after Finish(), so the finished builder, the SST metadata and the census (its first user, in diff 5) can share it. It is in this diff because Finish() hands the group buffer and its cache charge over to the summary. Adding the summary later would mean rewriting the charge code of Finish() in diff 5, so the charge code is written and tested once, here. - sst_dump prints a "Blob reference metadata" section after the raw user collected properties of an SST that has exact metadata: the reference count, the stored value bytes, the column family ID, the SST file number when built and the index block handle. An SST without the descriptor prints nothing more. Safety boundaries and unsupported cases: - Public API: this diff adds the option `enable_blob_indirection` with the C API functions `rocksdb_options_set_enable_blob_indirection` and `rocksdb_options_get_enable_blob_indirection`. It also inserts a new CacheEntryRole value before kMisc, which changes the numeric value of kMisc and kNumCacheEntryRoles. - With the option off, new blob values and SSTs are written in the legacy form. Legacy blob indexes stay readable. - One behavior change applies with the option off too: the five readers listed under "Separate change" fully validate wide-column entities with blob columns and their blob indexes (see "Separate change" above). - Exact reference metadata has one format (version 1) with a fixed 72-byte descriptor. The decoders reject any other version as Corruption. - The metadata blocks are protected by the table's block checksum only. With checksum = kNoChecksum, corrupt metadata is detected only when it breaks the format or order checks. - The index decoder checks each decoded count against the remaining input size before it reserves memory. So a small corrupt index cannot make it allocate a large vector. - Inlined-TTL and same-file blob indexes are not collected and cannot be relocated. Encoding, count and checksum errors return an error; a bad reference is never skipped. - Option gate (see Terms): outside tests no DB writes data in the new format. The gate checks only the option value, never persisted state. Diff 14 removes it, once data written with the option stays readable through the end of the stack and the safety checks for turning the option on are in place. - Downgrading the binary after data was written with the option on is not supported. Validation and known limitations: - Collector memory, measured in a release build that drives the collector directly (bench/collector-chunk-20261111/README.md; host not recorded; 16-byte keys, 1 KB values, 62K to 1M references per SST in 1 to 16 groups): the buffers used about 8.3 to 9 bytes per reference for uncompressed values and 12.8 to 13.5 bytes for compressed values, including the chunk headers. With 1000 groups in a 62K-reference SST, they used about 14 bytes per uncompressed reference. - Chunk size, from the same measurement: for an SST with 1M references, the chunk size changed memory by less than 5% between 256 and 4096 bytes. For an SST with 62K references in 1000 groups, the partly filled chunks dominated: about 14 bytes per reference at 512 bytes, 22 at 1024 and 72 at 4096. 256 bytes used 3-4% more memory than 512 and made Finish() up to 1.6 times slower with 100 or more groups. - Reference index block size, measured in census db_bench runs (source report and host not recorded; 16-byte keys, 200-byte values, 1 MB blob files, no compression, fillrandom and then a full compaction): about 1.7 KB per SST with 32 MB write buffers and a target file size of 8 MB or 2 MB, and about 10 KB per SST with 1 MB write buffers and a target of 2 MB or 200 KB. The size is the bytes read for the index blocks divided by the number of SSTs. - Unit tests cover the BlobIndex encodings and their rejection of bad input, the metadata layout and every corruption check, the collector (sorted path, unsorted fallback, memory charge), the flush integration, the option checks and option gate, wide-column validation, and canonical varints. See the Test Plan. - The collection buffer is not spilled to disk. One SST with very many blob references needs a matching amount of charged memory: the per-reference amount given under "Collection and memory", plus up to one partly used 512-byte chunk per group. While a fallback group is sorted, it also needs 16 bytes of sort key per reference and a second copy of its entries. For a test group of 200K references with every other one out of order, the test comment estimates about 44 bytes per reference, and the test checks that the peak charge is at most 56 bytes per reference. - Flush still writes a single output file. Until diff 7, one compaction output collects every reference it gets, charged to the block cache. Diff 7 cuts compaction outputs (including L0) at a reference cap and keeps each user key's history in one output. With the option gate, a large output is reachable only through the test hook. Stack context (same in diffs 2-14, except that diffs 10-14 add the diff 10 route rule to the install line; "<== this diff" marks this diff's part): Goal: remove garbage from blob files without rewriting any SST. A value gets a stable identity when it is first written. Indirect blob GC copies the live values of whole files into a relocation file and installs routes in the MANIFEST. Readers follow the routes. SSTs are not changed. How a read finds a value (P is the relocation file that holds the value now): ``` SST entry indirect BlobIndex { generation number, origin, original offset, size, CRCs } | v Version file table: generation number -> live file, or route old file -> P | | live file route found | | v v blob file: read at original offset relocation file P (always one hop): relocation file: look up key look up key (origin, original offset) (origin, original offset) ``` Lifecycle of one value (in this example, F is the blob file where the value is first written, and P is the relocation file that GC writes): ``` write new value goes to blob file F; index: origin = F, generation number = F | census per generation number: the SSTs and references that use it; | per physical file: live bytes, ranked by garbage ratio GC job copy the live values of whole input files (stored bytes, checked | with the stored CRC) into relocation file P, keyed by | (origin, original offset) install one MANIFEST edit: add P, route each input generation number still | in use to P, re-point the routes into the inputs to P, and remove | each input (blob file retirement for a blob file, relocation | file deletion for a relocation file) flatten compaction sets generation number = P in the SSTs it rewrites | route route retirement: route deletion records remove the routes that delete no SST uses ``` Stack map: | Diff | Adds | |---|---| | 1 | Finalize blob metadata outside the DB mutex (prep) | | 2 | Indirect BlobIndex (kIndirectBlob) and exact SST reference metadata <== this diff | | 3 | Relocation file format and stable read path | | 4 | MANIFEST routes and relocation state in the Version | | 5 | Exact reverse reference census | | 6 | MANIFEST apply outside the DB mutex, with a precondition re-check | | 7 | Coarse flush metadata, per-SST reference cap, epoch runs | | 8 | Indirect blob GC building blocks | | 9 | Indirect blob GC job | | 10 | Flatten generation numbers; route retirement | | 11 | Remote compaction validation | | 12 | Dynamic options and operations | | 13 | Lost relocation files | | 14 | RepairDB rebuilds relocation state; option gate removed | | 15 | db_stress and crash test support | Stack context: terms for later diffs: | Term | Meaning | |---|---| | census | Background count, per generation number, of the SSTs and references that use it, plus live bytes per physical file (diff 5) | | indirect origin | Normal blob file written with the option on; it is the origin of unflattened references | | flatten | Compaction rewrites the generation number to the relocation file that holds the value, so old routes can go. Only flatten makes generation number != origin (diff 10) | | blob file retirement | MANIFEST record (kBlobFileRetirement) that removes a collected blob file from the Version (diff 4) | | route deletion | MANIFEST record that removes a route that no SST uses. Route retirement is the background job that writes these records (diff 10) | Tracking issue: meta#15301 Differential Revision: D124652097
Summary: Pull Request resolved: meta#15540 Why the change is required: The goal of this stack is indirect blob GC, a blob GC that does not depend on SSTs: it removes garbage from blob files without rewriting any SST. Such a GC needs two things that RocksDB does not have: - A value identity that does not change when the value moves to another file. - A checked list of the blob references that each SST keeps. The list must include older internal versions of a key that the SST still keeps (for example for snapshots), and the blob columns of wide-column entities. This diff adds both. Later diffs build the census (an in-memory reverse index from generation number to the SSTs that use it), GC and repair on top of this list. This diff in one paragraph: With the new column family option enable_blob_indirection on, the blob file writer gives each new value an indirect BlobIndex (type 3, kIndirectBlob). It stores the value's stable identity (origin file, original offset) and its CRCs (an uncompressed value stores only the CRC of the uncompressed value; a compressed value also stores the CRC of its stored bytes). Also with the option on, the block-based table builder collects every blob reference that the SST keeps. It writes them as "exact reference metadata": data blocks, one reference index block, and a 72-byte descriptor table property. In this diff only the table builder writes this metadata, and only tests read it. A temporary option gate (see "option gate" under Terms) keeps the option off outside tests. Separately, wide-column entities with blob columns are now fully checked before their blob indexes are used, also with the option off. Parts of this diff: 1. Indirect BlobIndex (core): types 3 and 4 in db/blob/blob_index.h, and the blob file writer that writes type 3. 2. Exact reference metadata format and decoders (core): db/blob/blob_reference_metadata.{h,cc}. 3. Collector and table builder integration (core): BlobReferenceMetadataCollector, its block cache charge, and BlockBasedTableBuilder. 4. Option enable_blob_indirection and the option gate. Tests need it to turn the feature on. You can skip this part on a first read. 5. Wide-column validation (WideColumnSerialization::ForEachBlobIndex). The table builder uses it to give the blob columns of each entity to the collector. It also changes behavior with the option off (see New behavior). You can skip this part on a first read. 6. Small helpers: canonical varints (util/coding.h) and CacheReservationHandle::Resize(). You can skip these on a first read. How to read this diff: 1. Overview: read "Why" and "This diff in one paragraph" above, and "Terms used in this diff" right below. Then read the top comment of db/blob/blob_reference_metadata.h (terms, layout diagrams, run and entry layout, index layout). 2. Indirect BlobIndex (db/blob/blob_index.h) and canonical varints (util/coding.h GetCanonicalVarint32/64): read the overview comment and the kIndirectBlob / kIndirectFlattenedBlob code. Every later part uses these fields: origin, original offset, generation number, logical CRC and stored CRC. The new formats read varints only in the canonical form. 3. Blob file writer (db/blob/blob_file_builder.cc, blob_log_format.cc, blob_log_writer.cc): see how a new value gets a type-3 index and how its CRC is computed once. 4. Format types and decoders: the structs in db/blob/blob_reference_metadata.h, then in the .cc the constants, GapBase, DecodeEntryBody, DecodeRunEntriesAt, DecodeBlobReferenceIndex and BlobReferenceGroupDecoder. They use only BlobIndex and the canonical varints. Learn the on-disk format here, before you read the writer. 5. Collector and its cache charge: first read the new CacheEntryRole kBlobReferenceMetadataBuildingBuffer (include/rocksdb/cache.h) and CacheReservationHandle::Resize (cache/cache_reservation_manager.h). Then read BlobReferenceMetadataCollector in this order: the class comment, Add, AppendSorted, BuildAndWriteDataBlocks, EncodeIndex, FinishOnce. Then read the unsorted fallback (AppendUnsorted, SortUnsortedGroup, SortAndEncodeGroup). 6. Wide-column change (db/wide/wide_column_serialization.cc, ForEachBlobIndex and WalkV2Columns): it gives every blob index of an entity to a callback after it checks the whole entity. The table builder uses it to feed the collector. UpdateBoundaries, BlobGarbageMeter and the compaction iterator also use it, with the option on or off. 7. Option: enable_blob_indirection (include/rocksdb/advanced_options.h, options/cf_options.cc), and the option checks and gate in db/column_family.cc (ValidateOptions, TEST_SetAllowBlobIndirection), db/db_impl/db_impl_open.cc and db_impl_secondary.cc (the FIFO check for opens that skip ValidateOptions). 8. Table integration: table/block_based/block_based_table_factory.cc (the shared reservation manager), then block_based_table_builder.cc (Rep constructor, AddImpl, WriteBlobReferenceMetadata, EstimatedTailSize). See where the collector is created, fed (AddImpl(), which both Add() and TryAdd() call) and finished. block_based_table_reader.cc maps the new block name to a block type. Terms used in this diff: Stack terms: | Term | Meaning | |---|---| | origin | Blob file where the value was first written | | original offset | Value offset in the origin file | | stable identity | (origin, original offset). Never changes, even when the value moves | | generation number | The file number that a reference uses to find its value. A new value has generation number == origin. When indirect blob GC moves the value, the reference keeps its generation number, and a route sends the read to the relocation file | | indirect BlobIndex | A kIndirectBlob (type 3) or kIndirectFlattenedBlob (type 4) index. It stores the stable identity, so the value can move without an SST rewrite | | unflattened reference | Index with generation number == origin: the value was not moved, or it was moved but no compaction has rewritten its reference yet (see "flatten" in "Stack context: terms for later diffs") | | flattened reference | Index with generation number != origin (type 4). Only flatten (diff 10) writes it; this diff defines the encoding | | route | Maps an old file number (a file that GC has removed from the Version) to the relocation file that now holds its live values. One hop: a target is never itself routed. Stored in the MANIFEST (diff 4) | | relocation file | Block-based table keyed by (origin, original offset) that holds copied values. Each value is the stored bytes plus a 5-byte trailer (kSimpleGen2BlobTrailerSize) | | logical CRC | CRC32C of the uncompressed value (BlobIndex::stable_checksum()) | | stored CRC | CRC32C of the stored bytes plus the compression byte | | physical file | A file that holds value bytes: a normal blob file or a relocation file. A generation number can name a physical file, or it can be only the key of a route | | exact reference metadata | Per-SST sorted list of every blob reference the SST keeps, written when the SST is built. References with the same encoding class, generation number and origin form a group. The list is stored as data blocks, with an index of the groups. An SST that has it is an exact SST | | option gate | Until diff 14, ColumnFamilyData::ValidateOptions returns NotSupported for enable_blob_indirection on a read-write open, CreateColumnFamily and SetOptions, unless a test calls ColumnFamilyData::TEST_SetAllowBlobIndirection(true). It keeps the option off in production while the stack is not complete | Terms of this diff (all defined at the top of db/blob/blob_reference_metadata.h): | Term | Meaning | |---|---| | encoding class (class) | kDirect (a legacy kBlob or kBlobTTL index), unflattened, or flattened | | reference group (group) | All references of one SST with the same (encoding class, generation number, origin) | | run | The part of one group inside one data block | | record bytes | The bytes the value's record takes in the file named by its generation number | | gap | How far a record is from where it would be if the records were back to back. A gap of 0 means back to back | | descriptor | The 72-byte table property that points to the reference index block and holds the totals, the CF ID, the SST file number and the relocation-state fingerprint | | relocation-state fingerprint | 128-bit hash of the routes and relocation files of the Version that an SST was built from (fields route_snapshot_fingerprint_*). Written as 0 in this diff | | legacy | Written the way RocksDB writes it without blob indirection | Previous behavior: - A BlobIndex named only a physical blob file and an offset. A value could not move to another file without rewriting every SST that referenced it. - An SST had no metadata that listed its blob references. The only way to find them was to scan its user keys. - These readers get the blob indexes of a wide-column entity through ForEachBlobFileNumber: UpdateBoundaries, BlobGarbageMeter, the compaction iterator, WideColumnSerialization::RemoveBlobColumns, and the referenced_blob_bytes table property count in BlockBasedTableBuilder::Add. ForEachBlobFileNumber decoded each blob index, but it did not check the rest of the entity framing (name order, section sizes, exact input size), and it could call the callback before it found a later error. New behavior: Separate change (option on or off): - The same readers (UpdateBoundaries, BlobGarbageMeter, the compaction iterator, RemoveBlobColumns, and the referenced_blob_bytes count in BlockBasedTableBuilder::Add) use WideColumnSerialization::ForEachBlobIndex, with the option on or off. With the option on, the table builder also uses it to give the blob columns of each entity to the collector. For an entity with blob columns, it checks the full framing, then decodes and validates every blob index before any callback runs. So an entity with a malformed framing now fails the flush or compaction, even with the option off. The referenced_blob_bytes count itself does not fail the build on a malformed entity. It removes that property from the SST, as in base. Option and option gate: - This diff adds a new column family option, `enable_blob_indirection`. It is off by default, and SetOptions can change it. - The option gate (see Terms) keeps the option off outside tests until the stack is complete. ColumnFamilyData::ValidateOptions rejects the option with NotSupported on a read-write open, CreateColumnFamily and SetOptions. Read-only opens (DBImplReadOnly and CompactedDBImpl) and secondary opens (including DB::OpenAndCompact) do not run ValidateOptions; they check only the FIFO rule (see Implementation). Only tests can turn the option on, through the internal ColumnFamilyData::TEST_SetAllowBlobIndirection(). DBTestBase sets it. - When the option is on, ValidateOptions also requires enable_blob_files, the block-based table format, no blob direct write, and a compaction style other than FIFO. Blob file writer (option on): - The blob file writer encodes each new value with a stable identity (origin file, original offset) and two checksums: - the logical CRC: the CRC32C of the uncompressed value. - the stored CRC: the CRC32C of the exact stored (maybe compressed) bytes, extended with the compression byte. - For an uncompressed value the stored CRC is not written, because the reader can compute it from the logical CRC. - The checksums have no version field of their own. The BlobIndex type byte gives the format version: the two new types, 3 (kIndirectBlob) and 4 (kIndirectFlattenedBlob, see Implementation), are BlobIndex::kIndirectBlobFormatVersion (1). A later format uses new type values. - The read path does not check these checksums yet. The check starts in diff 3. Exact reference metadata (option on): 1. What: a sorted list of every blob reference that the SST keeps, in plain blob values and in the blob columns of wide-column entities. 2. Where: data blocks (target 16 KB each before compression; small blocks waste less I/O, because indirect blob GC reads a whole block to get a few groups) and one reference index block, written after the SST data blocks and before the SST tail, plus a 72-byte descriptor table property. 3. Who reads it: only tests in this diff. The census, GC and RepairDB read it in later diffs. 4. Details: the top comment of db/blob/blob_reference_metadata.h has the full format with diagrams, the record-bytes rules and the gap rule. - Not collected: SstFileWriter output (it has no DB file number) and any output whose CF is unknown (it has no CF ID). The descriptor needs both values, so these outputs stay in the legacy form. Checks on the metadata: - A repeated or overlapping reference in one group fails the SST build with Corruption. Readers check every count, order, size and total against the index and the descriptor, so corrupt metadata returns Corruption. The writer never makes a data block larger than kTargetDataBlockSize (16 KB), and readers accept up to kMaxDataBlockSize (64 KB). Details: "Readers" and "Integrity" at the top of db/blob/blob_reference_metadata.h, and BlobReferenceGroupDecoder. Collection and memory: - Each reference is encoded when it is added, in the on-disk entry encoding, into its group's chain of 512-byte chunks. Only a group that takes the unsorted fallback is encoded a second time, in Finish(). Inside a group, references normally arrive in rising offset order, so no reference is sorted; at the end only the groups are sorted. A group that gets a lower offset takes the unsorted fallback. The BlobReferenceMetadataCollector class comment explains the chunks, the chunk size and the fallback. - The collector keeps every reference of the SST in memory until the SST is finished, and charges this memory to the block cache under a new CacheEntryRole, kBlobReferenceMetadataBuildingBuffer. "Memory usage" and "Memory charge" in the BlobReferenceMetadataCollector class comment explain the parts of this memory and the charge rule. The measured amounts are under "Validation and known limitations". - There is no fixed limit on references. The collector fails with MemoryLimit above 2^20 groups or 2^20 data blocks in one SST; these are persisted format limits. It also fails with MemoryLimit when the reference index block could be larger than 64 MB (kMaxTopLevelMetadataBlockSize), which can happen from about 945K groups (64 MB / kMaxIndexEntrySize, 71 bytes per group), or from fewer groups when the SST also has many data blocks. - There is no charge when the table has no block cache or when charging for this role is disabled. Each block-based table factory holds one cache reservation manager for this role, shared by its builders and the SST summaries. If the charge fails, the SST build fails, so the flush or compaction fails with a background error. The buffers and their charge are released when the SST is finished or abandoned. Implementation: - BlobIndex adds two indirect encodings. The legacy encodings do not change. - Type 3, kIndirectBlob (unflattened, generation number == origin), stores the origin, the original offset, the stored size, the compression type, the logical CRC and, for compressed values, the stored CRC. It does not store the generation number. This saves the size of one varint file number per reference (about 3 to 4 bytes for typical file numbers; an estimate, not measured). - Type 4, kIndirectFlattenedBlob (flattened), stores the generation number first, then the same fields. Its generation number must be above the origin. The decoder rejects type 4 with generation number <= origin, so each reference has one encoding. - stored_checksum() computes the stored CRC of an uncompressed value when a caller asks for it: the logical CRC extended with the compression byte. A decode does not compute it, because most decodes (for example in compaction) never read it. - The encoders take the fields as one BlobIndex::IndirectBlobFields, set by field name, so values of the same type (file number, offset and size; the two CRCs) cannot be swapped. EncodeIndirectBlob() takes the generation number as a separate argument; EncodeUnflattenedIndirectBlob() uses the origin, so the common path cannot leave the generation number unset. - The new files blob_reference_metadata.{h,cc} define the exact reference metadata. The files also contain: - BlobReferenceMetadataCollector, the collector described above. It takes the SST identity (CF ID, SST file number and the relocation-state fingerprint) as one BlobReferenceSstIdentity, set by field name. - BlobReferenceGroupKey, the key of a group: (encoding class, generation number, origin). It has the order, the hash and the validity check that the collector map, the sort, the decoder and FindBlobReferenceGroup() use. BlobReference and BlobReferenceGroup keep their flat fields and return the key through group_key(). - The index codec and the descriptor table property. - BlobReferenceGroupDecoder, the only decoder of reference runs. It reads a group one data block at a time and holds every corruption check of a group walk. Readers that visit one group (found by a binary search of the index) use it, and so do readers that visit every block in order. - Callers in this diff: only the table builder runs the collector. The decoders (DecodeBlobReferenceIndex, FindBlobReferenceGroup, BlobReferenceGroupDecoder, ForEachBlobReference*) are used only by tests. kMaxDecodedBlobReferenceBytes is used only in compile-time checks of the count limits. The census (diff 5) and indirect blob GC (diffs 8 and 9) are their first production users. The census is also the first user of BlobReferenceSstSummary (see the last Implementation bullet). - Descriptor: the CF ID and the SST file number are the repair fields. They let RepairDB tell which column family and SST the list belongs to without a MANIFEST. The decoder rejects a descriptor whose repair fields are not set (unknown CF ID or SST file number 0). The descriptor also has a 128-bit relocation-state fingerprint (the route_snapshot_fingerprint fields). Remote compaction uses it in diff 11 to check that an SST built elsewhere used the same routes as the job that asked for it. In this diff the table builder always writes 0, and DecodeFrom() does not check it. The field is in the format now, so the descriptor keeps one format version (1) through the stack. While the option gate is on, no data in this format exists outside tests. - The collector validates each BlobIndex once, when it is added. Its Finish() writes the data blocks and then the reference index block through one block writer, and fills the descriptor, including the index handle. Destroying the collector frees its memory and then its charge, also when the build is abandoned. - BlockBasedTableBuilder creates the collector with the CF ID and SST file number from TableBuilderOptions. It passes the size of each entry's user key to the collector, so the collector can compute the exact size of each blob record. It adds the collector's EstimatedEncodedSize() to its tail size estimate: the blocks are written before the tail, but output file size planning uses this estimate. The reference blocks are written with BlockType::kUserDefinedIndex, which keeps their contents as opaque bytes. Readers read them without the block cache, so prepopulate_block_cache does not warm them into the block cache (WriteMaybeCompressedBlock() with warm_cache false). - To save CPU, compaction decodes each kTypeBlobIndex output value once, in CompactionOutputs::AddToOutput(). It passes the decode to TableBuilder::TryAdd() in the new field TableBuilderAddContext::decoded_blob_index, and to new overloads of BlobGarbageMeter::ProcessOutFlow() and FileMetaData::UpdateBoundaries(). BlockBasedTableBuilder decodes a kTypeBlobIndex value once for both the referenced_blob_bytes count and the collector, and uses the caller's decode when it has one. If the decode fails, no decode is passed, and each reader decodes the value itself and returns its usual error. This applies with the option on or off. The SST bytes are the same as when each reader decodes the value itself. - The collector converts each BlobIndex into a reference without deriving the stored CRC of an uncompressed value (ConvertBlobIndex() in db/blob/blob_reference_metadata.cc). The collector does not write that CRC, because a reader derives it again from the logical CRC. Deriving it would cost CPU for every reference of every output SST. BlobReference::FromBlobIndex() still derives it. - With the option on, the blob file builder computes the CRC32C of each stored value once. The blob log record CRC combines it with the key CRC (crc32c::Crc32cCombine). The stored CRC extends it with the compression byte. For an uncompressed value it is also the logical CRC. The bytes written do not change. - WideColumnSerialization::ForEachBlobIndex replaces ForEachBlobFileNumber. It reads the entity header and column types once. For a V1 entity (no column types and no blob columns), it returns right after the header. For a V2 entity without blob columns, it returns right after the type scan. It shares one V2 column walker (WalkV2Columns) with Deserialize(), so the entity is parsed in one pass. - Read-only opens (DBImplReadOnly, CompactedDBImpl) and secondary opens skip ValidateOptions, so they check the compaction style against the option once per column family: in DBImpl::Recover() for read-only opens, and at the start of OpenAsSecondaryImpl() for secondary opens. DB::OpenAndCompact opens through OpenAsSecondaryImpl(), so it gets the same check. - CacheReservationHandle gets Resize(), so each collector or summary can change its own part of the shared charge. - util/coding.h adds GetCanonicalVarint32/64, which reject overlong varint encodings. The new formats use them, so each value has one encoding. - BlobReferenceSstSummary is the in-memory list of an SST's groups and totals that Finish() can return. It has no caller yet: the table builder passes a null summary output. It is checked and does not change after Finish(), so the finished builder, the SST metadata and the census (its first user, in diff 5) can share it. It is in this diff because Finish() hands the group buffer and its cache charge over to the summary. Adding the summary later would mean rewriting the charge code of Finish() in diff 5, so the charge code is written and tested once, here. - sst_dump prints a "Blob reference metadata" section after the raw user collected properties of an SST that has exact metadata: the reference count, the stored value bytes, the column family ID, the SST file number when built and the index block handle. An SST without the descriptor prints nothing more. Safety boundaries and unsupported cases: - Public API: this diff adds the option `enable_blob_indirection` with the C API functions `rocksdb_options_set_enable_blob_indirection` and `rocksdb_options_get_enable_blob_indirection`. It also inserts a new CacheEntryRole value before kMisc, which changes the numeric value of kMisc and kNumCacheEntryRoles. - With the option off, new blob values and SSTs are written in the legacy form. Legacy blob indexes stay readable. - One behavior change applies with the option off too: the five readers listed under "Separate change" fully validate wide-column entities with blob columns and their blob indexes (see "Separate change" above). - Exact reference metadata has one format (version 1) with a fixed 72-byte descriptor. The decoders reject any other version as Corruption. - The metadata blocks are protected by the table's block checksum only. With checksum = kNoChecksum, corrupt metadata is detected only when it breaks the format or order checks. - The index decoder checks each decoded count against the remaining input size before it reserves memory. So a small corrupt index cannot make it allocate a large vector. - Inlined-TTL and same-file blob indexes are not collected and cannot be relocated. Encoding, count and checksum errors return an error; a bad reference is never skipped. - Option gate (see Terms): outside tests no DB writes data in the new format. The gate checks only the option value, never persisted state. Diff 14 removes it, once data written with the option stays readable through the end of the stack and the safety checks for turning the option on are in place. - Downgrading the binary after data was written with the option on is not supported. Validation and known limitations: - Collector memory, measured in a release build that drives the collector directly (bench/collector-chunk-20261111/README.md; host not recorded; 16-byte keys, 1 KB values, 62K to 1M references per SST in 1 to 16 groups): the buffers used about 8.3 to 9 bytes per reference for uncompressed values and 12.8 to 13.5 bytes for compressed values, including the chunk headers. With 1000 groups in a 62K-reference SST, they used about 14 bytes per uncompressed reference. - Chunk size, from the same measurement: for an SST with 1M references, the chunk size changed memory by less than 5% between 256 and 4096 bytes. For an SST with 62K references in 1000 groups, the partly filled chunks dominated: about 14 bytes per reference at 512 bytes, 22 at 1024 and 72 at 4096. 256 bytes used 3-4% more memory than 512 and made Finish() up to 1.6 times slower with 100 or more groups. - Reference index block size, measured in census db_bench runs (source report and host not recorded; 16-byte keys, 200-byte values, 1 MB blob files, no compression, fillrandom and then a full compaction): about 1.7 KB per SST with 32 MB write buffers and a target file size of 8 MB or 2 MB, and about 10 KB per SST with 1 MB write buffers and a target of 2 MB or 200 KB. The size is the bytes read for the index blocks divided by the number of SSTs. - Unit tests cover the BlobIndex encodings and their rejection of bad input, the metadata layout and every corruption check, the collector (sorted path, unsorted fallback, memory charge), the flush integration, the option checks and option gate, wide-column validation, and canonical varints. See the Test Plan. - The collection buffer is not spilled to disk. One SST with very many blob references needs a matching amount of charged memory: the per-reference amount given under "Collection and memory", plus up to one partly used 512-byte chunk per group. While a fallback group is sorted, it also needs 16 bytes of sort key per reference and a second copy of its entries. For a test group of 200K references with every other one out of order, the test comment estimates about 44 bytes per reference, and the test checks that the peak charge is at most 56 bytes per reference. - Flush still writes a single output file. Until diff 7, one compaction output collects every reference it gets, charged to the block cache. Diff 7 cuts compaction outputs (including L0) at a reference cap and keeps each user key's history in one output. With the option gate, a large output is reachable only through the test hook. Stack context (same in diffs 2-14, except that diffs 10-14 add the diff 10 route rule to the install line; "<== this diff" marks this diff's part): Goal: remove garbage from blob files without rewriting any SST. A value gets a stable identity when it is first written. Indirect blob GC copies the live values of whole files into a relocation file and installs routes in the MANIFEST. Readers follow the routes. SSTs are not changed. How a read finds a value (P is the relocation file that holds the value now): ``` SST entry indirect BlobIndex { generation number, origin, original offset, size, CRCs } | v Version file table: generation number -> live file, or route old file -> P | | live file route found | | v v blob file: read at original offset relocation file P (always one hop): relocation file: look up key look up key (origin, original offset) (origin, original offset) ``` Lifecycle of one value (in this example, F is the blob file where the value is first written, and P is the relocation file that GC writes): ``` write new value goes to blob file F; index: origin = F, generation number = F | census per generation number: the SSTs and references that use it; | per physical file: live bytes, ranked by garbage ratio GC job copy the live values of whole input files (stored bytes, checked | with the stored CRC) into relocation file P, keyed by | (origin, original offset) install one MANIFEST edit: add P, route each input generation number still | in use to P, re-point the routes into the inputs to P, and remove | each input (blob file retirement for a blob file, relocation | file deletion for a relocation file) flatten compaction sets generation number = P in the SSTs it rewrites | route route retirement: route deletion records remove the routes that delete no SST uses ``` Stack map: | Diff | Adds | |---|---| | 1 | Finalize blob metadata outside the DB mutex (prep) | | 2 | Indirect BlobIndex (kIndirectBlob) and exact SST reference metadata <== this diff | | 3 | Relocation file format and stable read path | | 4 | MANIFEST routes and relocation state in the Version | | 5 | Exact reverse reference census | | 6 | MANIFEST apply outside the DB mutex, with a precondition re-check | | 7 | Coarse flush metadata, per-SST reference cap, epoch runs | | 8 | Indirect blob GC building blocks | | 9 | Indirect blob GC job | | 10 | Flatten generation numbers; route retirement | | 11 | Remote compaction validation | | 12 | Dynamic options and operations | | 13 | Lost relocation files | | 14 | RepairDB rebuilds relocation state; option gate removed | | 15 | db_stress and crash test support | Stack context: terms for later diffs: | Term | Meaning | |---|---| | census | Background count, per generation number, of the SSTs and references that use it, plus live bytes per physical file (diff 5) | | indirect origin | Normal blob file written with the option on; it is the origin of unflattened references | | flatten | Compaction rewrites the generation number to the relocation file that holds the value, so old routes can go. Only flatten makes generation number != origin (diff 10) | | blob file retirement | MANIFEST record (kBlobFileRetirement) that removes a collected blob file from the Version (diff 4) | | route deletion | MANIFEST record that removes a route that no SST uses. Route retirement is the background job that writes these records (diff 10) | Tracking issue: meta#15301 Differential Revision: D124652097
Summary: Pull Request resolved: meta#15540 Why the change is required: The goal of this stack is indirect blob GC, a blob GC that does not depend on SSTs: it removes garbage from blob files without rewriting any SST. Such a GC needs two things that RocksDB does not have: - A value identity that does not change when the value moves to another file. - A checked list of the blob references that each SST keeps. The list must include older internal versions of a key that the SST still keeps (for example for snapshots), and the blob columns of wide-column entities. This diff adds both. Later diffs build the census (an in-memory reverse index from generation number to the SSTs that use it), GC and repair on top of this list. This diff in one paragraph: With the new column family option enable_blob_indirection on, the blob file writer gives each new value an indirect BlobIndex (type 3, kIndirectBlob). It stores the value's stable identity (origin file, original offset) and its CRCs (an uncompressed value stores only the CRC of the uncompressed value; a compressed value also stores the CRC of its stored bytes). Also with the option on, the block-based table builder collects every blob reference that the SST keeps. It writes them as "exact reference metadata": data blocks, one reference index block, and a 72-byte descriptor table property. In this diff only the table builder writes this metadata, and only tests read it. A temporary option gate (see "option gate" under Terms) keeps the option off outside tests. Separately, wide-column entities with blob columns are now fully checked before their blob indexes are used, also with the option off. Parts of this diff: 1. Indirect BlobIndex (core): types 3 and 4 in db/blob/blob_index.h, and the blob file writer that writes type 3. 2. Exact reference metadata format and decoders (core): db/blob/blob_reference_metadata.{h,cc}. 3. Collector and table builder integration (core): BlobReferenceMetadataCollector, its block cache charge, and BlockBasedTableBuilder. 4. Option enable_blob_indirection and the option gate. Tests need it to turn the feature on. You can skip this part on a first read. 5. Wide-column validation (WideColumnSerialization::ForEachBlobIndex). The table builder uses it to give the blob columns of each entity to the collector. It also changes behavior with the option off (see New behavior). You can skip this part on a first read. 6. Small helpers: canonical varints (util/coding.h) and CacheReservationHandle::Resize(). You can skip these on a first read. How to read this diff: 1. Overview: read "Why" and "This diff in one paragraph" above, and "Terms used in this diff" right below. Then read the top comment of db/blob/blob_reference_metadata.h (terms, layout diagrams, run and entry layout, index layout). 2. Indirect BlobIndex (db/blob/blob_index.h) and canonical varints (util/coding.h GetCanonicalVarint32/64): read the overview comment and the kIndirectBlob / kIndirectFlattenedBlob code. Every later part uses these fields: origin, original offset, generation number, logical CRC and stored CRC. The new formats read varints only in the canonical form. 3. Blob file writer (db/blob/blob_file_builder.cc, blob_log_format.cc, blob_log_writer.cc): see how a new value gets a type-3 index and how its CRC is computed once. 4. Format types and decoders: the structs in db/blob/blob_reference_metadata.h, then in the .cc the constants, GapBase, DecodeEntryBody, DecodeRunEntriesAt, DecodeBlobReferenceIndex and BlobReferenceGroupDecoder. They use only BlobIndex and the canonical varints. Learn the on-disk format here, before you read the writer. 5. Collector and its cache charge: first read the new CacheEntryRole kBlobReferenceMetadataBuildingBuffer (include/rocksdb/cache.h) and CacheReservationHandle::Resize (cache/cache_reservation_manager.h). Then read BlobReferenceMetadataCollector in this order: the class comment, Add, AppendSorted, BuildAndWriteDataBlocks, EncodeIndex, FinishOnce. Then read the unsorted fallback (AppendUnsorted, SortUnsortedGroup, SortAndEncodeGroup). 6. Wide-column change (db/wide/wide_column_serialization.cc, ForEachBlobIndex and WalkV2Columns): it gives every blob index of an entity to a callback after it checks the whole entity. The table builder uses it to feed the collector. UpdateBoundaries, BlobGarbageMeter and the compaction iterator also use it, with the option on or off. 7. Option: enable_blob_indirection (include/rocksdb/advanced_options.h, options/cf_options.cc), and the option checks and gate in db/column_family.cc (ValidateOptions, TEST_SetAllowBlobIndirection), db/db_impl/db_impl_open.cc and db_impl_secondary.cc (the FIFO check for opens that skip ValidateOptions). 8. Table integration: table/block_based/block_based_table_factory.cc (the shared reservation manager), then block_based_table_builder.cc (Rep constructor, AddImpl, WriteBlobReferenceMetadata, EstimatedTailSize). See where the collector is created, fed (AddImpl(), which both Add() and TryAdd() call) and finished. block_based_table_reader.cc maps the new block name to a block type. Terms used in this diff: Stack terms: | Term | Meaning | |---|---| | origin | Blob file where the value was first written | | original offset | Value offset in the origin file | | stable identity | (origin, original offset). Never changes, even when the value moves | | generation number | The file number that a reference uses to find its value. A new value has generation number == origin. When indirect blob GC moves the value, the reference keeps its generation number, and a route sends the read to the relocation file | | indirect BlobIndex | A kIndirectBlob (type 3) or kIndirectFlattenedBlob (type 4) index. It stores the stable identity, so the value can move without an SST rewrite | | unflattened reference | Index with generation number == origin: the value was not moved, or it was moved but no compaction has rewritten its reference yet (see "flatten" in "Stack context: terms for later diffs") | | flattened reference | Index with generation number != origin (type 4). Only flatten (diff 10) writes it; this diff defines the encoding | | route | Maps an old file number (a file that GC has removed from the Version) to the relocation file that now holds its live values. One hop: a target is never itself routed. Stored in the MANIFEST (diff 4) | | relocation file | Block-based table keyed by (origin, original offset) that holds copied values. Each value is the stored bytes plus a 5-byte trailer (kSimpleGen2BlobTrailerSize) | | logical CRC | CRC32C of the uncompressed value (BlobIndex::stable_checksum()) | | stored CRC | CRC32C of the stored bytes plus the compression byte | | physical file | A file that holds value bytes: a normal blob file or a relocation file. A generation number can name a physical file, or it can be only the key of a route | | exact reference metadata | Per-SST sorted list of every blob reference the SST keeps, written when the SST is built. References with the same encoding class, generation number and origin form a group. The list is stored as data blocks, with an index of the groups. An SST that has it is an exact SST | | option gate | Until diff 14, ColumnFamilyData::ValidateOptions returns NotSupported for enable_blob_indirection on a read-write open, CreateColumnFamily and SetOptions, unless a test calls ColumnFamilyData::TEST_SetAllowBlobIndirection(true). It keeps the option off in production while the stack is not complete | Terms of this diff (all defined at the top of db/blob/blob_reference_metadata.h): | Term | Meaning | |---|---| | encoding class (class) | kDirect (a legacy kBlob or kBlobTTL index), unflattened, or flattened | | reference group (group) | All references of one SST with the same (encoding class, generation number, origin) | | run | The part of one group inside one data block | | record bytes | The bytes the value's record takes in the file named by its generation number | | gap | How far a record is from where it would be if the records were back to back. A gap of 0 means back to back | | descriptor | The 72-byte table property that points to the reference index block and holds the totals, the CF ID, the SST file number and the relocation-state fingerprint | | relocation-state fingerprint | 128-bit hash of the routes and relocation files of the Version that an SST was built from (fields route_snapshot_fingerprint_*). Written as 0 in this diff | | legacy | Written the way RocksDB writes it without blob indirection | Previous behavior: - A BlobIndex named only a physical blob file and an offset. A value could not move to another file without rewriting every SST that referenced it. - An SST had no metadata that listed its blob references. The only way to find them was to scan its user keys. - These readers get the blob indexes of a wide-column entity through ForEachBlobFileNumber: UpdateBoundaries, BlobGarbageMeter, the compaction iterator, WideColumnSerialization::RemoveBlobColumns, and the referenced_blob_bytes table property count in BlockBasedTableBuilder::Add. ForEachBlobFileNumber decoded each blob index, but it did not check the rest of the entity framing (name order, section sizes, exact input size), and it could call the callback before it found a later error. New behavior: Separate change (option on or off): - The same readers (UpdateBoundaries, BlobGarbageMeter, the compaction iterator, RemoveBlobColumns, and the referenced_blob_bytes count in BlockBasedTableBuilder::Add) use WideColumnSerialization::ForEachBlobIndex, with the option on or off. With the option on, the table builder also uses it to give the blob columns of each entity to the collector. For an entity with blob columns, it checks the full framing, then decodes and validates every blob index before any callback runs. So an entity with a malformed framing now fails the flush or compaction, even with the option off. The referenced_blob_bytes count itself does not fail the build on a malformed entity. It removes that property from the SST, as in base. Option and option gate: - This diff adds a new column family option, `enable_blob_indirection`. It is off by default, and SetOptions can change it. - The option gate (see Terms) keeps the option off outside tests until the stack is complete. ColumnFamilyData::ValidateOptions rejects the option with NotSupported on a read-write open, CreateColumnFamily and SetOptions. Read-only opens (DBImplReadOnly and CompactedDBImpl) and secondary opens (including DB::OpenAndCompact) do not run ValidateOptions; they check only the FIFO rule (see Implementation). Only tests can turn the option on, through the internal ColumnFamilyData::TEST_SetAllowBlobIndirection(). DBTestBase sets it. - When the option is on, ValidateOptions also requires enable_blob_files, the block-based table format, no blob direct write, and a compaction style other than FIFO. Blob file writer (option on): - The blob file writer encodes each new value with a stable identity (origin file, original offset) and two checksums: - the logical CRC: the CRC32C of the uncompressed value. - the stored CRC: the CRC32C of the exact stored (maybe compressed) bytes, extended with the compression byte. - For an uncompressed value the stored CRC is not written, because the reader can compute it from the logical CRC. - The checksums have no version field of their own. The BlobIndex type byte gives the format version: the two new types, 3 (kIndirectBlob) and 4 (kIndirectFlattenedBlob, see Implementation), are BlobIndex::kIndirectBlobFormatVersion (1). A later format uses new type values. - The read path does not check these checksums yet. The check starts in diff 3. Exact reference metadata (option on): 1. What: a sorted list of every blob reference that the SST keeps, in plain blob values and in the blob columns of wide-column entities. 2. Where: data blocks (target 16 KB each before compression; small blocks waste less I/O, because indirect blob GC reads a whole block to get a few groups) and one reference index block, written after the SST data blocks and before the SST tail, plus a 72-byte descriptor table property. 3. Who reads it: only tests in this diff. The census, GC and RepairDB read it in later diffs. 4. Details: the top comment of db/blob/blob_reference_metadata.h has the full format with diagrams, the record-bytes rules and the gap rule. - Not collected: SstFileWriter output (it has no DB file number) and any output whose CF is unknown (it has no CF ID). The descriptor needs both values, so these outputs stay in the legacy form. Checks on the metadata: - A repeated or overlapping reference in one group fails the SST build with Corruption. Readers check every count, order, size and total against the index and the descriptor, so corrupt metadata returns Corruption. The writer never makes a data block larger than kTargetDataBlockSize (16 KB), and readers accept up to kMaxDataBlockSize (64 KB). Details: "Readers" and "Integrity" at the top of db/blob/blob_reference_metadata.h, and BlobReferenceGroupDecoder. Collection and memory: - Each reference is encoded when it is added, in the on-disk entry encoding, into its group's chain of 512-byte chunks. Only a group that takes the unsorted fallback is encoded a second time, in Finish(). Inside a group, references normally arrive in rising offset order, so no reference is sorted; at the end only the groups are sorted. A group that gets a lower offset takes the unsorted fallback. The BlobReferenceMetadataCollector class comment explains the chunks, the chunk size and the fallback. - The collector keeps every reference of the SST in memory until the SST is finished, and charges this memory to the block cache under a new CacheEntryRole, kBlobReferenceMetadataBuildingBuffer. "Memory usage" and "Memory charge" in the BlobReferenceMetadataCollector class comment explain the parts of this memory and the charge rule. The measured amounts are under "Validation and known limitations". - There is no fixed limit on references. The collector fails with MemoryLimit above 2^20 groups or 2^20 data blocks in one SST; these are persisted format limits. It also fails with MemoryLimit when the reference index block could be larger than 64 MB (kMaxTopLevelMetadataBlockSize), which can happen from about 945K groups (64 MB / kMaxIndexEntrySize, 71 bytes per group), or from fewer groups when the SST also has many data blocks. - There is no charge when the table has no block cache or when charging for this role is disabled. Each block-based table factory holds one cache reservation manager for this role, shared by its builders and the SST summaries. If the charge fails, the SST build fails, so the flush or compaction fails with a background error. The buffers and their charge are released when the SST is finished or abandoned. Implementation: - BlobIndex adds two indirect encodings. The legacy encodings do not change. - Type 3, kIndirectBlob (unflattened, generation number == origin), stores the origin, the original offset, the stored size, the compression type, the logical CRC and, for compressed values, the stored CRC. It does not store the generation number. This saves the size of one varint file number per reference (about 3 to 4 bytes for typical file numbers; an estimate, not measured). - Type 4, kIndirectFlattenedBlob (flattened), stores the generation number first, then the same fields. Its generation number must be above the origin. The decoder rejects type 4 with generation number <= origin, so each reference has one encoding. - stored_checksum() computes the stored CRC of an uncompressed value when a caller asks for it: the logical CRC extended with the compression byte. A decode does not compute it, because most decodes (for example in compaction) never read it. - The encoders take the fields as one BlobIndex::IndirectBlobFields, set by field name, so values of the same type (file number, offset and size; the two CRCs) cannot be swapped. EncodeIndirectBlob() takes the generation number as a separate argument; EncodeUnflattenedIndirectBlob() uses the origin, so the common path cannot leave the generation number unset. - The new files blob_reference_metadata.{h,cc} define the exact reference metadata. The files also contain: - BlobReferenceMetadataCollector, the collector described above. It takes the SST identity (CF ID, SST file number and the relocation-state fingerprint) as one BlobReferenceSstIdentity, set by field name. - BlobReferenceGroupKey, the key of a group: (encoding class, generation number, origin). It has the order, the hash and the validity check that the collector map, the sort, the decoder and FindBlobReferenceGroup() use. BlobReference and BlobReferenceGroup keep their flat fields and return the key through group_key(). - The index codec and the descriptor table property. - BlobReferenceGroupDecoder, the only decoder of reference runs. It reads a group one data block at a time and holds every corruption check of a group walk. Readers that visit one group (found by a binary search of the index) use it, and so do readers that visit every block in order. - Callers in this diff: only the table builder runs the collector. The decoders (DecodeBlobReferenceIndex, FindBlobReferenceGroup, BlobReferenceGroupDecoder, ForEachBlobReference*) are used only by tests. kMaxDecodedBlobReferenceBytes is used only in compile-time checks of the count limits. The census (diff 5) and indirect blob GC (diffs 8 and 9) are their first production users. The census is also the first user of BlobReferenceSstSummary (see the last Implementation bullet). - Descriptor: the CF ID and the SST file number are the repair fields. They let RepairDB tell which column family and SST the list belongs to without a MANIFEST. The decoder rejects a descriptor whose repair fields are not set (unknown CF ID or SST file number 0). The descriptor also has a 128-bit relocation-state fingerprint (the route_snapshot_fingerprint fields). Remote compaction uses it in diff 11 to check that an SST built elsewhere used the same routes as the job that asked for it. In this diff the table builder always writes 0, and DecodeFrom() does not check it. The field is in the format now, so the descriptor keeps one format version (1) through the stack. While the option gate is on, no data in this format exists outside tests. - The collector validates each BlobIndex once, when it is added. Its Finish() writes the data blocks and then the reference index block through one block writer, and fills the descriptor, including the index handle. Destroying the collector frees its memory and then its charge, also when the build is abandoned. - BlockBasedTableBuilder creates the collector with the CF ID and SST file number from TableBuilderOptions. It passes the size of each entry's user key to the collector, so the collector can compute the exact size of each blob record. It adds the collector's EstimatedEncodedSize() to its tail size estimate: the blocks are written before the tail, but output file size planning uses this estimate. The reference blocks are written with BlockType::kUserDefinedIndex, which keeps their contents as opaque bytes. Readers read them without the block cache, so prepopulate_block_cache does not warm them into the block cache (WriteMaybeCompressedBlock() with warm_cache false). - To save CPU, compaction decodes each kTypeBlobIndex output value once, in CompactionOutputs::AddToOutput(). It passes the decode to TableBuilder::TryAdd() in the new field TableBuilderAddContext::decoded_blob_index, and to new overloads of BlobGarbageMeter::ProcessOutFlow() and FileMetaData::UpdateBoundaries(). BlockBasedTableBuilder decodes a kTypeBlobIndex value once for both the referenced_blob_bytes count and the collector, and uses the caller's decode when it has one. If the decode fails, no decode is passed, and each reader decodes the value itself and returns its usual error. This applies with the option on or off. The SST bytes are the same as when each reader decodes the value itself. - The collector converts each BlobIndex into a reference without deriving the stored CRC of an uncompressed value (ConvertBlobIndex() in db/blob/blob_reference_metadata.cc). The collector does not write that CRC, because a reader derives it again from the logical CRC. Deriving it would cost CPU for every reference of every output SST. BlobReference::FromBlobIndex() still derives it. - With the option on, the blob file builder computes the CRC32C of each stored value once. The blob log record CRC combines it with the key CRC (crc32c::Crc32cCombine). The stored CRC extends it with the compression byte. For an uncompressed value it is also the logical CRC. The bytes written do not change. - WideColumnSerialization::ForEachBlobIndex replaces ForEachBlobFileNumber. It reads the entity header and column types once. For a V1 entity (no column types and no blob columns), it returns right after the header. For a V2 entity without blob columns, it returns right after the type scan. It shares one V2 column walker (WalkV2Columns) with Deserialize(), so the entity is parsed in one pass. - Read-only opens (DBImplReadOnly, CompactedDBImpl) and secondary opens skip ValidateOptions, so they check the compaction style against the option once per column family: in DBImpl::Recover() for read-only opens, and at the start of OpenAsSecondaryImpl() for secondary opens. DB::OpenAndCompact opens through OpenAsSecondaryImpl(), so it gets the same check. - CacheReservationHandle gets Resize(), so each collector or summary can change its own part of the shared charge. - util/coding.h adds GetCanonicalVarint32/64, which reject overlong varint encodings. The new formats use them, so each value has one encoding. - BlobReferenceSstSummary is the in-memory list of an SST's groups and totals that Finish() can return. It has no caller yet: the table builder passes a null summary output. It is checked and does not change after Finish(), so the finished builder, the SST metadata and the census (its first user, in diff 5) can share it. It is in this diff because Finish() hands the group buffer and its cache charge over to the summary. Adding the summary later would mean rewriting the charge code of Finish() in diff 5, so the charge code is written and tested once, here. - sst_dump prints a "Blob reference metadata" section after the raw user collected properties of an SST that has exact metadata: the reference count, the stored value bytes, the column family ID, the SST file number when built and the index block handle. An SST without the descriptor prints nothing more. Safety boundaries and unsupported cases: - Public API: this diff adds the option `enable_blob_indirection` with the C API functions `rocksdb_options_set_enable_blob_indirection` and `rocksdb_options_get_enable_blob_indirection`. It also inserts a new CacheEntryRole value before kMisc, which changes the numeric value of kMisc and kNumCacheEntryRoles. - With the option off, new blob values and SSTs are written in the legacy form. Legacy blob indexes stay readable. - One behavior change applies with the option off too: the five readers listed under "Separate change" fully validate wide-column entities with blob columns and their blob indexes (see "Separate change" above). - Exact reference metadata has one format (version 1) with a fixed 72-byte descriptor. The decoders reject any other version as Corruption. - The metadata blocks are protected by the table's block checksum only. With checksum = kNoChecksum, corrupt metadata is detected only when it breaks the format or order checks. - The index decoder checks each decoded count against the remaining input size before it reserves memory. So a small corrupt index cannot make it allocate a large vector. - Inlined-TTL and same-file blob indexes are not collected and cannot be relocated. Encoding, count and checksum errors return an error; a bad reference is never skipped. - Option gate (see Terms): outside tests no DB writes data in the new format. The gate checks only the option value, never persisted state. Diff 14 removes it, once data written with the option stays readable through the end of the stack and the safety checks for turning the option on are in place. - Downgrading the binary after data was written with the option on is not supported. Validation and known limitations: - Collector memory, measured in a release build that drives the collector directly (bench/collector-chunk-20261111/README.md; host not recorded; 16-byte keys, 1 KB values, 62K to 1M references per SST in 1 to 16 groups): the buffers used about 8.3 to 9 bytes per reference for uncompressed values and 12.8 to 13.5 bytes for compressed values, including the chunk headers. With 1000 groups in a 62K-reference SST, they used about 14 bytes per uncompressed reference. - Chunk size, from the same measurement: for an SST with 1M references, the chunk size changed memory by less than 5% between 256 and 4096 bytes. For an SST with 62K references in 1000 groups, the partly filled chunks dominated: about 14 bytes per reference at 512 bytes, 22 at 1024 and 72 at 4096. 256 bytes used 3-4% more memory than 512 and made Finish() up to 1.6 times slower with 100 or more groups. - Reference index block size, measured in census db_bench runs (source report and host not recorded; 16-byte keys, 200-byte values, 1 MB blob files, no compression, fillrandom and then a full compaction): about 1.7 KB per SST with 32 MB write buffers and a target file size of 8 MB or 2 MB, and about 10 KB per SST with 1 MB write buffers and a target of 2 MB or 200 KB. The size is the bytes read for the index blocks divided by the number of SSTs. - Unit tests cover the BlobIndex encodings and their rejection of bad input, the metadata layout and every corruption check, the collector (sorted path, unsorted fallback, memory charge), the flush integration, the option checks and option gate, wide-column validation, and canonical varints. See the Test Plan. - The collection buffer is not spilled to disk. One SST with very many blob references needs a matching amount of charged memory: the per-reference amount given under "Collection and memory", plus up to one partly used 512-byte chunk per group. While a fallback group is sorted, it also needs 16 bytes of sort key per reference and a second copy of its entries. For a test group of 200K references with every other one out of order, the test comment estimates about 44 bytes per reference, and the test checks that the peak charge is at most 56 bytes per reference. - Flush still writes a single output file. Until diff 7, one compaction output collects every reference it gets, charged to the block cache. Diff 7 cuts compaction outputs (including L0) at a reference cap and keeps each user key's history in one output. With the option gate, a large output is reachable only through the test hook. Stack context (same in diffs 2-14, except that diffs 10-14 add the diff 10 route rule to the install line; "<== this diff" marks this diff's part): Goal: remove garbage from blob files without rewriting any SST. A value gets a stable identity when it is first written. Indirect blob GC copies the live values of whole files into a relocation file and installs routes in the MANIFEST. Readers follow the routes. SSTs are not changed. How a read finds a value (P is the relocation file that holds the value now): ``` SST entry indirect BlobIndex { generation number, origin, original offset, size, CRCs } | v Version file table: generation number -> live file, or route old file -> P | | live file route found | | v v blob file: read at original offset relocation file P (always one hop): relocation file: look up key look up key (origin, original offset) (origin, original offset) ``` Lifecycle of one value (in this example, F is the blob file where the value is first written, and P is the relocation file that GC writes): ``` write new value goes to blob file F; index: origin = F, generation number = F | census per generation number: the SSTs and references that use it; | per physical file: live bytes, ranked by garbage ratio GC job copy the live values of whole input files (stored bytes, checked | with the stored CRC) into relocation file P, keyed by | (origin, original offset) install one MANIFEST edit: add P, route each input generation number still | in use to P, re-point the routes into the inputs to P, and remove | each input (blob file retirement for a blob file, relocation | file deletion for a relocation file) flatten compaction sets generation number = P in the SSTs it rewrites | route route retirement: route deletion records remove the routes that delete no SST uses ``` Stack map: | Diff | Adds | |---|---| | 1 | Finalize blob metadata outside the DB mutex (prep) | | 2 | Indirect BlobIndex (kIndirectBlob) and exact SST reference metadata <== this diff | | 3 | Relocation file format and stable read path | | 4 | MANIFEST routes and relocation state in the Version | | 5 | Exact reverse reference census | | 6 | MANIFEST apply outside the DB mutex, with a precondition re-check | | 7 | Coarse flush metadata, per-SST reference cap, epoch runs | | 8 | Indirect blob GC building blocks | | 9 | Indirect blob GC job | | 10 | Flatten generation numbers; route retirement | | 11 | Remote compaction validation | | 12 | Dynamic options and operations | | 13 | Lost relocation files | | 14 | RepairDB rebuilds relocation state; option gate removed | | 15 | db_stress and crash test support | Stack context: terms for later diffs: | Term | Meaning | |---|---| | census | Background count, per generation number, of the SSTs and references that use it, plus live bytes per physical file (diff 5) | | indirect origin | Normal blob file written with the option on; it is the origin of unflattened references | | flatten | Compaction rewrites the generation number to the relocation file that holds the value, so old routes can go. Only flatten makes generation number != origin (diff 10) | | blob file retirement | MANIFEST record (kBlobFileRetirement) that removes a collected blob file from the Version (diff 4) | | route deletion | MANIFEST record that removes a route that no SST uses. Route retirement is the background job that writes these records (diff 10) | Tracking issue: meta#15301 Differential Revision: D124652097
Summary: Pull Request resolved: meta#15540 Why the change is required: The goal of this stack is indirect blob GC, a blob GC that does not depend on SSTs: it removes garbage from blob files without rewriting any SST. Such a GC needs two things that RocksDB does not have: - A value identity that does not change when the value moves to another file. - A checked list of the blob references that each SST keeps. The list must include older internal versions of a key that the SST still keeps (for example for snapshots), and the blob columns of wide-column entities. This diff adds both. Later diffs build the census (an in-memory reverse index from generation number to the SSTs that use it), GC and repair on top of this list. This diff in one paragraph: With the new column family option enable_blob_indirection on, the blob file writer gives each new value an indirect BlobIndex (type 3, kIndirectBlob). It stores the value's stable identity (origin file, original offset) and its CRCs (an uncompressed value stores only the CRC of the uncompressed value; a compressed value also stores the CRC of its stored bytes). Also with the option on, the block-based table builder collects every blob reference that the SST keeps. It writes them as "exact reference metadata": data blocks, one reference index block, and a 72-byte descriptor table property. In this diff only the table builder writes this metadata, and only tests read it. A temporary option gate (see "option gate" under Terms) keeps the option off outside tests. Separately, wide-column entities with blob columns are now fully checked before their blob indexes are used, also with the option off. Parts of this diff: 1. Indirect BlobIndex (core): types 3 and 4 in db/blob/blob_index.h, and the blob file writer that writes type 3. 2. Exact reference metadata format and decoders (core): db/blob/blob_reference_metadata.{h,cc}. 3. Collector and table builder integration (core): BlobReferenceMetadataCollector, its block cache charge, and BlockBasedTableBuilder. 4. Option enable_blob_indirection and the option gate. Tests need it to turn the feature on. You can skip this part on a first read. 5. Wide-column validation (WideColumnSerialization::ForEachBlobIndex). The table builder uses it to give the blob columns of each entity to the collector. It also changes behavior with the option off (see New behavior). You can skip this part on a first read. 6. Small helpers: canonical varints (util/coding.h) and CacheReservationHandle::Resize(). You can skip these on a first read. How to read this diff: 1. Overview: read "Why" and "This diff in one paragraph" above, and "Terms used in this diff" right below. Then read the top comment of db/blob/blob_reference_metadata.h (terms, layout diagrams, run and entry layout, index layout). 2. Indirect BlobIndex (db/blob/blob_index.h) and canonical varints (util/coding.h GetCanonicalVarint32/64): read the overview comment and the kIndirectBlob / kIndirectFlattenedBlob code. Every later part uses these fields: origin, original offset, generation number, logical CRC and stored CRC. The new formats read varints only in the canonical form. 3. Blob file writer (db/blob/blob_file_builder.cc, blob_log_format.cc, blob_log_writer.cc): see how a new value gets a type-3 index and how its CRC is computed once. 4. Format types and decoders: the structs in db/blob/blob_reference_metadata.h, then in the .cc the constants, GapBase, DecodeEntryBody, DecodeRunEntriesAt, DecodeBlobReferenceIndex and BlobReferenceGroupDecoder. They use only BlobIndex and the canonical varints. Learn the on-disk format here, before you read the writer. 5. Collector and its cache charge: first read the new CacheEntryRole kBlobReferenceMetadataBuildingBuffer (include/rocksdb/cache.h) and CacheReservationHandle::Resize (cache/cache_reservation_manager.h). Then read BlobReferenceMetadataCollector in this order: the class comment, Add, AppendSorted, BuildAndWriteDataBlocks, EncodeIndex, FinishOnce. Then read the unsorted fallback (AppendUnsorted, SortUnsortedGroup, SortAndEncodeGroup). 6. Wide-column change (db/wide/wide_column_serialization.cc, ForEachBlobIndex and WalkV2Columns): it gives every blob index of an entity to a callback after it checks the whole entity. The table builder uses it to feed the collector. UpdateBoundaries, BlobGarbageMeter and the compaction iterator also use it, with the option on or off. 7. Option: enable_blob_indirection (include/rocksdb/advanced_options.h, options/cf_options.cc), and the option checks and gate in db/column_family.cc (ValidateOptions, TEST_SetAllowBlobIndirection), db/db_impl/db_impl_open.cc and db_impl_secondary.cc (the FIFO check for opens that skip ValidateOptions). 8. Table integration: table/block_based/block_based_table_factory.cc (the shared reservation manager), then block_based_table_builder.cc (Rep constructor, AddImpl, WriteBlobReferenceMetadata, EstimatedTailSize). See where the collector is created, fed (AddImpl(), which both Add() and TryAdd() call) and finished. block_based_table_reader.cc maps the new block name to a block type. Terms used in this diff: Stack terms: | Term | Meaning | |---|---| | origin | Blob file where the value was first written | | original offset | Value offset in the origin file | | stable identity | (origin, original offset). Never changes, even when the value moves | | generation number | The file number that a reference uses to find its value. A new value has generation number == origin. When indirect blob GC moves the value, the reference keeps its generation number, and a route sends the read to the relocation file | | indirect BlobIndex | A kIndirectBlob (type 3) or kIndirectFlattenedBlob (type 4) index. It stores the stable identity, so the value can move without an SST rewrite | | unflattened reference | Index with generation number == origin: the value was not moved, or it was moved but no compaction has rewritten its reference yet (see "flatten" in "Stack context: terms for later diffs") | | flattened reference | Index with generation number != origin (type 4). Only flatten (diff 10) writes it; this diff defines the encoding | | route | Maps an old file number (a file that GC has removed from the Version) to the relocation file that now holds its live values. One hop: a target is never itself routed. Stored in the MANIFEST (diff 4) | | relocation file | Block-based table keyed by (origin, original offset) that holds copied values. Each value is the stored bytes plus a 5-byte trailer (kSimpleGen2BlobTrailerSize) | | logical CRC | CRC32C of the uncompressed value (BlobIndex::stable_checksum()) | | stored CRC | CRC32C of the stored bytes plus the compression byte | | physical file | A file that holds value bytes: a normal blob file or a relocation file. A generation number can name a physical file, or it can be only the key of a route | | exact reference metadata | Per-SST sorted list of every blob reference the SST keeps, written when the SST is built. References with the same encoding class, generation number and origin form a group. The list is stored as data blocks, with an index of the groups. An SST that has it is an exact SST | | option gate | Until diff 14, ColumnFamilyData::ValidateOptions returns NotSupported for enable_blob_indirection on a read-write open, CreateColumnFamily and SetOptions, unless a test calls ColumnFamilyData::TEST_SetAllowBlobIndirection(true). It keeps the option off in production while the stack is not complete | Terms of this diff (all defined at the top of db/blob/blob_reference_metadata.h): | Term | Meaning | |---|---| | encoding class (class) | kDirect (a legacy kBlob or kBlobTTL index), unflattened, or flattened | | reference group (group) | All references of one SST with the same (encoding class, generation number, origin) | | run | The part of one group inside one data block | | record bytes | The bytes the value's record takes in the file named by its generation number | | gap | How far a record is from where it would be if the records were back to back. A gap of 0 means back to back | | descriptor | The 72-byte table property that points to the reference index block and holds the totals, the CF ID, the SST file number and the relocation-state fingerprint | | relocation-state fingerprint | 128-bit hash of the routes and relocation files of the Version that an SST was built from (fields route_snapshot_fingerprint_*). Written as 0 in this diff | | legacy | Written the way RocksDB writes it without blob indirection | Previous behavior: - A BlobIndex named only a physical blob file and an offset. A value could not move to another file without rewriting every SST that referenced it. - An SST had no metadata that listed its blob references. The only way to find them was to scan its user keys. - These readers get the blob indexes of a wide-column entity through ForEachBlobFileNumber: UpdateBoundaries, BlobGarbageMeter, the compaction iterator, WideColumnSerialization::RemoveBlobColumns, and the referenced_blob_bytes table property count in BlockBasedTableBuilder::Add. ForEachBlobFileNumber decoded each blob index, but it did not check the rest of the entity framing (name order, section sizes, exact input size), and it could call the callback before it found a later error. New behavior: Separate change (option on or off): - The same readers (UpdateBoundaries, BlobGarbageMeter, the compaction iterator, RemoveBlobColumns, and the referenced_blob_bytes count in BlockBasedTableBuilder::Add) use WideColumnSerialization::ForEachBlobIndex, with the option on or off. With the option on, the table builder also uses it to give the blob columns of each entity to the collector. For an entity with blob columns, it checks the full framing, then decodes and validates every blob index before any callback runs. So an entity with a malformed framing now fails the flush or compaction, even with the option off. The referenced_blob_bytes count itself does not fail the build on a malformed entity. It removes that property from the SST, as in base. Option and option gate: - This diff adds a new column family option, `enable_blob_indirection`. It is off by default, and SetOptions can change it. - The option gate (see Terms) keeps the option off outside tests until the stack is complete. ColumnFamilyData::ValidateOptions rejects the option with NotSupported on a read-write open, CreateColumnFamily and SetOptions. Read-only opens (DBImplReadOnly and CompactedDBImpl) and secondary opens (including DB::OpenAndCompact) do not run ValidateOptions; they check only the FIFO rule (see Implementation). Only tests can turn the option on, through the internal ColumnFamilyData::TEST_SetAllowBlobIndirection(). DBTestBase sets it. - When the option is on, ValidateOptions also requires enable_blob_files, the block-based table format, no blob direct write, and a compaction style other than FIFO. Blob file writer (option on): - The blob file writer encodes each new value with a stable identity (origin file, original offset) and two checksums: - the logical CRC: the CRC32C of the uncompressed value. - the stored CRC: the CRC32C of the exact stored (maybe compressed) bytes, extended with the compression byte. - For an uncompressed value the stored CRC is not written, because the reader can compute it from the logical CRC. - The checksums have no version field of their own. The BlobIndex type byte gives the format version: the two new types, 3 (kIndirectBlob) and 4 (kIndirectFlattenedBlob, see Implementation), are BlobIndex::kIndirectBlobFormatVersion (1). A later format uses new type values. - The read path does not check these checksums yet. The check starts in diff 3. Exact reference metadata (option on): 1. What: a sorted list of every blob reference that the SST keeps, in plain blob values and in the blob columns of wide-column entities. 2. Where: data blocks (target 16 KB each before compression; small blocks waste less I/O, because indirect blob GC reads a whole block to get a few groups) and one reference index block, written after the SST data blocks and before the SST tail, plus a 72-byte descriptor table property. 3. Who reads it: only tests in this diff. The census, GC and RepairDB read it in later diffs. 4. Details: the top comment of db/blob/blob_reference_metadata.h has the full format with diagrams, the record-bytes rules and the gap rule. - Not collected: SstFileWriter output (it has no DB file number) and any output whose CF is unknown (it has no CF ID). The descriptor needs both values, so these outputs stay in the legacy form. Checks on the metadata: - A repeated or overlapping reference in one group fails the SST build with Corruption. Readers check every count, order, size and total against the index and the descriptor, so corrupt metadata returns Corruption. The writer never makes a data block larger than kTargetDataBlockSize (16 KB), and readers accept up to kMaxDataBlockSize (64 KB). Details: "Readers" and "Integrity" at the top of db/blob/blob_reference_metadata.h, and BlobReferenceGroupDecoder. Collection and memory: - Each reference is encoded when it is added, in the on-disk entry encoding, into its group's chain of 512-byte chunks. Only a group that takes the unsorted fallback is encoded a second time, in Finish(). Inside a group, references normally arrive in rising offset order, so no reference is sorted; at the end only the groups are sorted. A group that gets a lower offset takes the unsorted fallback. The BlobReferenceMetadataCollector class comment explains the chunks, the chunk size and the fallback. - The collector keeps every reference of the SST in memory until the SST is finished, and charges this memory to the block cache under a new CacheEntryRole, kBlobReferenceMetadataBuildingBuffer. "Memory usage" and "Memory charge" in the BlobReferenceMetadataCollector class comment explain the parts of this memory and the charge rule. The measured amounts are under "Validation and known limitations". - There is no fixed limit on references. The collector fails with MemoryLimit above 2^20 groups or 2^20 data blocks in one SST; these are persisted format limits. It also fails with MemoryLimit when the reference index block could be larger than 64 MB (kMaxTopLevelMetadataBlockSize), which can happen from about 945K groups (64 MB / kMaxIndexEntrySize, 71 bytes per group), or from fewer groups when the SST also has many data blocks. - There is no charge when the table has no block cache or when charging for this role is disabled. Each block-based table factory holds one cache reservation manager for this role, shared by its builders and the SST summaries. If the charge fails, the SST build fails, so the flush or compaction fails with a background error. The buffers and their charge are released when the SST is finished or abandoned. Implementation: - BlobIndex adds two indirect encodings. The legacy encodings do not change. - Type 3, kIndirectBlob (unflattened, generation number == origin), stores the origin, the original offset, the stored size, the compression type, the logical CRC and, for compressed values, the stored CRC. It does not store the generation number. This saves the size of one varint file number per reference (about 3 to 4 bytes for typical file numbers; an estimate, not measured). - Type 4, kIndirectFlattenedBlob (flattened), stores the generation number first, then the same fields. Its generation number must be above the origin. The decoder rejects type 4 with generation number <= origin, so each reference has one encoding. - stored_checksum() computes the stored CRC of an uncompressed value when a caller asks for it: the logical CRC extended with the compression byte. A decode does not compute it, because most decodes (for example in compaction) never read it. - The encoders take the fields as one BlobIndex::IndirectBlobFields, set by field name, so values of the same type (file number, offset and size; the two CRCs) cannot be swapped. EncodeIndirectBlob() takes the generation number as a separate argument; EncodeUnflattenedIndirectBlob() uses the origin, so the common path cannot leave the generation number unset. - The new files blob_reference_metadata.{h,cc} define the exact reference metadata. The files also contain: - BlobReferenceMetadataCollector, the collector described above. It takes the SST identity (CF ID, SST file number and the relocation-state fingerprint) as one BlobReferenceSstIdentity, set by field name. - BlobReferenceGroupKey, the key of a group: (encoding class, generation number, origin). It has the order, the hash and the validity check that the collector map, the sort, the decoder and FindBlobReferenceGroup() use. BlobReference and BlobReferenceGroup keep their flat fields and return the key through group_key(). - The index codec and the descriptor table property. - BlobReferenceGroupDecoder, the only decoder of reference runs. It reads a group one data block at a time and holds every corruption check of a group walk. Readers that visit one group (found by a binary search of the index) use it, and so do readers that visit every block in order. - Callers in this diff: only the table builder runs the collector. The decoders (DecodeBlobReferenceIndex, FindBlobReferenceGroup, BlobReferenceGroupDecoder, ForEachBlobReference*) are used only by tests. kMaxDecodedBlobReferenceBytes is used only in compile-time checks of the count limits. The census (diff 5) and indirect blob GC (diffs 8 and 9) are their first production users. The census is also the first user of BlobReferenceSstSummary (see the last Implementation bullet). - Descriptor: the CF ID and the SST file number are the repair fields. They let RepairDB tell which column family and SST the list belongs to without a MANIFEST. The decoder rejects a descriptor whose repair fields are not set (unknown CF ID or SST file number 0). The descriptor also has a 128-bit relocation-state fingerprint (the route_snapshot_fingerprint fields). Remote compaction uses it in diff 11 to check that an SST built elsewhere used the same routes as the job that asked for it. In this diff the table builder always writes 0, and DecodeFrom() does not check it. The field is in the format now, so the descriptor keeps one format version (1) through the stack. While the option gate is on, no data in this format exists outside tests. - The collector validates each BlobIndex once, when it is added. Its Finish() writes the data blocks and then the reference index block through one block writer, and fills the descriptor, including the index handle. Destroying the collector frees its memory and then its charge, also when the build is abandoned. - BlockBasedTableBuilder creates the collector with the CF ID and SST file number from TableBuilderOptions. It passes the size of each entry's user key to the collector, so the collector can compute the exact size of each blob record. It adds the collector's EstimatedEncodedSize() to its tail size estimate: the blocks are written before the tail, but output file size planning uses this estimate. The reference blocks are written with BlockType::kUserDefinedIndex, which keeps their contents as opaque bytes. Readers read them without the block cache, so prepopulate_block_cache does not warm them into the block cache (WriteMaybeCompressedBlock() with warm_cache false). - To save CPU, compaction decodes each kTypeBlobIndex output value once, in CompactionOutputs::AddToOutput(). It passes the decode to TableBuilder::TryAdd() in the new field TableBuilderAddContext::decoded_blob_index, and to new overloads of BlobGarbageMeter::ProcessOutFlow() and FileMetaData::UpdateBoundaries(). BlockBasedTableBuilder decodes a kTypeBlobIndex value once for both the referenced_blob_bytes count and the collector, and uses the caller's decode when it has one. If the decode fails, no decode is passed, and each reader decodes the value itself and returns its usual error. This applies with the option on or off. The SST bytes are the same as when each reader decodes the value itself. - The collector converts each BlobIndex into a reference without deriving the stored CRC of an uncompressed value (ConvertBlobIndex() in db/blob/blob_reference_metadata.cc). The collector does not write that CRC, because a reader derives it again from the logical CRC. Deriving it would cost CPU for every reference of every output SST. BlobReference::FromBlobIndex() still derives it. - With the option on, the blob file builder computes the CRC32C of each stored value once. The blob log record CRC combines it with the key CRC (crc32c::Crc32cCombine). The stored CRC extends it with the compression byte. For an uncompressed value it is also the logical CRC. The bytes written do not change. - WideColumnSerialization::ForEachBlobIndex replaces ForEachBlobFileNumber. It reads the entity header and column types once. For a V1 entity (no column types and no blob columns), it returns right after the header. For a V2 entity without blob columns, it returns right after the type scan. It shares one V2 column walker (WalkV2Columns) with Deserialize(), so the entity is parsed in one pass. - Read-only opens (DBImplReadOnly, CompactedDBImpl) and secondary opens skip ValidateOptions, so they check the compaction style against the option once per column family: in DBImpl::Recover() for read-only opens, and at the start of OpenAsSecondaryImpl() for secondary opens. DB::OpenAndCompact opens through OpenAsSecondaryImpl(), so it gets the same check. - CacheReservationHandle gets Resize(), so each collector or summary can change its own part of the shared charge. - util/coding.h adds GetCanonicalVarint32/64, which reject overlong varint encodings. The new formats use them, so each value has one encoding. - BlobReferenceSstSummary is the in-memory list of an SST's groups and totals that Finish() can return. It has no caller yet: the table builder passes a null summary output. It is checked and does not change after Finish(), so the finished builder, the SST metadata and the census (its first user, in diff 5) can share it. It is in this diff because Finish() hands the group buffer and its cache charge over to the summary. Adding the summary later would mean rewriting the charge code of Finish() in diff 5, so the charge code is written and tested once, here. - sst_dump prints a "Blob reference metadata" section after the raw user collected properties of an SST that has exact metadata: the reference count, the stored value bytes, the column family ID, the SST file number when built and the index block handle. An SST without the descriptor prints nothing more. Safety boundaries and unsupported cases: - Public API: this diff adds the option `enable_blob_indirection` with the C API functions `rocksdb_options_set_enable_blob_indirection` and `rocksdb_options_get_enable_blob_indirection`. It also inserts a new CacheEntryRole value before kMisc, which changes the numeric value of kMisc and kNumCacheEntryRoles. - With the option off, new blob values and SSTs are written in the legacy form. Legacy blob indexes stay readable. - One behavior change applies with the option off too: the five readers listed under "Separate change" fully validate wide-column entities with blob columns and their blob indexes (see "Separate change" above). - Exact reference metadata has one format (version 1) with a fixed 72-byte descriptor. The decoders reject any other version as Corruption. - The metadata blocks are protected by the table's block checksum only. With checksum = kNoChecksum, corrupt metadata is detected only when it breaks the format or order checks. - The index decoder checks each decoded count against the remaining input size before it reserves memory. So a small corrupt index cannot make it allocate a large vector. - Inlined-TTL and same-file blob indexes are not collected and cannot be relocated. Encoding, count and checksum errors return an error; a bad reference is never skipped. - Option gate (see Terms): outside tests no DB writes data in the new format. The gate checks only the option value, never persisted state. Diff 14 removes it, once data written with the option stays readable through the end of the stack and the safety checks for turning the option on are in place. - Downgrading the binary after data was written with the option on is not supported. Validation and known limitations: - Collector memory, measured in a release build that drives the collector directly (bench/collector-chunk-20261111/README.md; host not recorded; 16-byte keys, 1 KB values, 62K to 1M references per SST in 1 to 16 groups): the buffers used about 8.3 to 9 bytes per reference for uncompressed values and 12.8 to 13.5 bytes for compressed values, including the chunk headers. With 1000 groups in a 62K-reference SST, they used about 14 bytes per uncompressed reference. - Chunk size, from the same measurement: for an SST with 1M references, the chunk size changed memory by less than 5% between 256 and 4096 bytes. For an SST with 62K references in 1000 groups, the partly filled chunks dominated: about 14 bytes per reference at 512 bytes, 22 at 1024 and 72 at 4096. 256 bytes used 3-4% more memory than 512 and made Finish() up to 1.6 times slower with 100 or more groups. - Reference index block size, measured in census db_bench runs (source report and host not recorded; 16-byte keys, 200-byte values, 1 MB blob files, no compression, fillrandom and then a full compaction): about 1.7 KB per SST with 32 MB write buffers and a target file size of 8 MB or 2 MB, and about 10 KB per SST with 1 MB write buffers and a target of 2 MB or 200 KB. The size is the bytes read for the index blocks divided by the number of SSTs. - Unit tests cover the BlobIndex encodings and their rejection of bad input, the metadata layout and every corruption check, the collector (sorted path, unsorted fallback, memory charge), the flush integration, the option checks and option gate, wide-column validation, and canonical varints. See the Test Plan. - The collection buffer is not spilled to disk. One SST with very many blob references needs a matching amount of charged memory: the per-reference amount given under "Collection and memory", plus up to one partly used 512-byte chunk per group. While a fallback group is sorted, it also needs 16 bytes of sort key per reference and a second copy of its entries. For a test group of 200K references with every other one out of order, the test comment estimates about 44 bytes per reference, and the test checks that the peak charge is at most 56 bytes per reference. - Flush still writes a single output file. Until diff 7, one compaction output collects every reference it gets, charged to the block cache. Diff 7 cuts compaction outputs (including L0) at a reference cap and keeps each user key's history in one output. With the option gate, a large output is reachable only through the test hook. Stack context (same in diffs 2-14, except that diffs 10-14 add the diff 10 route rule to the install line; "<== this diff" marks this diff's part): Goal: remove garbage from blob files without rewriting any SST. A value gets a stable identity when it is first written. Indirect blob GC copies the live values of whole files into a relocation file and installs routes in the MANIFEST. Readers follow the routes. SSTs are not changed. How a read finds a value (P is the relocation file that holds the value now): ``` SST entry indirect BlobIndex { generation number, origin, original offset, size, CRCs } | v Version file table: generation number -> live file, or route old file -> P | | live file route found | | v v blob file: read at original offset relocation file P (always one hop): relocation file: look up key look up key (origin, original offset) (origin, original offset) ``` Lifecycle of one value (in this example, F is the blob file where the value is first written, and P is the relocation file that GC writes): ``` write new value goes to blob file F; index: origin = F, generation number = F | census per generation number: the SSTs and references that use it; | per physical file: live bytes, ranked by garbage ratio GC job copy the live values of whole input files (stored bytes, checked | with the stored CRC) into relocation file P, keyed by | (origin, original offset) install one MANIFEST edit: add P, route each input generation number still | in use to P, re-point the routes into the inputs to P, and remove | each input (blob file retirement for a blob file, relocation | file deletion for a relocation file) flatten compaction sets generation number = P in the SSTs it rewrites | route route retirement: route deletion records remove the routes that delete no SST uses ``` Stack map: | Diff | Adds | |---|---| | 1 | Finalize blob metadata outside the DB mutex (prep) | | 2 | Indirect BlobIndex (kIndirectBlob) and exact SST reference metadata <== this diff | | 3 | Relocation file format and stable read path | | 4 | MANIFEST routes and relocation state in the Version | | 5 | Exact reverse reference census | | 6 | MANIFEST apply outside the DB mutex, with a precondition re-check | | 7 | Coarse flush metadata, per-SST reference cap, epoch runs | | 8 | Indirect blob GC building blocks | | 9 | Indirect blob GC job | | 10 | Flatten generation numbers; route retirement | | 11 | Remote compaction validation | | 12 | Dynamic options and operations | | 13 | Lost relocation files | | 14 | RepairDB rebuilds relocation state; option gate removed | | 15 | db_stress and crash test support | Stack context: terms for later diffs: | Term | Meaning | |---|---| | census | Background count, per generation number, of the SSTs and references that use it, plus live bytes per physical file (diff 5) | | indirect origin | Normal blob file written with the option on; it is the origin of unflattened references | | flatten | Compaction rewrites the generation number to the relocation file that holds the value, so old routes can go. Only flatten makes generation number != origin (diff 10) | | blob file retirement | MANIFEST record (kBlobFileRetirement) that removes a collected blob file from the Version (diff 4) | | route deletion | MANIFEST record that removes a route that no SST uses. Route retirement is the background job that writes these records (diff 10) | Tracking issue: meta#15301 Differential Revision: D124652097
4da30c3 to
937b04d
Compare
Summary: Pull Request resolved: meta#15540 Why the change is required: The goal of this stack is indirect blob GC, a blob GC that does not depend on SSTs: it removes garbage from blob files without rewriting any SST. Such a GC needs two things that RocksDB does not have: - A value identity that does not change when the value moves to another file. - A checked list of the blob references that each SST keeps. The list must include older internal versions of a key that the SST still keeps (for example for snapshots), and the blob columns of wide-column entities. This diff adds both. Later diffs build the census (an in-memory reverse index from generation number to the SSTs that use it), GC and repair on top of this list. This diff in one paragraph: With the new column family option enable_blob_indirection on, the blob file writer gives each new value an indirect BlobIndex (type 3, kIndirectBlob). It stores the value's stable identity (origin file, original offset) and its CRCs (an uncompressed value stores only the CRC of the uncompressed value; a compressed value also stores the CRC of its stored bytes). Also with the option on, the block-based table builder collects every blob reference that the SST keeps. It writes them as "exact reference metadata": data blocks, one reference index block, and a 72-byte descriptor table property. In this diff only the table builder writes this metadata, and only tests read it. A temporary option gate (see "option gate" under Terms) keeps the option off outside tests. Separately, wide-column entities with blob columns are now fully checked before their blob indexes are used, also with the option off. Parts of this diff: 1. Indirect BlobIndex (core): types 3 and 4 in db/blob/blob_index.h, and the blob file writer that writes type 3. 2. Exact reference metadata format and decoders (core): db/blob/blob_reference_metadata.{h,cc}. 3. Collector and table builder integration (core): BlobReferenceMetadataCollector, its block cache charge, and BlockBasedTableBuilder. 4. Option enable_blob_indirection and the option gate. Tests need it to turn the feature on. You can skip this part on a first read. 5. Wide-column validation (WideColumnSerialization::ForEachBlobIndex). The table builder uses it to give the blob columns of each entity to the collector. It also changes behavior with the option off (see New behavior). You can skip this part on a first read. 6. Small helpers: canonical varints (util/coding.h) and CacheReservationHandle::Resize(). You can skip these on a first read. How to read this diff: 1. Overview: read "Why" and "This diff in one paragraph" above, and "Terms used in this diff" right below. Then read the top comment of db/blob/blob_reference_metadata.h (terms, layout diagrams, run and entry layout, index layout). 2. Indirect BlobIndex (db/blob/blob_index.h) and canonical varints (util/coding.h GetCanonicalVarint32/64): read the overview comment and the kIndirectBlob / kIndirectFlattenedBlob code. Every later part uses these fields: origin, original offset, generation number, logical CRC and stored CRC. The new formats read varints only in the canonical form. 3. Blob file writer (db/blob/blob_file_builder.cc, blob_log_format.cc, blob_log_writer.cc): see how a new value gets a type-3 index and how its CRC is computed once. 4. Format types and decoders: the structs in db/blob/blob_reference_metadata.h, then in the .cc the constants, GapBase, DecodeEntryBody, DecodeRunEntriesAt, DecodeBlobReferenceIndex and BlobReferenceGroupDecoder. They use only BlobIndex and the canonical varints. Learn the on-disk format here, before you read the writer. 5. Collector and its cache charge: first read the new CacheEntryRole kBlobReferenceMetadataBuildingBuffer (include/rocksdb/cache.h) and CacheReservationHandle::Resize (cache/cache_reservation_manager.h). Then read BlobReferenceMetadataCollector in this order: the class comment, Add, AppendSorted, BuildAndWriteDataBlocks, EncodeIndex, FinishOnce. Then read the unsorted fallback (AppendUnsorted, SortUnsortedGroup, SortAndEncodeGroup). 6. Wide-column change (db/wide/wide_column_serialization.cc, ForEachBlobIndex and WalkV2Columns): it gives every blob index of an entity to a callback after it checks the whole entity. The table builder uses it to feed the collector. UpdateBoundaries, BlobGarbageMeter and the compaction iterator also use it, with the option on or off. 7. Option: enable_blob_indirection (include/rocksdb/advanced_options.h, options/cf_options.cc), and the option checks and gate in db/column_family.cc (ValidateOptions, TEST_SetAllowBlobIndirection), db/db_impl/db_impl_open.cc and db_impl_secondary.cc (the FIFO check for opens that skip ValidateOptions). 8. Table integration: table/block_based/block_based_table_factory.cc (the shared reservation manager), then block_based_table_builder.cc (Rep constructor, AddImpl, WriteBlobReferenceMetadata, EstimatedTailSize). See where the collector is created, fed (AddImpl(), which both Add() and TryAdd() call) and finished. block_based_table_reader.cc maps the new block name to a block type. Terms used in this diff: Stack terms: | Term | Meaning | |---|---| | origin | Blob file where the value was first written | | original offset | Value offset in the origin file | | stable identity | (origin, original offset). Never changes, even when the value moves | | generation number | The file number that a reference uses to find its value. A new value has generation number == origin. When indirect blob GC moves the value, the reference keeps its generation number, and a route sends the read to the relocation file | | indirect BlobIndex | A kIndirectBlob (type 3) or kIndirectFlattenedBlob (type 4) index. It stores the stable identity, so the value can move without an SST rewrite | | unflattened reference | Index with generation number == origin: the value was not moved, or it was moved but no compaction has rewritten its reference yet (see "flatten" in "Stack context: terms for later diffs") | | flattened reference | Index with generation number != origin (type 4). Only flatten (diff 10) writes it; this diff defines the encoding | | route | Maps an old file number (a file that GC has removed from the Version) to the relocation file that now holds its live values. One hop: a target is never itself routed. Stored in the MANIFEST (diff 4) | | relocation file | Block-based table keyed by (origin, original offset) that holds copied values. Each value is the stored bytes plus a 5-byte trailer (kSimpleGen2BlobTrailerSize) | | logical CRC | CRC32C of the uncompressed value (BlobIndex::stable_checksum()) | | stored CRC | CRC32C of the stored bytes plus the compression byte | | physical file | A file that holds value bytes: a normal blob file or a relocation file. A generation number can name a physical file, or it can be only the key of a route | | exact reference metadata | Per-SST sorted list of every blob reference the SST keeps, written when the SST is built. References with the same encoding class, generation number and origin form a group. The list is stored as data blocks, with an index of the groups. An SST that has it is an exact SST | | option gate | Until diff 14, ColumnFamilyData::ValidateOptions returns NotSupported for enable_blob_indirection on a read-write open, CreateColumnFamily and SetOptions, unless a test calls ColumnFamilyData::TEST_SetAllowBlobIndirection(true). It keeps the option off in production while the stack is not complete | Terms of this diff (all defined at the top of db/blob/blob_reference_metadata.h): | Term | Meaning | |---|---| | encoding class (class) | kDirect (a legacy kBlob or kBlobTTL index), unflattened, or flattened | | reference group (group) | All references of one SST with the same (encoding class, generation number, origin) | | run | The part of one group inside one data block | | record bytes | The bytes the value's record takes in the file named by its generation number | | gap | How far a record is from where it would be if the records were back to back. A gap of 0 means back to back | | descriptor | The 72-byte table property that points to the reference index block and holds the totals, the CF ID, the SST file number and the relocation-state fingerprint | | relocation-state fingerprint | 128-bit hash of the routes and relocation files of the Version that an SST was built from (fields route_snapshot_fingerprint_*). Written as 0 in this diff | | legacy | Written the way RocksDB writes it without blob indirection | Previous behavior: - A BlobIndex named only a physical blob file and an offset. A value could not move to another file without rewriting every SST that referenced it. - An SST had no metadata that listed its blob references. The only way to find them was to scan its user keys. - These readers get the blob indexes of a wide-column entity through ForEachBlobFileNumber: UpdateBoundaries, BlobGarbageMeter, the compaction iterator, WideColumnSerialization::RemoveBlobColumns, and the referenced_blob_bytes table property count in BlockBasedTableBuilder::Add. ForEachBlobFileNumber decoded each blob index, but it did not check the rest of the entity framing (name order, section sizes, exact input size), and it could call the callback before it found a later error. New behavior: Separate change (option on or off): - The same readers (UpdateBoundaries, BlobGarbageMeter, the compaction iterator, RemoveBlobColumns, and the referenced_blob_bytes count in BlockBasedTableBuilder::Add) use WideColumnSerialization::ForEachBlobIndex, with the option on or off. With the option on, the table builder also uses it to give the blob columns of each entity to the collector. For an entity with blob columns, it checks the full framing, then decodes and validates every blob index before any callback runs. So an entity with a malformed framing now fails the flush or compaction, even with the option off. The referenced_blob_bytes count itself does not fail the build on a malformed entity. It removes that property from the SST, as in base. Option and option gate: - This diff adds a new column family option, `enable_blob_indirection`. It is off by default, and SetOptions can change it. - The option gate (see Terms) keeps the option off outside tests until the stack is complete. ColumnFamilyData::ValidateOptions rejects the option with NotSupported on a read-write open, CreateColumnFamily and SetOptions. Read-only opens (DBImplReadOnly and CompactedDBImpl) and secondary opens (including DB::OpenAndCompact) do not run ValidateOptions; they check only the FIFO rule (see Implementation). Only tests can turn the option on, through the internal ColumnFamilyData::TEST_SetAllowBlobIndirection(). DBTestBase sets it. - When the option is on, ValidateOptions also requires enable_blob_files, the block-based table format, no blob direct write, and a compaction style other than FIFO. Blob file writer (option on): - The blob file writer encodes each new value with a stable identity (origin file, original offset) and two checksums: - the logical CRC: the CRC32C of the uncompressed value. - the stored CRC: the CRC32C of the exact stored (maybe compressed) bytes, extended with the compression byte. - For an uncompressed value the stored CRC is not written, because the reader can compute it from the logical CRC. - The checksums have no version field of their own. The BlobIndex type byte gives the format version: the two new types, 3 (kIndirectBlob) and 4 (kIndirectFlattenedBlob, see Implementation), are BlobIndex::kIndirectBlobFormatVersion (1). A later format uses new type values. - The read path does not check these checksums yet. The check starts in diff 3. Exact reference metadata (option on): 1. What: a sorted list of every blob reference that the SST keeps, in plain blob values and in the blob columns of wide-column entities. 2. Where: data blocks (target 16 KB each before compression; small blocks waste less I/O, because indirect blob GC reads a whole block to get a few groups) and one reference index block, written after the SST data blocks and before the SST tail, plus a 72-byte descriptor table property. 3. Who reads it: only tests in this diff. The census, GC and RepairDB read it in later diffs. 4. Details: the top comment of db/blob/blob_reference_metadata.h has the full format with diagrams, the record-bytes rules and the gap rule. - Not collected: SstFileWriter output (it has no DB file number) and any output whose CF is unknown (it has no CF ID). The descriptor needs both values, so these outputs stay in the legacy form. Checks on the metadata: - A repeated or overlapping reference in one group fails the SST build with Corruption. Readers check every count, order, size and total against the index and the descriptor, so corrupt metadata returns Corruption. The writer never makes a data block larger than kTargetDataBlockSize (16 KB), and readers accept up to kMaxDataBlockSize (64 KB). Details: "Readers" and "Integrity" at the top of db/blob/blob_reference_metadata.h, and BlobReferenceGroupDecoder. Collection and memory: - Each reference is encoded when it is added, in the on-disk entry encoding, into its group's chain of 512-byte chunks. Only a group that takes the unsorted fallback is encoded a second time, in Finish(). Inside a group, references normally arrive in rising offset order, so no reference is sorted; at the end only the groups are sorted. A group that gets a lower offset takes the unsorted fallback. The BlobReferenceMetadataCollector class comment explains the chunks, the chunk size and the fallback. - The collector keeps every reference of the SST in memory until the SST is finished, and charges this memory to the block cache under a new CacheEntryRole, kBlobReferenceMetadataBuildingBuffer. "Memory usage" and "Memory charge" in the BlobReferenceMetadataCollector class comment explain the parts of this memory and the charge rule. The measured amounts are under "Validation and known limitations". - There is no fixed limit on references. The collector fails with MemoryLimit above 2^20 groups or 2^20 data blocks in one SST; these are persisted format limits. It also fails with MemoryLimit when the reference index block could be larger than 64 MB (kMaxTopLevelMetadataBlockSize), which can happen from about 945K groups (64 MB / kMaxIndexEntrySize, 71 bytes per group), or from fewer groups when the SST also has many data blocks. - There is no charge when the table has no block cache or when charging for this role is disabled. Each block-based table factory holds one cache reservation manager for this role, shared by its builders and the SST summaries. If the charge fails, the SST build fails, so the flush or compaction fails with a background error. The buffers and their charge are released when the SST is finished or abandoned. Implementation: - BlobIndex adds two indirect encodings. The legacy encodings do not change. - Type 3, kIndirectBlob (unflattened, generation number == origin), stores the origin, the original offset, the stored size, the compression type, the logical CRC and, for compressed values, the stored CRC. It does not store the generation number. This saves the size of one varint file number per reference (about 3 to 4 bytes for typical file numbers; an estimate, not measured). - Type 4, kIndirectFlattenedBlob (flattened), stores the generation number first, then the same fields. Its generation number must be above the origin. The decoder rejects type 4 with generation number <= origin, so each reference has one encoding. - stored_checksum() computes the stored CRC of an uncompressed value when a caller asks for it: the logical CRC extended with the compression byte. A decode does not compute it, because most decodes (for example in compaction) never read it. - The encoders take the fields as one BlobIndex::IndirectBlobFields, set by field name, so values of the same type (file number, offset and size; the two CRCs) cannot be swapped. EncodeIndirectBlob() takes the generation number as a separate argument; EncodeUnflattenedIndirectBlob() uses the origin, so the common path cannot leave the generation number unset. - The new files blob_reference_metadata.{h,cc} define the exact reference metadata. The files also contain: - BlobReferenceMetadataCollector, the collector described above. It takes the SST identity (CF ID, SST file number and the relocation-state fingerprint) as one BlobReferenceSstIdentity, set by field name. - BlobReferenceGroupKey, the key of a group: (encoding class, generation number, origin). It has the order, the hash and the validity check that the collector map, the sort, the decoder and FindBlobReferenceGroup() use. BlobReference and BlobReferenceGroup keep their flat fields and return the key through group_key(). - The index codec and the descriptor table property. - BlobReferenceGroupDecoder, the only decoder of reference runs. It reads a group one data block at a time and holds every corruption check of a group walk. Readers that visit one group (found by a binary search of the index) use it, and so do readers that visit every block in order. - Callers in this diff: only the table builder runs the collector. The decoders (DecodeBlobReferenceIndex, FindBlobReferenceGroup, BlobReferenceGroupDecoder, ForEachBlobReference*) are used only by tests. kMaxDecodedBlobReferenceBytes is used only in compile-time checks of the count limits. The census (diff 5) and indirect blob GC (diffs 8 and 9) are their first production users. The census is also the first user of BlobReferenceSstSummary (see the last Implementation bullet). - Descriptor: the CF ID and the SST file number are the repair fields. They let RepairDB tell which column family and SST the list belongs to without a MANIFEST. The decoder rejects a descriptor whose repair fields are not set (unknown CF ID or SST file number 0). The descriptor also has a 128-bit relocation-state fingerprint (the route_snapshot_fingerprint fields). Remote compaction uses it in diff 11 to check that an SST built elsewhere used the same routes as the job that asked for it. In this diff the table builder always writes 0, and DecodeFrom() does not check it. The field is in the format now, so the descriptor keeps one format version (1) through the stack. While the option gate is on, no data in this format exists outside tests. - The collector validates each BlobIndex once, when it is added. Its Finish() writes the data blocks and then the reference index block through one block writer, and fills the descriptor, including the index handle. Destroying the collector frees its memory and then its charge, also when the build is abandoned. - BlockBasedTableBuilder creates the collector with the CF ID and SST file number from TableBuilderOptions. It passes the size of each entry's user key to the collector, so the collector can compute the exact size of each blob record. It adds the collector's EstimatedEncodedSize() to its tail size estimate: the blocks are written before the tail, but output file size planning uses this estimate. The reference blocks are written with BlockType::kUserDefinedIndex, which keeps their contents as opaque bytes. Readers read them without the block cache, so prepopulate_block_cache does not warm them into the block cache (WriteMaybeCompressedBlock() with warm_cache false). - To save CPU, compaction decodes each kTypeBlobIndex output value once, in CompactionOutputs::AddToOutput(). It passes the decode to TableBuilder::TryAdd() in the new field TableBuilderAddContext::decoded_blob_index, and to new overloads of BlobGarbageMeter::ProcessOutFlow() and FileMetaData::UpdateBoundaries(). BlockBasedTableBuilder decodes a kTypeBlobIndex value once for both the referenced_blob_bytes count and the collector, and uses the caller's decode when it has one. If the decode fails, no decode is passed, and each reader decodes the value itself and returns its usual error. This applies with the option on or off. The SST bytes are the same as when each reader decodes the value itself. - The collector converts each BlobIndex into a reference without deriving the stored CRC of an uncompressed value (ConvertBlobIndex() in db/blob/blob_reference_metadata.cc). The collector does not write that CRC, because a reader derives it again from the logical CRC. Deriving it would cost CPU for every reference of every output SST. BlobReference::FromBlobIndex() still derives it. - With the option on, the blob file builder computes the CRC32C of each stored value once. The blob log record CRC combines it with the key CRC (crc32c::Crc32cCombine). The stored CRC extends it with the compression byte. For an uncompressed value it is also the logical CRC. The bytes written do not change. - WideColumnSerialization::ForEachBlobIndex replaces ForEachBlobFileNumber. It reads the entity header and column types once. For a V1 entity (no column types and no blob columns), it returns right after the header. For a V2 entity without blob columns, it returns right after the type scan. It shares one V2 column walker (WalkV2Columns) with Deserialize(), so the entity is parsed in one pass. - Read-only opens (DBImplReadOnly, CompactedDBImpl) and secondary opens skip ValidateOptions, so they check the compaction style against the option once per column family: in DBImpl::Recover() for read-only opens, and at the start of OpenAsSecondaryImpl() for secondary opens. DB::OpenAndCompact opens through OpenAsSecondaryImpl(), so it gets the same check. - CacheReservationHandle gets Resize(), so each collector or summary can change its own part of the shared charge. - util/coding.h adds GetCanonicalVarint32/64, which reject overlong varint encodings. The new formats use them, so each value has one encoding. - BlobReferenceSstSummary is the in-memory list of an SST's groups and totals that Finish() can return. It has no caller yet: the table builder passes a null summary output. It is checked and does not change after Finish(), so the finished builder, the SST metadata and the census (its first user, in diff 5) can share it. It is in this diff because Finish() hands the group buffer and its cache charge over to the summary. Adding the summary later would mean rewriting the charge code of Finish() in diff 5, so the charge code is written and tested once, here. - sst_dump prints a "Blob reference metadata" section after the raw user collected properties of an SST that has exact metadata: the reference count, the stored value bytes, the column family ID, the SST file number when built and the index block handle. An SST without the descriptor prints nothing more. Safety boundaries and unsupported cases: - Public API: this diff adds the option `enable_blob_indirection` with the C API functions `rocksdb_options_set_enable_blob_indirection` and `rocksdb_options_get_enable_blob_indirection`. It also inserts a new CacheEntryRole value before kMisc, which changes the numeric value of kMisc and kNumCacheEntryRoles. - With the option off, new blob values and SSTs are written in the legacy form. Legacy blob indexes stay readable. - One behavior change applies with the option off too: the five readers listed under "Separate change" fully validate wide-column entities with blob columns and their blob indexes (see "Separate change" above). - Exact reference metadata has one format (version 1) with a fixed 72-byte descriptor. The decoders reject any other version as Corruption. - The metadata blocks are protected by the table's block checksum only. With checksum = kNoChecksum, corrupt metadata is detected only when it breaks the format or order checks. - The index decoder checks each decoded count against the remaining input size before it reserves memory. So a small corrupt index cannot make it allocate a large vector. - Inlined-TTL and same-file blob indexes are not collected and cannot be relocated. Encoding, count and checksum errors return an error; a bad reference is never skipped. - Option gate (see Terms): outside tests no DB writes data in the new format. The gate checks only the option value, never persisted state. Diff 14 removes it, once data written with the option stays readable through the end of the stack and the safety checks for turning the option on are in place. - Downgrading the binary after data was written with the option on is not supported. Validation and known limitations: - Collector memory, measured in a release build that drives the collector directly (bench/collector-chunk-20261111/README.md; host not recorded; 16-byte keys, 1 KB values, 62K to 1M references per SST in 1 to 16 groups): the buffers used about 8.3 to 9 bytes per reference for uncompressed values and 12.8 to 13.5 bytes for compressed values, including the chunk headers. With 1000 groups in a 62K-reference SST, they used about 14 bytes per uncompressed reference. - Chunk size, from the same measurement: for an SST with 1M references, the chunk size changed memory by less than 5% between 256 and 4096 bytes. For an SST with 62K references in 1000 groups, the partly filled chunks dominated: about 14 bytes per reference at 512 bytes, 22 at 1024 and 72 at 4096. 256 bytes used 3-4% more memory than 512 and made Finish() up to 1.6 times slower with 100 or more groups. - Reference index block size, measured in census db_bench runs (source report and host not recorded; 16-byte keys, 200-byte values, 1 MB blob files, no compression, fillrandom and then a full compaction): about 1.7 KB per SST with 32 MB write buffers and a target file size of 8 MB or 2 MB, and about 10 KB per SST with 1 MB write buffers and a target of 2 MB or 200 KB. The size is the bytes read for the index blocks divided by the number of SSTs. - Unit tests cover the BlobIndex encodings and their rejection of bad input, the metadata layout and every corruption check, the collector (sorted path, unsorted fallback, memory charge), the flush integration, the option checks and option gate, wide-column validation, and canonical varints. See the Test Plan. - The collection buffer is not spilled to disk. One SST with very many blob references needs a matching amount of charged memory: the per-reference amount given under "Collection and memory", plus up to one partly used 512-byte chunk per group. While a fallback group is sorted, it also needs 16 bytes of sort key per reference and a second copy of its entries. For a test group of 200K references with every other one out of order, the test comment estimates about 44 bytes per reference, and the test checks that the peak charge is at most 56 bytes per reference. - Flush still writes a single output file. Until diff 7, one compaction output collects every reference it gets, charged to the block cache. Diff 7 cuts compaction outputs (including L0) at a reference cap and keeps each user key's history in one output. With the option gate, a large output is reachable only through the test hook. Stack context (same in diffs 2-14, except that diffs 10-14 add the diff 10 route rule to the install line; "<== this diff" marks this diff's part): Goal: remove garbage from blob files without rewriting any SST. A value gets a stable identity when it is first written. Indirect blob GC copies the live values of whole files into a relocation file and installs routes in the MANIFEST. Readers follow the routes. SSTs are not changed. How a read finds a value (P is the relocation file that holds the value now): ``` SST entry indirect BlobIndex { generation number, origin, original offset, size, CRCs } | v Version file table: generation number -> live file, or route old file -> P | | live file route found | | v v blob file: read at original offset relocation file P (always one hop): relocation file: look up key look up key (origin, original offset) (origin, original offset) ``` Lifecycle of one value (in this example, F is the blob file where the value is first written, and P is the relocation file that GC writes): ``` write new value goes to blob file F; index: origin = F, generation number = F | census per generation number: the SSTs and references that use it; | per physical file: live bytes, ranked by garbage ratio GC job copy the live values of whole input files (stored bytes, checked | with the stored CRC) into relocation file P, keyed by | (origin, original offset) install one MANIFEST edit: add P, route each input generation number still | in use to P, re-point the routes into the inputs to P, and remove | each input (blob file retirement for a blob file, relocation | file deletion for a relocation file) flatten compaction sets generation number = P in the SSTs it rewrites | route route retirement: route deletion records remove the routes that delete no SST uses ``` Stack map: | Diff | Adds | |---|---| | 1 | Finalize blob metadata outside the DB mutex (prep) | | 2 | Indirect BlobIndex (kIndirectBlob) and exact SST reference metadata <== this diff | | 3 | Relocation file format and stable read path | | 4 | MANIFEST routes and relocation state in the Version | | 5 | Exact reverse reference census | | 6 | MANIFEST apply outside the DB mutex, with a precondition re-check | | 7 | Coarse flush metadata, per-SST reference cap, epoch runs | | 8 | Indirect blob GC building blocks | | 9 | Indirect blob GC job | | 10 | Flatten generation numbers; route retirement | | 11 | Remote compaction validation | | 12 | Dynamic options and operations | | 13 | Lost relocation files | | 14 | RepairDB rebuilds relocation state; option gate removed | | 15 | db_stress and crash test support | Stack context: terms for later diffs: | Term | Meaning | |---|---| | census | Background count, per generation number, of the SSTs and references that use it, plus live bytes per physical file (diff 5) | | indirect origin | Normal blob file written with the option on; it is the origin of unflattened references | | flatten | Compaction rewrites the generation number to the relocation file that holds the value, so old routes can go. Only flatten makes generation number != origin (diff 10) | | blob file retirement | MANIFEST record (kBlobFileRetirement) that removes a collected blob file from the Version (diff 4) | | route deletion | MANIFEST record that removes a route that no SST uses. Route retirement is the background job that writes these records (diff 10) | Tracking issue: meta#15301 Differential Revision: D124652097
…a#15323) Summary: Pull Request resolved: meta#15323 Background: This diff moves the consistency check of each new Version, and the saving of its blob files, out of the DB mutex. A Version is the in-memory list of the SST files and blob files of one column family (CF). VersionSet::LogAndApply builds each new Version with a VersionBuilder and installs it while it holds the DB mutex. Each VersionBuilder starts from a base Version: the current Version of its CF when the builder was made. In this diff, "finalizing the blob metadata" of a new Version means two things: adding the blob files to the new Version, and then checking the consistency of the whole new Version. Later diffs in the indirect blob GC stack add more blob state to each Version in the same step. So building a new Version now has two steps. Step 1 (under the DB mutex) adds the SST files and compact cursors. Step 2 (without the DB mutex) adds the blob files and checks the consistency of the new Version. If this work runs under the DB mutex, it delays unrelated DB operations, and the delay grows with the number of files. In this diff, the unlocked step is mostly the consistency check of the new Version. When force_consistency_checks is on (the default) or in debug builds, this check walks every SST. We estimate, without a measurement, that saving the blob metadata costs little in this diff, because it mostly copies shared pointers for the blob files. Later diffs add more blob work to the same unlocked step. Terms used in this diff: - ColumnFamilyData (CFD): the in-memory object of one CF. It is reference counted, and a dropped CF is freed when its last reference is released. - SuperVersion: the reference-counted set of memtables and the current Version that readers use. Each SuperVersion holds one CFD reference. - MANIFEST writer queue: LogAndApply puts one writer (a ManifestWriter) per CF into this queue. The writer at the head of the queue processes a batch of queued writers. The thread that owns that writer is the leader; the thread that called LogAndApply for a writer is its caller. - Writer callback: a function that a LogAndApply caller passes per writer (the `manifest_wcbs` argument, stored as ManifestWriter::manifest_write_callback). Flush uses it to remove the flushed memtables. - No-MANIFEST LogAndApply call: a call that passes dummy edits (VersionEdit::MarkNoManifestWriteDummy). It installs new Versions but writes no MANIFEST record. Only SetOptions uses it. In ProcessManifestWrites, such a batch has skip_manifest_write set. - Dummy flag: the mark set by VersionEdit::MarkNoManifestWriteDummy() on a dummy edit. A test can clear it on the SetOptions edits through the sync point DBImpl::SetOptions:dummy_edit, so that SetOptions runs through the MANIFEST path. - pre_cb: a callback that LogAndApply runs under the DB mutex when the caller's first writer reaches the head of the MANIFEST writer queue. - Options map: the option names and values that one SetOptions call gets for one column family handle. - Phasing anchor (compaction_phase_anchor_time): the time from which periodic catch-up compactions are spread out. SetOptions moves it to the current time when an options map sets periodic_compaction_seconds, even to its current value, and the call succeeds. It does not move if the call returns early because no option changes. See VersionSet::PrepareCompactionPhaseReanchor() in version_set.h. Why the change is required: - Blob metadata finalization must run without the DB mutex before later diffs add work to it. This diff moves finalization, and the consistency check of the new Version that follows it, out of the mutex. It changes no format. - SetOptions must change too. Before this diff, SetOptions installs the new options in pre_cb. It relies on LogAndApply keeping the DB mutex held until the new Version is installed. With this diff, the no-MANIFEST path of LogAndApply also releases the mutex during blob finalization. SetOptions builds its new Versions through the same ProcessManifestWrites code, so they need the same step 2 (the consistency check that walks every SST, and in later diffs the derived blob state). Holding the mutex only for SetOptions would keep that cost under the mutex, once for every CF that SetOptions touches. If SetOptions still installed the options in pre_cb, a reader could see the new options with the old Version. So SetOptions now only prepares the options in pre_cb, and LogAndApply publishes them together with the new Versions. The phasing anchor follows the same rule, so it now moves in two steps (prepare, then publish with the new Versions). - SetOptions now makes one LogAndApply call for all its CFs, so that the options and the Versions are published together. Because there is one call, the result is all-or-nothing: either every CF changes or none does. In one call, all writers of one CF share one new Version, built with the options of the first writer. So when two handles point to the same CF, SetOptions merges their options maps into one writer. Otherwise the second map would be lost. - Writer callbacks must run in the same DB mutex hold as the install, and at the end of the batch: - Existing race: before this diff, atomic flush removed the flushed memtables only after LogAndApply returned. Another thread can lead the batch and install the new Versions of this flush. Until the flush thread wakes up, takes the DB mutex and removes the memtables, other code under the DB mutex sees both the new SST files and the flushed memtables. This diff runs every callback in the mutex hold of the install, and atomic flush uses a callback too. This fixes the race. - End of the batch: as in base RocksDB, the callbacks run when the writers of the batch are marked done, with no DB mutex release in between. A Flush(wait=true) caller returns as soon as its memtables are removed. When the flush thread leads its batch, it installs its new SuperVersion right after LogAndApply() returns, without releasing the DB mutex. If more flushes finished during the MANIFEST write, MemTableList::TryInstallMemtableFlushResults() first commits them in another LogAndApply() round, which can release the DB mutex. A flush writer that another leader takes into its batch installs its SuperVersion after it takes the DB mutex again. Base RocksDB works the same way in both cases. Parts of this diff: 1. Unlocked blob finalization (the core): VersionBuilder saves a Version in two steps, and LogAndApply runs the second step without the DB mutex. 2. Consistency check of the builder's base Version once per builder. You can skip this part on a first read. It does not depend on the other parts. 3. All-or-nothing SetOptions: the options are prepared in pre_cb and published with the new Versions. The core needs it because the no-MANIFEST path now releases the DB mutex too, so that the new Versions of SetOptions get the same unlocked step 2 (see Why). 4. Writer callbacks run once per batch, when the writers are marked done, and atomic flush uses one. You can skip this part on a first read. Later diffs need it, and it does not depend on part 3. 5. Small items (see Implementation). You can skip these on a first read. - finish_batch signals every writer, because a SetOptions call now puts all its CFs in one batch, and the old search took quadratic time for that batch. - A null CFD in a batch returns Corruption. The check replaces an assert in code that this diff rewrites. How to read this diff: 1. Read the summary sections Why, Parts of this diff and "New behavior: building a Version". 2. Start with db/version_builder.h (SaveTableFilesTo, SaveBlobFilesAndCheckTo) and their bodies in db/version_builder.cc. They depend on nothing new. Then read base_checked_ and VersionBuilder::Apply in db/version_builder.cc (part 2). 3. Read ProcessManifestWrites in db/version_set.cc: the finalize_blob_files lambda, the two unlock paths (no-MANIFEST and MANIFEST), the comment and debug checks after mu->Lock() in the MANIFEST path, and the blob_finalization_failed branch. This is where step 2 of the save runs without the mutex. Only SetOptions uses the no-MANIFEST path. Releasing the mutex there is safe only because SetOptions now publishes its options at install time (see reading steps 5 to 7 below). 4. In the same function, read publish_batch_callbacks and finish_batch. Then read InstallMemtableAtomicFlushResults in db/memtable_list.cc (part 4). 5. Read the pieces that SetOptions needs: PrepareMutableCFOptions and PublishMutableCFOptions in db/column_family.h, ParamsForCfAtAnchor in db/periodic_compaction_phaser.h, and MutableCFOptionsPublication, the phasing anchor comment above PrepareCompactionPhaseReanchor, and the LogAndApply comment in db/version_set.h. 6. Read how these pieces go through LogAndApply in db/version_set.cc: the new ManifestWriter fields, how the new Version is built, the skip_manifest_write grouping, and the install block that publishes the options and the anchor. 7. Read DBImpl::SetOptions in db/db_impl/db_impl.cc. The comment at its top lists its steps. It groups the maps by CF, builds one dummy edit per CF, defines pre_cb (LogAndApply runs it under the DB mutex), makes one LogAndApply call, installs the new SuperVersions, and writes one OPTIONS file. Previous behavior: - LogAndApply called VersionBuilder::SaveTo under the DB mutex. SaveTo added the SST files, blob files and compact cursors to the new Version, and then checked its consistency. - VersionBuilder::Apply checked the consistency of the base Version on every edit. - SetOptions made one no-MANIFEST LogAndApply call per CF. Each call installed the options of its CF in pre_cb. When a map set periodic_compaction_seconds, SetOptions moved the phasing anchor to the current time before its first LogAndApply call (VersionSet::ReanchorCompactionPhase), so the anchor moved even when the call failed later. If a later CF failed, the earlier CFs kept their new options, and the next SetOptions call finished the update. - Each writer callback ran when its writer was taken off the queue at the end of the batch. Atomic flush updated its memtable lists after LogAndApply returned. New behavior: building a Version 1. Under the DB mutex, VersionBuilder::SaveTableFilesTo adds the SST files and compact cursors to each new Version. 2. LogAndApply releases the DB mutex. VersionBuilder::SaveBlobFilesAndCheckTo adds the blob files. Then it builds the list of dropped blob files (VersionStorageInfo::GetDroppedBlobFiles()). A dropped blob file is a missing blob file that an open with best_efforts_blob_db_repair recorded. Then it checks the consistency of the complete new Version (SST and blob state). The builder keeps its base Version referenced during this step. The batch stays at the head of the MANIFEST writer queue, so no other thread installs a Version for these CFs in the meantime. 3. LogAndApply takes the DB mutex again and installs the new Versions. In the no-MANIFEST path (used by SetOptions), Version::PrepareAppend (which calls VersionStorageInfo::PrepareForVersionAppend) also runs in step 2, without the mutex. The MANIFEST path already ran it without the mutex. If finalization fails, no MANIFEST record has been written. LogAndApply returns the error, deletes the unpublished Versions and keeps the current MANIFEST writer, the same as for a SaveTableFilesTo failure. It does not log "Error in committing version edit to MANIFEST" and does not force a new MANIFEST file. New behavior: consistency check of the builder's base Version VersionBuilder::Apply checks the consistency of the base Version only until the first check passes, so at most once per builder when the base Version is consistent. This is safe because the base Version never changes during the builder's life. It applies to all DBs and every builder user: LogAndApply, recovery and point-in-time recovery. Point-in-time recovery can undo the last edit with RollbackLastApply, but this does not change the base Version. With force_consistency_checks (true by default), a builder that applies many edits does not walk every SST of the base Version under the DB mutex for each edit. A failed check is not remembered, so the next Apply reports it again. New behavior: SetOptions The steps are listed in one place: the comment at the top of DBImpl::SetOptions (db/db_impl/db_impl.cc). The contract: - The call is all-or-nothing. If preparation fails for any CF, no CF's options change and no OPTIONS file is written. - Readers never see a mix of old and new options and Versions. LogAndApply publishes the options, the phasing anchor and the new Versions in one DB mutex hold, and SetOptions installs the new SuperVersions before it releases the mutex. - One call writes one OPTIONS file. A failed OPTIONS file write sets options_file_stale_, so the next call writes the file again. - Options maps given through different handles of the same CF go into one writer and are applied in map iteration order. - The batch of one SetOptions call holds only the writers of that call (see step 2 in the MutableCFOptionsPublication comment in version_set.h). Test-only MANIFEST path: a test can clear the dummy flag of the SetOptions edits (see Terms), so that SetOptions writes a real MANIFEST record. The options are then published only after that write succeeds. Grouping could then mix SetOptions writers with the writers of other callers. So a writer with prepared options is never grouped with another caller's writers, before or after: once the batch holds all writers of the leader's caller, it stops if the leader has prepared options, or before a next writer that has prepared options. New behavior: writer callbacks All writer callbacks of a batch run once, under the DB mutex, at the end of the batch, right before the writers are marked done. On success this is in the mutex hold of the install. The full contract (failure, early return, no user code or I/O) is in the comment of VersionSet::LogAndApply in version_set.h. Atomic flush passes one callback, on its first writer, that updates the memtable lists of the whole group. So the new Versions and the removal of their flushed memtables happen in one critical section. If LogAndApply returns before the batch starts (every CF of the group was dropped while its writers waited), no callback runs, and atomic flush updates the memtable lists itself after LogAndApply returns. Implementation: - VersionBuilder::SaveTo is split into SaveTableFilesTo (under the mutex) and SaveBlobFilesAndCheckTo (without the mutex). SaveTo still calls both. ProcessManifestWrites keeps the builder guards (BaseReferencedVersionBuilder) alive across the unlocked part. - ColumnFamilyData::PrepareMutableCFOptions parses and validates options without publishing them. ColumnFamilyData::PublishMutableCFOptions publishes them. - LogAndApply gets one new trailing parameter, a MutableCFOptionsPublication pointer (null by default). It holds the per-CF prepared options and the phasing anchor time. When it is set, the CFs must be distinct (checked in debug builds). The Version constructor takes optional PeriodicCompactionPhaseParams (std::nullopt by default), so a new Version can use the new anchor before it is committed. VersionSet::ReanchorCompactionPhase is replaced by PrepareCompactionPhaseReanchor (read the clock) and CommitCompactionPhaseReanchor (commit under the DB mutex). - finish_batch signals every writer of the batch. Only the first writer of each caller has a thread that waits on it, so the other signals do nothing. Before this diff, finish_batch searched the leader's own writers for each writer of the batch, so it could skip the signal for them. This search takes time quadratic in the number of writers of a large multi-CF call. Signaling every writer removes it. - A writer with a null CFD in a batch that is not a CF add or drop now returns Corruption. Before this diff, the code asserted, and release builds dereferenced the null pointer. Safety boundaries and unsupported cases: - No persisted format or public option changes. - Group commit of several writers for the same CF is unchanged. - SaveTableFilesTo still runs under the mutex, and its cost is linear in the number of SST files. So building a Version under the mutex is not constant time. - Not changed: an iterator, a GetEntityLazy result and a pinned GetMergeOperands result keep a dropped CF alive only through the CFD reference of their SuperVersion, as before. This diff adds a test for this, because lazy blob reads in later diffs depend on it. Validation and known limitations: - Unit tests cover SetOptions (all-or-nothing, two handles of the same CF, one OPTIONS file, no mixed view, phase anchor), unlocked blob finalization and its failure, the base Version consistency check, atomic flush when every column family is dropped while it waits, and reads that outlive a CF drop. See the Test Plan. - Performance: a db_bench run measured the DB mutex. With about 20K to 25K SSTs (W1 in the Test Plan) and frequent flush and compaction installs, the DB mutex hold time of one LogAndApply drops by about half (p50 6.67 ms to 3.13 ms, p99 13.5 ms to 5.85 ms). The total DB mutex wait drops by 20%, and reads per second rise by 6.7%. With 4 CFs and atomic flush (W2), the hold time drops by about a third and the read p99.9 latency drops by 24%. A small DB with about 110 SSTs at start (W3) shows no change. Most of the hold time that remains is SaveTableFilesTo, which still walks every SST under the mutex. - Most of the saving likely comes from consistency checks, which walk every SST (see Background): the check of the new Version now runs without the DB mutex, and the check of the base Version in Apply runs once per builder instead of once per edit. We estimate that saving the blob file metadata itself costs little. The benchmark did not measure these parts separately. Tracking issue: meta#15301 Differential Revision: D123408321
Summary: Pull Request resolved: meta#15540 Why the change is required: The goal of this stack is indirect blob GC, a blob GC that does not depend on SSTs: it removes garbage from blob files without rewriting any SST. Such a GC needs two things that RocksDB does not have: - A value identity that does not change when the value moves to another file. - A checked list of the blob references that each SST keeps. The list must include older internal versions of a key that the SST still keeps (for example for snapshots), and the blob columns of wide-column entities. This diff adds both. Later diffs build the census (an in-memory reverse index from generation number to the SSTs that use it), GC and repair on top of this list. This diff in one paragraph: With the new column family option enable_blob_indirection on, the blob file writer gives each new value an indirect BlobIndex (type 3, kIndirectBlob). It stores the value's stable identity (origin file, original offset) and its CRCs (an uncompressed value stores only the CRC of the uncompressed value; a compressed value also stores the CRC of its stored bytes). Also with the option on, the block-based table builder collects every blob reference that the SST keeps. It writes them as "exact reference metadata": data blocks, one reference index block, and a 72-byte descriptor table property. In this diff only the table builder writes this metadata, and only tests read it. A temporary option gate (see "option gate" under Terms) keeps the option off outside tests. Separately, wide-column entities with blob columns are now fully checked before their blob indexes are used, also with the option off. Parts of this diff: 1. Indirect BlobIndex (core): types 3 and 4 in db/blob/blob_index.h, and the blob file writer that writes type 3. 2. Exact reference metadata format and decoders (core): db/blob/blob_reference_metadata.{h,cc}. 3. Collector and table builder integration (core): BlobReferenceMetadataCollector, its block cache charge, and BlockBasedTableBuilder. 4. Option enable_blob_indirection and the option gate. Tests need it to turn the feature on. You can skip this part on a first read. 5. Wide-column validation (WideColumnSerialization::ForEachBlobIndex). The table builder uses it to give the blob columns of each entity to the collector. It also changes behavior with the option off (see New behavior). You can skip this part on a first read. 6. Small helpers: canonical varints (util/coding.h) and CacheReservationHandle::Resize(). You can skip these on a first read. How to read this diff: 1. Overview: read "Why" and "This diff in one paragraph" above, and "Terms used in this diff" right below. Then read the top comment of db/blob/blob_reference_metadata.h (terms, layout diagrams, run and entry layout, index layout). 2. Indirect BlobIndex (db/blob/blob_index.h) and canonical varints (util/coding.h GetCanonicalVarint32/64): read the overview comment and the kIndirectBlob / kIndirectFlattenedBlob code. Every later part uses these fields: origin, original offset, generation number, logical CRC and stored CRC. The new formats read varints only in the canonical form. 3. Blob file writer (db/blob/blob_file_builder.cc, blob_log_format.cc, blob_log_writer.cc): see how a new value gets a type-3 index and how its CRC is computed once. 4. Format types and decoders: the structs in db/blob/blob_reference_metadata.h, then in the .cc the constants, GapBase, DecodeEntryBody, DecodeRunEntriesAt, DecodeBlobReferenceIndex and BlobReferenceGroupDecoder. They use only BlobIndex and the canonical varints. Learn the on-disk format here, before you read the writer. 5. Collector and its cache charge: first read the new CacheEntryRole kBlobReferenceMetadataBuildingBuffer (include/rocksdb/cache.h) and CacheReservationHandle::Resize (cache/cache_reservation_manager.h). Then read BlobReferenceMetadataCollector in this order: the class comment, Add, AppendSorted, BuildAndWriteDataBlocks, EncodeIndex, FinishOnce. Then read the unsorted fallback (AppendUnsorted, SortUnsortedGroup, SortAndEncodeGroup). 6. Wide-column change (db/wide/wide_column_serialization.cc, ForEachBlobIndex and WalkV2Columns): it gives every blob index of an entity to a callback after it checks the whole entity. The table builder uses it to feed the collector. UpdateBoundaries, BlobGarbageMeter and the compaction iterator also use it, with the option on or off. 7. Option: enable_blob_indirection (include/rocksdb/advanced_options.h, options/cf_options.cc), and the option checks and gate in db/column_family.cc (ValidateOptions, TEST_SetAllowBlobIndirection), db/db_impl/db_impl_open.cc and db_impl_secondary.cc (the FIFO check for opens that skip ValidateOptions). 8. Table integration: table/block_based/block_based_table_factory.cc (the shared reservation manager), then block_based_table_builder.cc (Rep constructor, AddImpl, WriteBlobReferenceMetadata, EstimatedTailSize). See where the collector is created, fed (AddImpl(), which both Add() and TryAdd() call) and finished. block_based_table_reader.cc maps the new block name to a block type. Terms used in this diff: Stack terms: | Term | Meaning | |---|---| | origin | Blob file where the value was first written | | original offset | Value offset in the origin file | | stable identity | (origin, original offset). Never changes, even when the value moves | | generation number | The file number that a reference uses to find its value. A new value has generation number == origin. When indirect blob GC moves the value, the reference keeps its generation number, and a route sends the read to the relocation file | | indirect BlobIndex | A kIndirectBlob (type 3) or kIndirectFlattenedBlob (type 4) index. It stores the stable identity, so the value can move without an SST rewrite | | unflattened reference | Index with generation number == origin: the value was not moved, or it was moved but no compaction has rewritten its reference yet (see "flatten" in "Stack context: terms for later diffs") | | flattened reference | Index with generation number != origin (type 4). Only flatten (diff 10) writes it; this diff defines the encoding | | route | Maps an old file number (a file that GC has removed from the Version) to the relocation file that now holds its live values. One hop: a target is never itself routed. Stored in the MANIFEST (diff 4) | | relocation file | Block-based table keyed by (origin, original offset) that holds copied values. Each value is the stored bytes plus a 5-byte trailer (kSimpleGen2BlobTrailerSize) | | logical CRC | CRC32C of the uncompressed value (BlobIndex::stable_checksum()) | | stored CRC | CRC32C of the stored bytes plus the compression byte | | physical file | A file that holds value bytes: a normal blob file or a relocation file. A generation number can name a physical file, or it can be only the key of a route | | exact reference metadata | Per-SST sorted list of every blob reference the SST keeps, written when the SST is built. References with the same encoding class, generation number and origin form a group. The list is stored as data blocks, with an index of the groups. An SST that has it is an exact SST | | option gate | Until diff 14, ColumnFamilyData::ValidateOptions returns NotSupported for enable_blob_indirection on a read-write open, CreateColumnFamily and SetOptions, unless a test calls ColumnFamilyData::TEST_SetAllowBlobIndirection(true). It keeps the option off in production while the stack is not complete | Terms of this diff (all defined at the top of db/blob/blob_reference_metadata.h): | Term | Meaning | |---|---| | encoding class (class) | kDirect (a legacy kBlob or kBlobTTL index), unflattened, or flattened | | reference group (group) | All references of one SST with the same (encoding class, generation number, origin) | | run | The part of one group inside one data block | | record bytes | The bytes the value's record takes in the file named by its generation number | | gap | How far a record is from where it would be if the records were back to back. A gap of 0 means back to back | | descriptor | The 72-byte table property that points to the reference index block and holds the totals, the CF ID, the SST file number and the relocation-state fingerprint | | relocation-state fingerprint | 128-bit hash of the routes and relocation files of the Version that an SST was built from (fields route_snapshot_fingerprint_*). Written as 0 in this diff | | legacy | Written the way RocksDB writes it without blob indirection | Previous behavior: - A BlobIndex named only a physical blob file and an offset. A value could not move to another file without rewriting every SST that referenced it. - An SST had no metadata that listed its blob references. The only way to find them was to scan its user keys. - These readers get the blob indexes of a wide-column entity through ForEachBlobFileNumber: UpdateBoundaries, BlobGarbageMeter, the compaction iterator, WideColumnSerialization::RemoveBlobColumns, and the referenced_blob_bytes table property count in BlockBasedTableBuilder::Add. ForEachBlobFileNumber decoded each blob index, but it did not check the rest of the entity framing (name order, section sizes, exact input size), and it could call the callback before it found a later error. New behavior: Separate change (option on or off): - The same readers (UpdateBoundaries, BlobGarbageMeter, the compaction iterator, RemoveBlobColumns, and the referenced_blob_bytes count in BlockBasedTableBuilder::Add) use WideColumnSerialization::ForEachBlobIndex, with the option on or off. With the option on, the table builder also uses it to give the blob columns of each entity to the collector. For an entity with blob columns, it checks the full framing, then decodes and validates every blob index before any callback runs. So an entity with a malformed framing now fails the flush or compaction, even with the option off. The referenced_blob_bytes count itself does not fail the build on a malformed entity. It removes that property from the SST, as in base. Option and option gate: - This diff adds a new column family option, `enable_blob_indirection`. It is off by default, and SetOptions can change it. - The option gate (see Terms) keeps the option off outside tests until the stack is complete. ColumnFamilyData::ValidateOptions rejects the option with NotSupported on a read-write open, CreateColumnFamily and SetOptions. Read-only opens (DBImplReadOnly and CompactedDBImpl) and secondary opens (including DB::OpenAndCompact) do not run ValidateOptions; they check only the FIFO rule (see Implementation). Only tests can turn the option on, through the internal ColumnFamilyData::TEST_SetAllowBlobIndirection(). DBTestBase sets it. - When the option is on, ValidateOptions also requires enable_blob_files, the block-based table format, no blob direct write, and a compaction style other than FIFO. Blob file writer (option on): - The blob file writer encodes each new value with a stable identity (origin file, original offset) and two checksums: - the logical CRC: the CRC32C of the uncompressed value. - the stored CRC: the CRC32C of the exact stored (maybe compressed) bytes, extended with the compression byte. - For an uncompressed value the stored CRC is not written, because the reader can compute it from the logical CRC. - The checksums have no version field of their own. The BlobIndex type byte gives the format version: the two new types, 3 (kIndirectBlob) and 4 (kIndirectFlattenedBlob, see Implementation), are BlobIndex::kIndirectBlobFormatVersion (1). A later format uses new type values. - The read path does not check these checksums yet. The check starts in diff 3. Exact reference metadata (option on): 1. What: a sorted list of every blob reference that the SST keeps, in plain blob values and in the blob columns of wide-column entities. 2. Where: data blocks (target 16 KB each before compression; small blocks waste less I/O, because indirect blob GC reads a whole block to get a few groups) and one reference index block, written after the SST data blocks and before the SST tail, plus a 72-byte descriptor table property. 3. Who reads it: only tests in this diff. The census, GC and RepairDB read it in later diffs. 4. Details: the top comment of db/blob/blob_reference_metadata.h has the full format with diagrams, the record-bytes rules and the gap rule. - Not collected: SstFileWriter output (it has no DB file number) and any output whose CF is unknown (it has no CF ID). The descriptor needs both values, so these outputs stay in the legacy form. Checks on the metadata: - A repeated or overlapping reference in one group fails the SST build with Corruption. Readers check every count, order, size and total against the index and the descriptor, so corrupt metadata returns Corruption. The writer never makes a data block larger than kTargetDataBlockSize (16 KB), and readers accept up to kMaxDataBlockSize (64 KB). Details: "Readers" and "Integrity" at the top of db/blob/blob_reference_metadata.h, and BlobReferenceGroupDecoder. Collection and memory: - Each reference is encoded when it is added, in the on-disk entry encoding, into its group's chain of 512-byte chunks. Only a group that takes the unsorted fallback is encoded a second time, in Finish(). Inside a group, references normally arrive in rising offset order, so no reference is sorted; at the end only the groups are sorted. A group that gets a lower offset takes the unsorted fallback. The BlobReferenceMetadataCollector class comment explains the chunks, the chunk size and the fallback. - The collector keeps every reference of the SST in memory until the SST is finished, and charges this memory to the block cache under a new CacheEntryRole, kBlobReferenceMetadataBuildingBuffer. "Memory usage" and "Memory charge" in the BlobReferenceMetadataCollector class comment explain the parts of this memory and the charge rule. The measured amounts are under "Validation and known limitations". - There is no fixed limit on references. The collector fails with MemoryLimit above 2^20 groups or 2^20 data blocks in one SST; these are persisted format limits. It also fails with MemoryLimit when the reference index block could be larger than 64 MB (kMaxTopLevelMetadataBlockSize), which can happen from about 945K groups (64 MB / kMaxIndexEntrySize, 71 bytes per group), or from fewer groups when the SST also has many data blocks. - There is no charge when the table has no block cache or when charging for this role is disabled. Each block-based table factory holds one cache reservation manager for this role, shared by its builders and the SST summaries. If the charge fails, the SST build fails, so the flush or compaction fails with a background error. The buffers and their charge are released when the SST is finished or abandoned. Implementation: - BlobIndex adds two indirect encodings. The legacy encodings do not change. - Type 3, kIndirectBlob (unflattened, generation number == origin), stores the origin, the original offset, the stored size, the compression type, the logical CRC and, for compressed values, the stored CRC. It does not store the generation number. This saves the size of one varint file number per reference (about 3 to 4 bytes for typical file numbers; an estimate, not measured). - Type 4, kIndirectFlattenedBlob (flattened), stores the generation number first, then the same fields. Its generation number must be above the origin. The decoder rejects type 4 with generation number <= origin, so each reference has one encoding. - stored_checksum() computes the stored CRC of an uncompressed value when a caller asks for it: the logical CRC extended with the compression byte. A decode does not compute it, because most decodes (for example in compaction) never read it. - The encoders take the fields as one BlobIndex::IndirectBlobFields, set by field name, so values of the same type (file number, offset and size; the two CRCs) cannot be swapped. EncodeIndirectBlob() takes the generation number as a separate argument; EncodeUnflattenedIndirectBlob() uses the origin, so the common path cannot leave the generation number unset. - The new files blob_reference_metadata.{h,cc} define the exact reference metadata. The files also contain: - BlobReferenceMetadataCollector, the collector described above. It takes the SST identity (CF ID, SST file number and the relocation-state fingerprint) as one BlobReferenceSstIdentity, set by field name. - BlobReferenceGroupKey, the key of a group: (encoding class, generation number, origin). It has the order, the hash and the validity check that the collector map, the sort, the decoder and FindBlobReferenceGroup() use. BlobReference and BlobReferenceGroup keep their flat fields and return the key through group_key(). - The index codec and the descriptor table property. - BlobReferenceGroupDecoder, the only decoder of reference runs. It reads a group one data block at a time and holds every corruption check of a group walk. Readers that visit one group (found by a binary search of the index) use it, and so do readers that visit every block in order. - Callers in this diff: only the table builder runs the collector. The decoders (DecodeBlobReferenceIndex, FindBlobReferenceGroup, BlobReferenceGroupDecoder, ForEachBlobReference*) are used only by tests. kMaxDecodedBlobReferenceBytes is used only in compile-time checks of the count limits. The census (diff 5) and indirect blob GC (diffs 8 and 9) are their first production users. The census is also the first user of BlobReferenceSstSummary (see the last Implementation bullet). - Descriptor: the CF ID and the SST file number are the repair fields. They let RepairDB tell which column family and SST the list belongs to without a MANIFEST. The decoder rejects a descriptor whose repair fields are not set (unknown CF ID or SST file number 0). The descriptor also has a 128-bit relocation-state fingerprint (the route_snapshot_fingerprint fields). Remote compaction uses it in diff 11 to check that an SST built elsewhere used the same routes as the job that asked for it. In this diff the table builder always writes 0, and DecodeFrom() does not check it. The field is in the format now, so the descriptor keeps one format version (1) through the stack. While the option gate is on, no data in this format exists outside tests. - The collector validates each BlobIndex once, when it is added. Its Finish() writes the data blocks and then the reference index block through one block writer, and fills the descriptor, including the index handle. Destroying the collector frees its memory and then its charge, also when the build is abandoned. - BlockBasedTableBuilder creates the collector with the CF ID and SST file number from TableBuilderOptions. It passes the size of each entry's user key to the collector, so the collector can compute the exact size of each blob record. It adds the collector's EstimatedEncodedSize() to its tail size estimate: the blocks are written before the tail, but output file size planning uses this estimate. The reference blocks are written with BlockType::kUserDefinedIndex, which keeps their contents as opaque bytes. Readers read them without the block cache, so prepopulate_block_cache does not warm them into the block cache (WriteMaybeCompressedBlock() with warm_cache false). - To save CPU, compaction decodes each kTypeBlobIndex output value once, in CompactionOutputs::AddToOutput(). It passes the decode to TableBuilder::TryAdd() in the new field TableBuilderAddContext::decoded_blob_index, and to new overloads of BlobGarbageMeter::ProcessOutFlow() and FileMetaData::UpdateBoundaries(). BlockBasedTableBuilder decodes a kTypeBlobIndex value once for both the referenced_blob_bytes count and the collector, and uses the caller's decode when it has one. If the decode fails, no decode is passed, and each reader decodes the value itself and returns its usual error. This applies with the option on or off. The SST bytes are the same as when each reader decodes the value itself. - The collector converts each BlobIndex into a reference without deriving the stored CRC of an uncompressed value (ConvertBlobIndex() in db/blob/blob_reference_metadata.cc). The collector does not write that CRC, because a reader derives it again from the logical CRC. Deriving it would cost CPU for every reference of every output SST. BlobReference::FromBlobIndex() still derives it. - With the option on, the blob file builder computes the CRC32C of each stored value once. The blob log record CRC combines it with the key CRC (crc32c::Crc32cCombine). The stored CRC extends it with the compression byte. For an uncompressed value it is also the logical CRC. The bytes written do not change. - WideColumnSerialization::ForEachBlobIndex replaces ForEachBlobFileNumber. It reads the entity header and column types once. For a V1 entity (no column types and no blob columns), it returns right after the header. For a V2 entity without blob columns, it returns right after the type scan. It shares one V2 column walker (WalkV2Columns) with Deserialize(), so the entity is parsed in one pass. - Read-only opens (DBImplReadOnly, CompactedDBImpl) and secondary opens skip ValidateOptions, so they check the compaction style against the option once per column family: in DBImpl::Recover() for read-only opens, and at the start of OpenAsSecondaryImpl() for secondary opens. DB::OpenAndCompact opens through OpenAsSecondaryImpl(), so it gets the same check. - CacheReservationHandle gets Resize(), so each collector or summary can change its own part of the shared charge. - util/coding.h adds GetCanonicalVarint32/64, which reject overlong varint encodings. The new formats use them, so each value has one encoding. - BlobReferenceSstSummary is the in-memory list of an SST's groups and totals that Finish() can return. It has no caller yet: the table builder passes a null summary output. It is checked and does not change after Finish(), so the finished builder, the SST metadata and the census (its first user, in diff 5) can share it. It is in this diff because Finish() hands the group buffer and its cache charge over to the summary. Adding the summary later would mean rewriting the charge code of Finish() in diff 5, so the charge code is written and tested once, here. - sst_dump prints a "Blob reference metadata" section after the raw user collected properties of an SST that has exact metadata: the reference count, the stored value bytes, the column family ID, the SST file number when built and the index block handle. An SST without the descriptor prints nothing more. Safety boundaries and unsupported cases: - Public API: this diff adds the option `enable_blob_indirection` with the C API functions `rocksdb_options_set_enable_blob_indirection` and `rocksdb_options_get_enable_blob_indirection`. It also inserts a new CacheEntryRole value before kMisc, which changes the numeric value of kMisc and kNumCacheEntryRoles. - With the option off, new blob values and SSTs are written in the legacy form. Legacy blob indexes stay readable. - One behavior change applies with the option off too: the five readers listed under "Separate change" fully validate wide-column entities with blob columns and their blob indexes (see "Separate change" above). - Exact reference metadata has one format (version 1) with a fixed 72-byte descriptor. The decoders reject any other version as Corruption. - The metadata blocks are protected by the table's block checksum only. With checksum = kNoChecksum, corrupt metadata is detected only when it breaks the format or order checks. - The index decoder checks each decoded count against the remaining input size before it reserves memory. So a small corrupt index cannot make it allocate a large vector. - Inlined-TTL and same-file blob indexes are not collected and cannot be relocated. Encoding, count and checksum errors return an error; a bad reference is never skipped. - Option gate (see Terms): outside tests no DB writes data in the new format. The gate checks only the option value, never persisted state. Diff 14 removes it, once data written with the option stays readable through the end of the stack and the safety checks for turning the option on are in place. - Downgrading the binary after data was written with the option on is not supported. Validation and known limitations: - Collector memory, measured in a release build that drives the collector directly (bench/collector-chunk-20261111/README.md; host not recorded; 16-byte keys, 1 KB values, 62K to 1M references per SST in 1 to 16 groups): the buffers used about 8.3 to 9 bytes per reference for uncompressed values and 12.8 to 13.5 bytes for compressed values, including the chunk headers. With 1000 groups in a 62K-reference SST, they used about 14 bytes per uncompressed reference. - Chunk size, from the same measurement: for an SST with 1M references, the chunk size changed memory by less than 5% between 256 and 4096 bytes. For an SST with 62K references in 1000 groups, the partly filled chunks dominated: about 14 bytes per reference at 512 bytes, 22 at 1024 and 72 at 4096. 256 bytes used 3-4% more memory than 512 and made Finish() up to 1.6 times slower with 100 or more groups. - Reference index block size, measured in census db_bench runs (source report and host not recorded; 16-byte keys, 200-byte values, 1 MB blob files, no compression, fillrandom and then a full compaction): about 1.7 KB per SST with 32 MB write buffers and a target file size of 8 MB or 2 MB, and about 10 KB per SST with 1 MB write buffers and a target of 2 MB or 200 KB. The size is the bytes read for the index blocks divided by the number of SSTs. - Unit tests cover the BlobIndex encodings and their rejection of bad input, the metadata layout and every corruption check, the collector (sorted path, unsorted fallback, memory charge), the flush integration, the option checks and option gate, wide-column validation, and canonical varints. See the Test Plan. - The collection buffer is not spilled to disk. One SST with very many blob references needs a matching amount of charged memory: the per-reference amount given under "Collection and memory", plus up to one partly used 512-byte chunk per group. While a fallback group is sorted, it also needs 16 bytes of sort key per reference and a second copy of its entries. For a test group of 200K references with every other one out of order, the test comment estimates about 44 bytes per reference, and the test checks that the peak charge is at most 56 bytes per reference. - Flush still writes a single output file. Until diff 7, one compaction output collects every reference it gets, charged to the block cache. Diff 7 cuts compaction outputs (including L0) at a reference cap and keeps each user key's history in one output. With the option gate, a large output is reachable only through the test hook. Stack context (same in diffs 2-14, except that diffs 10-14 add the diff 10 route rule to the install line; "<== this diff" marks this diff's part): Goal: remove garbage from blob files without rewriting any SST. A value gets a stable identity when it is first written. Indirect blob GC copies the live values of whole files into a relocation file and installs routes in the MANIFEST. Readers follow the routes. SSTs are not changed. How a read finds a value (P is the relocation file that holds the value now): ``` SST entry indirect BlobIndex { generation number, origin, original offset, size, CRCs } | v Version file table: generation number -> live file, or route old file -> P | | live file route found | | v v blob file: read at original offset relocation file P (always one hop): relocation file: look up key look up key (origin, original offset) (origin, original offset) ``` Lifecycle of one value (in this example, F is the blob file where the value is first written, and P is the relocation file that GC writes): ``` write new value goes to blob file F; index: origin = F, generation number = F | census per generation number: the SSTs and references that use it; | per physical file: live bytes, ranked by garbage ratio GC job copy the live values of whole input files (stored bytes, checked | with the stored CRC) into relocation file P, keyed by | (origin, original offset) install one MANIFEST edit: add P, route each input generation number still | in use to P, re-point the routes into the inputs to P, and remove | each input (blob file retirement for a blob file, relocation | file deletion for a relocation file) flatten compaction sets generation number = P in the SSTs it rewrites | route route retirement: route deletion records remove the routes that delete no SST uses ``` Stack map: | Diff | Adds | |---|---| | 1 | Finalize blob metadata outside the DB mutex (prep) | | 2 | Indirect BlobIndex (kIndirectBlob) and exact SST reference metadata <== this diff | | 3 | Relocation file format and stable read path | | 4 | MANIFEST routes and relocation state in the Version | | 5 | Exact reverse reference census | | 6 | MANIFEST apply outside the DB mutex, with a precondition re-check | | 7 | Coarse flush metadata, per-SST reference cap, epoch runs | | 8 | Indirect blob GC building blocks | | 9 | Indirect blob GC job | | 10 | Flatten generation numbers; route retirement | | 11 | Remote compaction validation | | 12 | Dynamic options and operations | | 13 | Lost relocation files | | 14 | RepairDB rebuilds relocation state; option gate removed | | 15 | db_stress and crash test support | Stack context: terms for later diffs: | Term | Meaning | |---|---| | census | Background count, per generation number, of the SSTs and references that use it, plus live bytes per physical file (diff 5) | | indirect origin | Normal blob file written with the option on; it is the origin of unflattened references | | flatten | Compaction rewrites the generation number to the relocation file that holds the value, so old routes can go. Only flatten makes generation number != origin (diff 10) | | blob file retirement | MANIFEST record (kBlobFileRetirement) that removes a collected blob file from the Version (diff 4) | | route deletion | MANIFEST record that removes a route that no SST uses. Route retirement is the background job that writes these records (diff 10) | Tracking issue: meta#15301 Differential Revision: D124652097
Summary:
Why the change is required:
The goal of this stack is indirect blob GC, a blob GC that does not depend on SSTs: it removes garbage from blob files without rewriting any SST. Such a GC needs two things that RocksDB does not have:
This diff adds both. Later diffs build the census (an in-memory reverse index from generation number to the SSTs that use it), GC and repair on top of this list.
This diff in one paragraph:
With the new column family option enable_blob_indirection on, the blob file writer gives each new value an indirect BlobIndex (type 3, kIndirectBlob). It stores the value's stable identity (origin file, original offset) and its CRCs (an uncompressed value stores only the CRC of the uncompressed value; a compressed value also stores the CRC of its stored bytes). Also with the option on, the block-based table builder collects every blob reference that the SST keeps. It writes them as "exact reference metadata": data blocks, one reference index block, and a 72-byte descriptor table property. In this diff only the table builder writes this metadata, and only tests read it. A temporary option gate (see "option gate" under Terms) keeps the option off outside tests. Separately, wide-column entities with blob columns are now fully checked before their blob indexes are used, also with the option off.
Parts of this diff:
How to read this diff:
Terms used in this diff:
Stack terms:
Terms of this diff (all defined at the top of db/blob/blob_reference_metadata.h):
Previous behavior:
New behavior:
Separate change (option on or off):
Option and option gate:
enable_blob_indirection. It is off by default, and SetOptions can change it.Blob file writer (option on):
Exact reference metadata (option on):
Checks on the metadata:
Collection and memory:
Implementation:
Safety boundaries and unsupported cases:
enable_blob_indirectionwith the C API functionsrocksdb_options_set_enable_blob_indirectionandrocksdb_options_get_enable_blob_indirection. It also inserts a new CacheEntryRole value before kMisc, which changes the numeric value of kMisc and kNumCacheEntryRoles.Validation and known limitations:
Stack context (same in diffs 2-14, except that diffs 10-14 add the diff 10 route rule to the install line; "<== this diff" marks this diff's part):
Goal: remove garbage from blob files without rewriting any SST. A value gets a stable identity when it is first written. Indirect blob GC copies the live values of whole files into a relocation file and installs routes in the MANIFEST. Readers follow the routes. SSTs are not changed.
How a read finds a value (P is the relocation file that holds the value now):
Lifecycle of one value (in this example, F is the blob file where the value is first written, and P is the relocation file that GC writes):
Stack map:
Stack context: terms for later diffs:
Tracking issue: #15301
Differential Revision: D124652097