Repository navigation
Conversation
BACKUP of CAS tables to S3 on the same endpoint
CAS backup blobs less 5mb: a CAS blob whose source object is not greater than 5 MB is copied through buffers, because S3 accepts a byte range on `UploadPartCopy` only from a source object greater than 5 MB. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`copyS3File` lost its offset argument: a whole-object copy and a ranged copy are now two separate functions. A CAS blob is `[envelope][payload]`, so its copy is always ranged. Pass the blob size from the view plan as the source object size. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…fers S3 accepts a byte range in `UploadPartCopy` only from a source object greater than 5 MB, so such a blob is read through the `CAS` read path. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
k-morozov
changed the title
Upstream now has a ranged S3 copy: `copyS3FileRange` copies only `[offset, offset + size)` of a source object with `UploadPartCopy`, and falls back to buffers when the source object is 5 MB or smaller or when multipart copy is not available (ceeb177, 5c28acb). Before it, #2415 needed its own path for CAS blobs, because a blob is `[header][payload]` and a whole-object copy would put the header into the backup. With the upstream ranged copy, a CAS blob can go through the common `BackupWriterS3::copyFileFromDisk` path. Only the offset, the source object size and the fallback reader change: the offset includes the blob header, and the reader reads the raw object, so both use object coordinates. `whole_object` is now computed from this offset, so a CAS blob never takes `CopyObject`. This lets us drop the extra code that #2415 added to upstream files: - remove `tryNativeCopyFromContentAddressedDisk` and the 5 MB check; `copyS3FileRange` routes small blobs and stores without multipart copy to buffers itself; - remove `DataSourceDescription::files_are_whole_objects` and `canUseNativeCopyWith`, and restore the upstream `sameKind` and `==` checks and data source description initializers; - guard the native copy with `isContentAddressed` instead: the S3 and Azure backup readers, the Azure backup writer and `DiskObjectStorage::copyFile`. A CAS file goes through buffers when it is stored in the manifest (empty key) or is not a blob. Add trace logs with the reason for both cases, and for a ranged copy that cannot be native in `copyS3File`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A blob that cannot be copied inside `S3` is now read from its object in the pool, only the payload range, by the fallback of `copyS3FileRange`, not through the `CAS` read path. Also say that only the endpoint has to match for the copy inside `S3` (the backup can be in another bucket), and that a `CAS` disk cannot be a backup destination. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related: #2415
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Fixed BACKUP of a table on a CAS disk to an S3(...) or Disk(...) destination on the same S3 endpoint as the pool.
Documentation entry for user-facing changes
...
CI/CD Options
Exclude tests:
Regression jobs to run: