Antalya 26.8 BACKUP of CAS tables to S3 on the same endpoint

docs/en/antalya/cas/bucket-requirements.md

@@ -23,6 +23,7 @@ these conditions is refused rather than trusted.
 | Unconditional complete-object publication | `Backend::publishBlob` | An absent or condemned content-addressed body is replaced atomically; native stores may use multipart |
 | Native same-store copy when `cas_staging_backend = s3` | `IObjectStorage::copyObject` with `ObjectStorageCopyMode::NativeOnly` | The first absent staged publication may copy its complete object without a client-side fallback |
 | Exact-token delete | `Backend::deleteExact` | GC must delete only the incarnation it condemned, never a replacement |
+| Ranged same-store copy, for backups only | `UploadPartCopy` through `copyS3File` | A `BACKUP` to an `S3(...)` destination on the pool's own endpoint copies a blob's payload without its header. Optional, but only when the lack is recognized: `GCS` lacks it by its current XML API, so `Client::supportsMultiPartCopy` reports `false` there and the copy goes through the ClickHouse server, which is slower but correct. A store that instead refuses `UploadPartCopy` with an error other than `AccessDenied` fails the `BACKUP`; set `s3_allow_multipart_copy = 0` there |
 | Ranged `GET` | `Backend::get` / `Backend::getStream` with a `Range` | Opening one column file of a part costs one bounded read, not a whole-object fetch |
 | `LIST` with a resumable cursor | `Backend::list` | GC discovery and the orphan-manifest sweep page through the pool without a separate index |
 | No versioning / no delete markers | probed by `runCapabilityProbe`; `created_delete_marker` on `DeleteOutcome` | A delete marker over a live key would break exact-token semantics — GC would archive instead of reclaim |

docs/en/antalya/cas/index.md

@@ -94,4 +94,5 @@ per disk, so adopting it never requires migrating an existing deployment.
 | [Architecture overview](/antalya/cas/architecture/) | The object model, the Git analogy, and the safety invariants |
 | [Correctness](/antalya/cas/architecture/correctness) | How the design was verified: TLA+ models, counterexamples, soak methodology |
 | [Design history](/antalya/cas/architecture/design-history) | What earlier designs were tried and rejected, and why |
+| [Backup](/antalya/cas/operations/backup) | How `BACKUP` and `RESTORE` behave on a content-addressed disk |
 | [Roadmap](/antalya/cas/roadmap) | What is shipped, planned, and deliberately not pursued |

docs/en/antalya/cas/operations/backup.md

@@ -0,0 +1,160 @@
+---
+description: 'How BACKUP and RESTORE work for a table on a content-addressed disk: what holds the data during a backup, when the copy runs inside S3, and what is not supported yet.'
+sidebar_label: 'Backup'
+sidebar_position: 5
+slug: /antalya/cas/operations/backup
+title: 'CAS Operations — Backup'
+doc_type: 'guide'
+---
+
+# Operations — backup {#backup}
+
+Ordinary `BACKUP` and `RESTORE` work for a table on a content-addressed (`CAS`) disk. This page
+covers what happens during one, how it differs from a plain disk, and what the limits are.
+
+The `CAS`-native backup model — `snapshot` / `mirror` / `fetch` / `restore` — is designed but not
+implemented and is not wired into the SQL surface. See the [roadmap](/antalya/cas/roadmap#backups).
+
+## What is supported {#supported}
+
+```sql
+BACKUP TABLE t TO S3('https://bucket.s3.amazonaws.com/backups/b1', 'key', 'secret');
+RESTORE TABLE t AS t_restored FROM S3('https://bucket.s3.amazonaws.com/backups/b1', 'key', 'secret');
+```
+
+The destination can be `S3`, `Disk`, `File`, or an archive, but not a `CAS` disk (see
+[limitations](#limitations)). A backup can be restored onto a disk of any type, because it holds the
+table's files rather than the pool's objects.
+
+**An `Atomic` database is required.** That has been the default since 20.x. On the deprecated
+`Ordinary` engine the backup fails with `SUPPORT_IS_DISABLED`: that path pins files with temporary
+hard links, which object storage does not have.
+
+## What holds the data during a backup {#holding}
+
+On a plain disk a backup pins files against deletion with a hard link. `CAS` uses a different
+mechanism — pointer holding:
+
+- the backup holds a `shared_ptr` to the table and to each part;
+- the outdated-part cleanup skips those parts;
+- while a part is alive so is its [ref](/antalya/cas/architecture/manifests-and-refs#ref-table) — the
+  name under which the part is registered in its namespace's ref table, and through which it points
+  at its manifest;
+- while the ref is alive, garbage collection sees the manifest and its blobs as reachable.
+
+This mechanism lives in the process's memory and **does not survive a server restart**. An
+interrupted backup leaves nothing behind in the pool, but it also stops protecting the data once the
+process is gone. For durable pinning there is `FREEZE`, which publishes a real ref.
+
+## How the bytes move {#copy-path}
+
+Which path runs depends on the destination.
+
+**A destination outside the pool** — the common case: another `S3` endpoint, a local disk, an
+archive. Files are read through the `CAS` read path and written to the destination. Pool deduplication is lost:
+what was one blob shared by several replicas becomes ordinary files in the backup.
+
+**An `S3(...)` destination on the same `S3` endpoint as the pool** — the copy of a blob then runs
+inside the `S3` store itself, without sending the bytes through ClickHouse. Only the endpoint has to
+match: the backup can be in another bucket. The server issues one `UploadPartCopy` per upload part,
+so a payload larger than one part takes several.
+
+```sql
+BACKUP TABLE t TO S3('http://s3.example.com/bucket/backups/b1', 'key', 'secret');
+```
+
+Here the pool is under `http://s3.example.com/bucket/pool/`.
+
+The files of a part fall into two categories:
+
+| Category | Example | How it is copied |
+|---|---|---|
+| Blob | `data.bin`, marks, `primary.idx` | an `UploadPartCopy` naming a byte range — only the payload moves, without the blob's internal header; a blob of 5 MiB or smaller goes through buffers, see below |
+| Inside the manifest | `checksums.txt`, `count.txt`, `columns.txt` | read through the `CAS` read path and written through ClickHouse's buffers |
+
+A blob object is `[header][payload]`, so a file never starts at the beginning of its object. Every
+copy of a blob therefore names the payload range. A copy of the whole object would put the header
+into the backup, and a later restore would read wrong data.
+
+If a byte range cannot be copied inside `S3`, `CAS` does not fall back to copying the whole object —
+the payload is read and written through ClickHouse instead. That is slower, but correct. ClickHouse
+reads only the payload range of the blob object, directly from the pool, and uploads it to the
+backup. It takes that path on its own in four cases: the blob object is 5 MiB or smaller; the query
+sets `s3_allow_multipart_copy = 0`; the store is recognized as `GCS`, whose current XML API has no
+`UploadPartCopy`; or `UploadPartCopy` is refused with `AccessDenied`.
+
+The size limit comes from `S3`: it accepts a byte range in `UploadPartCopy` only if the source object
+is greater than 5 MB, and answers `InvalidRequest` otherwise. The limit is on the whole blob object,
+not on the upload part, so `s3_min_upload_part_size` does not change it. In a table with small parts
+most blobs are below the limit, so most of its files take this path.
+
+A store that refuses `UploadPartCopy` with any other error fails the `BACKUP` instead, because
+ClickHouse cannot tell a missing capability from a transient failure. On such a store, set
+`s3_allow_multipart_copy = 0` on the `BACKUP` query.
+
+| Turn off the copy inside `S3` | Turn off the range copy |
+|---|---|
+| `SETTINGS allow_s3_native_copy = 0` in the `BACKUP` query | the query setting `s3_allow_multipart_copy = 0` |
+
+```sql
+BACKUP TABLE t TO S3(...) SETTINGS allow_s3_native_copy = 0;
+```
+
+**A `Disk(...)` destination** never copies inside `S3`, even when the disk is an `s3` or `s3_plain`
+disk on the same endpoint as the pool:
+
+```sql
+BACKUP TABLE t TO Disk('backups_s3', 'b1');
+```
+
+Every file is read through the `CAS` read path and written through ClickHouse's buffers. The
+`allow_s3_native_copy` and `s3_allow_multipart_copy` settings have no effect on this path.
+
+## Restore {#restore}
+
+Each part is materialized in **one disk transaction** and published as one manifest and one ref. A
+partially restored part can never appear in the pool: either the whole part is published or nothing
+is.
+
+Restore onto a `CAS` disk never copies objects inside `S3`, even when the backup is on the same
+endpoint as the pool. Each file of the part is read from the backup and written through the `CAS`
+write path, because only that path can build the manifest and the blobs. A restore onto a plain disk
+works as usual and can copy inside `S3`.
+
+Restored data is packed afresh — on a `CAS` disk it gets new blobs and new refs. Deduplication
+against data already in the pool works as usual: identical content hashes to the same blob and is
+not written twice.
+
+## `FREEZE` is not a backup {#freeze}
+
+`ALTER TABLE ... FREEZE` works on `CAS` and publishes parts into a separate shadow namespace, which
+is a garbage-collection root in its own right. `DROP PARTITION` removes the live refs and leaves the
+snapshot alone; `SYSTEM UNFREEZE` removes only the shadow refs.
+
+It is still not a snapshot of a table: there is no SQL metadata, no single commit marker, no
+portable object with a listing and a restore API, and its lifetime is tied to a manual `UNFREEZE`.
+It is a useful building block, not a replacement for `BACKUP`.
+
+## Limitations {#limitations}
+
+- The `CAS`-native backup model (`snapshot` / `mirror` / `fetch`) is not implemented.
+- The `Ordinary` database engine is not supported.
+- Pool deduplication is lost in the backup: its size follows the logical files, not the unique blobs.
+- Pointer holding does not survive a server restart.
+- The copy inside the store works only for an `S3(...)` destination on an `S3` or `S3`-compatible
+  store, and only when the destination is on the same endpoint as the pool. Other destinations,
+  including `Disk(...)`, get the copy through ClickHouse's buffers.
+- A blob is copied inside `S3` only with multipart copy (`UploadPartCopy`), because only it can name a
+  byte range. A blob object of 5 MiB or smaller always goes through ClickHouse's buffers, because
+  `S3` accepts a byte-range copy only from a source object greater than 5 MB. With
+  `s3_allow_multipart_copy = 0`, on a store recognized as `GCS`, and when `UploadPartCopy` is refused
+  with `AccessDenied`, every blob goes through ClickHouse's buffers instead. A store that refuses
+  `UploadPartCopy` with another error fails the `BACKUP`; set `s3_allow_multipart_copy = 0` there.
+- Restore onto a `CAS` disk always writes through ClickHouse, see [restore](#restore).
+- A disk-level write of a part file onto a `CAS` disk outside of a part transaction is rejected with
+  `NOT_IMPLEMENTED` and the message `Autocommit writes are not supported for content part files`: a
+  `CAS` disk publishes a part's manifest and ref at commit, so it has nowhere to put a single
+  autocommitted file.
+- A `CAS` disk cannot be a backup destination: `BACKUP TABLE t TO Disk('<cas disk>', 'b1')` is rejected
+  the same way. A backup's own layout mirrors the table's data directory, so its files sit under a part
+  directory as well and count as part files.

docs/en/antalya/cas/roadmap.md

@@ -91,6 +91,10 @@ positioning.
 
 ## Backups {#backups}
 
+Ordinary `BACKUP` and `RESTORE` already work for a table on a `CAS` disk — see
+[backup](/antalya/cas/operations/backup) for how they behave and what the limits are. What follows is
+about the `CAS`-native model, which is a different thing.
+
 A `snapshot` / `mirror` / `fetch` / `restore` design is **approved but not implemented**. The
 model is deliberately git-shaped: `snapshot` is instant and free (like `git tag` — it references
 existing manifests, copies nothing); `mirror` is a continuous pull from a production pool into a

src/Backups/BackupIO_AzureBlobStorage.cpp

@@ -126,7 +126,8 @@ void BackupReaderAzureBlobStorage::copyFileToDisk(const String & path_in_backup,
     auto destination_data_source_description = destination_disk->getDataSourceDescription();
     LOG_TRACE(log, "Source description {}, destination description {}", data_source_description.description, destination_data_source_description.description);
     if (destination_data_source_description.object_storage_type == ObjectStorageType::Azure
-        && destination_data_source_description.is_encrypted == encrypted_in_backup)
+        && destination_data_source_description.is_encrypted == encrypted_in_backup
+        && !destination_disk->isContentAddressed())
     {
         LOG_TRACE(log, "Copying {} from AzureBlobStorage to disk {}", path_in_backup, destination_disk->getName());
         auto write_blob_function = [&](const Strings & dst_blob_path, WriteMode mode, const std::optional<ObjectAttributes> &) -> size_t
@@ -203,7 +204,8 @@ void BackupWriterAzureBlobStorage::copyFileFromDisk(
     auto source_data_source_description = src_disk->getDataSourceDescription();
     LOG_TRACE(log, "Source description {}, destination description {}", source_data_source_description.description, data_source_description.description);
     if (source_data_source_description.object_storage_type == ObjectStorageType::Azure
-        && source_data_source_description.is_encrypted == copy_encrypted)
+        && source_data_source_description.is_encrypted == copy_encrypted
+        && !src_disk->isContentAddressed())
     {
         /// getBlobPath() can return more than 2 elements if the file is stored as multiple objects in AzureBlobStorage container.
         /// In this case we can't use the native copy.

src/Backups/BackupIO_S3.cpp

@@ -16,6 +16,8 @@
 #include <IO/S3/Credentials.h>
 #include <IO/S3/getObjectInfo.h>
 #include <Disks/IDisk.h>
+#include <Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/ContentAddressedExchange.h>
+#include <Disks/DiskObjectStorage/ObjectStorages/IObjectStorage.h>
 
 #include <Poco/Util/AbstractConfiguration.h>
 
@@ -462,7 +464,8 @@ void BackupReaderS3::copyToDiskImpl(const String & path_in_backup, size_t offset
     /// We don't check for `has_throttling` here because the native copy almost doesn't use network.
     auto destination_data_source_description = destination_disk->getDataSourceDescription();
     if (destination_data_source_description.sameKind(data_source_description)
-        && (destination_data_source_description.is_encrypted == encrypted_in_backup))
+        && (destination_data_source_description.is_encrypted == encrypted_in_backup)
+        && !destination_disk->isContentAddressed())
     {
         LOG_TRACE(log, "Copying {} from S3 to disk {}", path_in_backup, destination_disk->getName());
         auto write_blob_function = [&](const Strings & blob_path, WriteMode mode, const std::optional<ObjectAttributes> & object_attributes) -> size_t
@@ -572,6 +575,40 @@ void BackupWriterS3::copyFileFromDisk(
         /// In this case we can't use the native copy.
         if (auto blob_path = src_disk->getBlobPath(src_path); blob_path.size() == 2)
         {
+            std::optional<IContentAddressedExchange::BlobViewPlan> ca_plan;
+            if (src_disk->isContentAddressed())
+            {
+                if (blob_path[0].empty())
+                {
+                    LOG_TRACE(
+                        log,
+                        "Copying file {} from content-addressed disk {} to S3 through buffers: the file is stored in the manifest",
+                        src_path,
+                        src_disk->getName());
+                    BackupWriterDefault::copyFileFromDisk(path_in_backup, src_disk, src_path, copy_encrypted, start_pos, length);
+                    return;
+                }
+
+                auto * ca = tryGetContentAddressedExchange(src_disk);
+                if (!ca)
+                    throw Exception(
+                        ErrorCodes::LOGICAL_ERROR,
+                        "Disk {} is content-addressed but does not implement IContentAddressedExchange",
+                        src_disk->getName());
+
+                ca_plan = ca->getBlobViewPlan(src_path);
+                if (!ca_plan)
+                {
+                    LOG_TRACE(
+                        log,
+                        "Copying file {} from content-addressed disk {} to S3 through buffers: the file is not a blob",
+                        src_path,
+                        src_disk->getName());
+                    BackupWriterDefault::copyFileFromDisk(path_in_backup, src_disk, src_path, copy_encrypted, start_pos, length);
+                    return;
+                }
+            }
+
             LOG_TRACE(log, "Copying file {} from disk {} to S3", src_path, src_disk->getName());
 
             auto src_client = disk_client_factory.getOrCreate(src_disk);
@@ -580,6 +617,9 @@ void BackupWriterS3::copyFileFromDisk(
             {
                 LOG_TRACE(log, "Falling back to copy file {} from disk {} to S3 through buffers", src_path, src_disk->getName());
 
+                if (ca_plan)
+                    return src_disk->getObjectStorage()->readObject(ca_plan->object, read_settings);
+
                 if (copy_encrypted)
                     return src_disk->readEncryptedFile(src_path, read_settings);
 
@@ -589,9 +629,10 @@ void BackupWriterS3::copyFileFromDisk(
             /// A copy is whole-object only if it covers the entire source; `start_pos != 0` is not a sound
             /// test, because a prefix [0, length) of a bigger file is a range too. `length` counts the bytes
             /// actually copied, so an encrypted source must be measured with its encrypted size.
-            const size_t source_size
-                = copy_encrypted ? src_disk->getEncryptedFileSize(src_path) : src_disk->getFileSize(src_path);
-            const bool whole_object = (start_pos == 0) && (length == source_size);
+            const size_t src_offset = ca_plan ? ca_plan->payload_offset + start_pos : start_pos;
+            const size_t source_size = ca_plan ? ca_plan->object.bytes_size
+                : (copy_encrypted ? src_disk->getEncryptedFileSize(src_path) : src_disk->getFileSize(src_path));
+            const bool whole_object = (src_offset == 0) && (length == source_size);
 
             if (whole_object)
                 copyS3File(
@@ -601,7 +642,7 @@ void BackupWriterS3::copyFileFromDisk(
                     s3_settings.request_settings, read_settings, blob_storage_log, runner, create_read_buffer);
             else
                 copyS3FileRange(
-                    src_client, /* src_bucket */ blob_path[1], /* src_key */ blob_path[0], start_pos, length,
+                    src_client, /* src_bucket */ blob_path[1], /* src_key */ blob_path[0], src_offset, length,
                     /* src_object_size= */ source_size,
                     /* dest_s3_client */ client, /* dest_bucket */ s3_uri.bucket,
                     /* dest_key */ fs::path(s3_uri.key) / path_in_backup,