diff --git a/.gitignore b/.gitignore index be965a52..a3944970 100644 --- a/.gitignore +++ b/.gitignore @@ -24,6 +24,7 @@ REPO_READY_AUDIT.md # MkDocs build output site/ +.venv-docs/ # pg_regress expected outputs — tracked; the global *.out would exclude them. !extension/coldfront/test/expected/*.out diff --git a/CLAUDE.md b/CLAUDE.md index b59dfccb..0eaf4df7 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -85,7 +85,7 @@ Snapshots of upstream state for the next major moves. Re-verify before acting on - duckdb-iceberg's 2.0-line commit path is rewritten (a retry loop with `RefreshFromCatalog` firing reactively after a 409, rather than proactively before the first attempt). Our bakery patch's anchors (`RefreshExistingManifestList`, `GetTransactionRequest`) no longer exist there — the proactive pre-commit refresh needs reimplementing, not dropping. - Two of our interop patches (manifest-list `format-version`, lowercase parquet `file_format`) are still needed against the 2.0-line `main`. - The DuckDB temp-spill collision above (`duckdb#15173`, `pg_duckdb#887`) is fixed on the 2.0 line: spill files are instance-qualified, so the per-backend temp directory stops being a correctness requirement once the stack moves to 2.0. -- Partitioned `INSERT` and `CREATE TABLE ... PARTITIONED BY (...)` already work at our current `ICEBERG_REF`; `docs/architecture.md` and the extension's partition-column notice calling this unsupported are stale and should be corrected independently of the 2.0 move. +- Partitioned `INSERT` and `CREATE TABLE ... PARTITIONED BY (...)` already work at our current `ICEBERG_REF`; `docs/architecture_guides/index.md` and the extension's partition-column notice calling this unsupported are stale and should be corrected independently of the 2.0 move. **DuckDB 1.5.6 (`v1.5-variegata` repin)** (checked 2026-10-04 at head `5dcf5070`): moving to the branch head forces a DuckDB newer than 1.5.4 for the whole stack — pg_duckdb loads extensions only for its exact DuckDB version, and pg_duckdb PR #1071 bumps to 1.5.5 while nothing upstream has tried 1.5.6 yet. All five `docker/iceberg-*-v15.patch` files still apply cleanly at the head and none of their fixes have landed upstream; `referenced_data_file` is still reader-only and `RETURNING` on a cold `INSERT`/`UPDATE`/`DELETE`/`MERGE` is still refused. The gains are vended-credential refresh, equality-delete scans with extra columns, truncated UTF-8 string bounds, an Azure SAS endpoint-suffix fix, and a `write.metadata.metrics` backport — treat a repin as a DuckDB stack move (base republish, full matrix), not a quick ref bump. @@ -109,8 +109,8 @@ Snapshots of upstream state for the next major moves. Re-verify before acting on ## Documentation - User docs live in `docs/` as a MkDocs site (pgEdge MkDocs structure); `README.md` is the GitHub landing/TOC. Build with `mkdocs build --strict` — it must pass with no warnings. - Keep `README.md` and `docs/index.md` near-identical, EXCEPT `docs/index.md` is user-facing and omits the developer-specific content that belongs only in a GitHub README: CI badges, build-from-source internals, the project-structure tree, dependency/build tables, and author/contributor info. -- Three indexes point at the pages in `docs/`, each with its own focus: the `mkdocs.yml` `nav:` (everything, including changelog, license and `formal/`), the `README.md` Documentation table (one terse row per guide), and the `docs/index.md` Next Steps list (prose, user-facing, top-level guides only; the per-mode deep dives are reached through `docs/architecture.md`, which lists them). Adding, renaming, or removing a guide under `docs/` means updating all three in the same commit; adding a deep dive means the nav, the README table, and the list in `docs/architecture.md`. `mkdocs build --strict` passes on a page no index links, so it will not catch this: check by listing `docs/*.md` against each index. +- Three indexes point at the pages in `docs/`, each with its own focus: the `mkdocs.yml` `nav:` (everything, including changelog, license and `formal/`), the `README.md` Documentation table (one terse row per guide), and the `docs/index.md` Next Steps list (prose, user-facing, top-level guides only; the per-mode deep dives are reached through `docs/architecture_guides/index.md`, which lists them). Adding, renaming, or removing a guide under `docs/` means updating all three in the same commit; adding a deep dive means the nav, the README table, and the list in `docs/architecture_guides/index.md`. `mkdocs build --strict` passes on a page no index links, so it will not catch this: check by listing `docs/*.md` against each index. - Internal/scratch docs (bench, experimental, hardening, comparison, backlog, scale-test, E2E) are gitignored and NEVER pushed to GitHub; never reference them from published docs. - ARCHITECTURE docs are reference material, not work logs: brief, present-tense, no history ("we used to…", version churn), no rejected-alternative narration. Keep the *why* (rationale that explains the design); cut the journey. -- One home per topic, no duplication across docs: usage/modes/config/CLI and one-time setup in `docs/usage.md`; build-from-source and testing/CI in `docs/installation.md`; storage-backend setup in `docs/object_store.md`; deep dives in `docs/architecture*.md`. When moving a section, fix every cross-anchor that pointed at it. +- One home per topic, no duplication across docs: shared day-to-day usage and the partition CLI in `docs/using_coldfront/index.md`; each mode's own guide in `docs/usage_tiered.md`, `docs/usage_decoupled.md`, and `docs/usage_partitioner.md`; PostgreSQL/ColdFront config in `docs/configuration.md`; Lakekeeper install/config/bootstrap in `docs/configure_lakekeeper.md`; build-from-source and testing/CI in `docs/installation.md`; storage-backend setup in `docs/object_store.md`; deep dives in `docs/architecture_guides/index.md` and `docs/architecture_tiered.md`/`docs/architecture_decoupled.md`/`docs/architecture_vectors.md`. When moving a section, fix every cross-anchor that pointed at it. - Tracked build-journal docs (`DUCKDB_1.5_PATCHED.md`, `DUCKDB_1.5_UNPATCHED.md`) are kept as history and never deleted, but they are not part of the docs site: never link them from `README.md` or `docs/`. Point build internals at the real recipe, `docker/Dockerfile.duckdb15-base`. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index ae9ef0d8..c978262a 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -5,8 +5,8 @@ Thanks for your interest in improving ColdFront. ## Getting started - Build and run the stack: [Installation](docs/installation.md). -- Day-to-day usage and the two operating modes: [Usage](docs/usage.md). -- Architecture: [Architecture](docs/architecture.md) (plus the tiered/decoupled +- Day-to-day usage and the two operating modes: [Usage](docs/using_coldfront/index.md). +- Architecture: [Architecture](docs/architecture_guides/index.md) (plus the tiered/decoupled deep dives). ## Development workflow diff --git a/README.md b/README.md index f280ee32..93104291 100644 --- a/README.md +++ b/README.md @@ -7,24 +7,35 @@ The ColdFront documentation consists of the following guides: - [Introduction](docs/index.md) -- Getting Started - - [Getting Started with ColdFront](docs/walkthrough.md) +- Architecture Guides + - [Architecture Overview](docs/architecture_guides/index.md) + - [Tiered Operating Mode](docs/architecture_tiered.md) + - [Decoupled (Iceberg-Only) Operating Mode](docs/architecture_decoupled.md) + - [Vector Storage Architecture](docs/architecture_vectors.md) +- Installing and Configuring ColdFront + - [Installing ColdFront](docs/installation.md) + - [Configuring ColdFront](docs/configuration.md) + - [Configuring Lakekeeper](docs/configure_lakekeeper.md) + - [Configuring your Object Store](docs/object_store.md) + - [Configuring a Distributed Deployment](docs/distributed_setup.md) +- [Using ColdFront](docs/using_coldfront/index.md) + - [Using ColdFront in Tiered Mode](docs/usage_tiered.md) + - [Using ColdFront in Decoupled Mode](docs/usage_decoupled.md) + - [Using ColdFront in Standalone Partitioned Mode](docs/usage_partitioner.md) + - [Working with Embeddings](docs/usage_vectors.md) +- [Exploring ColdFront Modes](docs/exploring_coldfront/index.md) - [Exploring Tiered Storage](docs/walkthrough_tiered.md) - [Exploring Decoupled Mode](docs/walkthrough_decoupled.md) - [Exploring the Standalone Partitioner](docs/walkthrough_partitioner.md) - [Exploring Distributed Mode](docs/walkthrough_distributed.md) - - [Building ColdFront from Source](docs/installation.md) - - [Setting Up an Object Store](docs/object_store.md) -- Architecture - - [Architecture Overview](docs/architecture.md) - - [Tiered Mode](docs/architecture_tiered.md) - - [Decoupled Mode](docs/architecture_decoupled.md) - - [Vector Storage](docs/architecture_vectors.md) -- [Using ColdFront](docs/usage.md) -- [Storing and Searching Embeddings](docs/usage_vectors.md) -- [Compacting the Cold Tier](docs/compaction.md) +- [Connecting to ColdFront](docs/connecting.md) +- Administration + - [Understanding ColdFront Caveats](docs/caveats.md) + - [Using Vended Credentials](docs/vended_credentials.md) + - [Supported Column Types](docs/supported_types.md) + - [Compacting the Cold Tier](docs/compaction.md) - Developer Resources - - [Verifying the Bakery Protocol](docs/formal/README.md) + - [Formal Model - ColdFront Decoupled-Mode Bakery](docs/formal/README.md) - [Release Notes](docs/changelog.md) ColdFront keeps tables in PostgreSQL and cold data in Apache Iceberg (Parquet @@ -103,7 +114,7 @@ tier, so the application sees one relation: ## Installation -If you are new to ColdFront, run the [guided walkthrough](docs/walkthrough.md) +If you are new to ColdFront, run the [guided walkthrough](docs/exploring_coldfront/index.md) to see all three modes in action with copy-pasteable commands. ColdFront is open source under the PostgreSQL License and runs on PostgreSQL @@ -112,12 +123,14 @@ build with Spock, which the Docker image is based on, for distributed mode. The full build workflow lives in the **[Installation guide](docs/installation.md)**: build the thin ColdFront layer on top of the published DuckDB 1.5.x base image (or build the base yourself), -or install bare-metal. Then continue with the Quickstart below. +or install bare-metal. Then continue with the +[walkthrough](docs/exploring_coldfront/index.md). ### Setting Up on Cloud S3? -Once the image is built, the **[S3 setup guide](docs/object_store.md)** takes -you from an empty bucket to a working cold tier end-to-end. +When the image is built, the +**[Configuring your Object Store](docs/object_store.md)** guide takes you from +an empty bucket to a working cold tier end-to-end. ## Configuration @@ -133,128 +146,74 @@ settings above. The `import` command takes a deployment YAML, modeled on [config.example.yaml](config.example.yaml), and writes it into the server once. ColdFront has no configuration file. A YAML passed to any later run is checked against the server, and the run takes only `postgres.dsn` from the file. -For every setting, see the [One-Time Setup](docs/usage.md#one-time-setup) and -[Tuning Knobs](docs/usage.md#tuning-knobs) sections of the Using ColdFront -guide. - -## Quickstart - -Build the image (see the [Installation](docs/installation.md) guide) and bring -up the stack: - -```bash -docker compose --profile local-store up -d --build -``` - -Bootstrap Lakekeeper and create a warehouse (see the one-time setup in the -[Using ColdFront](docs/usage.md) guide), then create a table in psql: - -```sql -CREATE EXTENSION IF NOT EXISTS pg_duckdb; -CREATE EXTENSION IF NOT EXISTS coldfront; -SELECT coldfront.set_storage_secret('admin', 'adminsecret', 'seaweedfs:8333'); - --- Decoupled (iceberg-only) table, stored entirely in Iceberg on S3: -SELECT coldfront.create_iceberg_table('public', 'events', - '[{"name":"id","type":"bigint"},{"name":"ts","type":"timestamptz"},{"name":"note","type":"text"}]'::jsonb, - '{month(ts)}'); -INSERT INTO events VALUES (1, now(), 'hello'); -SELECT count(*) FROM events; -``` - -ColdFront can adopt a table that already exists in the Iceberg catalog rather -than creating it: `coldfront.adopt_iceberg_table()` reads its schema from the -catalog and gives it the same wrapper view and registry row, read-only unless -you pass `p_writable => true`. `coldfront.release_iceberg_table()` hands the -table back with the Iceberg table untouched. See -[Adopting a Table That Already Exists in the Catalog](docs/usage.md#adopting-a-table-that-already-exists-in-the-catalog). - -To remove a table again, `coldfront.drop_iceberg_table()` unregisters it and -drops the Iceberg table, deleting the stored objects only when asked to. See -[Dropping an Iceberg Table](docs/usage.md#dropping-an-iceberg-table-both-modes). - -For compliance environments that cannot store an object-store credential, -`coldfront.set_storage_secret_vended()` runs with no credential in the -database: Lakekeeper issues short-lived per-table credentials at access time. -See [Vended Credentials](docs/usage.md#vended-credentials). +For every setting, see the +[Configuring Lakekeeper](docs/configure_lakekeeper.md) guide and the +[Configuring PostgreSQL](docs/configuration.md#configuring-postgresql) +section of the Configuring ColdFront guide. ## Using ColdFront -The Quickstart covers a decoupled table from start to finish. The -[Using ColdFront](docs/usage.md) guide covers both modes in depth, the -standalone partition manager and its CLI, the storage backends, and the -distributed setup; the [walkthrough](docs/walkthrough.md) demos run each mode -on a sample table. +[Using ColdFront in Tiered Mode](docs/usage_tiered.md), +[Using ColdFront in Decoupled Mode](docs/usage_decoupled.md), and +[Using ColdFront in Standalone Partitioned Mode](docs/usage_partitioner.md) +document each mode in depth; the +[Using ColdFront](docs/using_coldfront/index.md) guide helps you pick one, and +the [walkthrough](docs/exploring_coldfront/index.md) demos run each mode on a +sample table. Operational topics shared across modes - +[caveats](docs/caveats.md), [vended credentials](docs/vended_credentials.md), +[supported column types](docs/supported_types.md), and +[compaction](docs/compaction.md) - are documented under Administration; to run +ColdFront on more than one node, see +[Configuring a Distributed Deployment](docs/distributed_setup.md). ## Documentation -The following table lists the ColdFront guides and what each one covers: +The following table lists the ColdFront guides and what each one +documents: | Doc | Contents | |---|---| -| [Walkthrough](docs/walkthrough.md) | Sets up the demo stack and runs ColdFront hands-on. | -| [Tiered storage demo](docs/walkthrough_tiered.md) | Adds ColdFront to an existing database and moves its cold data to object storage. | -| [Decoupled mode demo](docs/walkthrough_decoupled.md) | Stores a table in Iceberg from the first row and adopts a table that another engine wrote. | -| [Partitioner demo](docs/walkthrough_partitioner.md) | Manages PostgreSQL range partitions without any cold tier. | -| [Distributed demo](docs/walkthrough_distributed.md) | Points two PostgreSQL nodes at one shared lake. | -| [Embeddings](docs/usage_vectors.md) | Covers storing and searching embeddings with the pgvector interface. | -| [Usage](docs/usage.md) | Covers day-to-day use: both modes plus the standalone partition manager, one-time setup, reading and writing, supported types, the partition CLI, storage backends, distributed (mesh) setup, and tuning. | -| [Installation](docs/installation.md) | Covers building from source (Docker or bare-metal), and testing and CI. | -| [Configuring your Object Store](docs/object_store.md) | Gets ColdFront running on cloud S3 (virtual-hosted), end to end. | -| [Compaction](docs/compaction.md) | Covers cold-tier table maintenance: compaction, snapshot expiry, and orphan-file removal. | -| [Architecture](docs/architecture.md) | Describes the shared architecture and core mechanics. | -| [Architecture: tiered](docs/architecture_tiered.md) | Describes tiered mode (hot PG plus cold Iceberg) in depth. | -| [Architecture: decoupled](docs/architecture_decoupled.md) | Describes decoupled (iceberg-only) mode in depth. | -| [Architecture: vectors](docs/architecture_vectors.md) | Describes vector storage internals: type mapping, routing state, cluster assignment, and layout. | - -## Least-Privilege Application Roles - -Application roles need no superuser and no server-file access, yet they read -and write the cold tier through the same transparent view. Onboarding an -application role is a single call: - -```sql -SELECT coldfront.grant_app_access('alice'); -``` - -grant_app_access grants only the minimum the cold path needs: membership in -duckdb.postgres_role, SET on the duckdb.unsafe_allow_execution_inside_functions -parameter, schema USAGE, `SELECT` on the registry and the watermark table, DML -on the dual-write anchor table, DML on every registered view and the hot table -behind it, and USAGE and `SELECT` on the hot table's sequences. Those objects -are derived from the registry, not hardcoded, and the call also grants EXECUTE -on a fixed allow-list of runtime cold-path functions. The call is idempotent -and is not executable by PUBLIC, so an application role can never self-grant. -The role is never granted pg_read_server_files or pg_write_server_files, so it -has no host-file access. `CREATE ROLE` and `GRANT` both replicate over Spock, -so you onboard a role once on any node and it propagates across the mesh. - -The Docker image sets `duckdb.postgres_role` to `coldfront_duckdb` and creates -that role when it initializes a new data directory. To name a different role, -set the container's `COLDFRONT_DUCKDB_ROLE` environment variable before that -first start. An empty value keeps pg_duckdb's default, under which only -superusers run DuckDB, so grant_app_access then fails with an error. - -For how the non-superuser path works - the `SECURITY DEFINER` attach helpers, -the `PGC_SUSET` / `GUC_SUPERUSER_ONLY` config hardening, the -`duckdb.postgres_role` default that the image sets up, and how least privilege -holds across a Spock mesh - see -[Architecture: non-superuser app roles](docs/architecture.md#non-superuser-app-roles-least-privilege). +| [Architecture Overview](docs/architecture_guides/index.md) | Describes the shared architecture and core mechanics. | +| [Tiered Operating Mode](docs/architecture_tiered.md) | Describes tiered mode (hot PostgreSQL plus cold Iceberg) in depth. | +| [Decoupled (Iceberg-Only) Operating Mode](docs/architecture_decoupled.md) | Describes decoupled (Iceberg-only) mode in depth. | +| [Vector Storage Architecture](docs/architecture_vectors.md) | Describes vector storage internals: type mapping, routing state, cluster assignment, and layout. | +| [Installing ColdFront](docs/installation.md) | Documents installing from a package or building from source (Docker or bare-metal), and testing and CI. | +| [Configuring ColdFront](docs/configuration.md) | Documents configuring PostgreSQL and ColdFront's own configuration. | +| [Configuring Lakekeeper](docs/configure_lakekeeper.md) | Installs, configures, and bootstraps Lakekeeper, then installs the extensions each database needs. | +| [Configuring your Object Store](docs/object_store.md) | Gets ColdFront running end to end, for any of four supported backends: S3-compatible, virtual-hosted cloud S3, GCS, or Azure ADLS Gen2. | +| [Configuring a Distributed Deployment](docs/distributed_setup.md) | Documents running a multi-writer decoupled-mode Spock mesh against one shared lake. | +| [Using ColdFront](docs/using_coldfront/index.md) | Compares the three operating modes, documents the shared SQL surface, and links to each mode's guide. | +| [Using ColdFront in Tiered Mode](docs/usage_tiered.md) | Documents archiving aging partitions to Iceberg while recent rows stay in the PostgreSQL heap. | +| [Using ColdFront in Decoupled Mode](docs/usage_decoupled.md) | Documents tables that live entirely in Iceberg from the first row, and dropping or releasing a registered table. | +| [Using ColdFront in Standalone Partitioned Mode](docs/usage_partitioner.md) | Documents managing PostgreSQL partitions with no cold tier at all, and the partition CLI shared with tiered mode. | +| [Working with Embeddings](docs/usage_vectors.md) | Documents storing and searching embeddings with the pgvector interface. | +| [Exploring ColdFront Modes](docs/exploring_coldfront/index.md) | Sets up the demo stack and runs ColdFront hands-on. | +| [Exploring Tiered Storage](docs/walkthrough_tiered.md) | Adds ColdFront to an existing database and moves its cold data to object storage. | +| [Exploring Decoupled Mode](docs/walkthrough_decoupled.md) | Stores a table in Iceberg from the first row and adopts a table that another engine wrote. | +| [Exploring the Standalone Partitioner](docs/walkthrough_partitioner.md) | Manages PostgreSQL range partitions without any cold tier. | +| [Exploring Distributed Mode](docs/walkthrough_distributed.md) | Points two PostgreSQL nodes at one shared lake. | +| [Connecting to ColdFront](docs/connecting.md) | Documents onboarding an application role with least privilege. | +| [Understanding ColdFront Caveats](docs/caveats.md) | Documents cross-tier isolation, mixed transactions, and other edge cases shared between tiered and decoupled tables. | +| [Using Vended Credentials](docs/vended_credentials.md) | Documents running with no stored object-store credential, for AWS S3 and Azure ADLS Gen2. | +| [Supported Column Types](docs/supported_types.md) | Documents the PostgreSQL column types, including arrays, that a cold tier supports. | +| [Compacting the Cold Tier](docs/compaction.md) | Documents cold-tier table maintenance: compaction, snapshot expiry, and orphan-file removal. | + +## Connecting to ColdFront + +See [Connecting to ColdFront](docs/connecting.md) for onboarding an +application role with least privilege. ## Caveats -Iceberg on Azure ADLS Gen2 requires Blob soft-delete, container soft-delete, -and change feed (blob events) to be OFF on the storage account. Lakekeeper -warehouse creation otherwise fails with HTTP 409 ("This endpoint does not -support BlobStorageEvents or SoftDelete"). Disable those features on the -storage account before using it as a cold tier. +See [Understanding ColdFront Caveats](docs/caveats.md) for ColdFront +behaviors you should be aware of, including a storage-account setting +Azure ADLS Gen2 requires before you use it as a cold tier. A cold-tier write that reads an ordinary PostgreSQL table, such as an `UPDATE ... FROM` or a `MERGE` from one, stores a `numeric` `NaN` from that -table as `0` and stores an array renumbered to start at 1, with no error. The -[Using ColdFront](docs/usage.md#reading-a-postgresql-table-in-a-cold-tier-write) -guide lists the writes that do this and shows how to check a source table -first. +table as `0` and stores an array renumbered to start at 1, with no error. +[Supported Column Types](docs/supported_types.md) lists the writes that do +this and shows how to check a source table first. ## Project Structure @@ -291,12 +250,18 @@ pgedge-coldfront/ │ ├── entrypoint.sh │ └── seaweedfs-s3.json ← SeaweedFS S3 auth config (example) ├── docs/ ← MkDocs site (user docs; mkdocs.yml at repo root) -│ ├── index.md · walkthrough.md · installation.md +│ ├── index.md · installation.md · configuration.md +│ ├── configure_lakekeeper.md · object_store.md · distributed_setup.md +│ ├── usage_tiered.md · usage_decoupled.md · usage_partitioner.md +│ ├── usage_vectors.md · connecting.md · caveats.md +│ ├── vended_credentials.md · supported_types.md · compaction.md │ ├── walkthrough_tiered.md · walkthrough_decoupled.md │ ├── walkthrough_partitioner.md · walkthrough_distributed.md -│ ├── object_store.md · usage.md · compaction.md -│ ├── architecture.md · architecture_tiered.md · architecture_decoupled.md -│ ├── architecture_vectors.md · usage_vectors.md · changelog.md +│ ├── architecture_tiered.md · architecture_decoupled.md +│ ├── architecture_vectors.md · changelog.md · LICENSE.md +│ ├── architecture_guides/ ← Architecture Guides landing page (index.md) +│ ├── exploring_coldfront/ ← Exploring ColdFront Modes landing page (index.md) +│ ├── using_coldfront/ ← Using ColdFront landing page (index.md) │ └── formal/ ← TLA+ model of the bakery protocol (Bakery.tla) ├── examples/walkthrough/ ← interactive walkthrough (guide.sh, its compose stack and configs) ├── docker-compose.yml ← END-USER single-node stack (ports published) diff --git a/docs/architecture_decoupled.md b/docs/architecture_decoupled.md index 832417e6..0c298840 100644 --- a/docs/architecture_decoupled.md +++ b/docs/architecture_decoupled.md @@ -6,9 +6,9 @@ archiver. PostgreSQL becomes a stateless compute front-end. Lakekeeper and the underlying object store own the storage. Decoupled mode shares the same codebase, docker stack and extension as tiered mode. The shared mechanics - pg_duckdb Iceberg I/O, the rewrite hook, the bakery protocol, the registry - -are in [architecture.md](architecture.md); tiered mode is in -[architecture_tiered.md](architecture_tiered.md). This document covers what is -specific to decoupled mode. +are in the [Architecture Overview](architecture_guides/index.md); tiered mode +is in [Tiered Operating Mode](architecture_tiered.md). This document covers +what is specific to decoupled mode. ## What "Decoupled" Means @@ -25,10 +25,10 @@ this document, concern by concern: | `coldfront.tiered_views` row | The row is required for each managed table. | The row is required (with `is_iceberg_only = true`), and `create_iceberg_table()` registers it. | | Required at runtime | `pg_duckdb`, `coldfront`, Lakekeeper, an S3-compatible store or Azure ADLS Gen2 | `pg_duckdb`, `coldfront` (lazy catalog ATTACH, DML rewrite, and write serialization), Lakekeeper, an S3-compatible store or Azure ADLS Gen2 | -The coldfront extension provides the lazy catalog-attach glue: the C extension -hook intercepts the first query that touches a tiered view (read or write) and, -if the Iceberg catalog `ice` is not yet attached in this session, issues the -following against the GUCs `coldfront.warehouse` and +The `coldfront` extension provides the lazy catalog-attach glue: the C +extension hook intercepts the first query that touches a tiered view (read or +write) and, if the Iceberg catalog `ice` is not yet attached in this session, +issues the following against the GUCs `coldfront.warehouse` and `coldfront.lakekeeper_endpoint`: ```sql @@ -80,8 +80,7 @@ because the DSN can hold credentials. `set_storage_secret` installs the attaches `pglocal` when such a write runs. With the GUC unset, that write fails, because DuckDB has no `pglocal` catalog to resolve. The extension reads a `numeric` `NaN` as `0` and renumbers an array to start at 1, so such a write -can store a changed value, as -[Reading a PostgreSQL Table in a Cold-Tier Write](usage.md#reading-a-postgresql-table-in-a-cold-tier-write) +can store a changed value, as [Supported Column Types](supported_types.md) describes. After that, the first query touching a tiered view in any session lazily @@ -89,7 +88,7 @@ attaches the catalog and `ice.public.*` becomes available. ## Interface -With the coldfront extension loaded and the storage secret set, the wrapper +With the `coldfront` extension loaded and the storage secret set, the wrapper view supports the operations below. ### What Works @@ -101,7 +100,7 @@ notes: |---|---|---| | Lazy catalog ATTACH | The C hook calls `ensure_attached()` on the first query that touches a tiered view. | The ATTACH costs one round-trip on the first Iceberg query per session. | | CREATE TABLE | `SELECT duckdb.raw_query('CREATE TABLE ice.. (...)')` | The statement is DuckDB SQL in attached-catalog syntax. | -| INSERT | The C hook rewrites an `INSERT INTO ` whose source is `VALUES (…)`, `SELECT … FROM ` or `SELECT … FROM generate_series(…)` to one `SELECT coldfront._exec_iceberg_with_claim(, 'INSERT INTO ice.. …')`. Source-table references get the prefix `pglocal..`, so DuckDB's postgres extension streams the source via libpq into the Iceberg writer. That attachment needs `coldfront.local_pg_dsn` (see [Bootstrap Sequence](#bootstrap-sequence)). A `WITH` entry that modifies data is refused, because DuckDB 1.5 has no data-modifying `WITH`. An `INSERT` nested in a `WITH` entry is rewritten in place when it reads no other entry, since DuckDB does not see them. | Each INSERT produces a single Iceberg snapshot, regardless of row count. | +| INSERT | The C hook rewrites an `INSERT INTO ` whose source is `VALUES (…)`, `SELECT … FROM ` or `SELECT … FROM generate_series(…)` to one `SELECT coldfront._exec_iceberg_with_claim(, 'INSERT INTO ice.. …')`. Source-table references get the prefix `pglocal..
`, so DuckDB's `postgres` extension streams the source via libpq into the Iceberg writer. That attachment needs `coldfront.local_pg_dsn` (see [Bootstrap Sequence](#bootstrap-sequence)). A `WITH` entry that modifies data is refused, because DuckDB 1.5 has no data-modifying `WITH`. An `INSERT` nested in a `WITH` entry is rewritten in place when it reads no other entry, since DuckDB does not see them. | Each INSERT produces a single Iceberg snapshot, regardless of row count. | | UPDATE | The hook rewrites `UPDATE SET … WHERE …` to `SELECT coldfront._exec_iceberg_with_claim(, 'UPDATE ice.. SET ... WHERE ...')`. | Iceberg applies the update as merge-on-read. | | DELETE | The hook rewrites `DELETE FROM WHERE …` to `SELECT coldfront._exec_iceberg_with_claim(, 'DELETE FROM ice.. WHERE ...')`. | Iceberg records the delete in position-delete files. | | MERGE (PostgreSQL 17 and later) | The hook rewrites `MERGE INTO USING … ON … WHEN …` to `SELECT coldfront._exec_iceberg_with_claim(, 'MERGE INTO ice.. …')`, the source's PostgreSQL tables prefixed with `pglocal..
` and each `INSERT` action's `OVERRIDING` clause dropped. Nested in a `WITH` entry it is rewritten in place when it reads no other entry, since DuckDB does not see them. | duckdb-iceberg runs one `UPDATE` or `DELETE` action per statement, and no `RETURNING`. | @@ -130,7 +129,7 @@ PG-side wrapper view. ## Supported Column Types The supported column types are exactly the set that round-trips cleanly between -PG and Iceberg. Both modes take them from one type map, +PG and Iceberg. Tiered and decoupled tables take them from one type map, `coldfront._iceberg_storage_type`. Anything outside this list is rejected at table-creation time. @@ -167,7 +166,8 @@ which would lose precision or identity: - custom enums, `xml`, `tsvector`/`tsquery`, range types, and multirange types. - composite types. - arrays of `timestamptz`, `jsonb`, `json`, `interval` or a vector type, and - arrays of arrays; [Arrays](usage.md#arrays) gives the reasons. + arrays of arrays; [Supported Array Types](supported_types.md#supported-array-types) + gives the reasons. - pgvector's `sparsevec`. - any type not enumerated above. @@ -206,8 +206,8 @@ DELETE FROM events WHERE id = 1; The fourth argument, `p_partition_cols`, is the table's Iceberg partitioning as a `text[]` of `PARTITIONED BY` terms, passed to DuckDB as written; the terms, and how a term with a comma or a quoted name is written in the array literal, -are in the [Mode 2](usage.md#mode-2-decoupled-iceberg-only) section of the -Using ColdFront guide. DuckDB refuses an unknown transform, a bad argument or a +are in the [Using ColdFront in Decoupled Mode](usage_decoupled.md) guide. +DuckDB refuses an unknown transform, a bad argument or a column outside the schema at `CREATE TABLE`. The one check DuckDB leaves to the first `INSERT`, a time transform on a column that is not a timestamp or date (for `hour`, not a timestamp), `coldfront._partition_clause()` makes at the @@ -445,7 +445,7 @@ Adoption inherits three limits: ## Wrapper Helper: `coldfront.drop_iceberg_table()` -Drops the Iceberg table backing a registered relation, in either mode: +Drops the Iceberg table backing a registered tiered or decoupled relation: ```sql -- catalog entry and stored objects both go @@ -455,9 +455,9 @@ SELECT coldfront.drop_iceberg_table('public', 'events', true); SELECT coldfront.drop_iceberg_table('public', 'events', false); ``` -One verb covers both modes because in both the thing being dropped is the -Iceberg table. What that means for PostgreSQL differs, because the modes -differ, and the function says which path it took in a NOTICE: +One verb covers tiered and decoupled tables because, for each, the thing being +dropped is the Iceberg table. What that means for PostgreSQL differs, because +the modes differ, and the function says which path it took in a NOTICE: - For an iceberg-only table, the Iceberg table is the whole relation, so nothing remains. This is the inverse of `create_iceberg_table()`. @@ -522,8 +522,9 @@ Three properties are worth knowing: ## ACID Model -(This section summarizes material from [architecture.md](architecture.md) -§Concurrency and §Known Limitations applied to the decoupled scenario.) +(This section summarizes material from the +[Architecture Overview](architecture_guides/index.md) §Concurrency and §Known +Limitations applied to the decoupled scenario.) The following table shows the status of each ACID property in decoupled mode: @@ -705,9 +706,9 @@ pointing at the same Lakekeeper endpoint and S3 bucket: `spock.feedback_frequency` messages, and in answer to the walsender's keepalive, which on an idle link comes every `wal_sender_timeout/2`, so the claim refuses to run unless `wal_sender_timeout` is positive and - below twice the window; usage.md gives the default and the rule. An - *alive* - peer that has not acked is either deferring (R-A's defer rule, legitimate) + below twice the window; + [Timeouts](distributed_setup.md#timeouts) gives the default and the + rule. An *alive* peer that has not acked is either deferring (R-A's defer rule, legitimate) or about to ack - either way, waiting is correct. A same-node claim is released by the C XactCallback in [extension/coldfront/src/coldfront.c](https://github.com/pgEdge/ColdFront/blob/main/extension/coldfront/src/coldfront.c) @@ -729,8 +730,8 @@ pointing at the same Lakekeeper endpoint and S3 bucket: - For DDL, Spock's `ddl_sql` repset replicates `CREATE/ALTER/DROP` of the wrapper view, and the `default` repset replicates the `coldfront.tiered_views` registry row (see - [Distributed Setup](usage.md#distributed-setup-3-node-mesh-decoupled-mode)), - so one node provisions the table and replication enables every peer's hook. + [Configuring a Distributed Deployment](distributed_setup.md)), so one node provisions the + table and replication enables every peer's hook. A column `ALTER TABLE` reaches the peers the same way: the node that runs it changes the shared Iceberg table, and its rebuilt view and registry row replicate. @@ -750,7 +751,7 @@ settings: wal_level = logical shared_preload_libraries = 'snowflake,spock,pg_duckdb,coldfront' -# Set together with coldfront.peer_alive_window_ms (see usage.md). +# Set together with coldfront.peer_alive_window_ms (see distributed_setup.md). wal_sender_timeout = 15s # Sync-rep is NOT required by the bakery - R-A's ack barrier replaces it. @@ -797,8 +798,7 @@ The call is idempotent. It puts every ColdFront table that replicates by value in the node's replication sets: the two bakery tables, the registry and the watermark, the storage secret, the lifecycle config and the vector routing state (the list and the reason for each table are in the -[Distributed Setup](usage.md#what-coldfrontensure_replicated-does) section of -the Using ColdFront guide). +[Configuring a Distributed Deployment](distributed_setup.md#what-coldfrontensure_replicated-does) guide). If it has not run on a peer, that peer's ack `INSERT`s are local-only and never replicate back to the originating writer: every claim on the originator waits at the ack barrier for an ack that never arrives. @@ -810,8 +810,8 @@ mandatory in any multi-node setup. ## When to Use Decoupled vs Tiered -The two modes suit different workloads; the guidance below summarizes when each -one fits. +Decoupled and tiered modes suit different workloads; the guidance below +summarizes when each one fits. Decoupled (iceberg-only) is the right choice when: @@ -852,16 +852,16 @@ Decoupled mode has the following limitations: - A transaction block that writes a decoupled table cannot also write a PostgreSQL table unless `duckdb.unsafe_allow_mixed_transactions` is on: pg_duckdb refuses the PostgreSQL write, or the `COMMIT` when that write came - first. The [Caveats](usage.md#caveats) describe the rule and the risk of - setting the parameter. + first. The [Caveats](caveats.md) describe the rule and the risk of setting + the parameter. ## Next Steps To go further with ColdFront, consult the following documents: -- The [Architecture](architecture.md) overview describes the mechanics both +- The [Architecture](architecture_guides/index.md) overview describes the mechanics both modes share. - The [Formal Verification](formal/README.md) document describes the TLA+ model of the bakery protocol and how to check it. -- The [Using ColdFront](usage.md) guide covers creating decoupled tables and +- The [Using ColdFront](using_coldfront/index.md) guide covers creating decoupled tables and setting up a distributed mesh. diff --git a/docs/architecture.md b/docs/architecture_guides/index.md similarity index 92% rename from docs/architecture.md rename to docs/architecture_guides/index.md index 00500b4e..4a93bc98 100644 --- a/docs/architecture.md +++ b/docs/architecture_guides/index.md @@ -7,10 +7,11 @@ rewrites reads and writes to hit the right tier, and **all Iceberg I/O goes through `pg_duckdb` running in-process inside PostgreSQL** - no external query engine, no Go Iceberg libraries. -For mode-specific design, see [architecture_tiered.md](architecture_tiered.md) -(hot PG + cold Iceberg) · -[architecture_decoupled.md](architecture_decoupled.md) (all-Iceberg) · -[architecture_vectors.md](architecture_vectors.md) (vector storage). +For mode-specific design, see +[Tiered Operating Mode](../architecture_tiered.md) (hot PostgreSQL plus cold +Iceberg), +[Decoupled (Iceberg-Only) Operating Mode](../architecture_decoupled.md), and +[Vector Storage Architecture](../architecture_vectors.md). ## Operating Modes and Topologies @@ -33,8 +34,8 @@ path in the hook takes, mostly through `_exec_iceberg_with_claim`. Decoupled mode always classifies as `TIER_COLD` and never reaches `emit_hot`; vanilla and mesh differ only in how that claim serializes cold writes. This document covers the shared mechanics and the tiered path; see -[architecture_decoupled.md](architecture_decoupled.md) for the decoupled mode's -ACID model and distributed scaling story. +[Decoupled (Iceberg-Only) Operating Mode](../architecture_decoupled.md) for the +decoupled mode's ACID model and distributed scaling story. ### Read Target: Primary or Physical Standby @@ -104,9 +105,9 @@ The following table describes each component, its role, and its license: How rows move through this depends on the storage mode: the tiered hot heap + archiver + `UNION ALL` data-flow is in the -[Data Flow](architecture_tiered.md#data-flow) section of the Tiered Mode page; +[Data Flow](../architecture_tiered.md#data-flow) section of the Tiered Mode page; the all-Iceberg flow is in -[architecture_decoupled.md](architecture_decoupled.md). +[Decoupled (Iceberg-Only) Operating Mode](../architecture_decoupled.md). ## Core Mechanics: pg_duckdb @@ -161,7 +162,7 @@ into a fresh instance) and the bakery's transient claim tables (`claims` / re-established a restored node serves hot reads but fails cold I/O cleanly. `ci/ops.sh` Check 4 exercises exactly this. -The Iceberg catalog ATTACH is **lazy**: the coldfront C extension hook issues +The Iceberg catalog ATTACH is **lazy**: the `coldfront` C extension hook issues `ATTACH IF NOT EXISTS` against Lakekeeper - using the cluster's `coldfront.warehouse` and `coldfront.lakekeeper_endpoint` GUCs - on the **first query that touches a tiered view** (read or write), per DuckDB cached @@ -175,17 +176,17 @@ view, so a missing warehouse never blocks a connection opened before bootstrap. pg_duckdb force-disables DuckDB's `LocalFileSystem` for any role that lacks both `pg_read_server_files` and `pg_write_server_files` (see [Upstream requests](#pg_duckdb-non-superuser-localfilesystem-blocks-side-loaded-extensions)), -which would block the side-loaded iceberg/postgres DuckDB extensions from -loading on `ATTACH`. So `coldfront.ensure_attached()` / `ensure_pg_attached()` -are `SECURITY DEFINER` with a pinned `search_path`: the extension load + -`ATTACH` run elevated (gates key off `GetUserId()`, the effective user), and -because the DuckDB instance is per-backend the attach persists for the -session - every subsequent cold read / `_exec_iceberg_with_claim` then runs -as the **app role** over S3/httpfs, never touching `LocalFileSystem`. The app -role needs only `duckdb.postgres_role` membership, object grants, and SET on -the superuser-only `duckdb.unsafe_allow_execution_inside_functions` -parameter, which the cross-tier move needs; **no superuser, no -`pg_{read,write}_server_files`**. +which would block the side-loaded `iceberg` and `postgres` DuckDB extensions +from loading on `ATTACH`. So `coldfront.ensure_attached()` / +`ensure_pg_attached()` are `SECURITY DEFINER` with a pinned `search_path`: the +extension load + `ATTACH` run elevated (gates key off `GetUserId()`, the +effective user), and because the DuckDB instance is per-backend the attach +persists for the session - every subsequent cold read / +`_exec_iceberg_with_claim` then runs as the **app role** over S3/httpfs, never +touching `LocalFileSystem`. The app role needs only `duckdb.postgres_role` +membership, object grants, and SET on the superuser-only +`duckdb.unsafe_allow_execution_inside_functions` parameter, which the +cross-tier move needs; **no superuser, no `pg_{read,write}_server_files`**. Because the attach helpers run elevated, the deployment-config GUCs they consume (`coldfront.warehouse`, `coldfront.lakekeeper_endpoint`, @@ -211,7 +212,7 @@ with the privilege it requires. `_exec_iceberg_with_claim` deliberately stays `SECURITY INVOKER` - it runs the caller's cold DML, which must execute as the caller. This `SECURITY DEFINER` setting is **protocol-neutral**: it changes the PG execution privilege, not the claim/ack/lock/ticket protocol, re-verified -against [the TLA+ model](formal/Bakery.tla) (all safe configs pass; the race +against [the TLA+ model](../formal/Bakery.tla) (all safe configs pass; the race config still violates `NoLakekeeperConflict`). See README "Security"; the setting is asserted by the journey's @@ -244,7 +245,7 @@ WHERE r['ts'] < '2026-03-01'::timestamptz; ## Application Interface Applications use the transparent view exactly like a table. A -`post_parse_analyze_hook` in the coldfront extension intercepts +`post_parse_analyze_hook` in the `coldfront` extension intercepts `INSERT`/`UPDATE`/`DELETE`/`MERGE` whose target is a registered relation - resolved in `coldfront.tiered_views` by name (`schema_name`, `relname`) - and rewrites the parsed `Query` so it lands in the correct tier; cold-side writes @@ -256,9 +257,8 @@ The same parse-analyze hook prepares a `SELECT` that DuckDB will run, wherever in the statement the view is named (a CTE, a sub-select, a set-operation branch): `date_bin`, the `::jsonb` cast, `jsonb_array_length` and the JSON builders (`jsonb_build_object`, `jsonb_agg` and their `json_` twins) are -rewritten into spellings both engines accept (see the -[Supported Column Types](usage.md#supported-column-types) section of the Using -ColdFront guide). A +rewritten into spellings both engines accept (see +[Supported Column Types](../supported_types.md)). A `planner_hook` folds bound parameters into such a read before pg_duckdb plans it when a parameter sits where DuckDB cannot type a placeholder (a direct argument of a pg_duckdb function, any argument of a table function); the plan @@ -267,8 +267,7 @@ above any custom plan, so under the plan cache's cost-based selection the read is planned from its values on every execution. `plan_cache_mode = force_generic_plan` bypasses that selection and picks the value-less generic plan, which fails with `only works with DuckDB execution` -(see the [Supported Column Types](usage.md#supported-column-types) section of -the Using ColdFront guide). +(see [Supported Column Types](../supported_types.md)). The following table maps each operation to its interface and routing path: @@ -287,12 +286,12 @@ How the hook splits a write is mode-specific: - In tiered mode, the hook routes by the partition-column watermark - hot heap vs cold Iceberg, with dual-tier writes for ambiguous predicates. See - [Transparent `INSERT`](architecture_tiered.md#transparent-insert) and - [Transparent `UPDATE`/`DELETE`](architecture_tiered.md#transparent-updatedelete) + [Transparent `INSERT`](../architecture_tiered.md#transparent-insert) and + [Transparent `UPDATE`/`DELETE`](../architecture_tiered.md#transparent-updatedelete) in the Tiered Mode page. - In decoupled mode, the hook always classifies `TIER_COLD`, so every write is a single-tier Iceberg write. See - [architecture_decoupled.md](architecture_decoupled.md). + [Decoupled (Iceberg-Only) Operating Mode](../architecture_decoupled.md). ### Cold-Tier DML from Inside PL/pgSQL (Functions, DO Blocks, Triggers) @@ -339,13 +338,13 @@ ColdFront coordinates concurrent writes across the cluster as follows: lock acquisition (10 attempts, exponential backoff from 100 ms to 51.2 s). - Hot writes are replicated by Spock normally (standard PG DML). - Cold writes via `duckdb.raw_query()` from multiple nodes are serialized - PG-side by the **bakery protocol** in the coldfront extension - every + PG-side by the **bakery protocol** in the `coldfront` extension - every iceberg-only `INSERT`/`UPDATE`/`DELETE`/`MERGE` wraps in `coldfront._exec_iceberg_with_claim`, which holds a globally-ordered Snowflake ticket via the Spock-replicated `coldfront.claims` table and waits for its turn before issuing the iceberg commit. The bakery produces no 409s and needs no retry in the application. See the - [Concurrency](architecture_decoupled.md#concurrency-horizontal-scaling-the-bakery-protocol) + [Concurrency](../architecture_decoupled.md#concurrency-horizontal-scaling-the-bakery-protocol) section of the Decoupled Mode page for the full design and benchmarks. ### Cold-Write Strategy: Stock vs Patched duckdb-iceberg @@ -361,7 +360,7 @@ The following table shows the two orderings: | Ordering | Applies to | Behavior | |---|---|---| | stock (default) | every write on stock upstream duckdb-iceberg; a `DELETE`, `UPDATE` or `MERGE` (the cross-tier move and `vector_train`'s assignment included) on every build | The writer takes the serializer first, *then* uploads parquet and commits inside it. On an unpatched binary this is what keeps a second writer from capturing a stale parent. A write that references rows the table holds needs it on every build: its position deletes name data files by path, and a compaction or another such write committing between its scan and its commit would rewrite or remove those files, after which the deletes apply to nothing and the rows come back. | -| async (both GUCs `on`) | appends (`INSERT`, `COPY`, the archiver's exports, the tiered `INSERT`'s cold sink) on the patched build (`iceberg-bakery-aware-commit-refresh-v15.patch`) | The statement uploads its parquet and queues the claim; the extension's transaction callback takes it at PRE_COMMIT, just before pg_duckdb's own PRE_COMMIT callback commits the Iceberg transaction, and releases it at COMMIT. Uploads overlap on one node and across nodes, an open transaction blocks no other writer, and only the short commit POST is serialized. A new data file conflicts with nothing committed in between. An append to a clustered table is checked at commit against the live centroid generation and fails with `serialization_failure` if a retrain replaced it (see [architecture_vectors.md](architecture_vectors.md)). | +| async (both GUCs `on`) | appends (`INSERT`, `COPY`, the archiver's exports, the tiered `INSERT`'s cold sink) on the patched build (`iceberg-bakery-aware-commit-refresh-v15.patch`) | The statement uploads its parquet and queues the claim; the extension's transaction callback takes it at PRE_COMMIT, just before pg_duckdb's own PRE_COMMIT callback commits the Iceberg transaction, and releases it at COMMIT. Uploads overlap on one node and across nodes, an open transaction blocks no other writer, and only the short commit POST is serialized. A new data file conflicts with nothing committed in between. An append to a clustered table is checked at commit against the live centroid generation and fails with `serialization_failure` if a retrain replaced it (see [Vector Storage Architecture](../architecture_vectors.md)). | For every write but a clustered table's append, the code path and the application-visible behavior are identical, so the GUCs are a performance knob; @@ -383,20 +382,20 @@ for the build. ### Transparent DDL via coldfront -A `ProcessUtility_hook` in the coldfront extension intercepts DDL that targets -a registered tiered table's hot heap or a decoupled table's wrapper view. A -hot heap is matched by resolving the DDL target relation to an OID and +A `ProcessUtility_hook` in the `coldfront` extension intercepts DDL that +targets a registered tiered table's hot heap or a decoupled table's wrapper +view. A hot heap is matched by resolving the DDL target relation to an OID and comparing it with the OID of the registry's `hot_table` - never by string, so it is schema-agnostic - and a view by its schema and name, the registry's key. A view the hook rebuilds keeps the owner and the table-level grants the view had. The hook rebuilds the view and updates the registry as the extension's owner, once the ownership check PostgreSQL makes for the statement has passed. -The role running the statement needs what PostgreSQL requires for the -statement itself, `USAGE` on the schema of the relation it names included, but -neither ownership of the view nor `CREATE` on its schema, and the hook's own -registry lookups and updates need no privilege on the registry. The Iceberg -change, which is cold I/O, runs as that role, so a column change also needs the -cold access `grant_app_access` gives, `SELECT` on the registry included. The +The role running the statement needs what PostgreSQL requires for the statement +itself, `USAGE` on the schema of the relation it names included, but neither +ownership of the view nor `CREATE` on its schema, and the hook's own registry +lookups and updates need no privilege on the registry. The Iceberg change, +which is cold I/O, runs as that role, so a column change also needs the cold +access `grant_app_access` gives, `SELECT` on the registry included. The following table summarizes the DDL it handles: | DDL | Behavior | @@ -438,7 +437,7 @@ to be usable on a peer is covered next. The tiered-specific cross-node behavior - what replicates so a tiered table is usable on every peer, and why both the registry and the watermark join the replication set - is in the -[Tiered Tables in a Spock Mesh](architecture_tiered.md#tiered-tables-in-a-spock-mesh) +[Tiered Tables in a Spock Mesh](../architecture_tiered.md#tiered-tables-in-a-spock-mesh) section of the Tiered Mode page. ### Registry Keying: By Name, Not OID @@ -463,7 +462,7 @@ node-independent, so the registry row is identical on every node and the replication set copies it by value (an OID is node-local and could not be). That is what makes cross-node tiered tables work with no per-node re-resolution - see the -[Tiered Tables in a Spock Mesh](architecture_tiered.md#tiered-tables-in-a-spock-mesh) +[Tiered Tables in a Spock Mesh](../architecture_tiered.md#tiered-tables-in-a-spock-mesh) section of the Tiered Mode page. Lower-level operations that genuinely need an OID - catalog lookups, the DDL @@ -523,14 +522,14 @@ mode (the archiver those with a `hot_period`, the partitioner those without), and exit with an error when they find none; tables enter it via `register`, or from a YAML `archiver.tables` list fed to `import`. Both expose a management CLI - `register` / `list` / `set` / `remove` / `import` / `export` - documented -in [usage.md](usage.md#managing-partitioned-tables-cli). +in [Managing Partitioned Tables (CLI)](../using_coldfront/index.md#managing-partitioned-tables-cli). ## Known Limitations These apply to both storage modes. Tiered-only limitations (cold `RETURNING`, dual-tier command tag, crash-safety of permissive writes, partition-scheme constraints, autovacuum-vs-cutover) are in the -[Tiered-Specific Limitations](architecture_tiered.md#tiered-specific-limitations) +[Tiered-Specific Limitations](../architecture_tiered.md#tiered-specific-limitations) section of the Tiered Mode page. The cross-cutting limitations are: @@ -733,7 +732,8 @@ and the directory and reporting what that freed. That sweep runs once per session and costs single-digit milliseconds per spill file, so even a directory abandoned with 600 GB in it (about 600 files at DuckDB's file size) is a couple of seconds on the one statement that finds it. See the -[Tuning Knobs](usage.md#tuning-knobs) section of the Using ColdFront guide. +[Configuring PostgreSQL](../configuration.md#configuring-postgresql) section +of the Configuring ColdFront guide. The PID is what makes that reclaim possible, and is why ColdFront does not apply pg_duckdb#887, which names the directory with a random uuid and removes @@ -801,11 +801,12 @@ with fleet benchmarking rather than as a patch ColdFront maintains. To go further with ColdFront, consult the following documents: -- The [Tiered Mode](architecture_tiered.md) deep dive describes the archive +- The [Tiered Mode](../architecture_tiered.md) deep dive describes the archive pipeline, transparent DML, and tiered tables in a Spock mesh. -- The [Decoupled Mode](architecture_decoupled.md) deep dive describes +- The [Decoupled Mode](../architecture_decoupled.md) deep dive describes iceberg-only tables, their ACID model, and the bakery protocol. -- The [Vector Storage](architecture_vectors.md) deep dive describes how +- The [Vector Storage](../architecture_vectors.md) deep dive describes how embeddings are stored, assigned to clusters, and searched. -- The [Using ColdFront](usage.md) guide covers day-to-day use of both modes. +- The [Using ColdFront](../using_coldfront/index.md) guide covers day-to-day + use of the tiered, decoupled, and standalone partitioned modes. diff --git a/docs/architecture_tiered.md b/docs/architecture_tiered.md index db8721e5..c15ff686 100644 --- a/docs/architecture_tiered.md +++ b/docs/architecture_tiered.md @@ -1,4 +1,4 @@ -# ColdFront - Tiered Operating Mode +# Tiered Operating Mode Tiered mode keeps recent rows in the PostgreSQL heap and archives older rows to Apache Iceberg, presenting both as one table through a `UNION ALL` view. An @@ -6,9 +6,10 @@ archiver moves rows hot→cold on a cron. This document covers the **tiered-specific** design. The shared mechanics - pg_duckdb Iceberg I/O, the rewrite hook, the bakery protocol, the registry, DDL -handling, infrastructure - live in [architecture.md](architecture.md); the -all-Iceberg alternative is -[architecture_decoupled.md](architecture_decoupled.md). +handling, infrastructure - live in the +[Architecture Overview](architecture_guides/index.md); the all-Iceberg +alternative is +[Decoupled (Iceberg-Only) Operating Mode](architecture_decoupled.md). ## Data Flow @@ -66,7 +67,7 @@ A single Go binary runs via cron. The binary converts an existing partitioned table into a tiered table on first run, then manages the ongoing lifecycle. The archiver is a thin SQL orchestrator - no DuckDB/Iceberg/Arrow Go libraries; all Iceberg I/O goes through `pg_duckdb` (see the -[Core Mechanics](architecture.md#core-mechanics-pg_duckdb) section of the +[Core Mechanics](architecture_guides/index.md#core-mechanics-pg_duckdb) section of the Architecture overview). ### Prerequisites @@ -76,7 +77,7 @@ The archiver requires the following before its first run: - PostgreSQL 16+ with pg_duckdb. - Lakekeeper, bootstrapped with a warehouse. - a configured storage secret (see the - [Session Setup](architecture.md#session-setup) section of the Architecture + [Session Setup](architecture_guides/index.md#session-setup) section of the Architecture overview). - static `s3:` or `azure:` credentials in the archiver config, unless the cold store uses vended credentials. @@ -119,7 +120,7 @@ batches. An `INSERT` nested in a `WITH` entry is rewritten in place, and a The bootstrap also gives the hot table its guards: a check constraint, named `coldfront_guard_`, on each column whose type admits a value the cold tier cannot hold unchanged. A `numeric` column refuses `NaN`, which pg_duckdb's -scan and DuckDB's postgres extension both read as `0` and which Iceberg's +scan and DuckDB's `postgres` extension both read as `0` and which Iceberg's decimal lacks; a `date`, `timestamp` or `timestamptz` column refuses `infinity` and `-infinity`, which Iceberg lacks and duckdb-iceberg refuses to write. An array column keeps the shape of an Iceberg list, one dimension numbered from 1, @@ -170,7 +171,7 @@ six-phase pipeline: 2. The bulk export copies the partition PG → Iceberg under a captured snapshot, using the temp table bridge (see the - [Temp Table Bridge](architecture.md#temp-table-bridge-pg-iceberg) section of + [Temp Table Bridge](architecture_guides/index.md#temp-table-bridge-pg-iceberg) section of the Architecture overview) and a single bakery-claimed Iceberg `INSERT`. Each cycle has already created the Iceberg namespace and table, if missing, before the per-partition loop. @@ -281,7 +282,7 @@ the flat path. ## Transparent `INSERT` The `post_parse_analyze_hook` (see the -[Application Interface](architecture.md#application-interface) section of the +[Application Interface](architecture_guides/index.md#application-interface) section of the Architecture overview) intercepts `INSERT` on a registered tiered view and rewrites it into a single statement that splits the input by the partition-column watermark. The hot half is always a plain set-based @@ -525,7 +526,8 @@ a partition-column SET separately by the `coldfront.allow_mixed_writes` GUC: value. Before anything is archived (no cutoff) every row is hot, so a partition-column -`UPDATE` is a plain hot `UPDATE` in either mode. +`UPDATE` is a plain hot `UPDATE` whether `coldfront.allow_mixed_writes` is on +or off. ## Write Modes: Strict vs Permissive (`allow_mixed_writes`) @@ -556,7 +558,7 @@ them. Strict mode avoids this path entirely. A cold-only `UPDATE` or `DELETE` sets nothing, so a transaction block that holds one cannot also write a PostgreSQL table unless the session sets the -parameter itself; the [Caveats](usage.md#caveats) state the rule. +parameter itself; the [Caveats](caveats.md) state the rule. ### Strict (`off`) @@ -568,7 +570,7 @@ applications to supply a tier-deterministic WHERE clause. ## Tiered Tables in a Spock Mesh The bakery protocol that serializes cold writes cluster-wide is described in -the [Concurrency](architecture.md#concurrency-and-pgedge-spock-deployments) +the [Concurrency](architecture_guides/index.md#concurrency-and-pgedge-spock-deployments) section of the Architecture overview. This section covers what is specific to a *tiered* table across a mesh. @@ -586,8 +588,9 @@ So alongside the bakery substrate (`coldfront.claims` / `coldfront.claim_acks`), **both `coldfront.tiered_views` and `coldfront.archive_watermark` are in the Spock replication set**: `coldfront.ensure_replicated()`, the one-time per-node mesh step described in -[usage.md](usage.md#what-coldfrontensure_replicated-does), adds them with the -rest. The archiver runs on one node and registers a table, its view and its +[Configuring a Distributed Deployment](distributed_setup.md#what-coldfrontensure_replicated-does), +adds them with the rest. The archiver runs on one node and registers a +table, its view and its registry row, in one transaction, so a peer applies them together and only gets these rows by replication; without `tiered_views` a peer cannot read the cold tier, and `INSERT`/`UPDATE`/`DELETE`/DDL-blocking stop recognizing the view. @@ -595,7 +598,7 @@ tier, and `INSERT`/`UPDATE`/`DELETE`/DDL-blocking stop recognizing the view. Both tables are **name-keyed** - `tiered_views` by `(schema_name, relname)`, `archive_watermark` by `table_name` - so each row replicates verbatim and correct on every node, with no OID divergence to reason about. See the -[Registry Keying](architecture.md#registry-keying-by-name-not-oid) section of +[Registry Keying](architecture_guides/index.md#registry-keying-by-name-not-oid) section of the Architecture overview. ## Partition Scheme Compatibility @@ -712,7 +715,7 @@ not make deletes less expensive. These are specific to the dual-tier model. Cross-cutting limitations (the planner-level takeover, jsonb-as-json, single-node execution, S3 compatibility, one-time secret setup) are in the -[Known Limitations](architecture.md#known-limitations) section of the +[Known Limitations](architecture_guides/index.md#known-limitations) section of the Architecture overview. The dual-tier model has the following limitations: @@ -765,9 +768,9 @@ The dual-tier model has the following limitations: To go further with ColdFront, consult the following documents: -- The [Architecture](architecture.md) overview describes the mechanics both +- The [Architecture](architecture_guides/index.md) overview describes the mechanics both modes share. - The [Decoupled Mode](architecture_decoupled.md) deep dive describes the bakery protocol that serializes cold writes. -- The [Using ColdFront](usage.md) guide covers tiering a table and managing it +- The [Using ColdFront](using_coldfront/index.md) guide covers tiering a table and managing it with the partition CLI. diff --git a/docs/architecture_vectors.md b/docs/architecture_vectors.md index 36678457..ce581556 100644 --- a/docs/architecture_vectors.md +++ b/docs/architecture_vectors.md @@ -99,7 +99,8 @@ an assignment already stored keeps meaning what it meant, and the primary key rejects a repeated `centroid_id` within one generation. `coldfront.tiered_views.vec_columns` records which columns are clustered, in -order. The list cannot be derived afterwards in either mode: the view exposes +order. The list cannot be derived afterwards for a tiered or a decoupled +table: the view exposes `real[]` rather than the pgvector type, and a decoupled table names its types without holding them. @@ -566,5 +567,5 @@ To go further with ColdFront, consult the following documents: searching vector columns. - The [Compaction](compaction.md) guide covers the maintenance that restores cluster order. -- The [Architecture](architecture.md) overview describes the mechanics both +- The [Architecture](architecture_guides/index.md) overview describes the mechanics both modes share. diff --git a/docs/caveats.md b/docs/caveats.md new file mode 100644 index 00000000..9d12dbb8 --- /dev/null +++ b/docs/caveats.md @@ -0,0 +1,154 @@ +# Understanding ColdFront Caveats + +A tiered or decoupled table behaves like an ordinary PostgreSQL table, but +the hot/cold split and the cold tier's DuckDB execution show through in a +few places: SQL statements that need special handling (`jsonb` operators, +`WITH`, `MERGE`, `TRUNCATE`, `COPY`), transaction and isolation semantics +that differ from native PostgreSQL, and a few naming and infrastructure +requirements. Keep the following in mind when running tiered or decoupled +tables: + +- `jsonb` reads come back as `json`. The `->`/`->>` operators work, and + ColdFront translates the `::jsonb` cast, `jsonb_array_length`, + `jsonb_build_object` / `jsonb_agg` and `date_bin`; other jsonb operators and + functions are unsupported on cold or cross-tier reads. A hot-only read of the + view runs in PostgreSQL with full jsonb (see + [Supported Column Types](supported_types.md)). +- Cross-tier isolation differs from PG-native isolation. The cold tier keeps + one Iceberg snapshot for the whole transaction, even at READ COMMITTED, so a + commit by another session after the transaction's first cold read stays + invisible until the transaction ends. The hot tier follows PostgreSQL's own + isolation level, so at READ COMMITTED a later statement can see new hot rows + but not new cold ones. Within one transaction, a read of a tiered or + decoupled table sees the transaction's own writes on both tiers. The held + snapshot is also why the compactor's `--expire-older-than` must stay longer + than the longest transaction that reads the table (see + [Compacting the Cold Tier](compaction.md)). +- One transaction block cannot hold both a PostgreSQL write and a cold-tier + write unless `duckdb.unsafe_allow_mixed_transactions` is on. pg_duckdb + refuses the pair ("Writing to DuckDB and Postgres tables in the same + transaction block is not supported"), at the PostgreSQL write when it comes + second and at `COMMIT` when it came first. A cold-tier write here is a + tiered `UPDATE` or `DELETE` whose `WHERE` bounds the cold tier, or any write + to a decoupled table. The statements that write both tiers themselves (a + dual-tier `UPDATE` or `DELETE`, a tiered `INSERT` with cold rows, a + cross-tier move) set the parameter `LOCAL` for the rest of their + transaction, so the block they ran in accepts later writes of either kind. + You can set it yourself, `SET LOCAL duckdb.unsafe_allow_mixed_transactions = + on`, at your own risk: pg_duckdb commits the Iceberg snapshot at + `PRE_COMMIT`, so a backend crash between that and the PostgreSQL commit + record keeps the cold write and loses the PostgreSQL writes. +- In decoupled mode, pg_duckdb commits the Iceberg snapshot at PRE_COMMIT, so a + backend crash after that but before the PG commit record leaves the Iceberg + write committed and the PG side lost. A crash after the Parquet upload but + before the commit POST leaves unreferenced objects, which the compactor's + orphan pass reclaims. +- In decoupled mode, concurrent writes from multiple PG nodes are serialized + PG-side by the bakery protocol: every iceberg-only `INSERT` goes through + `coldfront._exec_iceberg_with_claim`, which holds a globally-ordered + Snowflake ticket and waits for its turn before committing to Lakekeeper. + There are no 409 conflicts and no app-level retries. The protocol is + Lamport-1978 mutex with the Ricart-Agrawala (1981) deferred-reply + optimization; claims and acks replicate as Spock rows and it stays safe under + Spock's asymmetric apply (modeled in + [docs/formal/Bakery.tla](https://github.com/pgEdge/ColdFront/blob/main/docs/formal/Bakery.tla)). + The bakery requires the `snowflake` extension, the `coldfront.loopback_dsn` + GUC (the node's loopback, a unix-socket DSN), and a one-time + `SELECT coldfront.ensure_replicated()` call on every node after Spock mesh + setup; see + [The Bakery Protocol](architecture_decoupled.md#concurrency-horizontal-scaling-the-bakery-protocol). + Sync-rep is **not** required. The throughput ceiling is Lakekeeper's commit + rate, not the writer count. +- For direct table access, `_events` is the hot heap (tiered mode only). + `ice.public.` is the Iceberg table - only addressable via + `iceberg_scan(...)` or `duckdb.raw_query('… ice.… …')`, never via + PG-native 3-part names. +- A tiered `INSERT` streams its cold rows: the cold projection runs in + PostgreSQL, in a second session the DuckDB `postgres` extension opens from + `coldfront.local_pg_dsn`, and its rows go straight into the Iceberg writer + in one pass. An omitted IDENTITY column takes `nextval()` on the hot table's + sequence, so cold ids share it with the hot side, and an omitted column with + a DEFAULT takes it. The hot rows are one set-based `INSERT`. The second + session reads the source at the statement's own snapshot, so both tiers see + one state of the tables whatever another session commits meanwhile, and a + REPEATABLE READ transaction reads its snapshot on both tiers. A source the + stream cannot read the same way twice (a volatile or stable function such as + `now()`, a temporary table, a `WITH` entry, a table the transaction has + written, a SERIALIZABLE transaction, or a second table-reading `INSERT` in + one READ COMMITTED transaction) goes through + `coldfront._cold_sink` instead, which renders each row in plpgsql and writes + Iceberg in batches of `coldfront.cold_write_batch_size` rows, as does a + server without `coldfront.local_pg_dsn`. The statement's `cold_rows` column + is `NULL` when the cold rows streamed. A value that reaches the second + session or DuckDB as text is rendered in a form every reader takes as the + same value, whatever the caller's `DateStyle`, `IntervalStyle` or + `extra_float_digits`. +- `COPY FROM` reads the rows with PostgreSQL's `COPY` reader and writes + them through that same `INSERT` path, `coldfront.cold_write_batch_size` rows + per `INSERT`. The format options are the reader's, and a supplied value for a + `GENERATED ALWAYS` identity column is kept, as `COPY` into a table keeps it. + `COPY ... WHERE` and the `FREEZE`, `ON_ERROR`, `REJECT_LIMIT` and `DEFAULT` + options are refused. +- An `INSERT`, `UPDATE` or `DELETE` nested in a `WITH` entry goes through the + same rewrite as a top-level one. With a watermark an `INSERT`'s row may go + cold, so `RETURNING` on it is refused, as on a top-level `INSERT`; a hot + `UPDATE` or `DELETE` keeps `RETURNING`, a cold or dual-tier one cannot + return rows or read another `WITH` entry (its DuckDB half does not see + them), and a cross-tier move cannot be a `WITH` entry. A statement may write + a tiered view once, and the nested write may not have a `WITH` clause of + its own. On a decoupled view the source runs in DuckDB, so a `WITH` entry + that modifies data is refused (DuckDB 1.5 has no data-modifying `WITH`), and + a nested `INSERT` may not read another `WITH` entry. A statement that holds + a nested write and also reads a tiered view runs in DuckDB as a whole, and + pg_duckdb refuses it ("DuckDB does not support modifying CTEs"); read the + hot table, or split the statement. +- `MERGE INTO `, on PostgreSQL 17 and later (16 allows `MERGE` on tables + alone), runs on the tier its `ON` condition bounds the partition column to + (`AND t.ts >= ''` for the hot tier, `AND t.ts < ''` for the + cold tier): a hot `MERGE` runs in PostgreSQL against the hot table + and keeps `RETURNING`, a cold one in DuckDB against the Iceberg table. Each + tier sees its own rows alone, so a `MERGE` that bounds neither tier is + refused, and an `INSERT` action's row must belong to the statement's tier: a + hot `MERGE` refuses a row below the cutoff, a cold one a row at or after it; + insert such rows with `INSERT`, which splits them. A cold `INSERT` action + must give the identity column a value (`OVERRIDING SYSTEM VALUE` for a + `GENERATED ALWAYS` one), a cold `MERGE` cannot return rows and runs one + `UPDATE` or `DELETE` action per statement (duckdb-iceberg's limit), and a + `WHEN NOT MATCHED BY SOURCE` action must bound the same tier in its own + condition. A `MERGE` that sets the partition column is refused (run the + change as an `UPDATE`), as is one on a table with a clustered vector column + and one whose source reads the view. A `MERGE` nested in a `WITH` entry + takes the same path as a nested `UPDATE` or `DELETE`. On a decoupled view + every `MERGE` runs in DuckDB. +- `TRUNCATE` on a registered relation, or on the hot table behind a tiered one, + fails with an error, because the cold rows in Iceberg would stay visible + through the view. +- A table or column with a cold tier cannot have a name that PostgreSQL leaves + unquoted but DuckDB parses as a keyword. With DuckDB 1.5.4 those names are + `anti`, `asof`, `at`, `by`, `describe`, `glob`, `lambda`, `pivot`, + `pivot_longer`, `pivot_wider`, `positional`, `qualify`, `semi`, `show`, + `summarize`, `unpack` and `unpivot`. pg_duckdb passes a table or column name + to DuckDB quoted the way PostgreSQL quotes it, so such a name reaches DuckDB + bare even when the query quotes it. Every query pg_duckdb runs against the + table or column then fails with a DuckDB `Parser Error` + ([duckdb/pg_duckdb#1019](https://github.com/duckdb/pg_duckdb/issues/1019)). + ColdFront refuses such a column name wherever a column gets a cold tier: + tiered registration, an archive pass that tiers or expires data, + `coldfront.create_iceberg_table()`, `coldfront.adopt_iceberg_table()`, and an + `ADD COLUMN` or `RENAME COLUMN` on a table that already has a cold tier. + ColdFront refuses such a table name at tiered registration, in + `coldfront.create_iceberg_table()` and `coldfront.adopt_iceberg_table()`, and + in a `RENAME` of the view or the hot table of a table that already has a cold + tier. Partition-only registration, without a hot period, accepts a table with + such a name or column. ColdFront does not refuse a DuckDB keyword that the + DuckDB parser accepts as a name, such as `columns`. +- Iceberg on Azure ADLS Gen2 requires Blob soft-delete, container soft-delete, + and change feed (blob events) to be OFF on the storage account. Lakekeeper + warehouse creation otherwise fails with HTTP 409 ("This endpoint does not + support BlobStorageEvents or SoftDelete"). Disable those features on the + storage account before using it as a cold tier. +- A cold-tier write that reads an ordinary PostgreSQL table, such as an + `UPDATE ... FROM` or a `MERGE` from one, stores a `numeric` `NaN` from that + table as `0` and stores an array renumbered to start at 1, with no error. + [Supported Column Types](supported_types.md) lists the writes that do this + and shows how to check a source table first. diff --git a/docs/changelog.md b/docs/changelog.md index 3a1b5435..8e6ab8c8 100644 --- a/docs/changelog.md +++ b/docs/changelog.md @@ -370,7 +370,7 @@ this project adheres to data file it can apply to is left out of the rewrite. - A `numeric` `NaN` in a tiered table read back as `0` through the view once the table had a cold tier, and was archived as `0`: pg_duckdb and DuckDB's - postgres extension read `NaN` as zero, and Iceberg's decimal has no `NaN`. + `postgres` extension read `NaN` as zero, and Iceberg's decimal has no `NaN`. The hot table now refuses `NaN` when it is written, and an archive pass refuses a table that already holds one, with an error that names the column's check constraint. @@ -436,7 +436,7 @@ use. older data to Apache Iceberg on a watermark, presented to the application as a single unified view. - Decoupled mode stores a table entirely in Iceberg from the first row, with - PostgreSQL holding a thin wrapper view and the coldfront extension handling + PostgreSQL holding a thin wrapper view and the `coldfront` extension handling every data-modifying statement on that view. - Horizontal scale-out for decoupled mode across multiple PostgreSQL nodes sharing one Lakekeeper catalog and one object store, serialised by the bakery diff --git a/docs/compaction.md b/docs/compaction.md index c8840938..cc8daa3a 100644 --- a/docs/compaction.md +++ b/docs/compaction.md @@ -1,4 +1,4 @@ -# COMPACTOR - Cold-Tier Table Maintenance +# Compacting the Cold Tier `cmd/compactor` keeps a cold-tier Iceberg table healthy: it compacts each partition's small Parquet files into fewer large ones, expires old snapshots, @@ -15,7 +15,12 @@ claim. [apache/iceberg-go]: https://github.com/apache/iceberg-go -## Usage +Build the compactor binary with `make compactor`, which vets, lints +(golangci-lint required), tests, and builds `./bin/compactor`: + +```bash +make compactor +``` Run the compactor against the database server, naming the table to maintain: @@ -118,21 +123,3 @@ releases it when its PostgreSQL transaction commits, so its commit cannot interleave with a cold write's commit to the same table. Snapshot maintenance is the engine's job, not the catalog's: Lakekeeper does no Iceberg snapshot or orphan maintenance. - -## Requirements - -The compactor (`cmd/compactor`, apache/iceberg-go) reads the cold-tier Iceberg -tables directly. Build the binary with `make compactor`, which vets, lints -(golangci-lint required), tests, and builds `./bin/compactor`. - -## Next Steps - -To go further with ColdFront, consult the following documents: - -- The [Using ColdFront](usage.md) guide covers the server configuration the - compactor shares with the archiver, and the deployment YAML `import` writes - into it. -- The [Vector Storage](architecture_vectors.md) deep dive describes the sort - key the compactor applies to clustered tables. -- The [Architecture](architecture.md) overview describes the bakery claim each - compaction step takes. diff --git a/docs/configuration.md b/docs/configuration.md new file mode 100644 index 00000000..be1e0e5a --- /dev/null +++ b/docs/configuration.md @@ -0,0 +1,118 @@ +# Configuring ColdFront + +After installing ColdFront packages or building from source, several +components still need configuring before the cold tier comes online: + +- PostgreSQL and ColdFront's configuration is detailed in this + guide, regardless of which installation path you followed. +- installing, configuring, and bootstrapping Lakekeeper, and the + one-time per-database setup that ties everything together, + documented in the [Configuring Lakekeeper](configure_lakekeeper.md) + guide. +- your object store, documented in + [Configuring your Object Store](object_store.md). + +## Configuring PostgreSQL + +A package installation does not configure PostgreSQL. Add the following +settings to `postgresql.conf`, then restart PostgreSQL: + +```ini +shared_preload_libraries = 'pg_duckdb,coldfront' +duckdb.extension_directory = '/usr/lib/pgedge/coldfront/duckdb-extensions' +duckdb.allow_unsigned_extensions = true +duckdb.autoinstall_known_extensions = false +duckdb.postgres_role = 'coldfront_duckdb' +coldfront.iceberg_async_parquet = on +coldfront.iceberg_bakery_patch = on +coldfront.warehouse = '' +coldfront.lakekeeper_endpoint = 'http://:8181/catalog' +coldfront.local_pg_dsn = 'host=/var/run/postgresql dbname= user=' +``` + +Together, `duckdb.extension_directory`, `duckdb.allow_unsigned_extensions`, and +`duckdb.autoinstall_known_extensions` point pg_duckdb at the directory +where `pgedge-coldfront-duckdb-extensions` installs ColdFront's patched +DuckDB extensions, rather than letting it fetch the unpatched upstream +build - the same substitution that reintroduces HTTP 409 failures under +concurrent cold writes. The patched extensions are unsigned, so +`duckdb.allow_unsigned_extensions` must stay on; and because a missing +extension file would otherwise trigger that same fallback, +`duckdb.autoinstall_known_extensions` stays off. + +`duckdb.postgres_role` names the role whose members may run DuckDB. Without +it, only superusers can, and `coldfront.grant_app_access()` fails. Create the +role before you grant access to an application role: + +```sql +CREATE ROLE coldfront_duckdb NOLOGIN; +``` + +The two `coldfront.iceberg_*` settings make a node upload the Parquet files of +an `INSERT` or `COPY` outside its write lock (the bakery claim on a Spock mesh, +an advisory lock on a single node) and serialize only the catalog commit. The +packaged duckdb-iceberg includes the patch that this ordering requires, and the +[Configuring a Distributed Deployment](distributed_setup.md) guide describes +the mesh settings. + +ColdFront also exposes the following settings, which adjust write +behavior and execution; tune them as needed: + +| Setting | Description | +|---|---| +| `coldfront.allow_mixed_writes` | Controls tiered-mode `UPDATE`/`DELETE` whose WHERE cannot be proven to target one tier. `on` emits a dual-tier CTE; `off` rejects with an error and a hint. Not relevant in decoupled mode (every write is single-tier by definition). The default is `on`. | +| `coldfront.cold_write_batch_size` | Sets how many cold rows a tiered `INSERT` gathers (see [Caveats](caveats.md)) before writing them to Iceberg as one `INSERT`. A larger value writes fewer, larger Parquet files; the remainder always flushes, so a small write stays one file. The default is `10000`, with a minimum of `1`. | +| `coldfront.vector_probe` | Sets whether a recognized similarity search reads only the clusters nearest its query vector. `off` gives an exact scan of the whole corpus. Affects only a table with a trained vector column (see [Working with Embeddings](usage_vectors.md)). The default is `on`. | +| `coldfront.vector_nprobe` | Sets how many clusters such a search reads, overriding the column's own `nprobe`; at or above the column's `nlist` the search is exhaustive. The default is `0`, which uses the configured value. | +| `duckdb.force_execution` | Benchmark before enabling: on a mixed workload it helps `count(distinct)` and similar but regresses index lookups, top-K with PK ordering, and JSON access. The default is `off`. | +| `duckdb.temporary_directory` | Sets where DuckDB spills. Each backend gets its own subdirectory there, named after its process id, so concurrent spills cannot collide; one left by a departed backend is reclaimed. See [DuckDB: Spill Files Are Not Namespaced per Instance](architecture_guides/index.md#duckdb-spill-files-are-not-namespaced-per-instance). | +| `duckdb.max_temp_directory_size` | A cap per connection, not a cluster total. For a total budget, divide it by the concurrent sessions, or give the temp path its own filesystem or quota. The default is 90% of free space per session. | + +## Writing ColdFront's Configuration + +Server-level settings live in `postgresql.conf`, each managed table's +configuration is a row in `coldfront.partition_config`, and the +cold-store credential lives in `coldfront.storage_secret`. + +Write the configuration with `coldfront.set_storage_secret()` and the +`register` command. `set_storage_secret()` writes the following to +`coldfront.storage_secret`; the +[Configuring your Object Store](object_store.md) guide shares the exact +values for each supported backend: + +- `` - your object store's access key. +- `` - the matching secret. +- `` - your object store's endpoint. + +```sql +SELECT coldfront.set_storage_secret('', '', ''); +``` + +`register` writes the following to `coldfront.partition_config`: + +- `
` - the table name. +- `` - the partition cadence, e.g. `monthly`. +- `` - how long a partition stays hot before the archiver + moves it to the cold tier. + +```bash +./bin/archiver register --table
\ + --period --hot-period "" +``` + +Or do both in one step by importing a deployment YAML: + +```bash +archiver import --config deploy.yaml +``` + +The `pgedge-coldfront` package installs an example deployment YAML at +`/etc/pgedge/coldfront/config.yaml`. The file is only an example; edit it and pass it to `import` - no +ColdFront tool reads it unless `--config` names it. +After an import, the server holds the configuration. A later run that is given +a YAML checks the file against the server and refuses to run if any value +differs. The only value such a run takes from the file is `postgres.dsn`, which +connects when `--dsn` is unset. The +[Managing Partitioned Tables (CLI)](using_coldfront/index.md#managing-partitioned-tables-cli) +section of the Using ColdFront in Standalone Partitioned Mode guide describes +`register`, `import`, and `export`. diff --git a/docs/configure_lakekeeper.md b/docs/configure_lakekeeper.md new file mode 100644 index 00000000..2e566689 --- /dev/null +++ b/docs/configure_lakekeeper.md @@ -0,0 +1,253 @@ +# Configuring Lakekeeper + +ColdFront requires three services configured once per deployment: +PostgreSQL with the pg_duckdb and `coldfront` extensions, Lakekeeper, and +any object store (SeaweedFS, MinIO, AWS S3, GCS, or Azure ADLS Gen2). +This guide brings up and bootstraps Lakekeeper, then installs and +configures the PostgreSQL side; do this before creating any table. + +Lakekeeper is not part of ColdFront's own build; it is a separate +upstream project ([github.com/lakekeeper/lakekeeper](https://github.com/lakekeeper/lakekeeper)). +Choose how you are running it below. + +## With Docker + +Bring up the end-user stack first (the example uses SeaweedFS, gated +behind the `local-store` compose profile; host ports are published so +the `localhost` commands below work directly): + +```bash +docker compose --profile local-store up -d --build +``` + +Lakekeeper is now reachable at `localhost:8181` with no further setup. +Continue with [Bootstrapping Lakekeeper](#bootstrapping-lakekeeper), and use +`http://seaweedfs:8333` as the store endpoint. + +## Installing Lakekeeper + +On bare metal, how you get the `lakekeeper` binary in place depends on +your installation path; continue with Preparing and Starting Lakekeeper +below either way. + +### Performing a Package Installation + +The `pgedge-lakekeeper` package installs the `lakekeeper` binary and a +`lakekeeper` systemd service, but it does not enable or start the +service. + +### Installing the Upstream Binary + +Download the matching upstream prebuilt release directly - there is no +source build; the release ships a single stripped binary (glibc 2.34 or +later required): + +```bash +ARCH=$(uname -m) # x86_64 or aarch64 +LAKEKEEPER_VERSION=0.13.1 +curl -LO "https://github.com/lakekeeper/lakekeeper/releases/download/v${LAKEKEEPER_VERSION}/lakekeeper-${ARCH}-unknown-linux-gnu.tar.gz" +tar xzf "lakekeeper-${ARCH}-unknown-linux-gnu.tar.gz" +sudo install -m 0755 lakekeeper /usr/bin/lakekeeper +``` + +Create the system user the steps below run as: + +```bash +sudo groupadd -r lakekeeper +sudo useradd -r -g lakekeeper -d /var/lib/lakekeeper -s /sbin/nologin \ + -c "Lakekeeper Iceberg REST Catalog" lakekeeper +``` + +This repository's own packaging templates make convenient starting +points for the environment file and systemd unit, since you already +have the repo cloned to build the `coldfront` extension: +`packaging/lakekeeper/common/lakekeeper.env` and +`packaging/lakekeeper/common/lakekeeper.service`. Copy them into place, with +the environment file readable only by root and the `lakekeeper` group, as the +package installs it: + +```bash +sudo install -D -m 0640 -g lakekeeper packaging/lakekeeper/common/lakekeeper.env \ + /etc/lakekeeper/lakekeeper.env +sudo install -m 0644 packaging/lakekeeper/common/lakekeeper.service \ + /etc/systemd/system/lakekeeper.service +sudo systemctl daemon-reload +``` + +## Preparing and Starting Lakekeeper + +Lakekeeper stores its catalog in a PostgreSQL 15 or later database and +reads its settings from `/etc/lakekeeper/lakekeeper.env`. The +following steps prepare and start the service, whichever way you +installed it: + +1. Create a role and a database for the catalog: + + ```sql + CREATE ROLE lakekeeper LOGIN PASSWORD 'change-me'; + CREATE DATABASE lakekeeper OWNER lakekeeper; + ``` + + The migration in step 3 creates the `uuid-ossp`, `pgcrypto`, `pg_trgm`, + `btree_gin`, and `btree_gist` extensions, so either the role must be + allowed to run `CREATE EXTENSION` or a superuser must create them first. + +2. In `/etc/lakekeeper/lakekeeper.env`, set + `LAKEKEEPER__PG_DATABASE_URL_WRITE` to the database's connection string and + `LAKEKEEPER__PG_ENCRYPTION_KEY` to a random secret, such as the output of + `openssl rand -base64 32`. Lakekeeper encrypts stored credentials with the + key, so keep the key stable and backed up. Every node that shares the + catalog needs the same key. + +3. Run the one-time database migration as the `lakekeeper` user. The + command reads the environment file as that user, because the file is not + readable by other accounts: + + ```bash + sudo -u lakekeeper sh -c \ + 'set -a; . /etc/lakekeeper/lakekeeper.env; set +a; exec /usr/bin/lakekeeper migrate' + ``` + +4. Secure the listener before you start the service. By default, Lakekeeper + listens on port 8181 on every address and, with no authorization backend, + accepts every request. Do not start it that way: + + - If every client runs on the same host, set the following in + `/etc/lakekeeper/lakekeeper.env`: + + ```bash + LAKEKEEPER__BIND_IP=127.0.0.1 + ``` + + - If clients connect from other hosts, configure + [authentication](https://docs.lakekeeper.io/docs/latest/authentication/) + and an enforcing + [authorization](https://docs.lakekeeper.io/docs/latest/authorization/) + backend, and limit access to port 8181 with a firewall. + +5. Enable and start the service: + + ```bash + sudo systemctl enable --now lakekeeper + ``` + +## Bootstrapping Lakekeeper + +Bootstrap Lakekeeper, create the warehouse, and pre-create the Iceberg +namespace. In the warehouse command, replace `` with the +address Lakekeeper uses to reach your store: `http://seaweedfs:8333` for the +SeaweedFS store in the Docker stack, or `http://localhost:8333` for SeaweedFS +running on the same host. For another store, change the `storage-profile` and +`storage-credential` as shown in +[Configuring your Object Store](object_store.md#configuring-lakekeeper). + +```bash +# 1. Bootstrap Lakekeeper +curl -X POST http://localhost:8181/management/v1/bootstrap \ + -H "Content-Type: application/json" -d '{"accept-terms-of-use":true}' + +# 2. Create the warehouse and save its ID (replace as +# described above) +WID=$(curl -s -X POST http://localhost:8181/management/v1/warehouse \ + -H "Content-Type: application/json" -d '{ + "warehouse-name": "wh", + "storage-profile": { + "type": "s3", "bucket": "iceberg", "region": "us-east-1", + "endpoint": "", "path-style-access": true, + "flavor": "s3-compat", "sts-enabled": false, + "remote-signing-enabled": false + }, + "storage-credential": { + "type": "s3", "credential-type": "access-key", + "aws-access-key-id": "admin", "aws-secret-access-key": "adminsecret" + } + }' \ + | grep -oE '"warehouse-id":"[^"]+"' | cut -d'"' -f4) + +# 3. Create the Iceberg namespace in the new warehouse. +# REQUIRED for decoupled (iceberg-only) mode: the Iceberg CREATE SCHEMA is +# deferred to transaction COMMIT but the CREATE TABLE POST is sent +# immediately, so coldfront.create_iceberg_table - which runs both in one +# transaction - would fail with HTTP 404 against a cold warehouse. +# Pre-creating the namespace here (its own committed REST call) makes the +# function's in-txn CREATE SCHEMA IF NOT EXISTS a no-op so the table create +# succeeds. The archiver (tiered mode) creates the namespace itself and does +# not need this. +if [ -z "$WID" ]; then + echo "No warehouse ID - check that the warehouse create succeeded." +else + curl -X POST "http://localhost:8181/catalog/v1/$WID/namespaces" \ + -H "Content-Type: application/json" -d '{"namespace":["public"]}' +fi +``` + +## Configuring PostgreSQL for Lakekeeper + +When Lakekeeper is bootstrapped, install the extensions and set the +cold-tier credentials, once per database. Replace `` with the +store's address as PostgreSQL reaches it, without the `http://` prefix: +`seaweedfs:8333` in the Docker stack, or `localhost:8333` for SeaweedFS on the +same host. If you use `coldfront.local_pg_dsn`, described below, set it before +you run these statements: `set_storage_secret()` installs the DuckDB +`postgres` extension that the setting needs only when the setting is present. + +```sql +CREATE EXTENSION IF NOT EXISTS pg_duckdb; +CREATE EXTENSION IF NOT EXISTS coldfront; + +SELECT coldfront.set_storage_secret('admin', 'adminsecret', ''); +``` + +The secret is stored in the `coldfront.storage_secret` table (excluded from +`pg_dump`; in a Spock mesh it replicates by value once added to the default +repset - one-time mesh setup step 4) and materialized as a DuckDB PERSISTENT +SECRET that loads at instance init. There is **no per-session setup**: the +Iceberg catalog `ice` attaches **lazily** by the coldfront C hook on the first +query that touches a tiered/decoupled view (read or write). + +Every `set_storage_secret()` argument after the secret is optional. +`p_endpoint` defaults to `NULL`, `p_region` to `'us-east-1'`, `p_url_style` to +`'path'` (the other value is `'vhost'`), and `p_use_ssl` to `false`. +`p_url_style` and `p_use_ssl` apply only when an endpoint is given. A TLS +endpoint such as GCS's needs `p_use_ssl => true`, and AWS S3 omits the endpoint +and sets `p_region` to the bucket's Region. + +The image writes the server settings ColdFront needs into `postgresql.conf` +when it initializes a new data directory. A server installed from packages or +built another way sets them itself, as the +[package](configuration.md#configuring-postgresql) and +[bare-metal](installation.md#building-coldfront-on-bare-metal) sections of +installation.md show: + +- `shared_preload_libraries = 'pg_duckdb,coldfront'` loads both extensions at + server start. coldfront refuses to load any other way: `CREATE EXTENSION + coldfront` fails on a server that does not preload it, with an error that + names the setting. +- `coldfront.warehouse` and `coldfront.lakekeeper_endpoint` name the Lakekeeper + warehouse and its catalog endpoint, which the Iceberg catalog `ice` attaches + to. Only a superuser can set them, and while either is empty the catalog does + not attach. +- `coldfront.local_pg_dsn` is the connection string DuckDB uses to read + PostgreSQL tables from this server, which a decoupled table's + `INSERT … SELECT` from a PostgreSQL table needs, as does a cold write to a + table with a clustered vector column. A tiered `INSERT` streams its cold + rows through it, and renders them row by row when it is not set. Only a + superuser can set or read it. + +An application that connects as a non-superuser needs its role granted access +with `coldfront.grant_app_access()`, which takes an existing role. Replace +`` with the name of the role your application connects as: + +```sql +SELECT coldfront.grant_app_access(''); +``` + +`coldfront.grant_app_access()` derives the grants from the registry when it +runs, so run it again after you create or adopt a table, or tier one for the +first time. The call fails while `duckdb.postgres_role` is unset, because only +superusers can then run DuckDB. +[Connecting to ColdFront](connecting.md#least-privilege-application-roles) +lists what it grants. + +For a real cloud-S3 setup, see +[Configuring your Object Store](object_store.md). diff --git a/docs/connecting.md b/docs/connecting.md new file mode 100644 index 00000000..44a31ee6 --- /dev/null +++ b/docs/connecting.md @@ -0,0 +1,46 @@ +# Connecting to ColdFront + +An application connects to ColdFront as an ordinary PostgreSQL client; the +only ColdFront-specific step is onboarding its role using the principle of +least privilege. + +## Least-Privilege Application Roles + +Application roles need no superuser and no server-file access, yet they read +and write the cold tier through the same transparent view. Onboarding an +application role requires a single call. Replace `` with the name of +an existing role that your application connects as: + +```sql +SELECT coldfront.grant_app_access(''); +``` + +grant_app_access grants only the minimum the cold path needs: + +- membership in duckdb.postgres_role. +- SET on the duckdb.unsafe_allow_execution_inside_functions parameter. +- schema USAGE. +- `SELECT` on the registry and the watermark table. +- DML on the dual-write anchor table. +- DML on every registered view and the hot table behind it. +- USAGE and `SELECT` on the hot table's sequences. + +The tables and views above are derived from the registry, not hardcoded; +the call also grants EXECUTE on a fixed allow-list of runtime cold-path +functions. The call is idempotent +and is not executable by PUBLIC, so an application role can never self-grant. +The role is never granted pg_read_server_files or pg_write_server_files, so it +has no host-file access. `CREATE ROLE` and `GRANT` both replicate over Spock, +so you onboard a role once on any node and it propagates across the mesh. + +The Docker image sets `duckdb.postgres_role` to `coldfront_duckdb` and creates +that role when it initializes a new data directory. To name a different role, +set the container's `COLDFRONT_DUCKDB_ROLE` environment variable before that +first start. An empty value keeps pg_duckdb's default, under which only +superusers run DuckDB, so grant_app_access then fails with an error. + +For how the non-superuser path works - the `SECURITY DEFINER` attach helpers, +the `PGC_SUSET` / `GUC_SUPERUSER_ONLY` config hardening, the +`duckdb.postgres_role` default that the image sets up, and how least privilege +holds across a Spock mesh - see +[Architecture: non-superuser app roles](architecture_guides/index.md#non-superuser-app-roles-least-privilege). diff --git a/docs/distributed_setup.md b/docs/distributed_setup.md new file mode 100644 index 00000000..b6725688 --- /dev/null +++ b/docs/distributed_setup.md @@ -0,0 +1,201 @@ +# Configuring a Distributed Deployment + +For multi-writer iceberg workloads, run ColdFront on PostgreSQL nodes in a +Spock mesh against the same Lakekeeper + S3. The bakery serializes commits +PostgreSQL-side so writers never collide at the catalog. + +## Per-Node `postgresql.conf` + +The configuration below applies to each node in the mesh: + +```ini +wal_level = logical +shared_preload_libraries = 'snowflake,spock,pg_duckdb,coldfront' + +# The bakery rules a peer dead when its walsender's last reply is older than +# coldfront.peer_alive_window_ms (default 10 s), and an idle peer replies only +# to the walsender's keepalive, every wal_sender_timeout/2. The claim refuses +# to run unless wal_sender_timeout is positive and below twice the window. +wal_sender_timeout = 15s + +# Spock prereq + DDL replication. The ddl_sql repset (and with it the +# one-node provisioning promise for wrapper views) only works with DDL +# replication on; allow_ddl_from_functions covers DDL issued inside +# coldfront's plpgsql helpers. +track_commit_timestamp = on +spock.enable_ddl_replication = on +spock.allow_ddl_from_functions = on +spock.include_ddl_repset = on + +# Only on a server that has this setting (PostgreSQL 16.15, 17.11 and 18.6 in +# pgEdge's builds): +# it lists the libraries allowed as logical decoding output plugins, and its +# default leaves out Spock's, so no subscription can create its slot. A server +# without the setting refuses to start with this line in place. +output_plugin_libraries = 'pgoutput, test_decoding, spock_output' + +# Per-node - any distinct integer 1..1023 (must be unique per node; the value +# is otherwise arbitrary - the bakery matches acks by spock node name, not by id). +snowflake.node = 1 + +# DSN of the node's loopback, which runs the bakery's autonomous claim +# INSERT/DELETE. It must name a unix socket: every onboarded role can read +# this setting, so it must never need a password. The claim session only +# touches the coldfront.claims/claim_acks heap tables, so it never attaches +# the Iceberg catalog (the lazy catalog-attach hook fires only on a tiered +# view). application_name=coldfront_loopback marks the session as bakery +# traffic. Only a superuser can set it. +coldfront.loopback_dsn = 'host=/tmp dbname=coldfront user=coldfront application_name=coldfront_loopback' + +coldfront.warehouse = 'wh' +coldfront.lakekeeper_endpoint = 'http://lakekeeper:8181/catalog' + +# Upload the Parquet outside the bakery claim and serialize only the catalog +# commit. Set both, and only where duckdb-iceberg includes ColdFront's patch (the +# ColdFront image does). With either off, the upload is serialized too. +coldfront.iceberg_async_parquet = on +coldfront.iceberg_bakery_patch = on +``` + +The bakery has no peer-ack timeout. The +[Ricart-Agrawala (R-A)](formal/README.md) deferred-reply protocol's only +failure mode is a dead peer (would wait forever), and a liveness check +inside the wait loop closes it: +the bakery treats a peer whose `pg_stat_replication.reply_time` is older than +`coldfront.peer_alive_window_ms` as already acked, whatever the walsender's +state (a peer that has just reconnected is catching up and still alive). The +peer's apply worker sends that reply after it applies, every +`spock.feedback_frequency` messages, and in answer to the walsender's +keepalive; `wal_receiver_status_interval` plays no part, since Spock's apply +worker does not read it. The window and `wal_sender_timeout` go together; set +them as [Timeouts](#timeouts) below describes. An alive peer that has +not acked is either deferring legitimately (R-A's defer rule) or about to +ack, and waiting for it is correct, not a failure. + +A claim whose owner is gone (a hard backend crash) is reaped without operator +action: by that node's next cold write, to any table, by a peer's arriving +claim, or by the waiting peer's poke, a no-op `UPDATE` of its own claim row +about once a second for as long as it waits. That poke is the only replication +traffic the bakery generates while a writer waits, and there is none when +nothing waits. See *Orphan reaping* in +[The Bakery Protocol](architecture_decoupled.md#concurrency-horizontal-scaling-the-bakery-protocol). + +Sync-rep (`synchronous_standby_names`) is **not required** by the bakery - the +R-A ack barrier (the wait, before a claim commits, for every live peer to +acknowledge it) is what serializes iceberg commits. You can still enable +sync-rep cluster-wide if you want stronger durability for non-bakery writes, +but it plays no part in iceberg-commit serialization. + +Run this one-time mesh setup in this order on every node, because step 3 calls +`spock.repset_add_table`, which requires the local Spock node to already exist: + +```sql +-- 1. Extensions, in dependency order. snowflake is a bakery prereq +-- (claim tickets come from snowflake.nextval). +CREATE EXTENSION IF NOT EXISTS snowflake; +CREATE EXTENSION IF NOT EXISTS spock; +CREATE EXTENSION IF NOT EXISTS pg_duckdb; +CREATE EXTENSION IF NOT EXISTS coldfront; + +-- 2. Spock node + full-mesh subscriptions (each node has N-1 subs). Name each +-- subscription sub__from_ (on n1: sub_n1_from_n2): the +-- bakery's peer-liveness check looks the walsender up by that name, and a peer +-- whose subscription is named any other way counts as dead and is treated as +-- already acked. +SELECT spock.node_create('n', 'host= user=coldfront dbname=coldfront port=5432'); + +-- on n1 (n2 and n3 symmetric): +SELECT spock.sub_create('sub_n1_from_n2', 'host= user=coldfront dbname=coldfront port=5432', + ARRAY['default','default_insert_only','ddl_sql'], + false, false, '{}', '0', false); +SELECT spock.sub_create('sub_n1_from_n3', 'host= user=coldfront dbname=coldfront port=5432', + ARRAY['default','default_insert_only','ddl_sql'], + false, false, '{}', '0', false); + +-- 3. **Required** on every node, after its subscriptions exist and before +-- the first cold write, storage secret or table registration: put every +-- ColdFront table that replicates by value in this node's replication sets. +SELECT coldfront.ensure_replicated(); +``` + +The subscriptions set +`synchronize_structure := false, synchronize_data := false` because the tables +already exist on every node from the `coldfront` extension, so no initial copy +is needed. + +## Timeouts + +The following table lists the two `postgresql.conf` settings that decide +how long a cold write waits for a peer; set them together, on every node: + +| Setting | Notes | +|---|---| +| `wal_sender_timeout` | Recommended `15s`, set in `postgresql.conf` on every node (the image sets it if you are running in mesh mode). The node's walsender asks each peer for a reply after half of it (7.5 s) and drops a peer that has not replied for the whole of it. | +| `coldfront.peer_alive_window_ms` | Recommended `10000` (the default), set in `postgresql.conf`, by a superuser only. The bakery treats a peer whose last reply is older than this as dead and stops waiting for its ack. | + +The rule: half of `wal_sender_timeout` must stay below the window, with a +round trip to spare; the claim enforces the first part. An idle peer replies +only when the walsender asks, so the +PostgreSQL default of 60 s against a 10 s window would rule a live idle peer +dead; a claim refuses to run with such settings and names both. With the +recommended values a writer waits at most 10 s for a peer that is really +dead, and the walsender disconnects a peer silent for 15 s, which then +reconnects on its own. + +On a slow or lossy WAN raise both together and keep the rule, for example +`wal_sender_timeout = 30s` with `coldfront.peer_alive_window_ms = 20000`. +Both take effect on a reload (`ALTER SYSTEM SET ...;` then +`SELECT pg_reload_conf();`), no restart: walsenders re-read the timeout, and +each claim reads the window once. + +## What `coldfront.ensure_replicated()` Does + +The call is the one ColdFront-specific step of the mesh setup. It adds eight +tables to the node's replication sets (`claim_acks` to `default_insert_only`, +the rest to `default`), each keyed by name, so a row is identical on every node +and replicates by value: + +| Table | What replicates, and why a peer needs it | +|---|---| +| `coldfront.claims`, `coldfront.claim_acks` | The bakery's tickets and acknowledgements. A peer acknowledges an originator's claim by inserting into `claim_acks` on its own node, and the ack reaches the originator only if the table is in the peer's set. `claim_acks` goes in the insert-only set, because each node deletes only the ack copies it holds. | +| `coldfront.tiered_views` | The registry. A peer's hook recognizes a view by its row; without it the peer's writes through the view fail in PostgreSQL ("cannot insert into view") and its reads cannot attach the cold tier. | +| `coldfront.archive_watermark` | The hot/cold cutoff that routes each tiered write. | +| `coldfront.storage_secret` | The cold-store credential, so one `set_storage_secret` call reaches every node. | +| `coldfront.partition_config` | The per-table lifecycle. The archiver and the partitioner add this table themselves as well, because the partitioner runs without the extension. | +| `coldfront.vector_config`, `coldfront.vector_centroids` | The cluster routing state, so every node resolves a vector to the same cluster. | + +Two tables stay local by design and are not added: `coldfront.deferred_acks` +(each node's queue of the acks it owes) and `coldfront._dummy_dml_target`. + +The call runs on every node because a replication set is a property of the +provider: each node sends the rows of the tables in its own set, and a node +that skipped the step keeps its acks, registry rows and secret to itself. It +cannot run at `CREATE EXTENSION` time, because `spock.repset_add_table` needs +the local Spock node, and nothing runs it later on a node's behalf: +provisioning a table, setting the secret and writing cold rows all assume it +has run. It is idempotent, so running it again is harmless, and a node added to +the mesh later runs the same three steps. + +A skipped step shows up late and away from the node that skipped it. If a peer +never ran it, the originator of every cold write waits at the ack barrier for +an ack that never arrives. If the node that registers a table never ran it, +the view reaches each peer as replicated DDL but the registry row does not, so +the peer's writes through the view fail with "cannot insert into view". A +secret set on such a node stays on that node. + +Verify the step on every node. The query lists the eight tables: + +```sql +SELECT c.relname + FROM spock.replication_set rs + JOIN spock.replication_set_table rst ON rst.set_id = rs.set_id + JOIN pg_class c ON c.oid = rst.set_reloid + WHERE rs.set_name = 'default' AND c.relnamespace = 'coldfront'::regnamespace + ORDER BY 1; +``` + +Then verify the mesh before benchmarking - insert a sentinel claim on each node +and read it back from every other node. All N×(N-1) directions must show the +row before traffic starts. `ci/journey.sh` `story_mesh_substrate` is a +copyable reference: it checks the membership above on every node and then the +sentinels. diff --git a/docs/walkthrough.md b/docs/exploring_coldfront/index.md similarity index 61% rename from docs/walkthrough.md rename to docs/exploring_coldfront/index.md index 2b4cba88..4371350e 100644 --- a/docs/walkthrough.md +++ b/docs/exploring_coldfront/index.md @@ -1,43 +1,63 @@ --- -cwd: ../ +cwd: ../../ --- -# Getting Started with ColdFront +# Exploring ColdFront - Getting Started -If you are new to ColdFront, run the guided walkthrough. +This walkthrough is the foundation step for the mode-specific guides that +follow: -The walkthrough is a self-contained, step-by-step tour of ColdFront's three -operating modes - tiered storage (hot PostgreSQL + cold Iceberg), decoupled -mode (Iceberg-only from the first row), and the standalone partitioner - plus a -distributed demo that runs two nodes over one shared lake. Every command on -this page is real: click it to run in Codespaces, or copy it into a local -shell. +- [Exploring Tiered Storage](../walkthrough_tiered.md) +- [Exploring Decoupled Mode](../walkthrough_decoupled.md) +- [Exploring the Standalone Partitioner](../walkthrough_partitioner.md) +- [Exploring Distributed Mode](../walkthrough_distributed.md) + +It is a self-contained, step-by-step tour of ColdFront's three operating +modes, using Docker containers deployed with the Compose files in the +ColdFront repository. The walkthrough demonstrates: + +- tiered storage, which keeps recent data in PostgreSQL and archives older + data to cold Iceberg. +- decoupled mode, which stores a table entirely in Iceberg from the first + row. +- the standalone partitioner, which manages PostgreSQL partitions with no + cold tier at all. +- a distributed demo that runs two nodes over one shared lake. > **Beta software** - ColdFront is beta software under active development. Do > not use it in production. Interfaces, on-disk formats, and behavior may > change without notice, and data loss is possible. -## Running in GitHub Codespaces +!!! tip "Running in GitHub Codespaces" -[Open this repo in a Codespace](https://github.com/codespaces/new?repo=pgEdge/coldfront) -and everything is preinstalled - Docker, psql, and the -[Runme extension](https://marketplace.visualstudio.com/items?itemName=stateful.runme) -that turns each code block below into a runnable cell. Click **Run** on each -cell as you read. The first cell builds the Docker images (two to five -minutes); everything after it is quick. + [Open this repo in a Codespace](https://github.com/codespaces/new?repo=pgEdge/coldfront) + and select `Get Started` to deploy this walkthrough in a Codespace + environment. Docker, psql, and the + [Runme extension](https://marketplace.visualstudio.com/items?itemName=stateful.runme) + are preinstalled, and turn each code block below into a runnable cell. + Click **Run** on each cell as you read. The first cell builds the Docker + images (two to five minutes); each subsequent step is quicker. -If you prefer a terminal, `bash examples/walkthrough/guide.sh` runs the same -demos as an interactive guide. +If you prefer a terminal, run +[`examples/walkthrough/guide.sh`](https://github.com/pgEdge/ColdFront/blob/main/examples/walkthrough/guide.sh). +It brings up the stack, then presents the same four demos as an +interactive menu - tiered, decoupled, partitioner, and distributed - +plus options to reset the demo tables or tear the stack down. -## Running on Your Own Machine +## Walking Through on your own System -You need Docker 24+ with Compose V2 (the `docker compose` plugin, not the -legacy `docker-compose` binary), roughly 3 GB of free disk inside Docker's -virtual disk (the images alone are about 2.5 GB), and `curl`, `bash`, and -`psql` on the host. The interactive guide also needs `ss` on Linux or `lsof` on -macOS, which it uses to find free ports. +Before performing the walkthrough, confirm that your system meets the +following prerequisites: -The fastest path is the one-liner, which downloads the walkthrough files and -launches the interactive guide: +- Docker 24+ with Compose V2 (the `docker compose` plugin, not the legacy + `docker-compose` binary). +- roughly 3 GB of free disk inside Docker's virtual disk (the images alone + are about 2.5 GB). +- `curl`, `bash`, and `psql` reside on the host. +- `ss` on Linux or `lsof` on macOS, which the interactive guide uses to find + free ports. + +The following curl command downloads the walkthrough files and launches the +interactive guide: ```bash {"ignore":"true"} curl -fsSL \ @@ -74,7 +94,7 @@ Setup runs before the demos begin. Setup starts the containers, waits for PostgreSQL and Lakekeeper to accept connections, and creates the Lakekeeper warehouse and namespace. The stack includes the following services: -- PostgreSQL 16, 17, or 18 with the pg_duckdb and coldfront extensions. +- PostgreSQL 16, 17, or 18 with the pg_duckdb and `coldfront` extensions. - SeaweedFS, a local S3-compatible object store standing in for a real cloud bucket. - Lakekeeper, the Iceberg REST catalog that tracks table metadata and file @@ -113,8 +133,8 @@ curl -sf -X POST http://localhost:8181/management/v1/bootstrap \ -H 'Content-Type: application/json' \ -d '{"accept-terms-of-use":true}' -# Create the warehouse (retried by guide.sh until 200) -curl -sf -X POST http://localhost:8181/management/v1/warehouse \ +# Create the warehouse and save its ID (retried by guide.sh until 200) +WID=$(curl -sf -X POST http://localhost:8181/management/v1/warehouse \ -H 'Content-Type: application/json' \ -d '{ "warehouse-name": "wh", @@ -134,23 +154,25 @@ curl -sf -X POST http://localhost:8181/management/v1/warehouse \ "aws-access-key-id": "admin", "aws-secret-access-key": "adminsecret" } - }' + }' \ + | grep -oE '"warehouse-id":"[^"]+"' | cut -d'"' -f4) # Seed the public namespace -WID=$(curl -s http://localhost:8181/management/v1/warehouse \ - | grep -oE '"warehouse-id":"[^"]+"' \ - | head -1 | cut -d'"' -f4) -curl -sf -X POST \ - "http://localhost:8181/catalog/v1/${WID}/namespaces" \ - -H 'Content-Type: application/json' \ - -d '{"namespace":["public"]}' +if [ -z "$WID" ]; then + echo "No warehouse ID - check that the warehouse create succeeded." +else + curl -sf -X POST \ + "http://localhost:8181/catalog/v1/${WID}/namespaces" \ + -H 'Content-Type: application/json' \ + -d '{"namespace":["public"]}' +fi ``` ### Using a Cloud Object Store The walkthrough uses SeaweedFS by default. To use a cloud store instead, replace the warehouse JSON above, the `set_storage_secret` call in -[Step 5](walkthrough_tiered.md#pointing-coldfront-at-the-object-store) of the +[Step 5](../walkthrough_tiered.md#pointing-coldfront-at-the-object-store) of the tiered demo, and the `s3:` block in `examples/walkthrough/config/archiver.yaml`, which `import` writes into the server as the storage secret. @@ -165,10 +187,8 @@ store: | GCS (HMAC) | `SELECT coldfront.set_storage_secret(p_key_id => '', p_secret => '', p_endpoint => 'storage.googleapis.com', p_region => 'us-east-1', p_url_style => 'path', p_use_ssl => true);` | | Azure ADLS Gen2 | `SELECT coldfront.set_storage_secret_azure('AccountName=;AccountKey=;EndpointSuffix=core.windows.net');` | -The [Configuring your Object Store](object_store.md) guide shows the matching -warehouse JSON for AWS S3; for GCS and Azure, see the -[Storage Backends](usage.md#storage-backends) section of the Using ColdFront -guide. +The [Configuring your Object Store](../object_store.md) guide shows the +matching Lakekeeper warehouse JSON for each of these backends. ## Tearing Down the Stack @@ -184,7 +204,7 @@ The `-v` flag removes the named volumes: `pgdata` (PostgreSQL), `lkdata` (the Lakekeeper catalog) and `s3data` (the object store). Omit the flag to keep the data for a later session. -If you ran the [distributed demo](walkthrough_distributed.md), tear down its +If you ran the [distributed demo](../walkthrough_distributed.md), tear down its separate mesh stack too: ```bash @@ -200,13 +220,13 @@ switches back to the single-node stack. ## Next Steps -With the stack running, continue with the following guides: +With the stack running, continue with the following walkthroughs: -- The [Tiered Storage Demo](walkthrough_tiered.md) adds ColdFront to an +- The [Tiered Storage Demo](../walkthrough_tiered.md) adds ColdFront to an existing database and moves its cold data to object storage. -- The [Decoupled Mode Demo](walkthrough_decoupled.md) stores a table in Iceberg +- The [Decoupled Mode Demo](../walkthrough_decoupled.md) stores a table in Iceberg from the first row and adopts a table that another engine wrote. -- The [Partitioner Demo](walkthrough_partitioner.md) manages PostgreSQL range +- The [Partitioner Demo](../walkthrough_partitioner.md) manages PostgreSQL range partitions without any cold tier. -- The [Distributed Demo](walkthrough_distributed.md) points two PostgreSQL +- The [Distributed Demo](../walkthrough_distributed.md) points two PostgreSQL nodes at one shared lake. diff --git a/docs/index.md b/docs/index.md index 4d21720b..9c5eabb6 100644 --- a/docs/index.md +++ b/docs/index.md @@ -1,10 +1,12 @@ # pgEdge ColdFront -ColdFront keeps tables in PostgreSQL and cold data in Apache Iceberg (Parquet -on S3-compatible, Azure, or GCS storage), and the cold tier is both readable -and writable through the same SQL with no application changes. The application -queries every table as an ordinary PostgreSQL relation, and both operating -modes present the same standard SQL surface. +ColdFront keeps tables in PostgreSQL and, in its tiered and decoupled modes, +keeps cold data in Apache Iceberg (Parquet on S3-compatible, Azure, or GCS +storage). The cold tier is both readable and writable through the same SQL, +with no application changes. The application +queries every table as an ordinary PostgreSQL relation, with standard +`SELECT`, `INSERT`, `UPDATE`, and `DELETE`, and `MERGE` on PostgreSQL 17 and +later. !!! warning "Beta software" @@ -12,26 +14,33 @@ modes present the same standard SQL surface. production. Interfaces, on-disk formats, and behavior may change without notice, and data loss is possible. -ColdFront provides two operating modes: +ColdFront provides three operating modes: - Tiered mode keeps recent data in native PostgreSQL partitions and archives older data to Iceberg on a watermark; the application reads a single unified view, and the archiver moves rows from hot to cold on a schedule. - Decoupled mode stores the table entirely in Iceberg from the first row; - PostgreSQL holds a thin wrapper view and a registry row, and the coldfront + PostgreSQL holds a thin wrapper view and a registry row, and the `coldfront` extension handles every data-modifying statement on that view. +- Standalone partitioned mode manages time- or id-based PostgreSQL range + partitions with no cold tier; the `partitioner` binary creates upcoming + partitions and detaches and drops partitions past retention, on stock + PostgreSQL with no DuckDB or Iceberg. -Both modes coexist within one database, and you choose the mode per table at -creation time. The SQL surface is identical for both modes: standard `SELECT`, -`INSERT`, `UPDATE`, and `DELETE` against the relation, and `MERGE` on -PostgreSQL 17 and later. +Tables in different modes can share one database, and you choose the mode for +each table: tiered when you register an existing partitioned table with a hot +period, standalone partitioned when you register one without a hot period, +and decoupled when you create or adopt an Iceberg table. Decoupled mode scales out horizontally across many PostgreSQL nodes that share one Lakekeeper catalog and one object store. The bakery protocol in the -coldfront extension serializes Iceberg commits on the PostgreSQL side using +`coldfront` extension serializes Iceberg commits on the PostgreSQL side using Spock-replicated Snowflake tickets, so concurrent writers never collide at the -catalog. The protocol implements Lamport mutual exclusion with the -Ricart-Agrawala deferred-reply optimization, and the +catalog. The protocol implements +[Lamport mutual exclusion](https://en.wikipedia.org/wiki/Lamport%27s_distributed_mutual_exclusion_algorithm) +with the +[Ricart-Agrawala](https://en.wikipedia.org/wiki/Ricart%E2%80%93Agrawala_algorithm) +deferred-reply optimization, and the [formal model](formal/README.md) verifies its safety with TLA+. ## How It Works @@ -75,112 +84,37 @@ tier, so the application sees one relation: cron job: moves partitions out of the hot window: hot -> cold ``` -## Quickstart +!!! tip "New to ColdFront?" -Build the image (see the [Installation](installation.md) guide) and bring up -the stack: + If you are new to ColdFront, review the guided + [walkthroughs](exploring_coldfront/index.md) to see all three modes in action + with copy-pasteable commands. -```bash -docker compose --profile local-store up -d --build -``` - -Bootstrap Lakekeeper and create a warehouse (see the one-time setup in the -[Using ColdFront](usage.md) guide), then create a table in psql: - -```sql -CREATE EXTENSION IF NOT EXISTS pg_duckdb; -CREATE EXTENSION IF NOT EXISTS coldfront; -SELECT coldfront.set_storage_secret('admin', 'adminsecret', 'seaweedfs:8333'); - --- Decoupled (iceberg-only) table, stored entirely in Iceberg on S3: -SELECT coldfront.create_iceberg_table('public', 'events', - '[{"name":"id","type":"bigint"},{"name":"ts","type":"timestamptz"},{"name":"note","type":"text"}]'::jsonb, - '{month(ts)}'); -INSERT INTO events VALUES (1, now(), 'hello'); -SELECT count(*) FROM events; -``` - -ColdFront can adopt a table that already exists in the Iceberg catalog rather -than creating it: `coldfront.adopt_iceberg_table()` reads its schema from the -catalog and gives it the same wrapper view and registry row, read-only unless -you pass `p_writable => true`. `coldfront.release_iceberg_table()` hands the -table back with the Iceberg table untouched. See -[Adopting a Table That Already Exists in the Catalog](usage.md#adopting-a-table-that-already-exists-in-the-catalog). +## Exploring ColdFront with Walkthroughs -To remove a table again, `coldfront.drop_iceberg_table()` unregisters it and -drops the Iceberg table, deleting the stored objects only when asked to. See -[Dropping an Iceberg Table](usage.md#dropping-an-iceberg-table-both-modes). - -For compliance environments that cannot store an object-store credential, -`coldfront.set_storage_secret_vended()` runs with no credential in the -database: Lakekeeper issues short-lived per-table credentials at access time. -See [Vended Credentials](usage.md#vended-credentials). - -## Least-Privilege Application Roles - -Application roles need no superuser and no server-file access, yet they read -and write the cold tier through the same transparent view. Onboarding an -application role is a single call: - -```sql -SELECT coldfront.grant_app_access('alice'); -``` +Use the walkthroughs in the [Exploring ColdFront](exploring_coldfront/index.md) +section to learn more about ColdFront. Perform the steps on the +[Exploring ColdFront - Getting Started](exploring_coldfront/index.md) page +first, which brings up the stack; then continue to the detailed guide for +the mode you want to try: -grant_app_access grants only the minimum the cold path needs: membership in -duckdb.postgres_role, SET on the duckdb.unsafe_allow_execution_inside_functions -parameter, schema USAGE, `SELECT` on the registry and the watermark table, DML -on the dual-write anchor table, DML on every registered view and the hot table -behind it, and USAGE and `SELECT` on the hot table's sequences. Those objects -are derived from the registry, not hardcoded, and the call also grants EXECUTE -on a fixed allow-list of runtime cold-path functions. The call is idempotent -and is not executable by PUBLIC, so an application role can never self-grant. -The role is never granted pg_read_server_files or pg_write_server_files, so it -has no host-file access. `CREATE ROLE` and `GRANT` both replicate over Spock, -so you onboard a role once on any node and it propagates across the mesh. - -The Docker image sets `duckdb.postgres_role` to `coldfront_duckdb` and creates -that role when it initializes a new data directory. To name a different role, -set the container's `COLDFRONT_DUCKDB_ROLE` environment variable before that -first start. An empty value keeps pg_duckdb's default, under which only -superusers run DuckDB, so grant_app_access then fails with an error. - -For how the non-superuser path works - the `SECURITY DEFINER` attach helpers, -the `PGC_SUSET` / `GUC_SUPERUSER_ONLY` config hardening, the -`duckdb.postgres_role` default that the image sets up, and how least privilege -holds across a Spock mesh - see -[Architecture: non-superuser app roles](architecture.md#non-superuser-app-roles-least-privilege). - -## Caveats - -Iceberg on Azure ADLS Gen2 requires Blob soft-delete, container soft-delete, -and change feed (blob events) to be OFF on the storage account. Lakekeeper -warehouse creation otherwise fails with HTTP 409 ("This endpoint does not -support BlobStorageEvents or SoftDelete"). Disable those features on the -storage account before using it as a cold tier. - -A cold-tier write that reads an ordinary PostgreSQL table, such as an -`UPDATE ... FROM` or a `MERGE` from one, stores a `numeric` `NaN` from that -table as `0` and stores an array renumbered to start at 1, with no error. The -[Using ColdFront](usage.md#reading-a-postgresql-table-in-a-cold-tier-write) -guide lists the writes that do this and shows how to check a source table -first. +- [Exploring Tiered Storage](walkthrough_tiered.md) +- [Exploring Decoupled Mode](walkthrough_decoupled.md) +- [Exploring the Standalone Partitioner](walkthrough_partitioner.md) +- [Exploring Distributed Mode](walkthrough_distributed.md) ## Next Steps -If you are new to ColdFront, run the [guided walkthrough](walkthrough.md) to -see all three modes in action with copy-pasteable commands. - -To go further with ColdFront, consult the following guides: - -- The [Installation](installation.md) guide covers building ColdFront and - bringing up the stack. -- The [Using ColdFront](usage.md) guide covers both modes, the standalone - partition manager, supported types, and tuning. -- The [Embeddings](usage_vectors.md) guide covers storing and searching - embeddings through the pgvector interface. -- The [Configuring your Object Store](object_store.md) guide takes you from an - empty bucket to a working cold tier on cloud S3. -- The [Compaction](compaction.md) guide covers cold-tier maintenance: - compaction, snapshot expiry, and orphan-file removal. -- The [Architecture](architecture.md) overview explains the shared mechanics - and links to the per-mode deep dives. +For details about installing and configuring ColdFront, consult the +following guides: + +- The [Installation](installation.md) guide details installing ColdFront + from a package or building it from source. +- The [Configuring ColdFront](configuration.md) guide documents PostgreSQL + and ColdFront's own configuration. +- The [Configuring Lakekeeper](configure_lakekeeper.md) guide installs, + configures, and bootstraps it, then installs the extensions + each database needs. +- The [Configuring your Object Store](object_store.md) guide takes you from + an empty bucket or container to a working cold tier, for any of four + supported backends. diff --git a/docs/installation.md b/docs/installation.md index 9deaf814..22db0ac2 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -1,161 +1,100 @@ -# Installing and Configuring ColdFront +# Installing ColdFront -This guide covers installing ColdFront from the published package or -building it from source, and configuring it once installed. +This guide details installing ColdFront with a pgEdge package or +building it from source. After installing ColdFront, see +[Configuring ColdFront](configuration.md) for detailed information about +setting up PostgreSQL, Lakekeeper, and ColdFront itself. ## Using a Package to Install ColdFront ColdFront's packages come from the pgEdge package repository. Before you -install them, configure the repository by following the page for your platform -in the Enterprise Postgres documentation: +can use a package, you must configure the pgEdge repository by following +the steps in the Enterprise Postgres documentation: - For RHEL, Oracle Linux, AlmaLinux, and Rocky Linux, see [Configuring the Repository on Enterprise Linux](https://docs.pgedge.com/enterprise/el/configure-repo/). - For Debian and Ubuntu, see [Configuring the Repository on Debian and Ubuntu](https://docs.pgedge.com/enterprise/debian/configure-repo/). -Those pages cover the platform prerequisites, disabling any PGDG repository, -and creating the pgEdge repository. +The documentation details the platform requirements, disabling the PGDG +repository, and creating the pgEdge repository. -With the repository in place, install the ColdFront extension package for -your PostgreSQL major version. It depends on the pgEdge PostgreSQL server, -pg_duckdb, and ColdFront's DuckDB extensions, so the package manager installs -those as well. The following table shows each package's purpose and its name -on RHEL-family and Debian-family systems: +If you already have a pgEdge Enterprise Postgres server installed that you +would like to use, configure the pgEdge repository and install only ColdFront, +continuing with +[Installing the ColdFront Packages](#installing-the-coldfront-packages). +ColdFront packages from pgEdge require pgEdge's PostgreSQL build. -| Package | RHEL, Rocky Linux, AlmaLinux | Ubuntu, Debian | -|---|---|---| -| ColdFront extension | `pgedge-coldfront_` | `pgedge-postgresql--coldfront` | -| ColdFront's DuckDB extensions | `pgedge-coldfront-duckdb-extensions` | `pgedge-coldfront-duckdb-extensions` | -| pg_duckdb | `pgedge-pg-duckdb_` | `pgedge-postgresql--pg-duckdb` | -| ColdFront command-line tools (archiver, partitioner, compactor) | `pgedge-coldfront` | `pgedge-coldfront` | -| Lakekeeper | `pgedge-lakekeeper` | `pgedge-lakekeeper` | +### Installing PostgreSQL with Packages -For PostgreSQL 18 on RHEL, Rocky Linux, or AlmaLinux: +Before installing ColdFront, you will need to install pgEdge Enterprise Postgres +and start the server. On RHEL, Rocky Linux, or AlmaLinux, the package does not +create a database cluster, so create and start one as root: ```bash -sudo dnf install -y pgedge-coldfront_18 +sudo dnf install -y pgedge-enterprise-postgres_18 +sudo /usr/pgsql-18/bin/postgresql-18-setup initdb +sudo systemctl enable --now postgresql-18 ``` -On Ubuntu or Debian: +On Ubuntu or Debian, install the package, then check for a cluster with +`sudo pg_lsclusters`. If it lists no `18/main` cluster, create and start one: ```bash sudo apt update -sudo apt install -y pgedge-postgresql-18-coldfront +sudo apt install -y pgedge-enterprise-postgres-18 +sudo pg_lsclusters +sudo pg_createcluster 18 main --start # only if pg_lsclusters lists none ``` -The extension package does not install the command-line tools or Lakekeeper. -Install `pgedge-coldfront` and `pgedge-lakekeeper` separately on the hosts -that need them. +On either platform, confirm that the server is running before you continue: -### Configuring PostgreSQL +```bash +sudo -u postgres psql -c "SELECT version();" +``` -A package installation does not configure PostgreSQL. Add the following -settings to `postgresql.conf`, then restart PostgreSQL: +The command prints the PostgreSQL version. A connection error means the server +is not running; check `sudo systemctl status postgresql-18` on RHEL-family +systems or `sudo pg_lsclusters` on Ubuntu and Debian. -```ini -shared_preload_libraries = 'pg_duckdb,coldfront' -duckdb.extension_directory = '/usr/lib/pgedge/coldfront/duckdb-extensions' -duckdb.allow_unsigned_extensions = true -duckdb.autoinstall_known_extensions = false -coldfront.iceberg_async_parquet = on -coldfront.iceberg_bakery_patch = on -coldfront.warehouse = '' -coldfront.lakekeeper_endpoint = 'http://:8181/catalog' -coldfront.local_pg_dsn = 'host=/var/run/postgresql dbname= user=' +For details, see the pgEdge Enterprise Postgres installation guide for +[Enterprise Linux](https://docs.pgedge.com/enterprise/el/installing/) or +[Debian and Ubuntu](https://docs.pgedge.com/enterprise/debian/installing/). + +### Installing the ColdFront Packages + +With PostgreSQL running, install the ColdFront extension package for your +PostgreSQL major version, the ColdFront command-line tools (the archiver, the +partitioner, and the compactor), and Lakekeeper (the Iceberg catalog). The +extension package depends on pg_duckdb and ColdFront's DuckDB extensions, so +the package manager installs those as well. The following table shows each +package's purpose and its name on RHEL-family and Debian-family systems: + +| Package | RHEL, Rocky Linux, AlmaLinux | Ubuntu, Debian | +|---|---|---| +| pgEdge Enterprise Postgres | `pgedge-enterprise-postgres_` | `pgedge-enterprise-postgres-` | +| ColdFront extension | `pgedge-coldfront_` | `pgedge-postgresql--coldfront` | +| ColdFront's DuckDB extensions | `pgedge-coldfront-duckdb-extensions` | `pgedge-coldfront-duckdb-extensions` | +| pg_duckdb | `pgedge-pg-duckdb_` | `pgedge-postgresql--pg-duckdb` | +| ColdFront command-line tools (archiver, partitioner, compactor) | `pgedge-coldfront` | `pgedge-coldfront` | +| Lakekeeper | `pgedge-lakekeeper` | `pgedge-lakekeeper` | + +For PostgreSQL 18 on RHEL, Rocky Linux, or AlmaLinux: + +```bash +sudo dnf install -y pgedge-coldfront_18 pgedge-coldfront pgedge-lakekeeper ``` -The three `duckdb.*` settings make pg_duckdb load ColdFront's patched DuckDB -extensions from the directory where `pgedge-coldfront-duckdb-extensions` -installs them. Without these settings, pg_duckdb downloads the unpatched -upstream extensions, and concurrent cold writes can then fail with HTTP 409. -The patched extensions are unsigned, so `duckdb.allow_unsigned_extensions` must -be on. Keeping `duckdb.autoinstall_known_extensions` off stops DuckDB from -downloading an unpatched upstream copy when an extension file is missing. - -The two `coldfront.iceberg_*` settings make a node upload the Parquet files of -an `INSERT` or `COPY` outside its write lock (the bakery claim on a Spock mesh, -an advisory lock on a single node) and serialize only the catalog commit. The -packaged duckdb-iceberg includes the patch that this ordering requires, and the -[Distributed Setup](usage.md#distributed-setup-3-node-mesh-decoupled-mode) -section of the Using ColdFront guide describes the mesh settings. - -The [One-Time Setup](usage.md#one-time-setup) section of the Using ColdFront -guide describes the `coldfront.*` settings. Follow that section from the -Lakekeeper bootstrap onward to create the warehouse, the extensions, and the -cold-store credential. - -### Setting Up Lakekeeper - -The `pgedge-lakekeeper` package installs the `lakekeeper` binary and a -`lakekeeper` systemd service, but it does not enable or start the service. -Lakekeeper stores its catalog in a PostgreSQL 15 or later database and reads -its settings from `/etc/lakekeeper/lakekeeper.env`. The following steps -prepare and start the service: - -1. Create a role and a database for the catalog: - - ```sql - CREATE ROLE lakekeeper LOGIN PASSWORD 'change-me'; - CREATE DATABASE lakekeeper OWNER lakekeeper; - ``` - - The migration in step 3 creates the `uuid-ossp`, `pgcrypto`, `pg_trgm`, - `btree_gin`, and `btree_gist` extensions, so either the role must be - allowed to run `CREATE EXTENSION` or a superuser must create them first. - -2. In `/etc/lakekeeper/lakekeeper.env`, set - `LAKEKEEPER__PG_DATABASE_URL_WRITE` to the database's connection string and - `LAKEKEEPER__PG_ENCRYPTION_KEY` to a random secret, such as the output of - `openssl rand -base64 32`. Lakekeeper encrypts stored credentials with the - key, so keep the key stable and backed up. Every node that shares the - catalog needs the same key. - -3. Run the one-time database migration as the `lakekeeper` user: - - ```bash - set -a; . /etc/lakekeeper/lakekeeper.env; set +a - sudo -E -u lakekeeper /usr/bin/lakekeeper migrate - ``` - -4. Enable and start the service: - - ```bash - sudo systemctl enable --now lakekeeper - ``` - -Lakekeeper listens on port 8181 on every address by default. Without an -authorization backend in `lakekeeper.env`, the catalog accepts every request, -so configure authentication and authorization before you expose the service -beyond a trusted network. - -### Configuring ColdFront - -ColdFront has no configuration file, because the database holds its -configuration. The server settings are the `postgresql.conf` lines above, each -managed table is a row in `coldfront.partition_config`, and the cold-store -credential is in `coldfront.storage_secret`. The archiver, partitioner, and -compactor connect the way psql does, from the libpq environment (`PGHOST`, -`PGDATABASE`, `PGUSER`, `PGPASSWORD`, `PGSERVICE`) or `--dsn`, and read every -other setting from the server. - -You write that configuration with `coldfront.set_storage_secret()` and the -`register` command, or in one step by importing a deployment YAML: +For PostgreSQL 18 on Ubuntu or Debian: ```bash -archiver import --config deploy.yaml +sudo apt install -y pgedge-postgresql-18-coldfront pgedge-coldfront \ + pgedge-lakekeeper ``` -The `pgedge-coldfront` package installs an example deployment YAML at -`/etc/pgedge/coldfront/config.yaml`. That file is only an example to edit and -pass to `import`, and no ColdFront tool reads it unless `--config` names it. -After an import, the server holds the configuration. A later run that is given -a YAML checks the file against the server and refuses to run if any value -differs. The only value such a run takes from the file is `postgres.dsn`, which -connects when `--dsn` is unset. The -[Managing Partitioned Tables (CLI)](usage.md#managing-partitioned-tables-cli) -section of the Using ColdFront guide describes `register`, `import`, and -`export`. +The `pgedge-lakekeeper` package installs Lakekeeper but does not start it; you +configure and start it as described in +[Configuring Lakekeeper](configure_lakekeeper.md). ## Building ColdFront from Source @@ -164,14 +103,20 @@ Docker on top of the published base image or on bare metal. ColdFront runs on a **DuckDB 1.5.x** stack: PostgreSQL + pg_duckdb (DuckDB 1.5.4) and a **patched** duckdb-iceberg that includes ColdFront's five -patches - the bakery-aware commit-refresh patch (the no-409 guarantee for -concurrent cold-tier writers), two strict-reader interop patches (so -apache/iceberg-go, the cold-tier compactor, can read the manifests -duckdb-iceberg writes), a port of an upstream fix that computes the time -partitions of a `timestamptz` column in UTC, and a port of another that numbers -a new table schema above the highest schema id. No released pg_duckdb tag -includes DuckDB 1.5.x yet, so the stack is built from a pinned upstream PR plus -ColdFront's patches - all from sources you can fetch. +patches: + +- the bakery-aware commit-refresh patch (the no-409 guarantee for + concurrent cold-tier writers). +- two strict-reader interop patches, so apache/iceberg-go, the cold-tier + compactor, can read the manifests duckdb-iceberg writes. +- a port of an upstream fix that computes the time partitions of a + `timestamptz` column in UTC. +- a port of another that numbers a new table schema above the highest + schema id. + +No released pg_duckdb tag includes DuckDB 1.5.x yet, so the stack is built +from a pinned upstream PR plus ColdFront's patches - all from sources you +can fetch. ### Prerequisites @@ -182,12 +127,13 @@ The following table lists the prerequisites for each build path: | Docker build (below) | Docker, network access (GitHub, ghcr.io, quay.io, curl.se, and the distribution's RPM repositories), a few GB of disk and RAM, and 30-60 minutes for the base compile. | | The archiver and partitioner (all paths) | Go 1.26.5+ (pinned in [go.mod](https://github.com/pgEdge/ColdFront/blob/main/go.mod)) and `make` (`make build` produces `./bin/archiver` and `./bin/partitioner`). | | The compactor and the CI gate | golangci-lint for `make compactor`; `./run-ci-local.sh` also needs Docker and mkdocs with mkdocs-material. | -| Bare metal (near the end of this section) | `pg_config`, PostgreSQL server dev headers, libpq client headers and library (libpq-dev / libpq-devel), `make`, and `gcc`. | +| Bare metal (near the end of this section) | A PostgreSQL 16, 17, or 18 server, `pg_config`, PostgreSQL server dev headers, libpq client headers and library (libpq-dev / libpq-devel), `make`, and `gcc`. | -### What the Build Produces +### Preparing the DuckDB Base Image -`docker/Dockerfile.duckdb15-base` is the recipe; it fetches the requirements, -applies ColdFront's patches, and compiles a set of components. The following +[`docker/Dockerfile.duckdb15-base`](https://github.com/pgEdge/ColdFront/blob/main/docker/Dockerfile.duckdb15-base) +contains the recipe for the base image: it fetches the requirements, applies +ColdFront's patches, and compiles a set of components. The following table shows each component and its source: | Component | Source | @@ -253,7 +199,7 @@ expensive, stable compiles (pg_duckdb on DuckDB 1.5.4 + the patched duckdb-iceberg) live in the prebuilt **base**, published to `ghcr.io/pgedge/coldfront-duckdb-base:pg{16,17,18}`; the **app** build ([`docker/Dockerfile.duckdb15`](https://github.com/pgEdge/ColdFront/blob/main/docker/Dockerfile.duckdb15)) -`FROM`s it and compiles the coldfront extension in seconds. If you build the +`FROM`s it and compiles the `coldfront` extension in seconds. If you build the base yourself (step 1) the app layer `FROM`s your local image; otherwise it `FROM`s the published `ghcr.io/pgedge/coldfront-duckdb-base:pg`. Rebuild the published base via the @@ -267,8 +213,8 @@ publishes it. The floating `pg` tag moves to the same image only when the base-image workflow is dispatched from `main` with `push=true` and both architectures. -Then follow the [One-Time Setup](usage.md#one-time-setup) section of the Using -ColdFront guide: bootstrap Lakekeeper, create a table, tier it, and verify. +Then follow the [Configuring Lakekeeper](configure_lakekeeper.md) guide: +bootstrap it, create a table, tier it, and verify. !!! note "pg_duckdb pin" @@ -325,7 +271,7 @@ and seed the `public` namespace: curl -sf -X POST http://localhost:8181/management/v1/bootstrap \ -H 'Content-Type: application/json' -d '{"accept-terms-of-use":true}' -curl -s -X POST http://localhost:8181/management/v1/warehouse \ +WID=$(curl -s -X POST http://localhost:8181/management/v1/warehouse \ -H 'Content-Type: application/json' -d '{ "warehouse-name":"wh", "storage-profile":{"type":"s3","bucket":"iceberg","region":"us-east-1", @@ -333,12 +279,15 @@ curl -s -X POST http://localhost:8181/management/v1/warehouse \ "flavor":"s3-compat","sts-enabled":false,"remote-signing-enabled":false}, "storage-credential":{"type":"s3","credential-type":"access-key", "aws-access-key-id":"admin","aws-secret-access-key":"adminsecret"} - }' - -WID=$(curl -s http://localhost:8181/management/v1/warehouse \ - | grep -oE '"warehouse-id":"[^"]+"' | head -1 | cut -d'"' -f4) -curl -s -X POST "http://localhost:8181/catalog/v1/$WID/namespaces" \ - -H 'Content-Type: application/json' -d '{"namespace":["public"]}' + }' \ + | grep -oE '"warehouse-id":"[^"]+"' | cut -d'"' -f4) + +if [ -z "$WID" ]; then + echo "No warehouse ID - check that the warehouse create succeeded." +else + curl -s -X POST "http://localhost:8181/catalog/v1/$WID/namespaces" \ + -H 'Content-Type: application/json' -d '{"namespace":["public"]}' +fi ``` Create the extensions, set the cold-store secret, create a decoupled table, @@ -359,12 +308,12 @@ SQL A row count of 1 read back through Iceberg confirms the full path. For a real cloud store, drop the `local-store` profile, point the warehouse at your own -bucket, and follow the [One-Time Setup](usage.md#one-time-setup) section of the -Using ColdFront guide for the full tier-and-verify journey. +bucket, and follow the [Configuring Lakekeeper](configure_lakekeeper.md) guide +for the full tier-and-verify journey. ## Building ColdFront on Bare Metal -The coldfront extension is a standard PGXS C extension: +The `coldfront` extension is a standard PGXS C extension: ```bash cd extension/coldfront @@ -383,6 +332,8 @@ does. Then set the following in `postgresql.conf`: shared_preload_libraries = 'pg_duckdb,coldfront' # The locally built iceberg extension is unsigned. duckdb.allow_unsigned_extensions = true +duckdb.autoinstall_known_extensions = false +duckdb.postgres_role = 'coldfront_duckdb' coldfront.warehouse = '' coldfront.lakekeeper_endpoint = 'http://:8181/catalog' coldfront.local_pg_dsn = 'host=/var/run/postgresql dbname= user=' @@ -394,9 +345,17 @@ coldfront.iceberg_bakery_patch = on coldfront refuses to load any other way: `CREATE EXTENSION coldfront` fails on a server that does not preload it, with an error that names the setting. -See the [Tuning Knobs](usage.md#tuning-knobs) section of the Using ColdFront -guide for the remaining GUCs, and the README for the optional non-superuser -role that the image sets up. +`duckdb.postgres_role` names the role whose members may run DuckDB. Without +it, only superusers can, and `coldfront.grant_app_access()` fails. Create the +role before you grant access to an application role: + +```sql +CREATE ROLE coldfront_duckdb NOLOGIN; +``` + +See the [Configuring PostgreSQL](configuration.md#configuring-postgresql) +section of the Configuring ColdFront guide for the remaining GUCs, and +the README for the optional non-superuser role that the image sets up. ## Testing & CI @@ -462,8 +421,13 @@ SeaweedFS-only. To go further with ColdFront, consult the following guides: -- The [Walkthrough](walkthrough.md) guide runs the demo stack hands-on. -- The [Using ColdFront](usage.md) guide covers the one-time setup and both - modes. -- The [Configuring your Object Store](object_store.md) guide connects the cold - tier to AWS S3. +- The [Configuring ColdFront](configuration.md) guide details PostgreSQL + and ColdFront's own configuration. +- The [Configuring Lakekeeper](configure_lakekeeper.md) guide installs, + configures, and bootstraps it, then installs the extensions each + database needs. +- The [Walkthrough](exploring_coldfront/index.md) guide runs the demo stack hands-on. +- The [Using ColdFront](using_coldfront/index.md) guide documents the tiered, + decoupled, and standalone partitioned modes. +- The [Configuring your Object Store](object_store.md) + guide connects the cold tier to AWS S3. diff --git a/docs/object_store.md b/docs/object_store.md index dc46723b..2aa09c4e 100644 --- a/docs/object_store.md +++ b/docs/object_store.md @@ -1,87 +1,99 @@ # Configuring your Object Store -This walkthrough takes you from an empty S3 bucket to a working ColdFront cold -tier in one sitting. You stand up the ColdFront stack (PostgreSQL + the -Lakekeeper Iceberg catalog) with Docker, point it at your bucket, and write -rows that land as Apache Iceberg tables in S3 - readable straight back through -Postgres. - -This walkthrough targets a real cloud S3 service that uses virtual-hosted -addressing. For a path-style S3-compatible store (MinIO, SeaweedFS) or GCS, see -the [Storage Backends](usage.md#storage-backends) section of the Using -ColdFront guide instead. - -No prior ColdFront knowledge is assumed. Copy and paste the commands from top -to bottom. The examples use the following placeholders throughout: bucket -`my-iceberg-bucket`, region `eu-west-1`, key `AKIAEXAMPLE...`, secret -`` - substitute your own. +This walkthrough guides you from an empty bucket or container to a +working ColdFront cold tier. You stand up the ColdFront stack - +PostgreSQL and the Lakekeeper Iceberg catalog - with Docker, and point +it at your object store. ColdFront supports four cold-store backends: +S3-compatible stores (SeaweedFS, MinIO), virtual-hosted cloud S3 (AWS), +Google Cloud Storage, and Azure ADLS Gen2. Configure **exactly one** of +them for a given deployment; this guide shows the bucket/container +setup, the Lakekeeper warehouse, and the ColdFront credential for each. --- -## 1. Prerequisites +## Prerequisites + +Before you begin, decide which backend you are using, since the +bucket/container details, credential type, and Lakekeeper warehouse +profile differ by backend: + +- **S3-compatible stores** (SeaweedFS, MinIO) need a bucket, the + store's endpoint URL, and a long-term access key and secret. +- **Virtual-hosted cloud S3** (AWS) needs a bucket, the bucket's real + Region, and a long-term access key and secret. +- **Google Cloud Storage** needs a bucket and an + [HMAC key pair](https://cloud.google.com/storage/docs/authentication/hmackeys) + (this guide configures GCS as an S3 backend pointed at GCS's + S3-interoperability endpoint, rather than Lakekeeper's native `gcs` + profile). +- **Azure ADLS Gen2** needs a storage account with hierarchical namespace + enabled (for example, `az storage account create --hns true`), a filesystem + (container) in that account, and the account's access key. A StorageV2 + account without hierarchical namespace is not an ADLS Gen2 account, and + Lakekeeper's `adls` warehouse fails on it. + +For any backend that takes a long-term key (S3-compatible, virtual-hosted +S3, and GCS): + +> **Only long-term keys work.** Lakekeeper's warehouse credential has no +> field for a session token, so single sign-on (SSO) or temporary +> session-token credentials do **not** work here. Use a permanent +> access-key pair with no expiry. +> +> This applies to the warehouse's own credential. A deployment that must +> not store any object-store credential in the database can use vended +> credentials instead, where the warehouse issues short-lived per-table +> credentials at access time. Vended credentials are available for AWS S3 +> and Azure ADLS Gen2, but not for GCS or other S3-compatible stores; see +> [Using Vended Credentials](vended_credentials.md). + +For AWS S3 and S3-compatible stores, grant the key permission to read, +write, and list the bucket (`GetObject` / `PutObject` / `DeleteObject` / +`ListBucket`) and to abort a failed multipart upload +(`AbortMultipartUpload`), as in the following example policy: -Before you begin, gather the following: - -- an S3 bucket (`my-iceberg-bucket` in the examples below). -- the bucket's real region (`eu-west-1` in the examples below). -- a long-term access key, meaning an access key id (`AKIAEXAMPLE...`) and its - secret (``). - - > **Only long-term keys work.** Lakekeeper's warehouse credential has no - > field for a session token, so single sign-on (SSO) or temporary - > session-token credentials do **not** work here. Use a permanent - > access-key pair with no expiry. - > - > This applies to the warehouse's own credential. A deployment that must - > not store any object-store credential in the database can use vended - > credentials instead, where the warehouse issues short-lived per-table - > credentials at access time; see [usage.md](usage.md#vended-credentials). - -- permission for the key to read, write and list the bucket (`GetObject` / - `PutObject` / `DeleteObject` / `ListBucket`) and to abort a failed multipart - upload (`AbortMultipartUpload`), as in the following example policy: - - ```json +```json +{ + "Version": "2012-10-17", + "Statement": [ { - "Version": "2012-10-17", - "Statement": [ - { - "Effect": "Allow", - "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:ListBucket", "s3:AbortMultipartUpload"], - "Resource": [ - "arn:aws:s3:::my-iceberg-bucket", - "arn:aws:s3:::my-iceberg-bucket/*" - ] - } + "Effect": "Allow", + "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:ListBucket", "s3:AbortMultipartUpload"], + "Resource": [ + "arn:aws:s3:::my-iceberg-bucket", + "arn:aws:s3:::my-iceberg-bucket/*" ] } - ``` + ] +} +``` + +DuckDB never aborts a failed multipart upload, so add an +`AbortIncompleteMultipartUpload` lifecycle rule to the bucket to delete +leftover parts. - DuckDB never aborts a failed multipart upload, so add an - `AbortIncompleteMultipartUpload` lifecycle rule to the bucket to delete - leftover parts. +You also need the ColdFront image, built once by following the steps in +[Installing ColdFront](installation.md), which notes the registry access the +base image needs. Run the commands below from the repo root. -- the ColdFront image, built once by following - [installation.md](installation.md), which notes the registry access the base - image needs. Run the commands below from the repo root. +No prior ColdFront knowledge is required to perform the steps on this +page; perform the commands in the order they are presented. The +examples use placeholder bucket, account, and credential values +throughout - substitute your own. --- -## 2. Bring Up the Stack (Postgres + Lakekeeper) +## Bringing Up the Stack (Postgres + Lakekeeper) -Start the stack from the repo root with a single Compose command: +Our first step begins by starting the stack from the repo root with +a single Compose command. The `docker-compose.yml` file ships in the +ColdFront repository root, so `docker compose` finds it automatically +when you run the command from there: ```bash docker compose up -d --build ``` -This starts **Postgres** (with `pg_duckdb` and `coldfront` preloaded), -**Lakekeeper** (the Iceberg REST catalog), Lakekeeper's **own catalog -Postgres**, and a one-shot **migrate** job. The command does **not** start the -bundled SeaweedFS - that is gated behind the `local-store` profile for -credential-free local evaluation. For cloud S3 you connect to the bucket -directly, so you do not need the bundled SeaweedFS. - If a local Postgres already owns port 5432, pick another host port: ```bash @@ -90,7 +102,7 @@ COLDFRONT_PG_PORT=55432 docker compose up -d --build If port 8181 is taken, set `COLDFRONT_LK_PORT` the same way to move Lakekeeper's host port. Then use that port in place of `8181` in every `curl` -command in Section 3. +command in [Configuring Lakekeeper](#configuring-lakekeeper) below. Wait for Postgres to report healthy (the container name is derived from your directory, so resolve it at runtime): @@ -99,15 +111,35 @@ directory, so resolve it at runtime): docker inspect -f '{{.State.Health.Status}}' "$(docker compose ps -q db)" # => healthy ``` ---- +The command brings up four containers: + +- PostgreSQL, with `pg_duckdb` and `coldfront` already preloaded. +- Lakekeeper, the Iceberg REST catalog. +- Lakekeeper's own catalog database. +- a one-shot migrate job that applies Lakekeeper's schema migrations + and then exits. -## 3. One-Time Lakekeeper Setup +The command does not start the bundled SeaweedFS object-store emulator, +which runs only under the `local-store` profile. For credential-free local +evaluation of the S3-compatible path, start the stack with the profile: -Run the following three calls once each. +```bash +docker compose --profile local-store up -d --build +``` -### 3a. Bootstrap Lakekeeper +If you are connecting to a real S3-compatible store, AWS S3, GCS, or Azure, +connect to it directly and leave the profile off. + +--- -Bootstrap Lakekeeper and accept its terms of use: +## Configuring Lakekeeper + +Next, bootstrap Lakekeeper, then create a warehouse and pre-create the +namespace your Iceberg tables will live in. The warehouse and +credential shape depend on which backend you chose in +[Prerequisites](#prerequisites); follow the matching subsection below. + +First, bootstrap Lakekeeper and accept its terms of use: ```bash curl -X POST http://localhost:8181/management/v1/bootstrap \ @@ -115,11 +147,50 @@ curl -X POST http://localhost:8181/management/v1/bootstrap \ -d '{"accept-terms-of-use":true}' ``` -### 3b. Create the S3 Warehouse +### S3-Compatible Stores (SeaweedFS, MinIO) + +Create a warehouse with a path-style profile pointed at your store's +endpoint: + +```bash +WID=$(curl -s -X POST http://localhost:8181/management/v1/warehouse \ + -H "Content-Type: application/json" \ + -d '{ + "warehouse-name": "wh", + "storage-profile": { + "type": "s3", + "bucket": "my-iceberg-bucket", + "key-prefix": "coldfront", + "region": "us-east-1", + "endpoint": "https://my-store.example.com:9000", + "path-style-access": true, + "flavor": "s3-compat", + "sts-enabled": false, + "remote-signing-enabled": false + }, + "storage-credential": { + "type": "s3", + "credential-type": "access-key", + "aws-access-key-id": "", + "aws-secret-access-key": "" + } + }' \ + | grep -oE '"warehouse-id":"[^"]+"' | cut -d'"' -f4) +echo "$WID" # => the new warehouse's ID +``` + +`region` defaults to `us-east-1` if your store does not use Region +semantics. Set the ColdFront credential with an explicit endpoint: -A warehouse tells Lakekeeper where on S3 your Iceberg tables live and which -credential to use. This is the **virtual-hosted cloud-S3** profile, and the -following flags matter: +```sql +SELECT coldfront.set_storage_secret('', '', + 'my-store.example.com:9000', 'us-east-1', 'path', true); +``` + +### Virtual-Hosted Cloud S3 (AWS) + +This is the **virtual-hosted cloud-S3** profile, and the following +flags matter: - omitting `endpoint` selects native per-Region virtual-hosted addressing over HTTPS. @@ -133,7 +204,7 @@ following flags matter: Create the warehouse with those flags set: ```bash -curl -X POST http://localhost:8181/management/v1/warehouse \ +WID=$(curl -s -X POST http://localhost:8181/management/v1/warehouse \ -H "Content-Type: application/json" \ -d '{ "warehouse-name": "wh", @@ -153,17 +224,34 @@ curl -X POST http://localhost:8181/management/v1/warehouse \ "aws-access-key-id": "AKIAEXAMPLE...", "aws-secret-access-key": "" } - }' -# => HTTP 201 + }' \ + | grep -oE '"warehouse-id":"[^"]+"' | cut -d'"' -f4) +echo "$WID" # => the new warehouse's ID ``` The `key-prefix` is an arbitrary path inside the bucket. `coldfront` is only an example. +Set the cold-tier credential with a `NULL` endpoint, which selects +DuckDB's native per-Region virtual-hosted addressing: + +```sql +SELECT coldfront.set_storage_secret('AKIAEXAMPLE...', '', NULL, 'eu-west-1'); +``` + +!!! warning "The 3rd argument (endpoint) must be NULL for a cloud S3 endpoint" + + A `NULL` endpoint selects DuckDB's native per-Region virtual-hosted + HTTPS + addressing. The 4th argument is your bucket's region, from which DuckDB + derives the endpoint. A non-NULL endpoint applies `url_style` and + `use_ssl`, which default to path-style over plain HTTP - do **not** use + that shape for cloud S3. + A vended-credentials variant is also available. To run ColdFront with no stored -credential ([usage.md](usage.md#vended-credentials)), the warehouse issues -per-table temporary credentials from AWS Security Token Service (STS) instead -of handing the client a static key. Two things change from the warehouse above. +credential (see [Using Vended Credentials](vended_credentials.md)), the +warehouse issues per-table temporary credentials from AWS Security Token +Service (STS) instead of handing the client a static key. Two things change +from the warehouse above. First, create an IAM role scoped to the bucket. Lakekeeper assumes the role per table and vends the resulting short-lived key, secret, and session token to @@ -206,37 +294,137 @@ The role's trust policy above requires `sts:ExternalId`, so the `external-id` in the warehouse credential must match the `` in that condition; Lakekeeper itself makes `external-id` mandatory only for `aws-system-identity` credentials. On the database side, replace the `set_storage_secret(...)` call -in Section 4 with `SELECT coldfront.set_storage_secret_vended();`. +above with `SELECT coldfront.set_storage_secret_vended();`. -### 3c. Pre-Create the `public` Namespace +### Google Cloud Storage -Resolve the warehouse id, then create the `public` namespace under it: +Lakekeeper also has a native `gcs` storage profile, but this guide uses an +`s3` storage profile pointed at GCS's S3-interoperability endpoint, with an +HMAC key pair standing in for an access key. That matches the S3-style +credential that `coldfront.set_storage_secret()` stores for GCS: ```bash -WID=$(curl -s http://localhost:8181/management/v1/warehouse \ - | grep -oE '"warehouse-id":"[^"]+"' | head -1 | cut -d'"' -f4) -[ -n "$WID" ] || { echo "no warehouse id - did step 3b return 201?"; exit 1; } +WID=$(curl -s -X POST http://localhost:8181/management/v1/warehouse \ + -H "Content-Type: application/json" \ + -d '{ + "warehouse-name": "wh", + "storage-profile": { + "type": "s3", + "bucket": "my-iceberg-bucket", + "key-prefix": "coldfront", + "region": "us-east-1", + "endpoint": "https://storage.googleapis.com", + "path-style-access": true, + "flavor": "s3-compat", + "sts-enabled": false, + "remote-signing-enabled": false + }, + "storage-credential": { + "type": "s3", + "credential-type": "access-key", + "aws-access-key-id": "", + "aws-secret-access-key": "" + } + }' \ + | grep -oE '"warehouse-id":"[^"]+"' | cut -d'"' -f4) +echo "$WID" # => the new warehouse's ID +``` + +Lakekeeper's native `gcs` profile is service-account only and is +**not** used here. Set the ColdFront credential the same way as any +other S3-compatible endpoint: + +```sql +SELECT coldfront.set_storage_secret('', '', + 'storage.googleapis.com', 'us-east-1', 'path', true); +``` -curl -X POST "http://localhost:8181/catalog/v1/$WID/namespaces" \ +GCS has no STS equivalent on the interoperability endpoint, so it has +no vended-credentials variant; it always uses this static HMAC key +pair. + +### Azure ADLS Gen2 + +Azure ADLS Gen2 uses its own Lakekeeper warehouse type, `adls`, naming +a storage account and a filesystem (container) within it: + +```bash +WID=$(curl -s -X POST http://localhost:8181/management/v1/warehouse \ -H "Content-Type: application/json" \ - -d '{"namespace":["public"]}' + -d '{ + "warehouse-name": "wh", + "storage-profile": { + "type": "adls", + "filesystem": "my-iceberg-fs", + "account-name": "myaccount" + }, + "storage-credential": { + "type": "az", + "credential-type": "shared-access-key", + "key": "" + } + }' \ + | grep -oE '"warehouse-id":"[^"]+"' | cut -d'"' -f4) +echo "$WID" # => the new warehouse's ID +``` + +The warehouse's `storage-credential` is Lakekeeper's own credential for +reaching the account; it is separate from the credential ColdFront +itself uses. Set ColdFront's credential with +`set_storage_secret_azure()` instead of `set_storage_secret()` - it +takes a CONFIG-provider connection string, with the account key riding +inside `AccountKey=…`: + +```sql +SELECT coldfront.set_storage_secret_azure( + 'DefaultEndpointsProtocol=https;AccountName=myaccount;AccountKey=;EndpointSuffix=core.windows.net'); +``` + +`set_storage_secret_azure()` writes the same `coldfront.storage_secret` row +(replicated, `pg_dump`-excluded) and materializes a `TYPE azure` PERSISTENT +SECRET. The Azure cold tier is subject to the soft-delete / change-feed +restriction in [Caveats](caveats.md). + +A vended-credentials variant is also available: set `sas-enabled` on the +warehouse (on by default) and Lakekeeper vends a per-container SAS +token; the warehouse credential can then be a `shared-access-key`, +`client-credentials`, or `azure-system-identity`. On the database side, +use `SELECT coldfront.set_storage_secret_vended('azure');` instead of +`set_storage_secret_azure()`. + +### Pre-Creating the Namespace + +Regardless of backend, pre-create the `public` namespace under the warehouse, +using the ID that the warehouse-create command saved in `WID`: + +```bash +if [ -z "$WID" ]; then + echo "No warehouse ID - check that the warehouse create succeeded." +else + curl -X POST "http://localhost:8181/catalog/v1/$WID/namespaces" \ + -H "Content-Type: application/json" \ + -d '{"namespace":["public"]}' +fi ``` !!! note "Why this step is required (decoupled mode)" - `coldfront.create_iceberg_table()` (Section 5) runs `CREATE SCHEMA` and - `CREATE TABLE` in one transaction. duckdb-iceberg defers the schema create - to `COMMIT` but sends the table-create POST immediately, so against a - namespace-less warehouse it fails with HTTP 404. Pre-creating `public` + `coldfront.create_iceberg_table()` (in + [Testing with Decoupled Mode](#testing-with-decoupled-mode) below) runs + `CREATE SCHEMA` and `CREATE TABLE` in one transaction. duckdb-iceberg + defers the schema create to `COMMIT` but sends the table-create POST + immediately, so against a namespace-less warehouse it fails with HTTP + 404. Pre-creating `public` makes the in-transaction `CREATE SCHEMA IF NOT EXISTS` a no-op. (Tiered mode's archiver creates the namespace itself, so this is only needed for the decoupled demo below.) --- -## 4. Database Setup +## Creating the Supporting Extensions -Open psql inside the Postgres container: +Next, connect to the server with psql and open a session inside the +Postgres container: ```bash docker exec -it "$(docker compose ps -q db)" psql -U coldfront -d coldfront @@ -258,32 +446,22 @@ CREATE EXTENSION IF NOT EXISTS coldfront; `CREATE EXTENSION` it once. Skip that step and the next call fails with `schema "coldfront" does not exist`. -Set the cold-tier S3 credential once. The signature is -`set_storage_secret(key_id, secret, endpoint, region, url_style, use_ssl)`; the -last four default to `NULL`, `'us-east-1'`, `'path'`, `false`: - -```sql -SELECT coldfront.set_storage_secret('AKIAEXAMPLE...', '', NULL, 'eu-west-1'); -``` - -!!! warning "The 3rd argument (endpoint) must be NULL for a cloud S3 endpoint" - - A `NULL` endpoint selects DuckDB's native per-Region virtual-hosted + HTTPS - addressing. The 4th argument is your bucket's region, from which DuckDB - derives the endpoint. The SeaweedFS form you may have seen elsewhere passes - a non-NULL endpoint and no region - do **not** use that shape for cloud S3. +Now run the `set_storage_secret` (or `set_storage_secret_azure` / +`set_storage_secret_vended`) call from whichever backend subsection you +followed in [Configuring Lakekeeper](#configuring-lakekeeper) above. --- -## 5. Fastest Demo - Decoupled Mode (The Table Lives Entirely in S3) +## Testing with Decoupled Mode A decoupled (iceberg-only) table has no Postgres hot tier; every row lives in -Iceberg on S3, and you read and write it through a normal-looking Postgres -relation. This is the quickest way to prove the whole path works: +Iceberg on your object store, and you read and write it through a +normal-looking Postgres relation. This is the quickest way to prove the whole +path works: ```sql SELECT coldfront.create_iceberg_table( - 'public', 's3_demo', + 'public', 'cold_demo', '[ {"name": "id", "type": "bigint"}, {"name": "ts", "type": "timestamptz"}, @@ -291,73 +469,88 @@ SELECT coldfront.create_iceberg_table( ]'::jsonb, '{month(ts)}'); -INSERT INTO public.s3_demo VALUES (1, now(), 'hello from S3'); -INSERT INTO public.s3_demo VALUES (2, now(), 'second row'); +INSERT INTO public.cold_demo VALUES (1, now(), 'hello from the cold tier'); +INSERT INTO public.cold_demo VALUES (2, now(), 'second row'); -SELECT count(*) AS n, max(note) AS last FROM public.s3_demo; +SELECT count(*) AS n, max(note) AS last FROM public.cold_demo; -- => n = 2, last = 'second row' ``` -That `count(*) = 2` is read back through `iceberg_scan` from your real S3 -bucket - the round trip is complete. +That `count(*) = 2` is read back through `iceberg_scan` from your real +object store - the round trip is complete. -### Tiered Mode - The Headline Feature +### Tiered Mode is the Headline Feature -Decoupled mode is the warm-up. ColdFront's real purpose is **tiered** tables: a -partitioned Postgres table whose hot partitions automatically age out to -Iceberg on S3 once they pass a retention window, after which reads -transparently union live Postgres data with cold S3 data and writes route to -the correct tier. +Decoupled mode is just the warm-up. ColdFront's real purpose is +**tiered** tables: a partitioned Postgres table whose hot partitions +automatically age out to the cold tier once they pass a retention +window, after which reads transparently union live Postgres data with +cold data and writes route to the correct tier. You drive tiered tables with the `archiver` binary against a small YAML config -(Postgres DSN, the `wh` warehouse, your S3 region/keys); each table's lifecycle -is registered with `archiver register`. The credential and warehouse you set up -above are exactly what it needs. See [usage.md](usage.md) for the archiver -config and the partition CLI. +(Postgres DSN, your warehouse name, and your backend's credentials); each +table's lifecycle is registered with `archiver register`. The credential and +warehouse you set up above are exactly what it needs. See +[Writing ColdFront's Configuration](configuration.md#writing-coldfronts-configuration) +for the archiver configuration and +[Managing Partitioned Tables (CLI)](using_coldfront/index.md#managing-partitioned-tables-cli) +for the partition CLI. --- -## 6. Verify in S3 and Troubleshoot +## Verifying the Configuration + +You can confirm the objects physically landed in your object store +using that backend's own client. The following table shows an example +command per backend: + +| Backend | Example verification command | +|---|---| +| S3-compatible / AWS S3 | `aws s3 ls s3://my-iceberg-bucket/coldfront/ --recursive` | +| Google Cloud Storage | `gsutil ls -r gs://my-iceberg-bucket/coldfront/` | +| Azure ADLS Gen2 | `az storage blob list --account-name myaccount --container-name my-iceberg-fs --prefix coldfront --output table` | -Confirm objects physically landed (an S3 client such as the `aws` CLI, region -exported): +For AWS S3, export your region first: ```bash export AWS_DEFAULT_REGION=eu-west-1 aws s3 ls s3://my-iceberg-bucket/coldfront/ --recursive ``` -Iceberg stores each table directly under your `key-prefix`, as -`coldfront//`, with a `data/` directory (parquet) and a `metadata/` -directory (metadata JSON, `*.avro` manifests, snapshot files) - UUID paths, not -your table name. (A namespace created on a Lakekeeper release before 0.13 keeps -its `coldfront///` layout.) "Where did `s3_demo` -go?" → look under the UUID path beneath your `key-prefix`. +Iceberg stores each table directly under your `key-prefix` (S3-family +backends) or filesystem root (Azure), as `coldfront//`, +with a `data/` directory (parquet) and a `metadata/` directory +(metadata JSON, `*.avro` manifests, snapshot files) - UUID paths, not +your table name. (A namespace created on a Lakekeeper release before +0.13 keeps its `coldfront///` layout.) +"Where did `cold_demo` go?" → look under the UUID path beneath your +`key-prefix`. Work through the following checklist if something failed: - Check that you ran `CREATE EXTENSION coldfront` once in your database. Preloading is not the same as creating the extension, and a missing extension produces `schema "coldfront" does not exist`. -- Check that `set_storage_secret(..., NULL, 'eu-west-1')` has `NULL` as its - third argument (native virtual-hosted addressing over HTTPS) and your real - region as its fourth. A non-NULL endpoint applies `url_style` and `use_ssl`, - which default to path-style over plain HTTP. A wrong region sends requests to - another Region's endpoint, which S3 rejects (with HTTP 400 for a bucket in a - Region launched after 2019-03-20). +- For S3-family backends, check that `set_storage_secret(...)` has the right + endpoint argument for your backend: `NULL` for virtual-hosted cloud S3, or an + explicit host for an S3-compatible store or GCS. A wrong region sends + requests to another Region's endpoint, which S3 rejects (with HTTP 400 for a + bucket in a Region launched after 2019-03-20). +- For Azure, check that the connection string passed to + `set_storage_secret_azure()` has the right `AccountName` and `AccountKey`. - Check that the namespace `public` exists in Lakekeeper before `create_iceberg_table` runs. Without the namespace, the decoupled create fails with HTTP 404. - Check that the key is a long-term key, not an SSO or temporary session-token - credential. + credential (S3-family backends). ## Next Steps To go further with ColdFront, consult the following guides: -- The [Using ColdFront](usage.md) guide covers the archiver config, the - partition CLI, and the other storage backends. -- The [Compaction](compaction.md) guide covers cold-tier maintenance on the - bucket. -- The [Architecture](architecture.md) overview describes how the cold tier is +- The [Using ColdFront](using_coldfront/index.md) guide details the archiver config, the + partition CLI, and vended credentials in more detail. +- The [Compaction](compaction.md) guide documents cold-tier maintenance on + the object store. +- The [Architecture](architecture_guides/index.md) overview describes how the cold tier is read and written. diff --git a/docs/stylesheets/extra.css b/docs/stylesheets/extra.css index c46afb00..93b9f038 100644 --- a/docs/stylesheets/extra.css +++ b/docs/stylesheets/extra.css @@ -15,3 +15,9 @@ --md-footer-logo-dark-mode: block; --md-footer-logo-light-mode: none; } + +/* Keep inline code (e.g. GUC names) from wrapping mid-word inside + narrow table columns; prose cells wrap normally. */ +.md-typeset table code { + white-space: nowrap; +} diff --git a/docs/supported_types.md b/docs/supported_types.md new file mode 100644 index 00000000..67c122f6 --- /dev/null +++ b/docs/supported_types.md @@ -0,0 +1,188 @@ +# Supported Column Types + +The following PostgreSQL column types are supported: + +- `bigint`. +- `integer`. +- `smallint`. +- `real`. +- `double precision`. +- `boolean`. +- `timestamp with time zone`. +- `timestamp without time zone`. +- `date`. +- `time without time zone`. +- `uuid`. +- `text`. +- `varchar(N)`. +- `char(N)`. +- `bytea`. +- `numeric(P,S)` (P ≤ 38). +- `jsonb` / `json`. +- `interval`. +- `vector(N)` / `halfvec(N)` (pgvector; see + [Working with Embeddings](usage_vectors.md)). +- one-dimensional arrays of most of these types (see + [Supported Array Types](#supported-array-types)). + +Other data types (unbounded `numeric`, `xml`, `tsvector`, range/multirange types, +custom enums, composite types, pgvector's `sparsevec`, `inet`, `cidr`, `oid`, +and the arrays that [Supported Array Types](#supported-array-types) lists +as refused at registration) are +rejected when a tiered table is registered, and when a decoupled table is +created. ColdFront refuses silent fallback to `varchar` - losing +precision/identity is worse than no support. `inet`/`cidr`/`oid` are +rejected because pg_duckdb cannot process them in any Iceberg-backed query, +so no cast makes them readable; store IP data as `text` and `oid` values as +`bigint` instead (you can still index/compare them; cast on the hot side +only if needed). + +A few supported types need a closer look: + +- Iceberg cannot hold a `numeric` `NaN`, or an `infinity` or `-infinity` + value in a `date`, `timestamp with time zone` or `timestamp without time + zone` column, as array elements too. ColdFront refuses those values with + a check constraint named `coldfront_guard_`; a write that goes to + the cold tier of a tiered or decoupled table is refused by DuckDB, unless the + write reads a + `numeric` `NaN` from a PostgreSQL table and stores it as `0`. +- `char(N)` is stored and read as `varchar`. Values, comparisons, and + `length()` round-trip losslessly, but a cold read comes back unpadded + with `pg_typeof varchar`, because pg_duckdb has no fixed-length `char` + type. Use `text` or `varchar` if blank-padded display matters. +- `json` and `jsonb` are stored as `varchar` in Iceberg (no native + primitive); reads come back as DuckDB's `json`, not the rich PG `jsonb` + type, because Iceberg-backed reads run entirely in DuckDB. +- `interval` is stored as `varchar` in Iceberg, as ` months + days seconds`, the text PostgreSQL and DuckDB both read as the + same value; it is view-cast back to the rich PG type on read. + +The archiver adds the `NaN`/infinity constraints at the first archive pass +that tiers a partition of the table, and validates them once that pass's +bootstrap transaction has committed. A value written before then is accepted, +and each archive pass stops with an error that names the constraint until the +row is fixed; the next pass then validates the constraint and tiers the +table. A column that an `ALTER TABLE` adds through the DDL hook gets its +constraint in the same statement, which PostgreSQL validates against every +row of the hot table, so an `ADD COLUMN` whose default the constraint +refuses, such as `DEFAULT 'NaN'`, fails. A renamed column's constraint takes +the new column name, and a column whose name is longer than 47 bytes gets a +constraint named after a hash of the name. + +A cold `UPDATE ... FROM` or `DELETE ... USING`, a `MERGE` whose source is a +PostgreSQL table, and an `INSERT ... SELECT` into a decoupled table read that +PostgreSQL table through DuckDB's `postgres` extension, which reads a `numeric` +`NaN` as `0`, also inside an array, and drops an array's lower bound. ColdFront +cannot refuse those values on that path: the write stores the changed values +with no error, and its conditions compare the changed values. A condition that +names `'NaN'` fails in such a write, because DuckDB does not accept `'NaN'` as +a `numeric` value. To keep those values out of the cold tier, check the source +table in PostgreSQL before the write, and fix or remove the rows that hold +them. + +The following query counts those rows in a source table `staging` with a +`numeric` column `amount`, a `numeric` array `amounts` and an array `tags`: + +```sql +SELECT count(*) + FROM staging + WHERE amount = 'NaN' + OR 'NaN' = ANY (amounts) + OR array_lower(tags, 1) <> 1; +``` + +List every `numeric` column, `numeric` array and array column of your source +table the same way; a count of `0` means that the write stores those columns +unchanged. + +Queries like `data->>'key'` and `data->'key'` work against `jsonb`, and +ColdFront translates the `::jsonb` cast, `jsonb_array_length`, +`jsonb_build_object` / `jsonb_agg` (and their `json_` twins, which DuckDB +also lacks) and `date_bin` on read: the builders become the `concat` / +`to_json` / `array_agg` form both engines evaluate identically (the result +is JSON-equal to jsonb's rendering, in compact form and in argument order), +and `date_bin` becomes DuckDB's `time_bucket`, which takes the same +arguments and agrees on every fixed-width bucket. The jsonb-only operators +(`?`, `@>`, `<@`, `#>`, `#>>`) and most jsonb functions (`jsonb_typeof`, +`jsonb_extract_path`, `jsonb_extract_path_text`, `jsonb_set`, `jsonb_path_*`, +`jsonb_each`, `jsonb_object_keys`) are not supported on tiered or decoupled +data: DuckDB either lacks them or its same-named function differs in +signature or result. Reach into the document with `->`/`->>`. + +These limits apply only to reads that go through a tiered or iceberg-only view, +which pg_duckdb plans entirely in DuckDB. Ordinary (non-tiered) PostgreSQL +tables are untouched by ColdFront and keep full jsonb support, as does the hot +partition table itself. + +Bound parameters (`$1` from a prepared statement, a driver's extended protocol, +or a plpgsql variable) work in such reads. DuckDB types most placeholders from +their context (`ts > $1`); one that is a direct argument of a DuckDB function +with several overloads (`time_bucket`'s origin, so `date_bin`'s) or any +argument of a table function (`generate_series`) it cannot, so ColdFront plans +such a read from the bound values on every execution instead of caching a +generic plan. Under `plan_cache_mode = force_generic_plan` those reads cannot +run: the forced generic plan has no values and fails with +`only works with DuckDB execution`. + +A hot-only read is the exception. When a `SELECT` reads a tiered view directly +(no join, CTE, or sub-query) and its `WHERE` provably restricts to the hot +tier, ColdFront rewrites it to read the hot partition table in plain +PostgreSQL: the full jsonb operator and function set works, and the query skips +DuckDB entirely. Such a read returns `data` as native `jsonb` rather than the +view's `json`. Reads that span tiers, are cold-only, or reach the view through +a join or sub-query stay in DuckDB, where the limits above apply. + +## Supported Array Types + +A one-dimensional array of a supported type tiers as an Iceberg list of that +type. The following table shows how each array type is stored and how the view +reads it back: + +| PG type | Iceberg/Parquet storage | Reads back as | +|---|---|---| +| `bigint[]`, `integer[]`, `real[]`, `double precision[]`, `boolean[]`, `date[]`, `time[]`, `timestamp[]`, `uuid[]`, `bytea[]`, `numeric(P,S)[]` | A list of the element's storage type. | The same type. | +| `smallint[]` | A list of `INTEGER`. | `integer[]`, as a `smallint` column reads as `integer`. | +| `text[]` | A list of `VARCHAR`. | `text[]`. | +| `varchar(N)[]`, `char(N)[]` | A list of `VARCHAR`. | `character varying[]`, with `char(N)` elements unpadded. | + +The following arrays are refused when a tiered table is registered and when a +decoupled table is created, with an error that names the column: + +- `timestamptz[]`, because pg_duckdb cannot read a `timestamptz[]` column of a + PostgreSQL table, which a tiered table's hot tier is; store `timestamp[]` in + UTC instead. +- `jsonb[]`, `json[]` and `interval[]`, because those types are stored as text + and read back through a cast that an array element does not get. +- arrays of `vector` or `halfvec`, because a vector is itself stored as a list, + and ColdFront stores an array as a list of scalar values. + +An Iceberg list has one dimension and numbers its elements from 1, so ColdFront +refuses an array with more than one dimension or with a lower bound other than +1, such as `'[0:2]={1,2,3}'`. The hot table's check constraint refuses such an +array when it is written. A write to the cold tier refuses it too: a tiered +`INSERT` with an error that names the column, and a bound parameter with an +error that names no column. A write that reads the array from a PostgreSQL +table stores it renumbered from 1 instead. A tiered table with a column +declared with more than one dimension, such as `integer[][]`, passes registration and is refused +at its first archive pass, because pg_duckdb reads a column by its declared +dimensions; a decoupled table refuses such a column when it is created. A +decoupled table's array type is spelled as its element type followed by `[]`, +such as `integer[]`; the spellings `integer[3]` and `integer ARRAY` are +refused. `ALTER TABLE ... ADD COLUMN` cannot add an array column to a tiered +table, because duckdb-iceberg does not add a nested column to an Iceberg table. + +A tiered `INSERT` accepts an array in any spelling, because its source runs in +PostgreSQL. A cold `UPDATE` and a decoupled `INSERT` run in DuckDB, which does +not read PostgreSQL's brace literal: a `'{a,b}'` literal reaches DuckDB as text +and fails to cast. Write the array as `ARRAY['a', 'b']`, or pass it as a bound +parameter. + +Reads through a view run in DuckDB once the table has a cold tier, and the +following array operations work there: + +- `x = ANY(arr)`, `arr @> ARRAY[...]` and `arr && ARRAY[...]`. +- subscripts and slices, which count from 1, such as `arr[1]` and `arr[1:2]`. +- `array_length(arr, 1)` and `unnest(arr)`. + +A brace literal and a bound array parameter fail in such a read, as they do for +a vector, and so does `cardinality()`, which DuckDB defines for maps only. diff --git a/docs/usage.md b/docs/usage.md deleted file mode 100644 index 9c6f5ff7..00000000 --- a/docs/usage.md +++ /dev/null @@ -1,1542 +0,0 @@ -# Using ColdFront - -ColdFront offers two operating modes. Pick one per table; both can coexist in -the same database. The following table compares the two modes: - -| | Tiered (hot + cold) | Decoupled (iceberg-only) | -|---|---|---| -| Where rows live | Recent rows are hot in the PG heap, and archived rows are cold in Iceberg. | Every row is in Iceberg. | -| Setup | Create a partitioned table and let the archiver convert it on the first run that finds a partition older than `hot_period`. | One SQL call, `coldfront.create_iceberg_table(...)`, sets up the table. | -| Archiver | The archiver is required; it runs from cron and moves old partitions to cold. | The archiver is not used. | -| Best when | The workload has a recent-row OLTP part that benefits from PG indexes and transactional ergonomics. | The workload is purely analytic or append-mostly, and you want zero PG storage and stateless compute. | - -Once the table exists, **the SQL surface is identical**: `SELECT`, `INSERT`, -`UPDATE`, `DELETE` all work normally against the relation name (e.g. `events`), -except that a write touching the cold tier rejects `RETURNING`. - -## Prerequisites (Both Modes) - -The setup needs three services: PostgreSQL with the pg_duckdb and coldfront -extensions, Lakekeeper, and any S3-compatible object store (SeaweedFS, MinIO, -GCS, etc.). The one-time setup below brings them up and bootstraps them. - -## One-Time Setup - -Bring up the end-user stack (the example uses SeaweedFS, gated behind the -`local-store` compose profile; host ports are published so the `localhost` -commands below work directly). For the image build itself, see -[installation.md](installation.md): - -```bash -docker compose --profile local-store up -d --build -``` - -Then bootstrap Lakekeeper, create the warehouse, and pre-create the Iceberg -namespace: - -```bash -# 1. Bootstrap Lakekeeper -curl -X POST http://localhost:8181/management/v1/bootstrap \ - -H "Content-Type: application/json" -d '{"accept-terms-of-use":true}' - -# 2. Create warehouse (adjust endpoint/credentials for your S3 store) -curl -X POST http://localhost:8181/management/v1/warehouse \ - -H "Content-Type: application/json" -d '{ - "warehouse-name": "wh", - "storage-profile": { - "type": "s3", "bucket": "iceberg", "region": "us-east-1", - "endpoint": "http://seaweedfs:8333", "path-style-access": true, - "flavor": "s3-compat", "sts-enabled": false, - "remote-signing-enabled": false - }, - "storage-credential": { - "type": "s3", "credential-type": "access-key", - "aws-access-key-id": "admin", "aws-secret-access-key": "adminsecret" - } - }' - -# 3. Create the Iceberg namespace in the new warehouse. -# REQUIRED for decoupled (iceberg-only) mode: the Iceberg CREATE SCHEMA is -# deferred to transaction COMMIT but the CREATE TABLE POST is sent -# immediately, so coldfront.create_iceberg_table - which runs both in one -# transaction - would fail with HTTP 404 against a cold warehouse. -# Pre-creating the namespace here (its own committed REST call) makes the -# function's in-txn CREATE SCHEMA IF NOT EXISTS a no-op so the table create -# succeeds. The archiver (tiered mode) creates the namespace itself and does -# not need this. -WID=$(curl -s http://localhost:8181/management/v1/warehouse \ - | grep -oE '"warehouse-id":"[^"]+"' | head -1 | cut -d'"' -f4) -curl -X POST "http://localhost:8181/catalog/v1/$WID/namespaces" \ - -H "Content-Type: application/json" -d '{"namespace":["public"]}' -``` - -Then install the extensions and set the cold-tier credentials, once per -database: - -```sql -CREATE EXTENSION IF NOT EXISTS pg_duckdb; -CREATE EXTENSION IF NOT EXISTS coldfront; - -SELECT coldfront.set_storage_secret('admin', 'adminsecret', 'seaweedfs:8333'); -``` - -The secret is stored in the `coldfront.storage_secret` table (excluded from -`pg_dump`; in a Spock mesh it replicates by value once added to the default -repset - one-time mesh setup step 4) and materialized as a DuckDB PERSISTENT -SECRET that loads at instance init. There is **no per-session setup**: the -Iceberg catalog `ice` attaches **lazily** by the coldfront C hook on the first -query that touches a tiered/decoupled view (read or write). - -Every `set_storage_secret()` argument after the secret is optional. -`p_endpoint` defaults to `NULL`, `p_region` to `'us-east-1'`, `p_url_style` to -`'path'` (the other value is `'vhost'`), and `p_use_ssl` to `false`. -`p_url_style` and `p_use_ssl` apply only when an endpoint is given. A TLS -endpoint such as GCS's needs `p_use_ssl => true`, and AWS S3 omits the endpoint -and sets `p_region` to the bucket's Region. - -The image writes the server settings ColdFront needs into `postgresql.conf` -when it initializes a new data directory. A server installed from packages or -built another way sets them itself, as the -[package](installation.md#configuring-postgresql) and -[bare-metal](installation.md#building-coldfront-on-bare-metal) sections of -installation.md show: - -- `shared_preload_libraries = 'pg_duckdb,coldfront'` loads both extensions at - server start. coldfront refuses to load any other way: `CREATE EXTENSION - coldfront` fails on a server that does not preload it, with an error that - names the setting. -- `coldfront.warehouse` and `coldfront.lakekeeper_endpoint` name the Lakekeeper - warehouse and its catalog endpoint, which the Iceberg catalog `ice` attaches - to. Only a superuser can set them, and while either is empty the catalog does - not attach. -- `coldfront.local_pg_dsn` is the connection string DuckDB uses to read - PostgreSQL tables from this server, which a decoupled table's - `INSERT … SELECT` from a PostgreSQL table needs, as does a cold write to a - table with a clustered vector column. A tiered `INSERT` streams its cold - rows through it, and renders them row by row when it is not set. Only a - superuser can set or read it. - Set it before calling `set_storage_secret()`, which installs the DuckDB - `postgres` extension this path loads only when the setting is present. - -An application that connects as a non-superuser needs its role granted access -with `coldfront.grant_app_access()`, which takes an existing role: - -```sql -SELECT coldfront.grant_app_access('alice'); -``` - -`coldfront.grant_app_access()` derives the grants from the registry when it -runs, so run it again after you create or adopt a table, or tier one for the -first time. The call fails while `duckdb.postgres_role` is unset, because only -superusers can then run DuckDB. -[Least-Privilege Application Roles](index.md#least-privilege-application-roles) -lists what it grants. - -For a real cloud-S3 setup, see [object_store.md](object_store.md). - -## Mode 1 - Tiered (Hot + Cold) - -Create a partitioned table normally: - -```sql -CREATE TABLE events ( - id bigint GENERATED ALWAYS AS IDENTITY, - ts timestamptz NOT NULL, - status text, - data jsonb, - PRIMARY KEY (id, ts) -) PARTITION BY RANGE (ts); - -CREATE TABLE p_2026_04 PARTITION OF events - FOR VALUES FROM ('2026-04-01') TO ('2026-05-01'); -``` - -The archiver connects the way psql does, from the libpq environment (`PGHOST`, -`PGDATABASE`, `PGUSER`, `PGPASSWORD`, `PGSERVICE`) or `--dsn`, and reads every -other setting from the server: the tables from `coldfront.partition_config`, -the cold-store credential from `coldfront.storage_secret`, and the catalog from -the `coldfront.warehouse` and `coldfront.lakekeeper_endpoint` settings. It -needs no file to run. - -A deployment YAML is a way to write that configuration into the server once, -with `import`: - -```yaml -s3: - endpoint: "seaweedfs:8333" - access_key: "admin" - secret_key: "adminsecret" -archiver: - tables: - - source_table: events - partition_period: monthly - hot_period: "1 month" -``` - -The `import` command writes the file into the server: - -```bash -./bin/archiver import --config config.yaml -``` - -`import` writes the `s3:` or `azure:` stanza through `set_storage_secret` or -`set_storage_secret_azure`, then upserts each table into -`coldfront.partition_config`, validated as `register` validates one table, so -the file and the server agree when it returns. A file may also name -`iceberg.warehouse` and `iceberg.lakekeeper_endpoint`; the import checks those -against the server settings, which live in `postgresql.conf`, before it writes -anything, and refuses the file if they differ. From then on the server is the -configuration: change it with `set`, `register`, `set_storage_secret` or -`postgresql.conf`. A YAML passed to any later run is checked, value by value, -against what the server holds, and any difference is refused with an error -that says where configuration lives. A file that agrees, such as the output of -`export`, runs. `s3.region` defaults to `us-east-1`; -[Storage Backends](#storage-backends) describes the other `s3:` and `azure:` -keys, and a deployment on [vended credentials](#vended-credentials) has no -stanza to import. - -Register a single table in `coldfront.partition_config` without a file: - -```bash -./bin/archiver register --table events \ - --period monthly --hot-period "1 month" -# optional: --retention "5 years" DROPs cold data past that age -# (must exceed --hot-period; omit = keep forever) -``` - -Run the archiver (typically via cron): - -```bash -./bin/archiver -``` - -`--version` prints the version that `make build` stamps from `git describe` -(the release tag, or the commit it was built from) and exits; a plain -`go build` prints `unknown`. The partitioner and compactor accept the same -flag. - -The archiver also accepts `--debug-export-delay`, a test-only Go duration -(such as `5s`) that holds each partition's capture window open between the -bulk export and the replay, so that a test can race concurrent writes into it. -Leave it unset in production. - -The first run that finds a partition older than `hot_period` renames `events` -→ `_events`, creates the unified view `events`, and registers it; until then -the archiver only premakes partitions, and the table stays a plain partitioned -table. From then on every cycle (1) tiers partitions older than `hot_period` -from hot PG to cold Iceberg and advances the watermark, and (2) if -`retention_period` is set, drops cold Iceberg rows older than it. The data -lifecycle is **hot → `hot_period` → cold → `retention_period` → gone**; -omit `retention_period` to keep cold data forever. - -### Inbound Foreign Keys - -A tiered table cannot be the target of an enforced foreign key from another -table over its archivable range. Archiving physically removes the rows from -PostgreSQL (export to Iceberg, then `DETACH` + `DROP` the partition), and -PostgreSQL cannot enforce a foreign key against the cold tier. So the `DETACH` -is refused (SQLSTATE 23503) and the archiver fails fast, naming the blocking -constraint. - -This is inherent, not an edge case: referencing rows rarely have a hot-only -lifecycle. They usually outlive the partition they point at, so the foreign key -still pins those rows in PostgreSQL by the time the partition ages into the -cold tier. - -The foreign key is fundamentally incompatible with archiving the rows it -references, so before archiving such a table, drop it: - -```sql -ALTER TABLE event_logs DROP CONSTRAINT event_logs_events_fkey; -``` - -Better still, do not point an enforced foreign key at a tiered table over its -archivable range in the first place. - -### 2-Level (LIST → RANGE) Tiered Tables - -A table partitioned `LIST (region) → RANGE (ts)` can be tiered too - the same -`sub_partition` block as the standalone partition manager (Mode 3), so a -partition-manager-managed table can be "upgraded" to tiered by pointing the -archiver at it: - -```yaml - - source_table: regional - partition_column: ts # the RANGE (time) column - required for 2-level - partition_period: monthly - hot_period: 1 month - # retention_period: 5 years # optional cold expiry (region-agnostic, by ts) - sub_partition: - values_source: "SELECT region FROM regions" -``` - -One Iceberg table holds every region (region is an ordinary column); the -archiver tiers leaves a whole `ts` period at a time across **all** regions -before advancing the shared hot/cold watermark, so a period only becomes cold -once it is cold for every region. `id` mode is not supported in tiered mode -(the cold tier is time-keyed). For why the ordering works that way, see the -[Two-level tiering](architecture_tiered.md#two-level-list-range-tiering) -section of the Tiered Mode page. - -## Mode 2 - Decoupled (Iceberg-Only) - -A single call creates a decoupled table: - -```sql -SELECT coldfront.create_iceberg_table( - p_schema => 'public', - p_table => 'events', - p_columns => '[ - {"name":"id", "type":"bigint"}, - {"name":"ts", "type":"timestamptz"}, - {"name":"status", "type":"text"}, - {"name":"data", "type":"jsonb"} - ]'::jsonb, - p_partition_cols => '{month(ts)}' -); -``` - -`p_partition_cols` partitions the Iceberg table, here by the month of `ts`: -each month's rows land in their own data files, and a query with a time filter -skips the months outside it before reading anything. Each element is one term -as DuckDB's `PARTITIONED BY` takes it: a column name, `year(col)`, `month(col)` -or `day(col)` on a timestamp or date column, `hour(col)` on a timestamp column, -`bucket(N, col)` or `truncate(W, col)`; `'{month(ts), region}'` partitions by -both. The argument is a PostgreSQL array literal, so a term that contains a -comma is double-quoted inside it, the comma being the array's delimiter: -`'{"bucket(16, id)"}'`. A column name that itself needs double quotes has them -backslashed inside such an element: `'{"month(\"Event Time\")"}'`. Leave the -argument out for an unpartitioned table. - -That single statement provisions: - -- `ice.public.events` on the attached Iceberg catalog. -- a PG-side wrapper view `public.events` with proper PG-typed columns. - -- a `coldfront.tiered_views` registry row, so that the coldfront C hook - intercepts every `INSERT`, `UPDATE`, `DELETE`, and `MERGE` on the view and - rewrites each one to a single `duckdb.raw_query(...)` against - `ice.public.events`. - - -In a mesh one node provisions: Spock's `ddl_sql` repset replicates the -`CREATE VIEW`, and the `default` repset replicates the name-keyed registry row, -which enables the write hook on every peer. Each node also needs the one-time -setup call `coldfront.ensure_replicated()` - see the one-time mesh setup -below. - -### Changing a Decoupled Table's Columns - -An `ALTER TABLE` on the wrapper view adds, drops, renames or widens a column -of the Iceberg table, and the view follows, with its owner and table-level -grants kept; a column-level grant on the view is not kept. Only the view's -owner, or a role holding its privileges, can change its columns, and that role -needs the cold access `grant_app_access` gives, since the Iceberg change runs -as that role. The following statements change the columns of the table created -above: - -```sql -ALTER TABLE public.events ADD COLUMN qty integer; -ALTER TABLE public.events ALTER COLUMN qty TYPE bigint; -ALTER TABLE public.events RENAME COLUMN qty TO quantity; -ALTER TABLE public.events DROP COLUMN quantity; -``` - -A column takes the same type map as at creation, though an array column cannot -be added, because duckdb-iceberg does not add a nested column to an Iceberg -table. A type change is limited to the widening Iceberg accepts: `integer` to -`bigint`, `real` to `double precision`, `date` to `timestamp`, and a wider -`numeric` precision. An added column takes a name and a type only: a default, -a constraint, a collation, or a storage or compression option is refused, as -is a `USING` or `COLLATE` clause on a type change, and a column change shares -an `ALTER TABLE` only with other column changes. A vector column is neither -added nor changed, a table adopted read-only refuses every column change, and -an object built on the view, such as another view, makes a column change -fail. In a mesh, the change runs on one node: the Iceberg table is shared, and -the rebuilt view and registry row replicate. - -### Adopting a Table That Already Exists in the Catalog - -A table another engine wrote needs no provisioning, only a wrapper view and a -registry row. `coldfront.adopt_iceberg_table()` reads the schema from the -catalog and builds both, so the call names the relation and nothing else: - -```sql -SELECT coldfront.adopt_iceberg_table( - p_schema => 'public', - p_table => 'orders', - p_namespace => 'lake' -); -``` - -`p_schema` is the PostgreSQL schema the wrapper view goes in, and it must -exist. `p_namespace` is the Iceberg namespace the table lives in, which need -not exist as a PostgreSQL schema; it defaults to `p_schema` when omitted. The -view takes the table's name. - -Adoption is read-only by default, so reading someone else's lake table cannot -become writing it by accident. Passing `p_writable => true` enables the same -`INSERT`, `UPDATE`, `DELETE` and `MERGE` rewrite a created table gets: - -```sql -SELECT coldfront.adopt_iceberg_table('public', 'orders', 'lake', - p_writable => true); -``` - -Because Iceberg records no PostgreSQL type, an adopted column reads as whatever -its storage type maps to; a `jsonb` column comes back as `text`. Pass `p_types` -to restore one, provided the override maps to the type the catalog already -stores: - -```sql -SELECT coldfront.adopt_iceberg_table('public', 'orders', 'lake', - p_writable => true, - p_types => '{"meta":"jsonb"}'::jsonb); -``` - -Adoption binds the name once. A second call under a registered name is refused -whatever its arguments; to enable writes or change an override, release the -table with `coldfront.release_iceberg_table()` and adopt it again. A -non-superuser deployment runs `coldfront.grant_app_access()` after adopting, -because the grant reads the registry at call time. - -In a mesh one node adopts. The wrapper view replicates through Spock's -`ddl_sql` repset and the registry row through the `default` repset (see the -one-time mesh setup below), so every peer reads and writes the table under the -same name, and a peer's own adopt is refused as already registered. Adoption -takes the table's bakery claim around that check and the registry write, so two -nodes adopting the same table at the same moment are serialized: the second -waits, then reads the first's row and refuses, rather than both registering the -one table. A writer outside ColdFront is outside the bakery and can still -collide at Lakekeeper. - -One relation can be registered per Iceberg table. A namespace that needs -quoting, such as `Lake-EU`, works. - -## Mode 3 - Standalone Partition Manager (No Cold Tier) - -**You do not need Iceberg at all.** If automated PostgreSQL partition -maintenance is all you want - declarative time- or id-based RANGE partitioning -with a premade forward window and automatic age-out of old partitions - -ColdFront's `partitioner` binary is the whole product: stock PostgreSQL (or a -Spock mesh), no cold tier, no DuckDB, no Iceberg, nothing to preload. Each -invocation makes one reconcile pass per managed table: premake the forward -window, ensure the partition covering *now* exists, and detach-then-drop -partitions past retention (`DETACH ... CONCURRENTLY`, never a bare `DROP` of -attached data). Build the partitioner with `make build` and run it from cron: - -```bash -go build -o bin/partitioner ./cmd/partitioner # or: make build -./bin/partitioner -``` - -A database set up this way can gain the cold tier later: -`CREATE EXTENSION coldfront` adopts the `coldfront.partition_config` the -partitioner created, registrations included. - -The partitioner connects from the libpq environment or `--dsn`, like psql, and -reads its tables from `coldfront.partition_config`; it needs no file. A -partition-only deployment has no cold tier: a table with a `hot_period` -belongs to the archiver, and the partitioner leaves it alone. - -Register each managed table in `coldfront.partition_config`: - -```bash -./bin/partitioner register --table events \ - --period monthly --retention "12 months" \ - --strategy drop # drop (DETACH+DROP, destroy; default) - # | detach (DETACH only, keep as a - # standalone table - data preserved) -``` - -### Operating It - -Schedule one pass per period or more often - a cron line, or a systemd -`oneshot` service plus timer (systemd then reports a failed pass as a failed -unit, so alerting is free): - -```text -17 * * * * postgres PGDATABASE=mydb /usr/local/bin/partitioner >> /var/log/coldfront-partitioner.log 2>&1 -``` - -Keep the following operational behavior in mind when scheduling it: - -- The exit code is `0` when every table reconciled (a self-healed *behind* - condition still exits `0`) and non-zero when at least one table failed - (`N table(s) failed`), each failure logged with a `[table]` prefix (the bare - table name) followed by the schema-qualified error. Alert on a non-zero exit - code. -- If a table already has a *past* partition but none covers *now* at the start - of a pass (a lagging cron - live inserts had no home), the pass heals the - table (creates the current partition), logs a `WARNING (self-healed)` line, - and still exits `0`. Monitor for that warning in the log rather than via the - exit code, and widen `future_partitions` or run more often. A fresh table - (only just-premade future partitions) is **not** behind: its first reconcile - succeeds cleanly. -- With the default `expiration_strategy: drop`, expiry is - `DETACH CONCURRENTLY` + `DROP TABLE` - the data is **gone**, so back up - before shrinking `retention_period`. Set `expiration_strategy: detach` to - instead leave the expired partition as a standalone table (detached from the - parent, data preserved) and reclaim it yourself. A partition is expired only - once its *entire* range is older than `now − retention_period`, computed - with calendar-accurate PostgreSQL interval arithmetic (a real month, leap - years correct). `detach` is partition-only - the tiered archiver always drops - after exporting to cold. - -### Primary Keys on Time-Partitioned Tables (id Mode) - -PostgreSQL requires a unique index to include the partition key, so a plain -`PRIMARY KEY (id)` is impossible on a table partitioned by a separate `ts` -column. Partition instead by RANGE on a **time-ordered id** and the partition -key *is* the key - a single-column `PRIMARY KEY (id)`: - -```yaml - - source_table: events - partition_column: id - partition_period: monthly - retention_period: 12 months - part_mode: id - id_scheme: snowflake # snowflake | uuidv7 -``` - -`uuidv7` reads the RFC 9562 leading-millisecond timestamp; `snowflake` decodes -the pgEdge Snowflake extension's layout. Either way the manager computes the -month/day partition bounds as id values, so id-order equals time-order and the -forward/retention schedule is unchanged. - -### Two-Level (LIST → RANGE) Sub-Partitioning - -For a table partitioned by `LIST (region)` whose children are themselves -`RANGE`-partitioned by time, add a `sub_partition` block. `values_source` is a -query returning the current level-1 values; the manager provisions and -maintains a RANGE sub-tree under each, creating sub-trees for newly appearing -values automatically: - -```yaml - - source_table: events - partition_column: ts # the level-2 RANGE column - partition_period: monthly - retention_period: 12 months - sub_partition: - values_source: "SELECT code FROM regions" -``` - -Each level-1 child is named `
_`, with the value lowercased and -every character outside `a-z`, `0-9` and `_` replaced by `_`. That name must -fit in 50 bytes, which leaves room for the daily leaf suffix within -PostgreSQL's 63-byte identifier limit. A pass that meets a longer name fails, -for tiered two-level tables as well. The partitioner and the archiver also -fail a pass in which two values map to the same name. - -## Dropping an Iceberg Table (Both Modes) - -`coldfront.drop_iceberg_table()` removes the Iceberg table backing a registered -relation. The function is the only sanctioned way to do so: a plain -`DROP TABLE` or `DROP VIEW` on a registered relation stays blocked, because it -would leave the cold tier behind with nothing pointing at it. - -The call takes the schema, the table, and an explicit purge decision: - -```sql --- drop the catalog entry and delete the stored objects -SELECT coldfront.drop_iceberg_table('public', 'events', true); - --- drop the catalog entry and leave the objects in the object store -SELECT coldfront.drop_iceberg_table('public', 'events', false); -``` - -What remains afterwards depends on the mode, and the call reports which path it -took: - -- in decoupled mode the Iceberg table is the whole relation, so nothing remains - in PostgreSQL. -- in tiered mode the Iceberg table is the cold tier, so the cold tier is - removed and the hot table returns under the relation's own name, as an - ordinary partitioned table holding the data that had not yet aged out. - -In tiered mode the call also deletes the table's `partition_config` row, which -stops the archiver from tiering it again on the next run. - -There is no default for the purge argument, because the two outcomes are -irreversible in opposite directions. Passing `true` deletes the Parquet and -metadata objects, which for the cold tier are the only copy of that data. -Passing `false` keeps those objects but removes the catalog entry that -ColdFront would need to reach them again, so nothing reclaims them afterwards; -use it when another system is taking ownership of the files. - -ColdFront has no guard of its own against dropping the wrong table. The -preventive control lives in Lakekeeper: table protection has to be enabled -manually, per table, and a protected table cannot be dropped at all, whether or -not purge is requested. This function cannot override a hold. - -Deletion is not instantaneous. The catalog entry disappears with the call, and -Lakekeeper's own background queue removes the objects shortly afterwards, so a -listing taken immediately after the call can still show them. - -### Handing an Adopted Table Back - -An adopted table is released rather than dropped, because ColdFront does not -own it. `coldfront.release_iceberg_table()` removes the wrapper view, the -registry row and the relation's vector configuration, and performs no Iceberg -I/O, so the table keeps every row and stays in the catalog: - -```sql -SELECT coldfront.release_iceberg_table('public', 'orders'); -``` - -Release refuses a tiered registration, because removing one would leave its -cold rows unreachable while the hot table returned under the relation's name. -`drop_iceberg_table()` refuses a relation adopted read-only, for the mirrored -reason: read access gives no authority to destroy. - -## Managing Partitioned Tables (CLI) - -ColdFront's configuration lives in the server. **Per-table lifecycle** lives in -`coldfront.partition_config`, a name-keyed table that replicates by value -across a Spock mesh (like `tiered_views`/`archive_watermark`), so every node -reads identical config - no per-node file syncing. The cold-store credential -lives in `coldfront.storage_secret` and the catalog in the -`coldfront.warehouse` and `coldfront.lakekeeper_endpoint` settings. The -**connection** alone comes from outside, from the libpq environment or -`--dsn`, as for psql. Manage the lifecycle table with the CLI below (both -`partitioner` and `archiver` expose these subcommands; with no subcommand they -do their normal reconcile/archive run). - -`register` is the primary way to manage tables: one command adds or adopts one -table, validated on the spot. `import` and `export` are bulk helpers for -(re)configuring a machine - write a deployment YAML into a fresh server, or -dump the live config to git and replay it on another node - not the day-to-day -path. A YAML passed to anything but `import` is checked against the server and -refused if it disagrees in any value. `register` and `import` run the full -validation. `set` checks every value of the row that it would leave; it re-runs -the rest of the validation only when it changes the partition period or column, -`hot_period`, `retention_period` or the sub-partition source. - -The data lifecycle is **hot PG → `hot_period` → cold Iceberg → -`retention_period` → dropped** (tiered) or **hot PG → `retention_period` -→ dropped** (partition-only). Setting `hot_period` makes a table tiered; -omitting it makes it partition-only. - -### What Registration Refuses - -`register` and `import` run the same validation, so a table that one rejects -cannot be added by the other; `set` re-runs it when it changes a validated -field. Registration fails when: - -- a setting has a value that the table's `CHECK` constraints refuse. The - refusal names the flag, or on `import` the key, and the values that the - setting accepts; `set` checks the row that its change would leave. - Registration refuses: - - - a `--period` other than `monthly` or `daily`. - - a `--premake` below 1. - - a `--part-mode` other than `timestamp` or `id`. - - an `--id-scheme` other than `uuidv7` or `snowflake`, or one without - `--part-mode id`. - - `--part-mode id` without an `--id-scheme`. - - a `--strategy` other than `drop` or `detach`. - - `--part-mode id` or `--strategy detach` with a `--hot-period`. - - a `--sub-values-source` without a `--column`. - - a table with neither a `--hot-period` nor a `--retention`. - -- `--column`, or `partition_column` on `import`, names a column that is not the - table's partition key. On a two-level table, registration refuses only a - column that the table lacks. The refusal names the flag or key and, for a - flat table, the table's partition key. -- any relation in the partition tree is `UNLOGGED`, because that data is - truncated after a crash and replicates nowhere, so archiving it would - preserve rows PostgreSQL never promised to keep. -- the table has a `DEFAULT` partition. Rows the `DEFAULT` partition catches - have no time bounds, so they can never be tiered or expired, and PostgreSQL - refuses `DETACH PARTITION ... CONCURRENTLY` for every partition of a table - that has one, which is how partitions are expired. The partition's mere - existence is enough; it does not have to hold any rows. Move any rows the - `DEFAULT` partition holds into real partitions and detach it. -- the name differs only by case from an already-registered table. PostgreSQL - keeps `public."Events"` and `public.events` apart, but DuckDB matches - identifiers case-insensitively even when they are quoted, so both names would - resolve to one Iceberg table and overwrite each other. -- the name starts with an underscore, which is reserved for the tiered hot - table. -- the name leaves no room for the generated partition suffix: 53 bytes for - monthly and 50 for daily, within PostgreSQL's 63-byte identifier limit. -- a tiered table has a column whose type has no Iceberg equivalent (see - [Supported Column Types](#supported-column-types)). Every column goes through - the same type map the cold tier itself uses, so the answer comes back at the - prompt rather than hours later from cron, and the refusal names the column. - Partition-only tables are exempt: nothing about them reaches Iceberg, so - their column types are PostgreSQL's business alone. -- a tiered table's name, or the name of one of its columns, is one DuckDB - parses as a keyword, such as `by`, `at` or `show` (see [Caveats](#caveats)). - The archiver checks the column names again before it tiers or expires data; - it refuses a column added after registration before any table or data reaches - Iceberg. - -Registration validates the table as it is at that moment, not continuously. -Adding a `DEFAULT` partition to an already-registered table is therefore not -caught then; the next archiver run fails while reading that table's partition -bounds. - -Other unusual bounds are supported rather than refused. Partitions open at -either end (`MINVALUE`, `MAXVALUE`, `infinity`) are handled, as is PostgreSQL's -full timestamp range, from 4713 BC through year 294276. - -`hot_period` and `retention_period` are native PostgreSQL `interval`s - use any -interval syntax (`1 month`, `90 days`, `1 year 2 mons`, `5 years`). Expiry -boundaries are computed with calendar-accurate interval arithmetic -(`now() - period`: real months, leap years), and `retention_period` must exceed -`hot_period` (validated at `register`/`set` time). - -The following table describes the CLI subcommands: - -| Command | Purpose | -|---|---| -| `register` | Adds or adopts a table, and validates that the PRIMARY KEY covers the partition key. | -| `list` | Shows the managed tables and their lifecycle. | -| `set` | Changes fields, or disables or enables a table with `--disable`/`--enable`. | -| `remove` | Stops managing a table; the table itself is left intact. | -| `import` | Writes a deployment YAML into the server once: its `archiver.tables` as rows, validated as `register` validates each table, and its `s3:` or `azure:` stanza as the storage secret. | -| `export` | Dumps the active (enabled) config to YAML or SQL, as a git-reviewable backup to replay on another node. | - -The following examples register, inspect, and change managed tables: - -```bash -# Partition-only: keep 3 future partitions, drop those older than 12 months. -partitioner register --table events --period monthly --retention "12 months" - -# Partition-only, but DETACH (preserve) expired partitions instead of dropping them. -partitioner register --table events --period monthly \ - --retention "12 months" --strategy detach - -# Tiered: tier to cold Iceberg after 1 month, then drop cold data after 5 years. -archiver register --table events --period monthly \ - --hot-period "1 month" --retention "5 years" - -# id mode - a real single-column PRIMARY KEY (id) on a snowflake-keyed table. -partitioner register --table events --period monthly \ - --column id --part-mode id --id-scheme snowflake --retention "1 year" - -# 2-level LIST(region) → RANGE(ts), tiered; region values come from a table. -archiver register --table regional --period monthly --column ts \ - --hot-period "1 month" --sub-values-source "SELECT region FROM regions" - -partitioner list # what's managed -partitioner set --table events --retention "24 months" -partitioner set --table events --disable # pause (keeps the row) -partitioner remove --table events # unregister, keep the table -partitioner import --config deploy.yaml # write a deployment YAML into the server once -partitioner export > managed.yaml # active config, git-reviewable (--format sql for INSERTs) -``` - -Run `partitioner` (or `archiver`) with `help` or `--help` for the command -overview, and with no arguments for its normal run; every subcommand has a -detailed `--help` with worked examples. The write commands accept -`--print-sql` (emit the SQL without running it - review/commit it; `import` -prints its table `INSERT`s); `register` and `import` also accept `--dry-run`. -`set --enable`/`--disable` (mutually exclusive) pause/resume a table without -removing it; a disabled table is skipped by reconcile and omitted from -`export`. Per table, only the cadence and a lifecycle boundary (`hot_period` or -`retention_period`) are required; `partition_column` is auto-detected from -`pg_catalog` for flat tables (required for 2-level). `register` checks the -lifecycle rules before it writes the row, and the table's `CHECK` constraints -enforce them again at write time. - -The subcommands also take the following connection and table flags: - -- `--dsn` takes a PostgreSQL connection string on every subcommand. Without - it the connection comes from the libpq environment, as for psql. -- `--config` names a deployment YAML. On `import` it is required, since it - names the file to write into the server. On any other subcommand the file is - checked against the server, value by value, and refused if it disagrees; its - `postgres.dsn` connects when `--dsn` is unset. -- `--schema` (default `public`) names the table's schema on `register`, `set` - and `remove`. -- `--premake` (default `3`) sets on `register` how many future partitions are - kept ahead of now, stored as `future_partitions`. - -`set` changes only the fields whose flags are passed: `--period`, `--column`, -`--premake`, `--hot-period`, `--retention`, `--sub-values-source` and -`--strategy`, each taking the same value as on `register`. An empty value -clears the field, so `--hot-period ""` makes a tiered table partition-only and -hands it to the partitioner. A change that leaves neither a `hot_period` nor a -`retention_period` is refused. - -For managing tables, the archiver and partitioner read the managed set only -from `coldfront.partition_config`. `register` adds one table; `import` adds -every table in a YAML `archiver.tables` list. Both write through the same -validation (the PK must cover the partition key; `retention` must exceed -`hot_period`), so an imported table is checked exactly as a registered one. -`export` dumps the active rows back to YAML or SQL for git. - -The archiver manages only the rows that have a `hot_period`, and the -partitioner only the rows without one, so each table belongs to one binary. A -run that finds none of its own rows exits non-zero with -`no tables in coldfront.partition_config`, naming the `register` and `import` -commands that add one. - -In an `archiver.tables` entry, `source_schema` names the table's schema and -defaults to `public`. `source_table` also accepts the `schema.table` form, and -an explicit `source_schema` takes precedence over that prefix. -`future_partitions` defaults to `3`, as `--premake` does on `register`. - -## Storage Backends - -Configure **exactly one** of the following cold-store backends, through -`set_storage_secret` or through the `s3:` or `azure:` stanza of a deployment -YAML, which `import` writes through the same setters. - -### S3 - -The S3 backend works with any S3-compatible store (SeaweedFS, MinIO). Set -`endpoint`, `use_ssl: true` for a TLS endpoint, and `url_style: path` (default) -or `vhost`. `region` defaults to `us-east-1`. - -### Virtual-Hosted Cloud S3 - -For virtual-hosted cloud S3 (AWS S3 is the canonical one), **omit `endpoint`** -(and the `endpoint` arg to `set_storage_secret`) so DuckDB uses the native -per-Region virtual-hosted + HTTPS addressing; set `region` to your bucket's -Region. DuckDB derives the endpoint from `region`, and a bucket in a Region -launched after 2019-03-20 (e.g. `ap-south-2`) rejects requests sent to another -Region's endpoint with HTTP 400. The Lakekeeper warehouse profile must be a -virtual-hosted `s3` profile (`flavor: aws`, `path-style-access: false`, no -custom endpoint); the full walkthrough is [object_store.md](object_store.md). - -### Google Cloud Storage - -Google Cloud Storage is *not a separate backend*: use `s3:` pointed at GCS's -S3-interoperability endpoint with an -[HMAC key pair](https://cloud.google.com/storage/docs/authentication/hmackeys) -(`endpoint: storage.googleapis.com`, `use_ssl: true`, -`access_key`/`secret_key` = the HMAC id/secret). Lakekeeper's warehouse uses an -`s3` profile (`flavor: s3-compat`, `path-style`) at the same endpoint. -ColdFront's test suite covers this setup end to end, reading and writing -Iceberg through the interoperability endpoint. Lakekeeper's native `gcs` -profile is service-account only and is **not** used. - -### Azure ADLS Gen2 - -Azure ADLS Gen2 is a supported cold-store backend; the access key rides inside -`connection_string`. Set the credential with `set_storage_secret_azure()` -instead of `set_storage_secret()` - it takes a CONFIG-provider connection -string. The storage-account access key rides inside `AccountKey=…`; the -DuckDB azure secret has no separate account-key parameter, so shared-key auth -lives entirely in the connection string: - -```sql -SELECT coldfront.set_storage_secret_azure( - 'DefaultEndpointsProtocol=https;AccountName=;AccountKey=;EndpointSuffix=core.windows.net'); -``` - -`set_storage_secret_azure()` writes the same `coldfront.storage_secret` row -(replicated, `pg_dump`-excluded) and materializes a `TYPE azure` PERSISTENT -SECRET. The Azure cold tier is subject to the soft-delete / change-feed -restriction in [Caveats](index.md#caveats). - -The storage account must have hierarchical namespace enabled, for example with -`az storage account create --hns true`. A StorageV2 account without -hierarchical namespace is not an ADLS Gen2 account, and Lakekeeper's `adls` -warehouse fails on it. - -## Vended Credentials - -Vended credentials let a deployment run with no object-store credential stored -in the database, in a DuckDB secret file, or in an archiver config; this suits -compliance environments that forbid persisting long-term keys. Lakekeeper -issues a short-lived, per-table credential (an S3 STS access key, secret, and -session token) at read and write time, and ColdFront uses it directly. The -long-term credential lives only in the Lakekeeper warehouse. - -Enable vended credentials with a single call that stores no credential: - -```sql -SELECT coldfront.set_storage_secret_vended(); -``` - -This writes a `coldfront.storage_secret` row marked as vended, so nothing is -materialized as a DuckDB secret; `ensure_attached()` then attaches the catalog -with credential vending turned on. The archiver and the compactor read the -same row, so a vended deployment has no `s3:` or `azure:` stanza to import and -no credential in any file. - -Vended mode targets the two clouds that issue scoped credentials: AWS S3 (STS) -and Azure ADLS Gen2 (SAS). The `set_storage_secret_vended()` call above enables -AWS S3; the same call with `'azure'` enables Azure through the identical path: - -```sql -SELECT coldfront.set_storage_secret_vended('azure'); -``` - -Vending requires a Lakekeeper warehouse configured to vend credentials: - -- On AWS S3, the warehouse uses `flavor: aws` with `sts-enabled: true`, an - `assume-role-arn` for a bucket-scoped IAM role, and an `external-id` on the - warehouse credential that the role's trust policy also requires. The full - warehouse and IAM-role setup is in - [object_store.md](object_store.md#3b-create-the-s3-warehouse). -- On Azure ADLS Gen2, the warehouse is an `adls` warehouse with `sas-enabled` - (on by default). Lakekeeper vends a per-container SAS token; the warehouse - credential can be a `shared-access-key`, `client-credentials`, or - `azure-system-identity`. - -Google Cloud Storage over the S3-interoperability endpoint has no STS to issue -short-lived credentials, so GCS stays on static HMAC credentials. - -The compactor runs fully under vended credentials: compaction, snapshot expiry, -and orphan-file reclaim all use the vended per-table credentials. - -Switching a running deployment between static and vended credentials changes -the attach mode, which is fixed per PostgreSQL backend at attach time; open a -new session (or restart the archiver and compactor, which are short-lived -processes) so the new mode takes effect. - -## Reading + Writing (Identical for Both Modes) - -The same SQL works against the relation name in either mode, as the following -examples show: - -```sql --- Reads - pg_duckdb handles the iceberg side; PG handles the heap side -SELECT count(*) FROM events; -SELECT id, status, data->>'k' FROM events WHERE ts >= '2026-04-01'; - --- Inserts, updates, and deletes all go through the coldfront C hook, --- which rewrites the query into one PG-set-based statement (hot side) --- + duckdb.raw_query calls (cold side). For iceberg-only mode every write --- goes cold; for tiered mode the hook splits by ts vs the watermark. -INSERT INTO events (ts, status, data) VALUES (now(), 'ok', '{"k":1}'); -UPDATE events SET status = 'fixed' WHERE id = 123; -DELETE FROM events WHERE ts < '2025-01-01'; - --- Bulk INSERT shapes: the source is read once, the hot rows are one --- set-based INSERT and the cold rows are written in batches: -INSERT INTO events (ts, status, data) VALUES (...), (...), (...); -INSERT INTO events (ts, status, data) SELECT ts, status, data FROM staging; -INSERT INTO events (ts, status, data) SELECT now() + i*'1s'::interval, 'ok', '{}' - FROM generate_series(1, 1000) i; --- A WITH clause works in both positions; a nested INSERT takes no RETURNING: -WITH moved AS (DELETE FROM staging RETURNING ts, status, data) -INSERT INTO events (ts, status, data) SELECT ts, status, data FROM moved; -WITH i AS (INSERT INTO events (ts, status, data) VALUES (now(), 'ok', '{}')) -SELECT 1; --- COPY FROM loads the same way, one INSERT per cold_write_batch_size rows: -COPY events (ts, status, data) FROM '/path/to/events.csv' WITH (FORMAT csv); - --- Transactions work; ROLLBACK undoes Iceberg writes too -BEGIN; - UPDATE events SET status = 'pending' WHERE id = 1; - SELECT status FROM events WHERE id = 1; -- sees 'pending' if row 1 is hot -ROLLBACK; -SELECT status FROM events WHERE id = 1; -- back to whatever it was -``` - -## Supported Column Types - -The following PostgreSQL column types are supported: - -`bigint` · `integer` · `smallint` · `real` · `double precision` · -`boolean` · `timestamp with time zone` · `timestamp without time zone` · -`date` · `time without time zone` · `uuid` · `text` · `varchar(N)` · -`char(N)` · `bytea` · `numeric(P,S)` (P ≤ 38) · `jsonb` / `json` · -`interval` · `vector(N)` / `halfvec(N)` (pgvector; see -[usage_vectors.md](usage_vectors.md)) · one-dimensional arrays of most of -these types (see [Arrays](#arrays)) - -Anything else (unbounded `numeric`, `xml`, `tsvector`, range/multirange types, -custom enums, composite types, pgvector's `sparsevec`, and the arrays that -[Arrays](#arrays) lists as refused at registration) is rejected when a tiered -table is registered, and when a decoupled table is created. ColdFront refuses -silent fallback to `varchar` - losing precision/identity is worse than no -support. - -Iceberg cannot hold a `numeric` `NaN`, or an `infinity` or `-infinity` in a -`date`, `timestamp` or `timestamptz` column, so ColdFront refuses those values, -as array elements too. A tiered table's hot table has a check constraint named -`coldfront_guard_` on each column of those types and on each array -column; an `INSERT` or `UPDATE` that stores one of the values fails with an -error that names the constraint. A write that goes to the cold tier, in either -mode, is refused by DuckDB, unless the write reads the value from a PostgreSQL -table, as -[Reading a PostgreSQL Table in a Cold-Tier Write](#reading-a-postgresql-table-in-a-cold-tier-write) -describes. - -The archiver adds the constraints at the first archive pass that tiers a -partition of the table, and validates them once that pass's bootstrap -transaction has committed. A value written before then is accepted, and each -archive pass stops with an error that names the constraint until the row is -fixed; the next pass then validates the constraint and tiers the table. A -column that an `ALTER TABLE` adds through the DDL hook gets its constraint in -the same statement, which PostgreSQL validates against every row of the hot -table, so an `ADD COLUMN` whose default the constraint refuses, such as -`DEFAULT 'NaN'`, fails. A renamed column's constraint takes the new column -name, and a column whose name is longer than 47 bytes gets a constraint named -after a hash of the name. - -`char(N)` is stored and read as `varchar`. The data round-trips losslessly: -values, comparisons, and `length()` match a hot PG table, where `length()` -already ignores `char(N)` trailing padding. The only difference is cosmetic: a -cold `char(N)` column reads back unpadded with `pg_typeof varchar`, because -pg_duckdb has no fixed-length `char` type. Use `text` or `varchar` if -blank-padded display matters. - -`json`, `jsonb` and `interval` are stored as `varchar` in Iceberg (no native -primitive), an `interval` as ` months days seconds`, -the text PostgreSQL and DuckDB both read as the same value. On read, `interval` is view-cast back to the rich PG type; `json` -and `jsonb` come back as DuckDB's `json` (the equivalent of PG's `jsonb`), not -the rich PG `jsonb` type, because Iceberg-backed reads run entirely in DuckDB. -Queries like `data->>'key'` and `data->'key'` work, and ColdFront translates -the `::jsonb` cast, `jsonb_array_length`, `jsonb_build_object` / `jsonb_agg` -(and their `json_` twins, which DuckDB also lacks) and `date_bin` on read: the -builders become the `concat` / `to_json` / `array_agg` form both engines -evaluate identically (the result is JSON-equal to jsonb's rendering, in compact -form and in argument order), and `date_bin` becomes DuckDB's `time_bucket`, -which takes the same arguments and agrees on every fixed-width bucket. The -jsonb-only operators (`?`, `@>`, `<@`, `#>`, `#>>`) and most jsonb functions -(`jsonb_typeof`, `jsonb_extract_path`, `jsonb_extract_path_text`, `jsonb_set`, -`jsonb_path_*`, `jsonb_each`, `jsonb_object_keys`) are not supported on tiered -or decoupled data: DuckDB either lacks them or its same-named function differs -in signature or result. Reach into the document with `->`/`->>`. - -These limits apply only to reads that go through a tiered or iceberg-only view, -which pg_duckdb plans entirely in DuckDB. Ordinary (non-tiered) PostgreSQL -tables are untouched by ColdFront and keep full jsonb support, as does the hot -partition table itself. - -Bound parameters (`$1` from a prepared statement, a driver's extended protocol, -or a plpgsql variable) work in such reads. DuckDB types most placeholders from -their context (`ts > $1`); one that is a direct argument of a DuckDB function -with several overloads (`time_bucket`'s origin, so `date_bin`'s) or any -argument of a table function (`generate_series`) it cannot, so ColdFront plans -such a read from the bound values on every execution instead of caching a -generic plan. Under `plan_cache_mode = force_generic_plan` those reads cannot -run: the forced generic plan has no values and fails with -`only works with DuckDB execution`. - -A hot-only read is the exception. When a `SELECT` reads a tiered view directly -(no join, CTE, or sub-query) and its `WHERE` provably restricts to the hot -tier, ColdFront rewrites it to read the hot partition table in plain -PostgreSQL: the full jsonb operator and function set works, and the query skips -DuckDB entirely. Such a read returns `data` as native `jsonb` rather than the -view's `json`. Reads that span tiers, are cold-only, or reach the view through -a join or sub-query stay in DuckDB, where the limits above apply. - -`inet`/`cidr`/`oid` are **not supported**: pg_duckdb cannot process them -(`inet` Oid 869, `oid` Oid 26) in any Iceberg-backed query, and every -cross-tier read is planned by pg_duckdb - so no cast makes them readable. Store -IP data as `text` and `oid` values as `bigint` (you can still index/compare -them; cast on the hot side only if needed). - -### Arrays - -A one-dimensional array of a supported type tiers as an Iceberg list of that -type. The following table shows how each array type is stored and how the view -reads it back: - -| PG type | Iceberg/Parquet storage | Reads back as | -|---|---|---| -| `bigint[]`, `integer[]`, `real[]`, `double precision[]`, `boolean[]`, `date[]`, `time[]`, `timestamp[]`, `uuid[]`, `bytea[]`, `numeric(P,S)[]` | A list of the element's storage type. | The same type. | -| `smallint[]` | A list of `INTEGER`. | `integer[]`, as a `smallint` column reads as `integer`. | -| `text[]` | A list of `VARCHAR`. | `text[]`. | -| `varchar(N)[]`, `char(N)[]` | A list of `VARCHAR`. | `character varying[]`, with `char(N)` elements unpadded. | - -The following arrays are refused when a tiered table is registered and when a -decoupled table is created, with an error that names the column: - -- `timestamptz[]`, because pg_duckdb cannot read a `timestamptz[]` column of a - PostgreSQL table, which a tiered table's hot tier is; store `timestamp[]` in - UTC instead. -- `jsonb[]`, `json[]` and `interval[]`, because those types are stored as text - and read back through a cast that an array element does not get. -- arrays of `vector` or `halfvec`, because a vector is itself stored as a list, - and ColdFront stores an array as a list of scalar values. - -An Iceberg list has one dimension and numbers its elements from 1, so ColdFront -refuses an array with more than one dimension or with a lower bound other than -1, such as `'[0:2]={1,2,3}'`. The hot table's check constraint refuses such an -array when it is written. A write to the cold tier refuses it too: a tiered -`INSERT` with an error that names the column, and a bound parameter with an -error that names no column. A write that reads the array from a PostgreSQL -table stores it renumbered from 1 instead, as -[Reading a PostgreSQL Table in a Cold-Tier Write](#reading-a-postgresql-table-in-a-cold-tier-write) -describes. A tiered table with a column declared with more than one dimension, -such as `integer[][]`, passes registration and is refused at its first archive -pass, because pg_duckdb reads a column by its declared dimensions; a decoupled -table refuses such a column when it is created. A decoupled table's array type -is spelled as its element type followed by `[]`, such as `integer[]`; the -spellings `integer[3]` and `integer ARRAY` are refused. -`ALTER TABLE ... ADD COLUMN` cannot add an array column to a tiered table, -because duckdb-iceberg does not add a nested column to an Iceberg table. - -A tiered `INSERT` accepts an array in any spelling, because its source runs in -PostgreSQL. A cold `UPDATE` and a decoupled `INSERT` run in DuckDB, which does -not read PostgreSQL's brace literal: a `'{a,b}'` literal reaches DuckDB as text -and fails to cast. Write the array as `ARRAY['a', 'b']`, or pass it as a bound -parameter. - -Reads through a view run in DuckDB once the table has a cold tier, and the -following array operations work there: - -- `x = ANY(arr)`, `arr @> ARRAY[...]` and `arr && ARRAY[...]`. -- subscripts and slices, which count from 1, such as `arr[1]` and `arr[1:2]`. -- `array_length(arr, 1)` and `unnest(arr)`. - -A brace literal and a bound array parameter fail in such a read, as they do for -a vector, and so does `cardinality()`, which DuckDB defines for maps only. - -### Reading a PostgreSQL Table in a Cold-Tier Write - -A cold-tier write that reads an ordinary PostgreSQL table can store a value -that differs from the value in that table, without raising an error. - -!!! warning "Such a write stores a numeric NaN as 0" - - When a cold-tier write reads a `numeric` value from a PostgreSQL table, the - write stores a `NaN` from that table in the cold tier as `0`. ColdFront - raises no error, so check the source table before the write, as this - section shows. - -The following writes read a PostgreSQL table this way: - -- an `INSERT ... SELECT` from a PostgreSQL table into a decoupled table. -- an `UPDATE ... FROM` or a `DELETE ... USING` that names a PostgreSQL table, - on a decoupled table or on the cold rows of a tiered table. -- a `MERGE` whose source is a PostgreSQL table, on a decoupled table or on the - cold tier of a tiered table. - -ColdFront runs these writes in DuckDB, and DuckDB reads the PostgreSQL table -through its `postgres` extension. As the extension reads the table, it changes -two kinds of value: - -- a `numeric` `NaN`, which the extension reads as `0`, also inside an array. -- an array that starts at an element other than 1, which the extension - renumbers from 1, so `'[0:1]={7,8}'` arrives as `{7,8}`. - -The write stores the changed value, and the conditions of the write compare the -changed value. PostgreSQL treats a `NaN` as greater than every number, so -`s.amount > 100` holds for a `NaN` in PostgreSQL, but in such a write the -condition sees `0` and does not hold. A condition that names `NaN`, such as -`s.amount <> 'NaN'`, fails in such a write, because DuckDB does not accept -`'NaN'` as a `numeric` value. - -The extension reads the following values unchanged: - -- a `real` or `double precision` `NaN`, `Infinity` or `-Infinity`. -- a `date`, `timestamp` or `timestamptz` `infinity` or `-infinity`, which the - write then refuses to store, as every write to the cold tier does. - -The following writes refuse these values instead: - -- an `INSERT` or `COPY` into a tiered table, because PostgreSQL runs its source - and ColdFront hands each `numeric` value to DuckDB as text, so DuckDB refuses - a `NaN` and ColdFront refuses an array that does not start at 1. -- a write to the hot rows of a tiered table, because PostgreSQL runs that - write, and the check constraints of the hot table refuse both values. -- a `NaN` written as a literal or as a bound parameter, because DuckDB refuses - `'NaN'` as a `numeric` value. - -To keep these values out of the cold tier, check the source table in PostgreSQL -before the write, and fix or remove the rows that the check finds. The -following query counts those rows in a source table `staging` with a `numeric` -column `amount`, a `numeric` array `amounts` and an array `tags`: - -```sql -SELECT count(*) - FROM staging - WHERE amount = 'NaN' - OR 'NaN' = ANY (amounts) - OR array_lower(tags, 1) <> 1; -``` - -In the query, list every `numeric` column of your source table as `amount` is -listed, every `numeric` array as `amounts` is, and every array column as `tags` -is. A count of `0` means that the write stores those columns unchanged. - -## Caveats - -Keep the following caveats in mind when running either mode: - -- A cold-tier write that reads an ordinary PostgreSQL table, such as an - `UPDATE ... FROM` or a `MERGE` from one, stores a `numeric` `NaN` from that - table as `0` and stores an array renumbered to start at 1, with no error. - [Reading a PostgreSQL Table in a Cold-Tier Write](#reading-a-postgresql-table-in-a-cold-tier-write) - lists the writes that do this and shows how to check a source table first. -- `jsonb` reads come back as `json`. The `->`/`->>` operators work, and - ColdFront translates the `::jsonb` cast, `jsonb_array_length`, - `jsonb_build_object` / `jsonb_agg` and `date_bin`; other jsonb operators and - functions are unsupported on cold or cross-tier reads. A hot-only read of the - view runs in PostgreSQL with full jsonb (see - [Supported Column Types](#supported-column-types)). -- Cross-tier isolation differs from PG-native isolation. The cold tier keeps - one Iceberg snapshot for the whole transaction, even at READ COMMITTED, so a - commit by another session after the transaction's first cold read stays - invisible until the transaction ends. The hot tier follows PostgreSQL's own - isolation level, so at READ COMMITTED a later statement can see new hot rows - but not new cold ones. Within one transaction, a read of a tiered or - decoupled table sees the transaction's own writes on both tiers. The held - snapshot is also why the compactor's `--expire-older-than` must stay longer - than the longest transaction that reads the table (see - [compaction.md](compaction.md)). -- One transaction block cannot hold both a PostgreSQL write and a cold-tier - write unless `duckdb.unsafe_allow_mixed_transactions` is on. pg_duckdb - refuses the pair ("Writing to DuckDB and Postgres tables in the same - transaction block is not supported"), at the PostgreSQL write when it comes - second and at `COMMIT` when it came first. A cold-tier write here is a - tiered `UPDATE` or `DELETE` whose `WHERE` bounds the cold tier, or any write - to a decoupled table. The statements that write both tiers themselves (a - dual-tier `UPDATE` or `DELETE`, a tiered `INSERT` with cold rows, a - cross-tier move) set the parameter `LOCAL` for the rest of their - transaction, so the block they ran in accepts later writes of either kind. - You can set it yourself, `SET LOCAL duckdb.unsafe_allow_mixed_transactions = - on`, at your own risk: pg_duckdb commits the Iceberg snapshot at - `PRE_COMMIT`, so a backend crash between that and the PostgreSQL commit - record keeps the cold write and loses the PostgreSQL writes. -- In decoupled mode, pg_duckdb commits the Iceberg snapshot at PRE_COMMIT, so a - backend crash after that but before the PG commit record leaves the Iceberg - write committed and the PG side lost. A crash after the Parquet upload but - before the commit POST leaves unreferenced objects, which the compactor's - orphan pass reclaims. -- In decoupled mode, concurrent writes from multiple PG nodes are serialized - PG-side by the bakery protocol: every iceberg-only `INSERT` goes through - `coldfront._exec_iceberg_with_claim`, which holds a globally-ordered - Snowflake ticket and waits for its turn before committing to Lakekeeper. - There are no 409 conflicts and no app-level retries. The protocol is - Lamport-1978 mutex with the Ricart-Agrawala (1981) deferred-reply - optimization; claims and acks replicate as Spock rows and it stays safe under - Spock's asymmetric apply (modeled in - [docs/formal/Bakery.tla](https://github.com/pgEdge/ColdFront/blob/main/docs/formal/Bakery.tla)). - The bakery requires the `snowflake` extension, the `coldfront.loopback_dsn` - GUC (the node's loopback, a unix-socket DSN), and a one-time - `SELECT coldfront.ensure_replicated()` call on every node after Spock mesh - setup; see - [architecture_decoupled.md](architecture_decoupled.md#concurrency-horizontal-scaling-the-bakery-protocol). - Sync-rep is **not** required. The throughput ceiling is Lakekeeper's commit - rate, not the writer count. -- For direct table access, `_events` is the hot heap (tiered mode only). - `ice.public.` is the Iceberg table - only addressable via - `iceberg_scan(...)` or `duckdb.raw_query('… ice.… …')`, never via - PG-native 3-part names. -- A tiered `INSERT` streams its cold rows: the cold projection runs in - PostgreSQL, in a second session the DuckDB `postgres` extension opens from - `coldfront.local_pg_dsn`, and its rows go straight into the Iceberg writer - in one pass. An omitted IDENTITY column takes `nextval()` on the hot table's - sequence, so cold ids share it with the hot side, and an omitted column with - a DEFAULT takes it. The hot rows are one set-based `INSERT`. The second - session reads the source at the statement's own snapshot, so both tiers see - one state of the tables whatever another session commits meanwhile, and a - REPEATABLE READ transaction reads its snapshot on both tiers. A source the - stream cannot read the same way twice (a volatile or stable function such as - `now()`, a temporary table, a `WITH` entry, a table the transaction has - written, a SERIALIZABLE transaction, or a second table-reading `INSERT` in - one READ COMMITTED transaction) goes through - `coldfront._cold_sink` instead, which renders each row in plpgsql and writes - Iceberg in batches of `coldfront.cold_write_batch_size` rows, as does a - server without `coldfront.local_pg_dsn`. The statement's `cold_rows` column - is `NULL` when the cold rows streamed. A value that reaches the second - session or DuckDB as text is rendered in a form every reader takes as the - same value, whatever the caller's `DateStyle`, `IntervalStyle` or - `extra_float_digits`. -- `COPY FROM` reads the rows with PostgreSQL's `COPY` reader and writes - them through that same `INSERT` path, `coldfront.cold_write_batch_size` rows - per `INSERT`. The format options are the reader's, and a supplied value for a - `GENERATED ALWAYS` identity column is kept, as `COPY` into a table keeps it. - `COPY ... WHERE` and the `FREEZE`, `ON_ERROR`, `REJECT_LIMIT` and `DEFAULT` - options are refused. -- An `INSERT`, `UPDATE` or `DELETE` nested in a `WITH` entry goes through the - same rewrite as a top-level one. With a watermark an `INSERT`'s row may go - cold, so `RETURNING` on it is refused, as on a top-level `INSERT`; a hot - `UPDATE` or `DELETE` keeps `RETURNING`, a cold or dual-tier one cannot - return rows or read another `WITH` entry (its DuckDB half does not see - them), and a cross-tier move cannot be a `WITH` entry. A statement may write - a tiered view once, and the nested write may not have a `WITH` clause of - its own. On a decoupled view the source runs in DuckDB, so a `WITH` entry - that modifies data is refused (DuckDB 1.5 has no data-modifying `WITH`), and - a nested `INSERT` may not read another `WITH` entry. A statement that holds - a nested write and also reads a tiered view runs in DuckDB as a whole, and - pg_duckdb refuses it ("DuckDB does not support modifying CTEs"); read the - hot table, or split the statement. -- `MERGE INTO `, on PostgreSQL 17 and later (16 allows `MERGE` on tables - alone), runs on the tier its `ON` condition bounds the partition column to - (`AND t.ts >= ''` for the hot tier, `AND t.ts < ''` for the - cold tier): a hot `MERGE` runs in PostgreSQL against the hot table - and keeps `RETURNING`, a cold one in DuckDB against the Iceberg table. Each - tier sees its own rows alone, so a `MERGE` that bounds neither tier is - refused, and an `INSERT` action's row must belong to the statement's tier: a - hot `MERGE` refuses a row below the cutoff, a cold one a row at or after it; - insert such rows with `INSERT`, which splits them. A cold `INSERT` action - must give the identity column a value (`OVERRIDING SYSTEM VALUE` for a - `GENERATED ALWAYS` one), a cold `MERGE` cannot return rows and runs one - `UPDATE` or `DELETE` action per statement (duckdb-iceberg's limit), and a - `WHEN NOT MATCHED BY SOURCE` action must bound the same tier in its own - condition. A `MERGE` that sets the partition column is refused (run the - change as an `UPDATE`), as is one on a table with a clustered vector column - and one whose source reads the view. A `MERGE` nested in a `WITH` entry - takes the same path as a nested `UPDATE` or `DELETE`. On a decoupled view - every `MERGE` runs in DuckDB. -- `TRUNCATE` on a registered relation, or on the hot table behind a tiered one, - fails with an error, because the cold rows in Iceberg would stay visible - through the view. -- A table or column with a cold tier cannot have a name that PostgreSQL leaves - unquoted but DuckDB parses as a keyword. With DuckDB 1.5.4 those names are - `anti`, `asof`, `at`, `by`, `describe`, `glob`, `lambda`, `pivot`, - `pivot_longer`, `pivot_wider`, `positional`, `qualify`, `semi`, `show`, - `summarize`, `unpack` and `unpivot`. pg_duckdb passes a table or column name - to DuckDB quoted the way PostgreSQL quotes it, so such a name reaches DuckDB - bare even when the query quotes it. Every query pg_duckdb runs against the - table or column then fails with a DuckDB `Parser Error` - ([duckdb/pg_duckdb#1019](https://github.com/duckdb/pg_duckdb/issues/1019)). - ColdFront refuses such a column name wherever a column gets a cold tier: - tiered registration, an archive pass that tiers or expires data, - `coldfront.create_iceberg_table()`, `coldfront.adopt_iceberg_table()`, and an - `ADD COLUMN` or `RENAME COLUMN` on a table that already has a cold tier. - ColdFront refuses such a table name at tiered registration, in - `coldfront.create_iceberg_table()` and `coldfront.adopt_iceberg_table()`, and - in a `RENAME` of the view or the hot table of a table that already has a cold - tier. Partition-only registration, without a hot period, accepts a table with - such a name or column. ColdFront does not refuse a DuckDB keyword that the - DuckDB parser accepts as a name, such as `columns`. - -## Distributed Setup (3-Node Mesh, Decoupled Mode) - -For multi-writer iceberg workloads, run ColdFront on N PG nodes in a Spock mesh -against the same Lakekeeper + S3. The bakery serializes commits PG-side so -writers never collide at the catalog. - -### Per-Node `postgresql.conf` - -The configuration below applies to each node in the mesh: - -```ini -wal_level = logical -shared_preload_libraries = 'snowflake,spock,pg_duckdb,coldfront' - -# The bakery rules a peer dead when its walsender's last reply is older than -# coldfront.peer_alive_window_ms (default 10 s), and an idle peer replies only -# to the walsender's keepalive, every wal_sender_timeout/2. The claim refuses -# to run unless wal_sender_timeout is positive and below twice the window. -wal_sender_timeout = 15s - -# Spock prereq + DDL replication. The ddl_sql repset (and with it the -# one-node provisioning promise for wrapper views) only works with DDL -# replication on; allow_ddl_from_functions covers DDL issued inside -# coldfront's plpgsql helpers. -track_commit_timestamp = on -spock.enable_ddl_replication = on -spock.allow_ddl_from_functions = on -spock.include_ddl_repset = on - -# Only on a server that has this setting (PostgreSQL 16.15, 17.11 and 18.6 in -# pgEdge's builds): -# it lists the libraries allowed as logical decoding output plugins, and its -# default leaves out Spock's, so no subscription can create its slot. A server -# without the setting refuses to start with this line in place. -output_plugin_libraries = 'pgoutput, test_decoding, spock_output' - -# Per-node - any distinct integer 1..1023 (must be unique per node; the value -# is otherwise arbitrary - the bakery matches acks by spock node name, not by id). -snowflake.node = 1 - -# DSN of the node's loopback, which runs the bakery's autonomous claim -# INSERT/DELETE. It must name a unix socket: every onboarded role can read -# this setting, so it must never need a password. The claim session only -# touches the coldfront.claims/claim_acks heap tables, so it never attaches -# the Iceberg catalog (the lazy catalog-attach hook fires only on a tiered -# view). application_name=coldfront_loopback marks the session as bakery -# traffic. Only a superuser can set it. -coldfront.loopback_dsn = 'host=/tmp dbname=coldfront user=coldfront application_name=coldfront_loopback' - -coldfront.warehouse = 'wh' -coldfront.lakekeeper_endpoint = 'http://lakekeeper:8181/catalog' - -# Upload the Parquet outside the bakery claim and serialize only the catalog -# commit. Set both, and only where duckdb-iceberg includes ColdFront's patch (the -# ColdFront image does). With either off, the upload is serialized too. -coldfront.iceberg_async_parquet = on -coldfront.iceberg_bakery_patch = on -``` - -The bakery has no peer-ack timeout. R-A's only failure mode is a dead peer -(would wait forever), and a liveness check inside the wait loop closes it: -the bakery treats a peer whose `pg_stat_replication.reply_time` is older than -`coldfront.peer_alive_window_ms` as already acked, whatever the walsender's -state (a peer that has just reconnected is catching up and still alive). The -peer's apply worker sends that reply after it applies, every -`spock.feedback_frequency` messages, and in answer to the walsender's -keepalive; `wal_receiver_status_interval` plays no part, since Spock's apply -worker does not read it. The window and `wal_sender_timeout` go together; set -them as [Timeouts](#timeouts) below describes. An alive peer that has -not acked is either deferring legitimately (R-A's defer rule) or about to -ack, and waiting for it is correct, not a failure. - -A claim whose owner is gone (a hard backend crash) is reaped without operator -action: by that node's next cold write, to any table, by a peer's arriving -claim, or by the waiting peer's poke, a no-op `UPDATE` of its own claim row -about once a second for as long as it waits. That poke is the only replication -traffic the bakery generates while a writer waits, and there is none when -nothing waits. See [architecture_decoupled.md](architecture_decoupled.md), -*Orphan reaping*. - -Sync-rep (`synchronous_standby_names`) is **not required** by the bakery - the -R-A ack barrier is what serializes iceberg commits. You can still enable -sync-rep cluster-wide if you want stronger durability for non-bakery writes, -but it plays no part in iceberg-commit serialization. - -Run this one-time mesh setup in this order on every node, because step 3 calls -`spock.repset_add_table`, which requires the local Spock node to already exist: - -```sql --- 1. Extensions, in dependency order. snowflake is a bakery prereq --- (claim tickets come from snowflake.nextval). -CREATE EXTENSION IF NOT EXISTS snowflake; -CREATE EXTENSION IF NOT EXISTS spock; -CREATE EXTENSION IF NOT EXISTS pg_duckdb; -CREATE EXTENSION IF NOT EXISTS coldfront; - --- 2. Spock node + full-mesh subscriptions (each node has N-1 subs). Name each --- subscription sub__from_ (on n1: sub_n1_from_n2): the --- bakery's peer-liveness check looks the walsender up by that name, and a peer --- whose subscription is named any other way counts as dead and is treated as --- already acked. -SELECT spock.node_create('n', 'host= user=coldfront dbname=coldfront port=5432'); - --- on n1 (n2 and n3 symmetric): -SELECT spock.sub_create('sub_n1_from_n2', 'host= user=coldfront dbname=coldfront port=5432', - ARRAY['default','default_insert_only','ddl_sql'], - false, false, '{}', '0', false); -SELECT spock.sub_create('sub_n1_from_n3', 'host= user=coldfront dbname=coldfront port=5432', - ARRAY['default','default_insert_only','ddl_sql'], - false, false, '{}', '0', false); - --- 3. **Required** on every node, after its subscriptions exist and before --- the first cold write, storage secret or table registration: put every --- ColdFront table that replicates by value in this node's replication sets. -SELECT coldfront.ensure_replicated(); -``` - -The subscriptions set -`synchronize_structure := false, synchronize_data := false` because the tables -already exist on every node from the coldfront extension, so no initial copy is -needed. - -### Timeouts - -The following table lists the two settings that decide how long a cold write -waits for a peer; set them together, on every node: - -| Setting | Recommended | Where | What it does | -|---|---|---|---| -| `wal_sender_timeout` | `15s` | `postgresql.conf` on every node; the image sets it in mesh mode | The node's walsender asks each peer for a reply after half of it (7.5 s) and drops a peer that has not replied for the whole of it. | -| `coldfront.peer_alive_window_ms` | `10000` (the default) | `postgresql.conf`, superuser-only | The bakery treats a peer whose last reply is older than this as dead and stops waiting for its ack. | - -The rule: half of `wal_sender_timeout` must stay below the window, with a -round trip to spare; the claim enforces the first part. An idle peer replies -only when the walsender asks, so the -PostgreSQL default of 60 s against a 10 s window would rule a live idle peer -dead; a claim refuses to run with such settings and names both. With the -recommended values a writer waits at most 10 s for a peer that is really -dead, and the walsender disconnects a peer silent for 15 s, which then -reconnects on its own. - -On a slow or lossy WAN raise both together and keep the rule, for example -`wal_sender_timeout = 30s` with `coldfront.peer_alive_window_ms = 20000`. -Both take effect on a reload (`ALTER SYSTEM SET ...;` then -`SELECT pg_reload_conf();`), no restart: walsenders re-read the timeout, and -each claim reads the window once. - -### What `coldfront.ensure_replicated()` Does - -The call is the one ColdFront-specific step of the mesh setup. It adds eight -tables to the node's replication sets (`claim_acks` to `default_insert_only`, -the rest to `default`), each keyed by name, so a row is identical on every node -and replicates by value: - -| Table | What replicates, and why a peer needs it | -|---|---| -| `coldfront.claims`, `coldfront.claim_acks` | The bakery's tickets and acknowledgements. A peer acknowledges an originator's claim by inserting into `claim_acks` on its own node, and the ack reaches the originator only if the table is in the peer's set. `claim_acks` goes in the insert-only set, because each node deletes only the ack copies it holds. | -| `coldfront.tiered_views` | The registry. A peer's hook recognizes a view by its row; without it the peer's writes through the view fail in PostgreSQL ("cannot insert into view") and its reads cannot attach the cold tier. | -| `coldfront.archive_watermark` | The hot/cold cutoff that routes each tiered write. | -| `coldfront.storage_secret` | The cold-store credential, so one `set_storage_secret` call reaches every node. | -| `coldfront.partition_config` | The per-table lifecycle. The archiver and the partitioner add this table themselves as well, because the partitioner runs without the extension. | -| `coldfront.vector_config`, `coldfront.vector_centroids` | The cluster routing state, so every node resolves a vector to the same cluster. | - -Two tables stay local by design and are not added: `coldfront.deferred_acks` -(each node's queue of the acks it owes) and `coldfront._dummy_dml_target`. - -The call runs on every node because a replication set is a property of the -provider: each node sends the rows of the tables in its own set, and a node -that skipped the step keeps its acks, registry rows and secret to itself. It -cannot run at `CREATE EXTENSION` time, because `spock.repset_add_table` needs -the local Spock node, and nothing runs it later on a node's behalf: -provisioning a table, setting the secret and writing cold rows all assume it -has run. It is idempotent, so running it again is harmless, and a node added to -the mesh later runs the same three steps. - -A skipped step shows up late and away from the node that skipped it. If a peer -never ran it, the originator of every cold write waits at the ack barrier for -an ack that never arrives. If the node that registers a table never ran it, -the view reaches each peer as replicated DDL but the registry row does not, so -the peer's writes through the view fail with "cannot insert into view". A -secret set on such a node stays on that node. - -Verify the step on every node. The query lists the eight tables: - -```sql -SELECT c.relname - FROM spock.replication_set rs - JOIN spock.replication_set_table rst ON rst.set_id = rs.set_id - JOIN pg_class c ON c.oid = rst.set_reloid - WHERE rs.set_name = 'default' AND c.relnamespace = 'coldfront'::regnamespace - ORDER BY 1; -``` - -Then verify the mesh before benching - insert a sentinel claim on each node -and read it back from every other node. All N×(N-1) directions must show the -row before traffic starts. `ci/journey.sh` `story_mesh_substrate` is a -copyable reference: it checks the membership above on every node and then the -sentinels. - -## Tuning Knobs - -The following GUCs adjust write behavior and execution; tune them as needed: - -- `coldfront.allow_mixed_writes` (bool, default `on`) controls what happens for - tiered-mode `UPDATE`/`DELETE` whose WHERE cannot be proven to target one - tier. `on` emits a dual-tier CTE; `off` rejects with an error and a hint. The - setting is not relevant in decoupled mode (every write is single-tier by - definition). -- `coldfront.cold_write_batch_size` (int, default `10000`, minimum `1`) sets - how many cold rows a tiered `INSERT` gathers (see [Caveats](#caveats)) before - it writes them to Iceberg as one `INSERT`. A larger value writes fewer, - larger Parquet files, and the remainder always flushes, so a small write - stays one file. -- `coldfront.vector_probe` (bool, default `on`) sets whether a recognized - similarity search reads only the clusters nearest its query vector. `off` - gives an exact scan of the whole corpus. The setting affects only a table - with a trained vector column ([usage_vectors.md](usage_vectors.md)). -- `coldfront.vector_nprobe` (int, default `0`) sets how many clusters such a - search reads, overriding the column's own `nprobe`. `0` uses the configured - value; at or above the column's `nlist` the search is exhaustive. -- Benchmark `duckdb.force_execution` (default off) before you turn it on: on a - mixed workload it helps `count(distinct)` and similar but regresses index - lookups, top-K with PK ordering, and JSON access. -- `duckdb.temporary_directory` sets where DuckDB spills. Each backend gets its - own subdirectory there, named after its process id, so concurrent spills - cannot collide; one left by a departed backend is reclaimed. See - [architecture.md](architecture.md#duckdb-spill-files-are-not-namespaced-per-instance). -- `duckdb.max_temp_directory_size` is a cap **per connection**, not a cluster - total, and when unset the cap is 90% of free space per session. For a total - budget, divide it by the concurrent sessions or give the temp path its own - filesystem or quota. - -## Next Steps - -To go further with ColdFront, consult the following guides: - -- The [Architecture](architecture.md) overview describes the tiered - architecture, the watermark, the archiver, transparent `UPDATE`/`DELETE`, and - concurrency. -- The [Decoupled Mode](architecture_decoupled.md) deep dive describes - decoupled-mode internals, the ACID model, and distributed scaling. -- The [Embeddings](usage_vectors.md) guide covers storing and searching - embeddings through the pgvector interface. -- The [Compaction](compaction.md) guide covers cold-tier maintenance: - compaction, snapshot expiry, and orphan-file removal. diff --git a/docs/usage_decoupled.md b/docs/usage_decoupled.md new file mode 100644 index 00000000..4ea24a41 --- /dev/null +++ b/docs/usage_decoupled.md @@ -0,0 +1,160 @@ +# Using ColdFront in Decoupled Mode + +Before you begin, configure PostgreSQL, Lakekeeper, and your object store by +following [Configuring ColdFront](configuration.md), [Configuring Lakekeeper](configure_lakekeeper.md), +and [Configuring your Object Store](object_store.md); every row of a +decoupled table lives in the cold tier those guides set up, from the first +row. This page walks through creating a decoupled table with +`coldfront.create_iceberg_table()`, changing its columns, and adopting a +table that another engine already wrote into the catalog. + +A single call creates a decoupled table: + +```sql +SELECT coldfront.create_iceberg_table( + p_schema => 'public', + p_table => 'events', + p_columns => '[ + {"name":"id", "type":"bigint"}, + {"name":"ts", "type":"timestamptz"}, + {"name":"status", "type":"text"}, + {"name":"data", "type":"jsonb"} + ]'::jsonb, + p_partition_cols => '{month(ts)}' +); +``` + +`p_partition_cols` partitions the Iceberg table, here by the month of `ts`: +each month's rows land in their own data files, and a query with a time filter +skips the months outside it before reading anything. Each element is one term +as DuckDB's `PARTITIONED BY` takes it: a column name, `year(col)`, `month(col)` +or `day(col)` on a timestamp or date column, `hour(col)` on a timestamp column, +`bucket(N, col)` or `truncate(W, col)`; `'{month(ts), region}'` partitions by +both. The argument is a PostgreSQL array literal, so a term that contains a +comma is double-quoted inside it, the comma being the array's delimiter: +`'{"bucket(16, id)"}'`. A column name that itself needs double quotes has them +backslashed inside such an element: `'{"month(\"Event Time\")"}'`. Leave the +argument out for an unpartitioned table. + +That single statement provisions: + +- `ice.public.events` on the attached Iceberg catalog. +- a PG-side wrapper view `public.events` with proper PG-typed columns. + +- a `coldfront.tiered_views` registry row, so that the coldfront C hook + intercepts every `INSERT`, `UPDATE`, `DELETE`, and `MERGE` on the view and + rewrites each one to a single `duckdb.raw_query(...)` against + `ice.public.events`. + + +In a mesh one node provisions: Spock's `ddl_sql` repset replicates the +`CREATE VIEW`, and the `default` repset replicates the name-keyed registry row, +which enables the write hook on every peer. Each node also needs the +one-time setup call `coldfront.ensure_replicated()` - see +[Configuring a Distributed Deployment](distributed_setup.md). + +## Changing a Decoupled Table's Columns + +An `ALTER TABLE` on the wrapper view adds, drops, renames or widens a column +of the Iceberg table, and the view follows, with its owner and table-level +grants kept; a column-level grant on the view is not kept. Only the view's +owner, or a role holding its privileges, can change its columns, and that role +needs the cold access `grant_app_access` gives, since the Iceberg change runs +as that role. The following statements change the columns of the table created +above: + +```sql +ALTER TABLE public.events ADD COLUMN qty integer; +ALTER TABLE public.events ALTER COLUMN qty TYPE bigint; +ALTER TABLE public.events RENAME COLUMN qty TO quantity; +ALTER TABLE public.events DROP COLUMN quantity; +``` + +A column takes the same type map as at creation, though an array column cannot +be added, because duckdb-iceberg does not add a nested column to an Iceberg +table. A type change is limited to the widening Iceberg accepts: `integer` to +`bigint`, `real` to `double precision`, `date` to `timestamp`, and a wider +`numeric` precision. An added column takes a name and a type only: a default, +a constraint, a collation, or a storage or compression option is refused, as +is a `USING` or `COLLATE` clause on a type change, and a column change shares +an `ALTER TABLE` only with other column changes. A vector column is neither +added nor changed, a table adopted read-only refuses every column change, and +an object built on the view, such as another view, makes a column change +fail. In a mesh, the change runs on one node: the Iceberg table is shared, and +the rebuilt view and registry row replicate. + +## Adopting a Table That Already Exists in the Catalog + +A table another engine wrote needs no provisioning, only a wrapper view and a +registry row. `coldfront.adopt_iceberg_table()` reads the schema from the +catalog and builds both, so the call names the relation and nothing else: + +```sql +SELECT coldfront.adopt_iceberg_table( + p_schema => 'public', + p_table => 'orders', + p_namespace => 'lake' +); +``` + +`p_schema` is the PostgreSQL schema the wrapper view goes in, and it must +exist. `p_namespace` is the Iceberg namespace the table lives in, which need +not exist as a PostgreSQL schema; it defaults to `p_schema` when omitted. The +view takes the table's name. + +Adoption is read-only by default, so reading someone else's lake table cannot +become writing it by accident. Passing `p_writable => true` enables the same +`INSERT`, `UPDATE`, `DELETE` and `MERGE` rewrite a created table gets: + +```sql +SELECT coldfront.adopt_iceberg_table('public', 'orders', 'lake', + p_writable => true); +``` + +Because Iceberg records no PostgreSQL type, an adopted column reads as whatever +its storage type maps to; a `jsonb` column comes back as `text`. Pass `p_types` +to restore one, provided the override maps to the type the catalog already +stores: + +```sql +SELECT coldfront.adopt_iceberg_table('public', 'orders', 'lake', + p_writable => true, + p_types => '{"meta":"jsonb"}'::jsonb); +``` + +Adoption binds the name once. A second call under a registered name is refused +whatever its arguments; to enable writes or change an override, release the +table with `coldfront.release_iceberg_table()` and adopt it again. A +non-superuser deployment runs `coldfront.grant_app_access()` after adopting, +because the grant reads the registry at call time. + +In a mesh one node adopts. The wrapper view replicates through Spock's +`ddl_sql` repset and the registry row through the `default` repset (see +[Configuring a Distributed Deployment](distributed_setup.md)), so every peer +reads and writes the table under the same name, and a peer's own +adopt is refused as already registered. Adoption takes the table's bakery +claim around that check and the registry write, so two nodes adopting the +same table at the same moment are serialized: the second waits, then reads +the first's row and refuses, rather than both registering the one table. A +writer outside ColdFront is outside the bakery and can still collide at +Lakekeeper. + +One relation can be registered per Iceberg table. A namespace that needs +quoting, such as `Lake-EU`, works. + +## Handing an Adopted Table Back + +An adopted table is released rather than +[dropped](using_coldfront/index.md#dropping-an-iceberg-table), +because ColdFront does not own it. `coldfront.release_iceberg_table()` removes +the wrapper view, the registry row and the relation's vector configuration, and +performs no Iceberg I/O, so the table keeps every row and stays in the catalog: + +```sql +SELECT coldfront.release_iceberg_table('public', 'orders'); +``` + +Release refuses a tiered registration, because removing one would leave its +cold rows unreachable while the hot table returned under the relation's name. +`drop_iceberg_table()` refuses a relation adopted read-only, for the mirrored +reason: read access gives no authority to destroy. diff --git a/docs/usage_partitioner.md b/docs/usage_partitioner.md new file mode 100644 index 00000000..ae47b5eb --- /dev/null +++ b/docs/usage_partitioner.md @@ -0,0 +1,126 @@ +# Using ColdFront in Standalone Partitioned Mode + +This mode needs only a working PostgreSQL instance, optionally in a Spock mesh; +it uses no Lakekeeper, no object store, no pg_duckdb, and no `coldfront` +extension. A database set up this way can gain the cold tier later by following +[Configuring ColdFront](configuration.md), +[Configuring Lakekeeper](configure_lakekeeper.md), and +[Configuring your Object Store](object_store.md). This page walks through +building and registering the `partitioner` binary, operating it day to day, +giving a time-partitioned table a real primary key, and two-level +sub-partitioning. The management CLI that the partitioner shares with the +archiver is described in +[Managing Partitioned Tables (CLI)](using_coldfront/index.md#managing-partitioned-tables-cli). + +**You do not need Iceberg at all.** If automated PostgreSQL partition +maintenance is all you want - declarative time- or id-based RANGE partitioning +with a premade forward window and automatic age-out of old partitions - +ColdFront's `partitioner` binary is the whole product: stock PostgreSQL (or a +Spock mesh), no cold tier, no DuckDB, no Iceberg, nothing to preload. Each +invocation makes one reconcile pass per managed table: premake the forward +window, ensure the partition covering *now* exists, and detach-then-drop +partitions past retention (`DETACH ... CONCURRENTLY`, never a bare `DROP` of +attached data). Build the partitioner with `make build` and run it from cron: + +```bash +go build -o bin/partitioner ./cmd/partitioner # or: make build +./bin/partitioner +``` + +A database set up this way can gain the cold tier later: +`CREATE EXTENSION coldfront` adopts the `coldfront.partition_config` the +partitioner created, registrations included. + +The partitioner connects from the libpq environment or `--dsn`, like psql, and +reads its tables from `coldfront.partition_config`; it needs no file. A +partition-only deployment has no cold tier: a table with a `hot_period` +belongs to the archiver, and the partitioner leaves it alone. + +Register each managed table in `coldfront.partition_config`: + +```bash +./bin/partitioner register --table events \ + --period monthly --retention "12 months" \ + --strategy drop # drop (DETACH+DROP, destroy; default) + # | detach (DETACH only, keep as a + # standalone table - data preserved) +``` + +## Operating the Partitioner + +Schedule one pass per period or more often - a cron line, or a systemd +`oneshot` service plus timer (systemd then reports a failed pass as a failed +unit, so alerting is free): + +```text +17 * * * * postgres PGDATABASE=mydb /usr/local/bin/partitioner >> /var/log/coldfront-partitioner.log 2>&1 +``` + +Keep the following operational behavior in mind when scheduling it: + +- The exit code is `0` when every table reconciled (a self-healed *behind* + condition still exits `0`) and non-zero when at least one table failed + (`N table(s) failed`), each failure logged with a `[table]` prefix (the bare + table name) followed by the schema-qualified error. Alert on a non-zero exit + code. +- If a table already has a *past* partition but none covers *now* at the start + of a pass (a lagging cron - live inserts had no home), the pass heals the + table (creates the current partition), logs a `WARNING (self-healed)` line, + and still exits `0`. Monitor for that warning in the log rather than via the + exit code, and widen `future_partitions` or run more often. A fresh table + (only just-premade future partitions) is **not** behind: its first reconcile + succeeds cleanly. +- With the default `expiration_strategy: drop`, expiry is + `DETACH CONCURRENTLY` + `DROP TABLE` - the data is **gone**, so back up + before shrinking `retention_period`. Set `expiration_strategy: detach` to + instead leave the expired partition as a standalone table (detached from the + parent, data preserved) and reclaim it yourself. A partition is expired only + once its *entire* range is older than `now − retention_period`, computed + with calendar-accurate PostgreSQL interval arithmetic (a real month, leap + years correct). `detach` is partition-only - the tiered archiver always drops + after exporting to cold. + +## Primary Keys on Time-Partitioned Tables (id Mode) + +PostgreSQL requires a unique index to include the partition key, so a plain +`PRIMARY KEY (id)` is impossible on a table partitioned by a separate `ts` +column. Partition instead by RANGE on a **time-ordered id** and the partition +key *is* the key - a single-column `PRIMARY KEY (id)`: + +```yaml + - source_table: events + partition_column: id + partition_period: monthly + retention_period: 12 months + part_mode: id + id_scheme: snowflake # snowflake | uuidv7 +``` + +`uuidv7` reads the RFC 9562 leading-millisecond timestamp; `snowflake` decodes +the pgEdge Snowflake extension's layout. Either way the manager computes the +month/day partition bounds as id values, so id-order equals time-order and the +forward/retention schedule is unchanged. + +## Two-Level (LIST → RANGE) Sub-Partitioning + +For a table partitioned by `LIST (region)` whose children are themselves +`RANGE`-partitioned by time, add a `sub_partition` block. `values_source` is a +query returning the current level-1 values; the manager provisions and +maintains a RANGE sub-tree under each, creating sub-trees for newly appearing +values automatically: + +```yaml + - source_table: events + partition_column: ts # the level-2 RANGE column + partition_period: monthly + retention_period: 12 months + sub_partition: + values_source: "SELECT code FROM regions" +``` + +Each level-1 child is named `
_`, with the value lowercased and +every character outside `a-z`, `0-9` and `_` replaced by `_`. That name must +fit in 50 bytes, which leaves room for the daily leaf suffix within +PostgreSQL's 63-byte identifier limit. A pass that meets a longer name fails, +for tiered two-level tables as well. The partitioner and the archiver also +fail a pass in which two values map to the same name. diff --git a/docs/usage_tiered.md b/docs/usage_tiered.md new file mode 100644 index 00000000..46730d1c --- /dev/null +++ b/docs/usage_tiered.md @@ -0,0 +1,134 @@ +# Using ColdFront in Tiered Mode + +Before you begin, configure PostgreSQL, Lakekeeper, and your object store by +following [Configuring ColdFront](configuration.md), [Configuring Lakekeeper](configure_lakekeeper.md), +and [Configuring your Object Store](object_store.md); tiered mode archives +aging partitions to the cold tier those guides set up. This page walks +through creating a tiered table, registering it with the archiver, running +the archiver, and the lifecycle that follows, then details two restrictions +tiered tables carry: inbound foreign keys and 2-level tiered tables. + +Create a partitioned table normally: + +```sql +CREATE TABLE events ( + id bigint GENERATED ALWAYS AS IDENTITY, + ts timestamptz NOT NULL, + status text, + data jsonb, + PRIMARY KEY (id, ts) +) PARTITION BY RANGE (ts); + +CREATE TABLE p_2026_04 PARTITION OF events + FOR VALUES FROM ('2026-04-01') TO ('2026-05-01'); +``` + +Then tell the archiver about the table, either with a deployment YAML: + +```yaml +s3: + endpoint: "seaweedfs:8333" + access_key: "admin" + secret_key: "adminsecret" +archiver: + tables: + - source_table: events + partition_period: monthly + hot_period: "1 month" +``` + +```bash +./bin/archiver import --config config.yaml +``` + +or by registering the table directly: + +```bash +./bin/archiver register --table events \ + --period monthly --hot-period "1 month" +# optional: --retention "5 years" DROPs cold data past that age +# (must exceed --hot-period; omit = keep forever) +``` + +See [Writing ColdFront's Configuration](configuration.md#writing-coldfronts-configuration) +for how `register` and `import` write this, and what the server checks, and +[Managing Partitioned Tables (CLI)](using_coldfront/index.md#managing-partitioned-tables-cli) +for `list`, `set`, `remove`, and `export`, which the archiver shares with the +partitioner. + +Run the archiver (typically via cron): + +```bash +./bin/archiver +``` + +`--version` prints the version that `make build` stamps from `git describe` +(the release tag, or the commit it was built from) and exits; a plain +`go build` prints `unknown`. The partitioner and compactor accept the same +flag. + +The archiver also accepts `--debug-export-delay`, a test-only Go duration +(such as `5s`) that holds each partition's capture window open between the +bulk export and the replay, so that a test can race concurrent writes into it. +Leave it unset in production. + +The first run that finds a partition older than `hot_period` renames `events` +→ `_events`, creates the unified view `events`, and registers it; until then +the archiver only premakes partitions, and the table stays a plain partitioned +table. From then on every cycle (1) tiers partitions older than `hot_period` +from hot PG to cold Iceberg and advances the watermark, and (2) if +`retention_period` is set, drops cold Iceberg rows older than it. The data +lifecycle is **hot → `hot_period` → cold → `retention_period` → gone**; +omit `retention_period` to keep cold data forever. + +To remove a tiered table's cold tier entirely rather than waiting out its +retention period, see [Dropping an Iceberg Table](using_coldfront/index.md#dropping-an-iceberg-table). + +## Inbound Foreign Keys + +A tiered table cannot be the target of an enforced foreign key from another +table over its archivable range. Archiving physically removes the rows from +PostgreSQL (export to Iceberg, then `DETACH` + `DROP` the partition), and +PostgreSQL cannot enforce a foreign key against the cold tier. So the `DETACH` +is refused (SQLSTATE 23503) and the archiver fails fast, naming the blocking +constraint. + +This is inherent, not an edge case: referencing rows rarely have a hot-only +lifecycle. They usually outlive the partition they point at, so the foreign key +still pins those rows in PostgreSQL by the time the partition ages into the +cold tier. + +The foreign key is fundamentally incompatible with archiving the rows it +references, so before archiving such a table, drop it: + +```sql +ALTER TABLE event_logs DROP CONSTRAINT event_logs_events_fkey; +``` + +Better still, do not point an enforced foreign key at a tiered table over its +archivable range in the first place. + +## 2-Level (LIST → RANGE) Tiered Tables + +A table partitioned `LIST (region) → RANGE (ts)` can be tiered too - the same +`sub_partition` block as [Using ColdFront in Standalone Partitioned Mode](usage_partitioner.md), +so a partition-manager-managed table can be "upgraded" to tiered by pointing +the archiver at it: + +```yaml + - source_table: regional + partition_column: ts # the RANGE (time) column - required for 2-level + partition_period: monthly + hot_period: 1 month + # retention_period: 5 years # optional cold expiry (region-agnostic, by ts) + sub_partition: + values_source: "SELECT region FROM regions" +``` + +One Iceberg table holds every region (region is an ordinary column); the +archiver tiers leaves a whole `ts` period at a time across **all** regions +before advancing the shared hot/cold watermark, so a period only becomes cold +once it is cold for every region. `id` mode is not supported in tiered mode +(the cold tier is time-keyed). For why the ordering works that way, see the +[Two-level tiering](architecture_tiered.md#two-level-list-range-tiering) +section of the Tiered Mode page. diff --git a/docs/usage_vectors.md b/docs/usage_vectors.md index df63b51a..709f5916 100644 --- a/docs/usage_vectors.md +++ b/docs/usage_vectors.md @@ -1,8 +1,8 @@ # Working with Embeddings ColdFront stores embeddings in the cold tier as Iceberg `list` and keeps -the pgvector interface you already write. A vector column works in both modes: -a tiered table whose recent rows stay in PostgreSQL, and a decoupled table that +the pgvector interface you already write. A vector column works in a tiered +table, whose recent rows stay in PostgreSQL, and in a decoupled table, which lives entirely in Iceberg. ## Creating a Table @@ -34,7 +34,7 @@ You do not need pgvector in a database that has none. A table that already exists in the catalog can be adopted instead, through `coldfront.adopt_iceberg_table()` (see -[Adopting a Table That Already Exists in the Catalog](usage.md#adopting-a-table-that-already-exists-in-the-catalog)). +[Adopting a Table That Already Exists in the Catalog](usage_decoupled.md#adopting-a-table-that-already-exists-in-the-catalog)). Adoption registers a clustered vector column for each Iceberg column named `_cf_vec_list_`, which is where ColdFront keeps each row's cluster. No `coldfront.vector_config` row and no centroids come back with an adopted @@ -432,5 +432,5 @@ To go further with ColdFront, consult the following documents: state, cluster assignment, and probe rewrite behind this guide. - The [Compaction](compaction.md) guide covers the maintenance that keeps clustered tables in cluster order. -- The [Using ColdFront](usage.md) guide covers the table setup both modes - share. +- The [Using ColdFront](using_coldfront/index.md) guide covers the table setup + that tiered and decoupled tables share. diff --git a/docs/using_coldfront/index.md b/docs/using_coldfront/index.md new file mode 100644 index 00000000..3d79d286 --- /dev/null +++ b/docs/using_coldfront/index.md @@ -0,0 +1,303 @@ +# Using ColdFront + +ColdFront offers three operating modes. Pick one per table; tables in +different modes can coexist in the same database. The following table +compares tiered and decoupled modes, which use the cold Iceberg tier: + +| | Tiered (hot + cold) | Decoupled (iceberg-only) | +|---|---|---| +| Where rows live | Recent rows are hot in the PG heap, and archived rows are cold in Iceberg. | Every row is in Iceberg. | +| Setup | Create a partitioned table and let the archiver convert it on the first run that finds a partition older than `hot_period`. | One SQL call, `coldfront.create_iceberg_table(...)`, sets up the table. | +| Archiver | The archiver is required; it runs from cron and moves old partitions to cold. | The archiver is not used. | +| Best when | The workload has a recent-row OLTP part that benefits from PG indexes and transactional ergonomics. | The workload is purely analytic or append-mostly, and you want zero PG storage and stateless compute. | + +Once a tiered or decoupled table exists, **the SQL surface is identical**: +`SELECT`, `INSERT`, `UPDATE`, `DELETE` all work normally against the relation +name (e.g. `events`), except that a write touching the cold tier rejects +`RETURNING`, as the following examples show: + +```sql +-- Reads - pg_duckdb handles the iceberg side; PG handles the heap side +SELECT count(*) FROM events; +SELECT id, status, data->>'k' FROM events WHERE ts >= '2026-04-01'; + +-- Inserts, updates, and deletes all go through the coldfront C hook, +-- which rewrites the query into one PG-set-based statement (hot side) +-- + duckdb.raw_query calls (cold side). For iceberg-only mode every write +-- goes cold; for tiered mode the hook splits by ts vs the watermark. +INSERT INTO events (ts, status, data) VALUES (now(), 'ok', '{"k":1}'); +UPDATE events SET status = 'fixed' WHERE id = 123; +DELETE FROM events WHERE ts < '2025-01-01'; + +-- Bulk INSERT shapes: the source is read once, the hot rows are one +-- set-based INSERT and the cold rows are written in batches: +INSERT INTO events (ts, status, data) VALUES (...), (...), (...); +INSERT INTO events (ts, status, data) SELECT ts, status, data FROM staging; +INSERT INTO events (ts, status, data) SELECT now() + i*'1s'::interval, 'ok', '{}' + FROM generate_series(1, 1000) i; +-- A WITH clause works in both positions; a nested INSERT takes no RETURNING: +WITH moved AS (DELETE FROM staging RETURNING ts, status, data) +INSERT INTO events (ts, status, data) SELECT ts, status, data FROM moved; +WITH i AS (INSERT INTO events (ts, status, data) VALUES (now(), 'ok', '{}')) +SELECT 1; +-- COPY FROM loads the same way, one INSERT per cold_write_batch_size rows: +COPY events (ts, status, data) FROM '/path/to/events.csv' WITH (FORMAT csv); + +-- Transactions work; ROLLBACK undoes Iceberg writes too +BEGIN; + UPDATE events SET status = 'pending' WHERE id = 1; + SELECT status FROM events WHERE id = 1; -- sees 'pending' if row 1 is hot +ROLLBACK; +SELECT status FROM events WHERE id = 1; -- back to whatever it was +``` + +Follow the guide for the mode you want: + +- [Using ColdFront in Tiered Mode](../usage_tiered.md) archives aging + partitions to Iceberg while recent rows stay in the PostgreSQL heap. +- [Using ColdFront in Decoupled Mode](../usage_decoupled.md) keeps every row + in Iceberg from the start. +- [Using ColdFront in Standalone Partitioned Mode](../usage_partitioner.md) + manages plain PostgreSQL partitions with no cold tier at all. + +Operational topics shared across modes - caveats, vended credentials, +supported types, and distributed (mesh) setup - are documented under +Administration, starting with [Caveats](../caveats.md). + +## Dropping an Iceberg Table + +`coldfront.drop_iceberg_table()` removes the Iceberg table backing a registered +relation. The function is the only sanctioned way to do so: a plain +`DROP TABLE` or `DROP VIEW` on a registered relation stays blocked, because it +would leave the cold tier behind with nothing pointing at it. + +The call takes the schema, the table, and an explicit purge decision: + +```sql +-- drop the catalog entry and delete the stored objects +SELECT coldfront.drop_iceberg_table('public', 'events', true); + +-- drop the catalog entry and leave the objects in the object store +SELECT coldfront.drop_iceberg_table('public', 'events', false); +``` + +What remains afterwards depends on the mode, and the call reports which path it +took: + +- in decoupled mode the Iceberg table is the whole relation, so nothing remains + in PostgreSQL. +- in [tiered mode](../usage_tiered.md) the Iceberg table is the cold tier, so the + cold tier is removed and the hot table returns under the relation's own + name, as an ordinary partitioned table holding the data that had not yet + aged out. + +In tiered mode the call also deletes the table's `partition_config` row, which +stops the archiver from tiering it again on the next run. + +There is no default for the purge argument, because the two outcomes are +irreversible in opposite directions. Passing `true` deletes the Parquet and +metadata objects, which for the cold tier are the only copy of that data. +Passing `false` keeps those objects but removes the catalog entry that +ColdFront would need to reach them again, so nothing reclaims them afterwards; +use it when another system is taking ownership of the files. + +ColdFront has no guard of its own against dropping the wrong table. The +preventive control lives in Lakekeeper: table protection has to be enabled +manually, per table, and a protected table cannot be dropped at all, whether or +not purge is requested. This function cannot override a hold. + +Deletion is not instantaneous. The catalog entry disappears with the call, and +Lakekeeper's own background queue removes the objects shortly afterwards, so a +listing taken immediately after the call can still show them. + +## Managing Partitioned Tables (CLI) + +ColdFront's configuration lives in the server. **Per-table lifecycle** lives in +`coldfront.partition_config`, a name-keyed table that replicates by value +across a Spock mesh (like `tiered_views`/`archive_watermark`), so every node +reads identical config - no per-node file syncing. The cold-store credential +lives in `coldfront.storage_secret` and the catalog in the +`coldfront.warehouse` and `coldfront.lakekeeper_endpoint` settings. The +**connection** alone comes from outside, from the libpq environment or +`--dsn`, as for psql. Manage the lifecycle table with the CLI below (both +`partitioner` and `archiver` expose these subcommands; with no subcommand they +do their normal reconcile/archive run). + +`register` is the primary way to manage tables: one command adds or adopts one +table, validated on the spot. `import` and `export` are bulk helpers for +(re)configuring a machine - write a deployment YAML into a fresh server, or +dump the live config to git and replay it on another node - not the day-to-day +path. A YAML passed to anything but `import` is checked against the server and +refused if it disagrees in any value. +`register` and `import` run the full +validation. `set` checks every value of the row that it would leave; it re-runs +the rest of the validation only when it changes the partition period or column, +`hot_period`, `retention_period` or the sub-partition source. + +The data lifecycle is **hot PG → `hot_period` → cold Iceberg → +`retention_period` → dropped** (tiered) or **hot PG → `retention_period` +→ dropped** (partition-only). Setting `hot_period` makes a table tiered; +omitting it makes it partition-only. + +### What Registration Refuses + +`register` and `import` run the same validation, so a table that one rejects +cannot be added by the other; `set` re-runs it when it changes a validated +field. Registration fails when: + +- a setting has a value that the table's `CHECK` constraints refuse. The + refusal names the flag, or on `import` the key, and the values that the + setting accepts; `set` checks the row that its change would leave. + Registration refuses: + + - a `--period` other than `monthly` or `daily`. + - a `--premake` below 1. + - a `--part-mode` other than `timestamp` or `id`. + - an `--id-scheme` other than `uuidv7` or `snowflake`, or one without + `--part-mode id`. + - `--part-mode id` without an `--id-scheme`. + - a `--strategy` other than `drop` or `detach`. + - `--part-mode id` or `--strategy detach` with a `--hot-period`. + - a `--sub-values-source` without a `--column`. + - a table with neither a `--hot-period` nor a `--retention`. + +- `--column`, or `partition_column` on `import`, names a column that is not the + table's partition key. On a two-level table, registration refuses only a + column that the table lacks. The refusal names the flag or key and, for a + flat table, the table's partition key. +- any relation in the partition tree is `UNLOGGED`, because that data is + truncated after a crash and replicates nowhere, so archiving it would + preserve rows PostgreSQL never promised to keep. +- the table has a `DEFAULT` partition. Rows the `DEFAULT` partition catches + have no time bounds, so they can never be tiered or expired, and PostgreSQL + refuses `DETACH PARTITION ... CONCURRENTLY` for every partition of a table + that has one, which is how partitions are expired. The partition's mere + existence is enough; it does not have to hold any rows. Move any rows the + `DEFAULT` partition holds into real partitions and detach it. +- the name differs only by case from an already-registered table. PostgreSQL + keeps `public."Events"` and `public.events` apart, but DuckDB matches + identifiers case-insensitively even when they are quoted, so both names would + resolve to one Iceberg table and overwrite each other. +- the name starts with an underscore, which is reserved for the tiered hot + table. +- the name leaves no room for the generated partition suffix: 53 bytes for + monthly and 50 for daily, within PostgreSQL's 63-byte identifier limit. +- a tiered table has a column whose type has no Iceberg equivalent (see + [Supported Column Types](../supported_types.md)). Every column + goes through the same type map the cold tier itself uses, so the answer + comes back at the prompt rather than hours later from cron, and the + refusal names the column. Partition-only tables are exempt: nothing about + them reaches Iceberg, so their column types are PostgreSQL's business + alone. +- a tiered table's name, or the name of one of its columns, is one DuckDB + parses as a keyword, such as `by`, `at` or `show` (see [Caveats](../caveats.md)). + The archiver checks the column names again before it tiers or expires data; + it refuses a column added after registration before any table or data reaches + Iceberg. + +Registration validates the table as it is at that moment, not continuously. +Adding a `DEFAULT` partition to an already-registered table is therefore not +caught then; the next archiver run fails while reading that table's partition +bounds. + +Other unusual bounds are supported rather than refused. Partitions open at +either end (`MINVALUE`, `MAXVALUE`, `infinity`) are handled, as is PostgreSQL's +full timestamp range, from 4713 BC through year 294276. + +`hot_period` and `retention_period` are native PostgreSQL `interval`s - use any +interval syntax (`1 month`, `90 days`, `1 year 2 mons`, `5 years`). Expiry +boundaries are computed with calendar-accurate interval arithmetic +(`now() - period`: real months, leap years), and `retention_period` must exceed +`hot_period` (validated at `register`/`set` time). + +The following table describes the CLI subcommands: + +| Command | Purpose | +|---|---| +| `register` | Adds or adopts a table, and validates that the PRIMARY KEY covers the partition key. | +| `list` | Shows the managed tables and their lifecycle. | +| `set` | Changes fields, or disables or enables a table with `--disable`/`--enable`. | +| `remove` | Stops managing a table; the table itself is left intact. | +| `import` | Writes a deployment YAML into the server once: its `archiver.tables` as rows, validated as `register` validates each table, and its `s3:` or `azure:` stanza as the storage secret. | +| `export` | Dumps the active (enabled) config to YAML or SQL, as a git-reviewable backup to replay on another node. | + +The following examples register, inspect, and change managed tables: + +```bash +# Partition-only: keep 3 future partitions, drop those older than 12 months. +partitioner register --table events --period monthly --retention "12 months" + +# Partition-only, but DETACH (preserve) expired partitions instead of dropping them. +partitioner register --table events --period monthly \ + --retention "12 months" --strategy detach + +# Tiered: tier to cold Iceberg after 1 month, then drop cold data after 5 years. +archiver register --table events --period monthly \ + --hot-period "1 month" --retention "5 years" + +# id mode - a real single-column PRIMARY KEY (id) on a snowflake-keyed table. +partitioner register --table events --period monthly \ + --column id --part-mode id --id-scheme snowflake --retention "1 year" + +# 2-level LIST(region) → RANGE(ts), tiered; region values come from a table. +archiver register --table regional --period monthly --column ts \ + --hot-period "1 month" --sub-values-source "SELECT region FROM regions" + +partitioner list # what's managed +partitioner set --table events --retention "24 months" +partitioner set --table events --disable # pause (keeps the row) +partitioner remove --table events # unregister, keep the table +partitioner import --config deploy.yaml # write a deployment YAML into the server once +partitioner export > managed.yaml # active config, git-reviewable (--format sql for INSERTs) +``` + +Run `partitioner` (or `archiver`) with `help` or `--help` for the command +overview, and with no arguments for its normal run; every subcommand has a +detailed `--help` with worked examples. The write commands accept +`--print-sql` (emit the SQL without running it - review/commit it; `import` +prints its table `INSERT`s); `register` and `import` also accept `--dry-run`. +`set --enable`/`--disable` (mutually exclusive) pause/resume a table without +removing it; a disabled table is skipped by reconcile and omitted from +`export`. Per table, only the cadence and a lifecycle boundary (`hot_period` or +`retention_period`) are required; `partition_column` is auto-detected from +`pg_catalog` for flat tables (required for 2-level). `register` checks the +lifecycle rules before it writes the row, and the table's `CHECK` constraints +enforce them again at write time. + +The subcommands also take the following connection and table flags: + +- `--dsn` takes a PostgreSQL connection string on every subcommand. Without + it the connection comes from the libpq environment, as for psql. +- `--config` names a deployment YAML. On `import` it is required, since it + names the file to write into the server. On any other subcommand the file is + checked against the server, value by value, and refused if it disagrees; its + `postgres.dsn` connects when `--dsn` is unset. +- `--schema` (default `public`) names the table's schema on `register`, `set` + and `remove`. +- `--premake` (default `3`) sets on `register` how many future partitions are + kept ahead of now, stored as `future_partitions`. + +`set` changes only the fields whose flags are passed: `--period`, `--column`, +`--premake`, `--hot-period`, `--retention`, `--sub-values-source` and +`--strategy`, each taking the same value as on `register`. An empty value +clears the field, so `--hot-period ""` makes a tiered table partition-only and +hands it to the partitioner. A change that leaves neither a `hot_period` nor a +`retention_period` is refused. + +For managing tables, the archiver and partitioner read the managed set only +from `coldfront.partition_config`. `register` adds one table; `import` adds +every table in a YAML `archiver.tables` list. Both write through the same +validation (the PK must cover the partition key; `retention` must exceed +`hot_period`), so an imported table is checked exactly as a registered one. +`export` dumps the active rows back to YAML or SQL for git. + +The archiver manages only the rows that have a `hot_period`, and the +partitioner only the rows without one, so each table belongs to one binary. A +run that finds none of its own rows exits non-zero with +`no tables in coldfront.partition_config`, naming the `register` and `import` +commands that add one. + +In an `archiver.tables` entry, `source_schema` names the table's schema and +defaults to `public`. `source_table` also accepts the `schema.table` form, and +an explicit `source_schema` takes precedence over that prefix. +`future_partitions` defaults to `3`, as `--premake` does on `register`. diff --git a/docs/vended_credentials.md b/docs/vended_credentials.md new file mode 100644 index 00000000..43f136b5 --- /dev/null +++ b/docs/vended_credentials.md @@ -0,0 +1,51 @@ +# Using Vended Credentials + +Vended credentials let a deployment run with no object-store credential stored +in the database, in a DuckDB secret file, or in an archiver config; this suits +compliance environments that forbid persisting long-term keys. Lakekeeper +issues a short-lived, per-table credential (an S3 STS access key, secret, and +session token) at read and write time, and ColdFront uses it directly. The +long-term credential lives only in the Lakekeeper warehouse. + +Enable vended credentials with a single call that stores no credential: + +```sql +SELECT coldfront.set_storage_secret_vended(); +``` + +This writes a `coldfront.storage_secret` row marked as vended, so nothing is +materialized as a DuckDB secret; `ensure_attached()` then attaches the catalog +with credential vending turned on. The archiver and the compactor read the +same row, so a vended deployment has no `s3:` or `azure:` stanza to import and +no credential in any file. + +Vended mode targets the two clouds that issue scoped credentials: AWS S3 (STS) +and Azure ADLS Gen2 (SAS). The `set_storage_secret_vended()` call above enables +AWS S3; the same call with `'azure'` enables Azure through the identical path: + +```sql +SELECT coldfront.set_storage_secret_vended('azure'); +``` + +Vending requires a Lakekeeper warehouse configured to vend credentials: + +- On AWS S3, the warehouse uses `flavor: aws` with `sts-enabled: true`, an + `assume-role-arn` for a bucket-scoped IAM role, and an `external-id` on the + warehouse credential that the role's trust policy also requires. The full + warehouse and IAM-role setup is in the Configuring Lakekeeper section of + [Configuring your Object Store](object_store.md#configuring-lakekeeper). +- On Azure ADLS Gen2, the warehouse is an `adls` warehouse with `sas-enabled` + (on by default). Lakekeeper vends a per-container SAS token; the warehouse + credential can be a `shared-access-key`, `client-credentials`, or + `azure-system-identity`. + +Google Cloud Storage over the S3-interoperability endpoint has no STS to issue +short-lived credentials, so GCS stays on static HMAC credentials. + +The compactor runs fully under vended credentials: compaction, snapshot expiry, +and orphan-file reclaim all use the vended per-table credentials. + +Switching a running deployment between static and vended credentials changes +the attach mode, which is fixed per PostgreSQL backend at attach time; open a +new session (or restart the archiver and compactor, which are short-lived +processes) so the new mode takes effect. diff --git a/docs/walkthrough_decoupled.md b/docs/walkthrough_decoupled.md index 6cec6a5e..9654094a 100644 --- a/docs/walkthrough_decoupled.md +++ b/docs/walkthrough_decoupled.md @@ -5,10 +5,10 @@ cwd: ../ Decoupled mode stores a table entirely in Iceberg from the first row. PostgreSQL holds a thin wrapper view and a registry entry. This is a fresh -table - there is no migration from the tiered demo, and the two modes are -independent. +table - there is no migration from the tiered demo, and tiered and decoupled +tables are independent. -Run the steps in [Setting Up the Stack](walkthrough.md#setting-up-the-stack) +Run the steps in [Setting Up the Stack](exploring_coldfront/index.md#setting-up-the-stack) before you start this demo. The demo also needs the extensions and storage secret from [Step 4](walkthrough_tiered.md#enabling-the-extensions) and [Step 5](walkthrough_tiered.md#pointing-coldfront-at-the-object-store) of the @@ -56,7 +56,7 @@ DELETE FROM events_lake WHERE id = 2; SELECT count(*) AS after_delete FROM events_lake; ``` -All four DML operations reach the Iceberg table transparently. The coldfront +All four DML operations reach the Iceberg table transparently. The `coldfront` extension intercepts each statement on the view and rewrites it to the Iceberg path via pg_duckdb. @@ -178,10 +178,10 @@ To go further with ColdFront, consult the following guides: - The [Partitioner Demo](walkthrough_partitioner.md) manages PostgreSQL range partitions without any cold tier. -- The [Using ColdFront](usage.md) guide covers both modes in depth, including - the full one-time setup, supported column types, the partition manager CLI, - and tuning options. -- The [Architecture](architecture.md) overview explains the shared mechanics +- The [Using ColdFront](using_coldfront/index.md) guide compares the tiered, + decoupled, and standalone partitioned modes, and covers dropping Iceberg + tables and the partition management CLI. +- The [Architecture](architecture_guides/index.md) overview explains the shared mechanics and links to the per-mode deep dives. -- The [Tearing Down the Stack](walkthrough.md#tearing-down-the-stack) section +- The [Tearing Down the Stack](exploring_coldfront/index.md#tearing-down-the-stack) section describes how to stop the stack and remove its data. diff --git a/docs/walkthrough_distributed.md b/docs/walkthrough_distributed.md index d1e1859f..3c5f2680 100644 --- a/docs/walkthrough_distributed.md +++ b/docs/walkthrough_distributed.md @@ -44,7 +44,7 @@ If port 5442, 5443, 8191, or 8343 is already in use on your host, set or `COLDFRONT_MESH_S3_PORT` before `up`. Next, run the `curl` block from -[Setting Up the Stack](walkthrough.md#setting-up-the-stack) with port 8191 in +[Setting Up the Stack](exploring_coldfront/index.md#setting-up-the-stack) with port 8191 in place of 8181. That creates the `wh` warehouse and `public` namespace on the mesh's own Lakekeeper. @@ -228,12 +228,12 @@ number and no fork. To go further with ColdFront, consult the following guides: -- The [Using ColdFront](usage.md) guide covers both modes in depth, including - the full one-time setup, supported column types, the partition manager CLI, - and tuning options. -- The [Architecture](architecture.md) overview explains the shared mechanics +- The [Using ColdFront](using_coldfront/index.md) guide compares the tiered, + decoupled, and standalone partitioned modes, and covers dropping Iceberg + tables and the partition management CLI. +- The [Architecture](architecture_guides/index.md) overview explains the shared mechanics and links to the per-mode deep dives. - The [Compaction](compaction.md) guide covers cold-tier maintenance: compaction, snapshot expiry, and orphan-file removal. -- The [Tearing Down the Stack](walkthrough.md#tearing-down-the-stack) section +- The [Tearing Down the Stack](exploring_coldfront/index.md#tearing-down-the-stack) section describes how to stop the stack and remove its data. diff --git a/docs/walkthrough_partitioner.md b/docs/walkthrough_partitioner.md index 992a74e7..065f1861 100644 --- a/docs/walkthrough_partitioner.md +++ b/docs/walkthrough_partitioner.md @@ -8,7 +8,7 @@ tier. If all you need is automated partition maintenance on stock PostgreSQL, the partitioner is the whole product - no Iceberg, no DuckDB, no archiver cold path. -Run the steps in [Setting Up the Stack](walkthrough.md#setting-up-the-stack) +Run the steps in [Setting Up the Stack](exploring_coldfront/index.md#setting-up-the-stack) before you start this demo. ## Creating the Demo Table @@ -66,7 +66,7 @@ WHERE inhparent = 'part_demo'::regclass; A table registered with `--strategy detach` keeps its expired partitions as standalone tables instead of dropping them. The -[Managing Partitioned Tables (CLI)](usage.md#managing-partitioned-tables-cli) +[Managing Partitioned Tables (CLI)](using_coldfront/index.md#managing-partitioned-tables-cli) section describes the partition CLI and shows more `register` examples. ## Next Steps @@ -75,8 +75,8 @@ To go further with ColdFront, consult the following guides: - The [Distributed Demo](walkthrough_distributed.md) points two PostgreSQL nodes at one shared lake. -- The [Using ColdFront](usage.md) guide covers both modes in depth, including - the full one-time setup, supported column types, the partition manager CLI, - and tuning options. -- The [Tearing Down the Stack](walkthrough.md#tearing-down-the-stack) section +- The [Using ColdFront](using_coldfront/index.md) guide compares the tiered, + decoupled, and standalone partitioned modes, and covers dropping Iceberg + tables and the partition management CLI. +- The [Tearing Down the Stack](exploring_coldfront/index.md#tearing-down-the-stack) section describes how to stop the stack and remove its data. diff --git a/docs/walkthrough_tiered.md b/docs/walkthrough_tiered.md index 62bc626b..65d9b09b 100644 --- a/docs/walkthrough_tiered.md +++ b/docs/walkthrough_tiered.md @@ -8,7 +8,7 @@ database full of data, add ColdFront to it, and let the archiver relocate the cold majority to object storage - without migrating to a new database or changing a line of application SQL. -Run the steps in [Setting Up the Stack](walkthrough.md#setting-up-the-stack) +Run the steps in [Setting Up the Stack](exploring_coldfront/index.md#setting-up-the-stack) before you start this demo. The following table shows the eleven steps this demo covers: @@ -33,7 +33,7 @@ Setup starts the infrastructure: PostgreSQL (your database), plus Lakekeeper and SeaweedFS. The cold-storage side sits idle until you point ColdFront at it in [Step 5](#pointing-coldfront-at-the-object-store). Run the `docker compose up` command shown in -[Setting Up the Stack](walkthrough.md#setting-up-the-stack). Then confirm that +[Setting Up the Stack](exploring_coldfront/index.md#setting-up-the-stack). Then confirm that the `db`, `lakekeeper-db`, and `lakekeeper` services report `healthy`: ```bash @@ -595,9 +595,9 @@ To go further with ColdFront, consult the following guides: - The [Decoupled Mode Demo](walkthrough_decoupled.md) stores a table in Iceberg from the first row and adopts a table that another engine wrote. -- The [Configuring your Object Store](object_store.md) guide takes you from an - empty bucket to a working cold tier on AWS S3. -- The [Architecture](architecture.md) overview explains the shared mechanics +- The [Configuring your Object Store](object_store.md) + guide takes you from an empty bucket to a working cold tier on AWS S3. +- The [Architecture](architecture_guides/index.md) overview explains the shared mechanics and links to the per-mode deep dives. -- The [Tearing Down the Stack](walkthrough.md#tearing-down-the-stack) section +- The [Tearing Down the Stack](exploring_coldfront/index.md#tearing-down-the-stack) section describes how to stop the stack and remove its data. diff --git a/mkdocs.yml b/mkdocs.yml index 162a029a..69ff8c8d 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -30,6 +30,7 @@ theme: - navigation.tracking - navigation.prune - navigation.top + - navigation.indexes - toc.follow palette: @@ -57,27 +58,45 @@ repo_url: https://github.com/pgEdge/ColdFront nav: - pgEdge ColdFront: index.md - - Installing and Configuring ColdFront: installation.md - - Getting Started: - - Getting Started with ColdFront: walkthrough.md + + - Architecture Guides: + - architecture_guides/index.md + - Tiered Operating Mode: architecture_tiered.md + - Decoupled (Iceberg-Only) Operating Mode: architecture_decoupled.md + - Vector Storage Architecture: architecture_vectors.md + + - Installing and Configuring ColdFront: + - Installing ColdFront: installation.md + - Configuring ColdFront: + - configuration.md + - Configuring Lakekeeper: configure_lakekeeper.md + - Configuring your Object Store: object_store.md + - Configuring a Distributed Deployment: distributed_setup.md + + - Using ColdFront: + - using_coldfront/index.md + - Using ColdFront in Tiered Mode: usage_tiered.md + - Using ColdFront in Decoupled Mode: usage_decoupled.md + - Using ColdFront in Standalone Partitioned Mode: usage_partitioner.md + - Working with Embeddings: usage_vectors.md + + - Exploring ColdFront Modes: + - exploring_coldfront/index.md - Exploring Tiered Storage: walkthrough_tiered.md - Exploring Decoupled Mode: walkthrough_decoupled.md - Exploring the Standalone Partitioner: walkthrough_partitioner.md - Exploring Distributed Mode: walkthrough_distributed.md - - Configuring your Object Store: object_store.md - - Architecture: - - pgEdge ColdFront - Architecture: architecture.md - - ColdFront - Tiered Operating Mode: architecture_tiered.md - - Decoupled (Iceberg-Only) Operating Mode: architecture_decoupled.md - - Vector Storage Architecture: architecture_vectors.md + - Connecting to ColdFront: connecting.md - - Using ColdFront: usage.md - - Working with Embeddings: usage_vectors.md - - COMPACTOR - Cold-Tier Table Maintenance: compaction.md + - Administration: + - Understanding ColdFront Caveats: caveats.md + - Using Vended Credentials: vended_credentials.md + - Supported Column Types: supported_types.md + - Compacting the Cold Tier: compaction.md - Developer Resources: - Formal Model - ColdFront Decoupled-Mode Bakery (TLA+/PlusCal): formal/README.md - - Changelog: changelog.md + - Release Notes - Changelog: changelog.md - License: LICENSE.md