From d0bf088ba96bbfbf4c75f5ac35bc01ed01e9e913 Mon Sep 17 00:00:00 2001 From: Susan Douglas Date: Wed, 14 Oct 2026 07:51:54 -0400 Subject: [PATCH 01/33] docs: mirror the MkDocs nav in the README table of contents --- README.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index 93104291..525dd2ad 100644 --- a/README.md +++ b/README.md @@ -6,9 +6,8 @@ The ColdFront documentation consists of the following guides: -- [Introduction](docs/index.md) -- Architecture Guides - - [Architecture Overview](docs/architecture_guides/index.md) +- [pgEdge ColdFront](docs/index.md) +- [Architecture Guides](docs/architecture_guides/index.md) - [Tiered Operating Mode](docs/architecture_tiered.md) - [Decoupled (Iceberg-Only) Operating Mode](docs/architecture_decoupled.md) - [Vector Storage Architecture](docs/architecture_vectors.md) @@ -35,8 +34,9 @@ The ColdFront documentation consists of the following guides: - [Supported Column Types](docs/supported_types.md) - [Compacting the Cold Tier](docs/compaction.md) - Developer Resources - - [Formal Model - ColdFront Decoupled-Mode Bakery](docs/formal/README.md) -- [Release Notes](docs/changelog.md) + - [Formal Model - ColdFront Decoupled-Mode Bakery (TLA+/PlusCal)](docs/formal/README.md) +- [Release Notes - Changelog](docs/changelog.md) +- [License](docs/LICENSE.md) ColdFront keeps tables in PostgreSQL and cold data in Apache Iceberg (Parquet on S3-compatible, Azure, or GCS storage), and the cold tier is both readable From 205e931fcc70b7a91296f8b08bdbf66f6638327a Mon Sep 17 00:00:00 2001 From: Susan Douglas Date: Wed, 14 Oct 2026 11:30:26 -0400 Subject: [PATCH 02/33] docs: restructure the README around installing and configuring --- README.md | 747 +++++++++++++++++++++++++++++++++++++++--------------- 1 file changed, 546 insertions(+), 201 deletions(-) diff --git a/README.md b/README.md index 525dd2ad..95c926f9 100644 --- a/README.md +++ b/README.md @@ -7,68 +7,112 @@ The ColdFront documentation consists of the following guides: - [pgEdge ColdFront](docs/index.md) -- [Architecture Guides](docs/architecture_guides/index.md) - - [Tiered Operating Mode](docs/architecture_tiered.md) - - [Decoupled (Iceberg-Only) Operating Mode](docs/architecture_decoupled.md) - - [Vector Storage Architecture](docs/architecture_vectors.md) +- [Architecture Guides](docs/architecture_guides/index.md) - Describes the + shared architecture and core mechanics. + - [Tiered Operating Mode](docs/architecture_tiered.md) - Describes tiered + mode (hot PostgreSQL plus cold Iceberg) in depth. + - [Decoupled (Iceberg-Only) Operating Mode](docs/architecture_decoupled.md) - + Describes decoupled (Iceberg-only) mode in depth. + - [Vector Storage Architecture](docs/architecture_vectors.md) - Describes + vector storage internals: type mapping, routing state, cluster + assignment, and layout. - Installing and Configuring ColdFront - - [Installing ColdFront](docs/installation.md) - - [Configuring ColdFront](docs/configuration.md) - - [Configuring Lakekeeper](docs/configure_lakekeeper.md) - - [Configuring your Object Store](docs/object_store.md) - - [Configuring a Distributed Deployment](docs/distributed_setup.md) -- [Using ColdFront](docs/using_coldfront/index.md) - - [Using ColdFront in Tiered Mode](docs/usage_tiered.md) - - [Using ColdFront in Decoupled Mode](docs/usage_decoupled.md) - - [Using ColdFront in Standalone Partitioned Mode](docs/usage_partitioner.md) - - [Working with Embeddings](docs/usage_vectors.md) -- [Exploring ColdFront Modes](docs/exploring_coldfront/index.md) - - [Exploring Tiered Storage](docs/walkthrough_tiered.md) - - [Exploring Decoupled Mode](docs/walkthrough_decoupled.md) - - [Exploring the Standalone Partitioner](docs/walkthrough_partitioner.md) - - [Exploring Distributed Mode](docs/walkthrough_distributed.md) -- [Connecting to ColdFront](docs/connecting.md) + - [Installing ColdFront](docs/installation.md) - Documents installing from + a package or building from source (Docker or bare-metal), and testing and + CI. + - [Configuring ColdFront](docs/configuration.md) - Documents configuring + PostgreSQL and ColdFront's own configuration. + - [Configuring Lakekeeper](docs/configure_lakekeeper.md) - Installs, + configures, and bootstraps Lakekeeper, then installs the extensions + each database needs. + - [Configuring your Object Store](docs/object_store.md) - Gets + ColdFront running end to end, for any of four supported backends: + S3-compatible, virtual-hosted cloud S3, GCS, or Azure ADLS Gen2. + - [Configuring a Distributed Deployment](docs/distributed_setup.md) - + Documents running a multi-writer decoupled-mode Spock mesh against + one shared lake. +- [Using ColdFront](docs/using_coldfront/index.md) - Compares the three + operating modes, documents the shared SQL surface, and links to each mode's + guide. + - [Using ColdFront in Tiered Mode](docs/usage_tiered.md) - Documents + archiving aging partitions to Iceberg while recent rows stay in the + PostgreSQL heap. + - [Using ColdFront in Decoupled Mode](docs/usage_decoupled.md) - Documents + tables that live entirely in Iceberg from the first row, and dropping or + releasing a registered table. + - [Using ColdFront in Standalone Partitioned Mode](docs/usage_partitioner.md) - + Documents managing PostgreSQL partitions with no cold tier at all, and + the partition CLI shared with tiered mode. + - [Working with Embeddings](docs/usage_vectors.md) - Documents storing and + searching embeddings with the pgvector interface. +- [Exploring ColdFront Modes](docs/exploring_coldfront/index.md) - Sets up the + demo stack and runs ColdFront hands-on. + - [Exploring Tiered Storage](docs/walkthrough_tiered.md) - Adds ColdFront + to an existing database and moves its cold data to object storage. + - [Exploring Decoupled Mode](docs/walkthrough_decoupled.md) - Stores a + table in Iceberg from the first row and adopts a table that another + engine wrote. + - [Exploring the Standalone Partitioner](docs/walkthrough_partitioner.md) - + Manages PostgreSQL range partitions without any cold tier. + - [Exploring Distributed Mode](docs/walkthrough_distributed.md) - Points + two PostgreSQL nodes at one shared lake. +- [Connecting to ColdFront](docs/connecting.md) - Documents onboarding an + application role with least privilege. - Administration - - [Understanding ColdFront Caveats](docs/caveats.md) - - [Using Vended Credentials](docs/vended_credentials.md) - - [Supported Column Types](docs/supported_types.md) - - [Compacting the Cold Tier](docs/compaction.md) + - [Understanding ColdFront Caveats](docs/caveats.md) - Documents cross-tier + isolation, mixed transactions, and other edge cases shared between tiered + and decoupled tables. + - [Using Vended Credentials](docs/vended_credentials.md) - Documents + running with no stored object-store credential, for AWS S3 and Azure ADLS + Gen2. + - [Supported Column Types](docs/supported_types.md) - Documents the + PostgreSQL column types, including arrays, that a cold tier supports. + - [Compacting the Cold Tier](docs/compaction.md) - Documents cold-tier + table maintenance: compaction, snapshot expiry, and orphan-file removal. - Developer Resources - [Formal Model - ColdFront Decoupled-Mode Bakery (TLA+/PlusCal)](docs/formal/README.md) - [Release Notes - Changelog](docs/changelog.md) - [License](docs/LICENSE.md) ColdFront keeps tables in PostgreSQL and cold data in Apache Iceberg (Parquet -on S3-compatible, Azure, or GCS storage), and the cold tier is both readable -and writable through the same SQL with no application changes. The application -queries every table as an ordinary PostgreSQL relation, and both operating -modes present the same standard SQL surface. +on S3-compatible, Azure, or GCS storage). The cold tier is both readable and +writable through the same SQL, with no application changes. The application +queries every table as an ordinary PostgreSQL relation, with standard +`SELECT`, `INSERT`, `UPDATE`, and `DELETE`, and `MERGE` on PostgreSQL 17 and +later. > [!WARNING] > ColdFront is beta software under active development. Do not use it in > production. Interfaces, on-disk formats, and behavior may change without > notice, and data loss is possible. -ColdFront provides two operating modes: +ColdFront provides three operating modes: - Tiered mode keeps recent data in native PostgreSQL partitions and archives older data to Iceberg on a watermark; the application reads a single unified view, and the archiver moves rows from hot to cold on a schedule. - Decoupled mode stores the table entirely in Iceberg from the first row; - PostgreSQL holds a thin wrapper view and a registry row, and the coldfront + PostgreSQL holds a thin wrapper view and a registry row, and the `coldfront` extension handles every data-modifying statement on that view. +- Standalone partitioned mode manages time- or id-based PostgreSQL range + partitions with no cold tier; the `partitioner` binary creates upcoming + partitions and detaches and drops partitions past retention, on stock + PostgreSQL with no DuckDB or Iceberg. -Both modes coexist within one database, and you choose the mode per table at -creation time. The SQL surface is identical for both modes: standard `SELECT`, -`INSERT`, `UPDATE`, and `DELETE` against the relation, and `MERGE` on -PostgreSQL 17 and later. +Tables in different modes can share one database, and you choose the mode for +each table: tiered when you register an existing partitioned table with a hot +period, standalone partitioned when you register one without a hot period, +and decoupled when you create or adopt an Iceberg table. Decoupled mode scales out horizontally across many PostgreSQL nodes that share one Lakekeeper catalog and one object store. The bakery protocol in the -coldfront extension serializes Iceberg commits on the PostgreSQL side using +`coldfront` extension serializes Iceberg commits on the PostgreSQL side using Spock-replicated Snowflake tickets, so concurrent writers never collide at the -catalog. The protocol implements Lamport mutual exclusion with the -Ricart-Agrawala deferred-reply optimization, and the +catalog. The protocol implements +[Lamport mutual exclusion](https://en.wikipedia.org/wiki/Lamport%27s_distributed_mutual_exclusion_algorithm) +with the +[Ricart-Agrawala](https://en.wikipedia.org/wiki/Ricart%E2%80%93Agrawala_algorithm) +deferred-reply optimization, and the [formal model](docs/formal/README.md) verifies its safety with TLA+. ## How It Works @@ -112,110 +156,468 @@ tier, so the application sees one relation: cron job: moves partitions out of the hot window: hot -> cold ``` -## Installation - -If you are new to ColdFront, run the [guided walkthrough](docs/exploring_coldfront/index.md) -to see all three modes in action with copy-pasteable commands. - -ColdFront is open source under the PostgreSQL License and runs on PostgreSQL -16, 17, and 18: stock PostgreSQL for a single node, and pgEdge's PostgreSQL -build with Spock, which the Docker image is based on, for distributed mode. The -full build workflow lives in the -**[Installation guide](docs/installation.md)**: build the thin ColdFront layer -on top of the published DuckDB 1.5.x base image (or build the base yourself), -or install bare-metal. Then continue with the -[walkthrough](docs/exploring_coldfront/index.md). - -### Setting Up on Cloud S3? - -When the image is built, the -**[Configuring your Object Store](docs/object_store.md)** guide takes you from -an empty bucket to a working cold tier end-to-end. - -## Configuration - -ColdFront reads its settings from two places. The server settings live in -`postgresql.conf`: `shared_preload_libraries = 'pg_duckdb,coldfront'`, -`coldfront.warehouse` and `coldfront.lakekeeper_endpoint`, plus -`snowflake.node` and `coldfront.loopback_dsn` on every node of a mesh. The -Docker image writes them on first start. The archiver, partitioner and -compactor connect from the libpq environment or `--dsn` and read everything -else from the server: each table's lifecycle in `coldfront.partition_config`, -the cold-store credential in `coldfront.storage_secret` and the catalog -settings above. The `import` command takes a deployment YAML, modeled on -[config.example.yaml](config.example.yaml), and writes it into the server once. -ColdFront has no configuration file. A YAML passed to any later run is checked -against the server, and the run takes only `postgres.dsn` from the file. -For every setting, see the -[Configuring Lakekeeper](docs/configure_lakekeeper.md) guide and the -[Configuring PostgreSQL](docs/configuration.md#configuring-postgresql) -section of the Configuring ColdFront guide. - -## Using ColdFront - -[Using ColdFront in Tiered Mode](docs/usage_tiered.md), -[Using ColdFront in Decoupled Mode](docs/usage_decoupled.md), and -[Using ColdFront in Standalone Partitioned Mode](docs/usage_partitioner.md) -document each mode in depth; the -[Using ColdFront](docs/using_coldfront/index.md) guide helps you pick one, and -the [walkthrough](docs/exploring_coldfront/index.md) demos run each mode on a -sample table. Operational topics shared across modes - -[caveats](docs/caveats.md), [vended credentials](docs/vended_credentials.md), -[supported column types](docs/supported_types.md), and -[compaction](docs/compaction.md) - are documented under Administration; to run -ColdFront on more than one node, see -[Configuring a Distributed Deployment](docs/distributed_setup.md). - -## Documentation - -The following table lists the ColdFront guides and what each one -documents: - -| Doc | Contents | -|---|---| -| [Architecture Overview](docs/architecture_guides/index.md) | Describes the shared architecture and core mechanics. | -| [Tiered Operating Mode](docs/architecture_tiered.md) | Describes tiered mode (hot PostgreSQL plus cold Iceberg) in depth. | -| [Decoupled (Iceberg-Only) Operating Mode](docs/architecture_decoupled.md) | Describes decoupled (Iceberg-only) mode in depth. | -| [Vector Storage Architecture](docs/architecture_vectors.md) | Describes vector storage internals: type mapping, routing state, cluster assignment, and layout. | -| [Installing ColdFront](docs/installation.md) | Documents installing from a package or building from source (Docker or bare-metal), and testing and CI. | -| [Configuring ColdFront](docs/configuration.md) | Documents configuring PostgreSQL and ColdFront's own configuration. | -| [Configuring Lakekeeper](docs/configure_lakekeeper.md) | Installs, configures, and bootstraps Lakekeeper, then installs the extensions each database needs. | -| [Configuring your Object Store](docs/object_store.md) | Gets ColdFront running end to end, for any of four supported backends: S3-compatible, virtual-hosted cloud S3, GCS, or Azure ADLS Gen2. | -| [Configuring a Distributed Deployment](docs/distributed_setup.md) | Documents running a multi-writer decoupled-mode Spock mesh against one shared lake. | -| [Using ColdFront](docs/using_coldfront/index.md) | Compares the three operating modes, documents the shared SQL surface, and links to each mode's guide. | -| [Using ColdFront in Tiered Mode](docs/usage_tiered.md) | Documents archiving aging partitions to Iceberg while recent rows stay in the PostgreSQL heap. | -| [Using ColdFront in Decoupled Mode](docs/usage_decoupled.md) | Documents tables that live entirely in Iceberg from the first row, and dropping or releasing a registered table. | -| [Using ColdFront in Standalone Partitioned Mode](docs/usage_partitioner.md) | Documents managing PostgreSQL partitions with no cold tier at all, and the partition CLI shared with tiered mode. | -| [Working with Embeddings](docs/usage_vectors.md) | Documents storing and searching embeddings with the pgvector interface. | -| [Exploring ColdFront Modes](docs/exploring_coldfront/index.md) | Sets up the demo stack and runs ColdFront hands-on. | -| [Exploring Tiered Storage](docs/walkthrough_tiered.md) | Adds ColdFront to an existing database and moves its cold data to object storage. | -| [Exploring Decoupled Mode](docs/walkthrough_decoupled.md) | Stores a table in Iceberg from the first row and adopts a table that another engine wrote. | -| [Exploring the Standalone Partitioner](docs/walkthrough_partitioner.md) | Manages PostgreSQL range partitions without any cold tier. | -| [Exploring Distributed Mode](docs/walkthrough_distributed.md) | Points two PostgreSQL nodes at one shared lake. | -| [Connecting to ColdFront](docs/connecting.md) | Documents onboarding an application role with least privilege. | -| [Understanding ColdFront Caveats](docs/caveats.md) | Documents cross-tier isolation, mixed transactions, and other edge cases shared between tiered and decoupled tables. | -| [Using Vended Credentials](docs/vended_credentials.md) | Documents running with no stored object-store credential, for AWS S3 and Azure ADLS Gen2. | -| [Supported Column Types](docs/supported_types.md) | Documents the PostgreSQL column types, including arrays, that a cold tier supports. | -| [Compacting the Cold Tier](docs/compaction.md) | Documents cold-tier table maintenance: compaction, snapshot expiry, and orphan-file removal. | - -## Connecting to ColdFront - -See [Connecting to ColdFront](docs/connecting.md) for onboarding an -application role with least privilege. - -## Caveats - -See [Understanding ColdFront Caveats](docs/caveats.md) for ColdFront -behaviors you should be aware of, including a storage-account setting -Azure ADLS Gen2 requires before you use it as a cold tier. - -A cold-tier write that reads an ordinary PostgreSQL table, such as an -`UPDATE ... FROM` or a `MERGE` from one, stores a `numeric` `NaN` from that -table as `0` and stores an array renumbered to start at 1, with no error. -[Supported Column Types](docs/supported_types.md) lists the writes that do -this and shows how to check a source table first. - -## Project Structure +ColdFront stores its settings in the server and has no configuration file. +Settings live in two places: + +- `postgresql.conf` holds the server settings: + - You must set `shared_preload_libraries`, `coldfront.warehouse`, and + `coldfront.lakekeeper_endpoint` on each server. + - `shared_preload_libraries` must include `pg_duckdb` and `coldfront`. + - You must also set `snowflake.node` and `coldfront.loopback_dsn` on each + node of a Spock mesh. +- Tables in the `coldfront` schema hold additional parameters: + - `coldfront.partition_config` holds each table's lifecycle. + - `coldfront.storage_secret` holds the cold-store credential. + +The archiver, partitioner, and compactor connect through the libpq environment +or `--dsn`, and read the settings from the server. To load a deployment YAML +into the server, run `archiver import --config ` once. Model the file on +[config.example.yaml](config.example.yaml). A later run checks any YAML you +pass against the server and uses only its `postgres.dsn`. + +To review a list of available settings, see: + +- The [Configuring Lakekeeper](docs/configure_lakekeeper.md) guide. +- The [Configuring PostgreSQL](docs/configuration.md#configuring-postgresql) + section of the Configuring ColdFront guide. + +## Installing and Configuring ColdFront + +ColdFront runs on PostgreSQL 16, 17, and 18. A single node needs only stock +PostgreSQL. Distributed mode needs pgEdge's PostgreSQL build with Spock, which +the Docker image is based on. You can install ColdFront from pgEdge packages or +build it from source in a Docker container or on bare metal. + +### Installing and Configuring ColdFront with Packages + +ColdFront packages come from the pgEdge package repository. First, configure +the repository for your platform, as described in the +[pgEdge Enterprise Postgres guide](https://docs.pgedge.com/enterprise/). Then, +install the ColdFront extension package for your PostgreSQL major version. The +package manager also installs the pgEdge PostgreSQL server, pg_duckdb, and +ColdFront's DuckDB extensions. + +For PostgreSQL 18 on RHEL, Rocky Linux, or AlmaLinux: + +```bash +sudo dnf install -y pgedge-coldfront_18 +``` + +For PostgreSQL 18 on Ubuntu or Debian: + +```bash +sudo apt update +sudo apt install -y pgedge-postgresql-18-coldfront +``` + +The extension package does not include the command-line tools or Lakekeeper. +Install `pgedge-coldfront` and `pgedge-lakekeeper` on the hosts that need +them. For repository setup and every package name, see +[Using a Package to Install ColdFront](docs/installation.md#using-a-package-to-install-coldfront). + +After installing the packages, use the following steps to configure PostgreSQL +and create the extensions: + +1. Add the following settings to `postgresql.conf`: + + ```ini + shared_preload_libraries = 'pg_duckdb,coldfront' + duckdb.extension_directory = '/usr/lib/pgedge/coldfront/duckdb-extensions' + duckdb.allow_unsigned_extensions = true + duckdb.autoinstall_known_extensions = false + coldfront.warehouse = '' + coldfront.lakekeeper_endpoint = 'http://:8181/catalog' + ``` + + The three `duckdb.*` settings make pg_duckdb load ColdFront's patched + DuckDB extensions. Without them, pg_duckdb can fetch the unpatched + upstream extensions instead. + +2. Restart PostgreSQL. `CREATE EXTENSION coldfront` fails on a server that + does not preload the extension. + +3. Connect to the database and create the extensions: + + ```sql + CREATE EXTENSION pg_duckdb; + CREATE EXTENSION coldfront; + ``` + +### Installing and Configuring ColdFront in a Docker Container + +The Docker build adds a thin ColdFront layer to a published base image. The +base image already includes pg_duckdb and the patched duckdb-iceberg +extension. You need Docker and network access. To build the command-line +tools, you also need Go 1.26.5 or later and golangci-lint. + +The Docker image writes the server settings into `postgresql.conf` on first +start. It reads the warehouse name and the Lakekeeper endpoint from the +`COLDFRONT_WAREHOUSE` and `COLDFRONT_LAKEKEEPER` environment variables in +`docker-compose.yml`. The defaults, `wh` and `http://lakekeeper:8181/catalog`, +match the warehouse that [Configuring Lakekeeper](docs/configure_lakekeeper.md) +creates. + +1. Clone the repository: + + ```bash + git clone https://github.com/pgEdge/coldfront.git + cd coldfront + ``` + +2. Build the command-line tools. `make build` builds the archiver and the + partitioner, and `make compactor` builds the compactor: + + ```bash + make build + make compactor + ``` + +3. Build the ColdFront layer and start the stack. The `local-store` profile + adds a SeaweedFS object store to the stack. The stack uses PostgreSQL 18 + unless you set `PG_MAJOR`: + + ```bash + docker compose --profile local-store up -d --build + ``` + +4. Connect to the database: + + ```bash + psql -h localhost -U coldfront -d coldfront + ``` + +5. Create the extensions: + + ```sql + CREATE EXTENSION pg_duckdb; + CREATE EXTENSION coldfront; + ``` + +The deployed image is for development and testing. It trusts every connection +without a password. For information about the image's environment variables +and a smoke test, see +[Building the Image on Docker](docs/installation.md#building-the-image-on-docker). + +### Installing and Configuring ColdFront on Bare Metal + +Before building ColdFront on bare metal, install the pg_duckdb PostgreSQL +extension (built on DuckDB 1.5.4) and a patched duckdb-iceberg DuckDB +extension. For the prerequisites and the patches, see +[Building ColdFront from Source](docs/installation.md#building-coldfront-from-source). +You also need `pg_config`, the PostgreSQL server development headers, the +libpq headers and library, `make`, `gcc`, Go 1.26.5 or later, and +golangci-lint. + +1. Clone the repository: + + ```bash + git clone https://github.com/pgEdge/coldfront.git + cd coldfront + ``` + +2. Build and install the `coldfront` extension. `pg_config` must be on your + `PATH`: + + ```bash + cd extension/coldfront + make && make install + ``` + +3. Build the command-line tools. `make build` builds the archiver and the + partitioner, and `make compactor` builds the compactor: + + ```bash + cd ../.. + make build + make compactor + ``` + +4. Copy the DuckDB extensions you built into the directory that pg_duckdb + loads them from. Copy `iceberg`, `avro`, `azure`, and `postgres_scanner` + into `$PGDATA/pg_duckdb/extensions/v1.5.4/linux_/`, where `` is + `amd64` or `arm64`. + +5. Add the following settings to `postgresql.conf`: + + ```ini + shared_preload_libraries = 'pg_duckdb,coldfront' + duckdb.allow_unsigned_extensions = true + coldfront.warehouse = '' + coldfront.lakekeeper_endpoint = 'http://:8181/catalog' + ``` + + The locally built `iceberg` extension is unsigned, so + `duckdb.allow_unsigned_extensions` must be on. + +6. Restart PostgreSQL. `CREATE EXTENSION coldfront` fails on a server that + does not preload the extension. + +7. Connect to the database and create the extensions: + + ```sql + CREATE EXTENSION pg_duckdb; + CREATE EXTENSION coldfront; + ``` + +For the full steps, see +[Building ColdFront on Bare Metal](docs/installation.md#building-coldfront-on-bare-metal). + +### Connecting ColdFront to Lakekeeper and Object Storage + +After configuring the server and creating the extensions, you're ready to +connect ColdFront to Lakekeeper and an object store. ColdFront stores the cold +tier in an object store, through a Lakekeeper warehouse. + +ColdFront can store its Iceberg tables in AWS S3, in an S3-compatible store +such as SeaweedFS or MinIO, in Google Cloud Storage, or in Azure ADLS Gen2. You +will need to connect ColdFront to a store once for each deployment. + +#### Connecting a Package or Bare-Metal Installation to Lakekeeper and an Object Store + +A package or bare-metal installation runs Lakekeeper and an object store on +the host. + +1. Start an object store. If you're walking through this installation to + evaluate ColdFront, you can use SeaweedFS, a free S3-compatible store, on + the host. The repository's `seaweedfs-s3.json` sets its credentials to + `admin` and `adminsecret`: + + ```bash + weed server -s3 -dir=/var/lib/seaweedfs \ + -s3.config=docker/seaweedfs-s3.json + ``` + + If you would like to evaluate ColdFront on AWS S3, Google Cloud Storage, + or Azure ADLS Gen2, create a bucket or container there instead. The + [Configuring your Object Store](docs/object_store.md) guide covers + configuration of each one, including production stores. + +2. Install Lakekeeper. With the pgEdge repository configured, install the + `pgedge-lakekeeper` package: + + ```bash + sudo dnf install -y pgedge-lakekeeper # RHEL, Rocky Linux, AlmaLinux + sudo apt install -y pgedge-lakekeeper # Ubuntu, Debian + ``` + + If you're not using the pgEdge repository, install the upstream binary as + described in + [Installing Lakekeeper](docs/configure_lakekeeper.md#installing-lakekeeper). + Then copy the environment file and the systemd unit from the ColdFront + repository: + + ```bash + sudo install -D -m 0640 packaging/lakekeeper/common/lakekeeper.env \ + /etc/lakekeeper/lakekeeper.env + sudo install -m 0644 packaging/lakekeeper/common/lakekeeper.service \ + /etc/systemd/system/lakekeeper.service + sudo systemctl daemon-reload + ``` + +3. Create a role and a database for the Lakekeeper catalog: + + ```sql + CREATE ROLE lakekeeper LOGIN PASSWORD 'change-me'; + CREATE DATABASE lakekeeper OWNER lakekeeper; + ``` + +4. In `/etc/lakekeeper/lakekeeper.env`, set the catalog's connection string + and an encryption key. Keep the key stable and backed up: + + ```bash + LAKEKEEPER__PG_DATABASE_URL_WRITE=postgresql://lakekeeper:change-me@127.0.0.1:5432/lakekeeper + LAKEKEEPER__PG_ENCRYPTION_KEY= + ``` + +5. Run the one-time migration, then start Lakekeeper: + + ```bash + set -a; . /etc/lakekeeper/lakekeeper.env; set +a + sudo -E -u lakekeeper /usr/bin/lakekeeper migrate + sudo systemctl enable --now lakekeeper + ``` + +6. Bootstrap Lakekeeper, create the `wh` warehouse on the SeaweedFS store, and + create the `public` namespace; decoupled mode requires this namespace. The + warehouse name must match `coldfront.warehouse`: + + ```bash + curl -X POST http://localhost:8181/management/v1/bootstrap \ + -H "Content-Type: application/json" -d '{"accept-terms-of-use":true}' + + curl -X POST http://localhost:8181/management/v1/warehouse \ + -H "Content-Type: application/json" -d '{ + "warehouse-name": "wh", + "storage-profile": { + "type": "s3", "bucket": "iceberg", "region": "us-east-1", + "endpoint": "http://localhost:8333", "path-style-access": true, + "flavor": "s3-compat", "sts-enabled": false, + "remote-signing-enabled": false + }, + "storage-credential": { + "type": "s3", "credential-type": "access-key", + "aws-access-key-id": "admin", "aws-secret-access-key": "adminsecret" + } + }' + + WID=$(curl -s http://localhost:8181/management/v1/warehouse \ + | grep -oE '"warehouse-id":"[^"]+"' | head -1 | cut -d'"' -f4) + curl -X POST "http://localhost:8181/catalog/v1/$WID/namespaces" \ + -H "Content-Type: application/json" -d '{"namespace":["public"]}' + ``` + + If you're using a different type of object store, change the + `storage-profile` and `storage-credential` as shown in the store's section + of [Configuring your Object Store](docs/object_store.md#configuring-lakekeeper). + +7. Store the SeaweedFS credential in the database: + + ```sql + SELECT coldfront.set_storage_secret('admin', 'adminsecret', + 'localhost:8333'); + ``` + + If you use `coldfront.local_pg_dsn`, set it before this call. Another store + takes its own arguments, and Azure ADLS Gen2 uses + `coldfront.set_storage_secret_azure()` instead. + +#### Connecting a Docker Installation to Lakekeeper and an Object Store + +The Docker stack includes a SeaweedFS store, so you can use it as the cold tier +to evaluate ColdFront without a cloud account. + +1. Bootstrap Lakekeeper: + + ```bash + curl -X POST http://localhost:8181/management/v1/bootstrap \ + -H "Content-Type: application/json" -d '{"accept-terms-of-use":true}' + ``` + +2. Create the `wh` warehouse on the SeaweedFS store: + + ```bash + curl -X POST http://localhost:8181/management/v1/warehouse \ + -H "Content-Type: application/json" -d '{ + "warehouse-name": "wh", + "storage-profile": { + "type": "s3", "bucket": "iceberg", "region": "us-east-1", + "endpoint": "http://seaweedfs:8333", "path-style-access": true, + "flavor": "s3-compat", "sts-enabled": false, + "remote-signing-enabled": false + }, + "storage-credential": { + "type": "s3", "credential-type": "access-key", + "aws-access-key-id": "admin", "aws-secret-access-key": "adminsecret" + } + }' + ``` + +3. Create the `public` namespace in the warehouse; decoupled mode requires + this namespace: + + ```bash + WID=$(curl -s http://localhost:8181/management/v1/warehouse \ + | grep -oE '"warehouse-id":"[^"]+"' | head -1 | cut -d'"' -f4) + curl -X POST "http://localhost:8181/catalog/v1/$WID/namespaces" \ + -H "Content-Type: application/json" -d '{"namespace":["public"]}' + ``` + +4. Store the SeaweedFS credential in the database: + + ```sql + SELECT coldfront.set_storage_secret('admin', 'adminsecret', + 'seaweedfs:8333'); + ``` + +#### Granting Access to Application Roles + +After connecting ColdFront to Lakekeeper and an object store, you must grant +access to each application role that connects as a non-superuser: + +```sql +SELECT coldfront.grant_app_access(''); +``` + +Run this command again after you create, adopt, or first tier a table. The +[Connecting to ColdFront](docs/connecting.md) guide describes the privileges +granted to the role. + +## Support & Resources + +After you install, configure, and connect ColdFront, use the following +resources to learn more: + +- To try each of the three modes with commands you can copy and run, follow + the [guided walkthrough](docs/exploring_coldfront/index.md). +- For information about using ColdFront, see + [Using ColdFront](docs/using_coldfront/index.md), which helps you choose a + mode and create your first table. Each mode has its own guide: + - [Tiered Mode](docs/usage_tiered.md) + - [Decoupled Mode](docs/usage_decoupled.md) + - [Standalone Partitioned Mode](docs/usage_partitioner.md) +- To run ColdFront on more than one node, see + [Configuring a Distributed Deployment](docs/distributed_setup.md). +- To review ColdFront caveats, see + [Understanding ColdFront Caveats](docs/caveats.md). + +For more information about pgEdge products, visit +[docs.pgedge.com](https://docs.pgedge.com). + +To report an issue with the software, visit +[GitHub Issues](https://github.com/pgEdge/ColdFront/issues). + +## For Developers + +We welcome your contributions to ColdFront. For more information, see +[CONTRIBUTING.md](CONTRIBUTING.md). The following sections describe +ColdFront's dependencies, its version numbers, and the layout of the +repository. + +### Author + +Jimmy Angelakos created ColdFront. + +### Dependencies + +The following table lists the services and components ColdFront runs against: + +| Component | Version | Purpose | +|-----------|---------|---------| +| PostgreSQL | 16, 17, or 18 | Provides the database with native partitioning; stock PostgreSQL serves a single node, and the Docker image and distributed mode use pgEdge's PostgreSQL build with Spock. | +| pg_duckdb | commit c04e6a2 (PR #1025), on DuckDB 1.5.4 | Runs Iceberg reads and writes through in-process DuckDB. | +| duckdb-iceberg | `v1.5-variegata` @ `5edc45f0`, patched | Provides Iceberg catalog and I/O for DuckDB, with ColdFront's five patches applied (see [docker/Dockerfile.duckdb15-base](docker/Dockerfile.duckdb15-base)). | +| Lakekeeper | latest | Provides the Iceberg REST catalog (a Rust binary). | +| S3-compatible store | any | Stores the cold data; SeaweedFS, MinIO, AWS S3, and GCS all work. | +| Azure ADLS Gen2 | any | Stores the cold data instead of an S3 store, through `set_storage_secret_azure` and an `adls` warehouse. | + +Building from source needs the Go toolchain (the version is pinned in +[go.mod](go.mod)). The Go module dependencies are the source of truth in +[go.mod](go.mod) / [go.sum](go.sum); the archiver and partitioner build as +static, CGO-free binaries on `pgx/v5`, and the compactor is a separate module +([cmd/compactor/go.mod](cmd/compactor/go.mod)) built on `apache/iceberg-go`. + +### Versioning + +ColdFront has two independent version numbers, each following its own +convention: + +- Release tags use three-part [Semantic Versioning](https://semver.org) + (`vMAJOR.MINOR.PATCH`, for example `v1.0.0`); Git tags, GitHub releases, + container image tags, and the changelog all use this form. Three parts are + required because ColdFront is a Go module, and the toolchain recognizes only + full `vX.Y.Z` tags as releases. The patch field keeps a bugfix-only release + (`v1.0.1`) distinct from a feature release (`v1.1.0`), which matters for a + data-writing extension, where a release that keeps the same behavior and adds + one safety fix must be easy to recognize. +- The PostgreSQL extension uses the conventional two-part version in its + control file (`default_version = '1.0'`) and upgrade-script filenames + (`coldfront--1.0--1.1.sql`), as is standard for PostgreSQL extensions. + +The two map cleanly: extension `1.0` ships inside release `v1.0.0`, and a patch +release may keep the same extension version or bump it with an upgrade script +when the SQL changes. + +### Project Structure The repository is laid out as follows: @@ -275,65 +677,8 @@ pgedge-coldfront/ └── DUCKDB_1.5_UNPATCHED.md ← building/running the base unpatched, and the consequences ``` -## Dependencies - -The following table lists the services and components ColdFront runs against: - -| Component | Version | Purpose | -|-----------|---------|---------| -| PostgreSQL | 16, 17, or 18 | Provides the database with native partitioning; stock PostgreSQL serves a single node, and the Docker image and distributed mode use pgEdge's PostgreSQL build with Spock. | -| pg_duckdb | commit c04e6a2 (PR #1025), on DuckDB 1.5.4 | Runs Iceberg reads and writes through in-process DuckDB. | -| duckdb-iceberg | `v1.5-variegata` @ `5edc45f0`, patched | Provides Iceberg catalog and I/O for DuckDB, with ColdFront's five patches applied (see [docker/Dockerfile.duckdb15-base](docker/Dockerfile.duckdb15-base)). | -| Lakekeeper | latest | Provides the Iceberg REST catalog (a Rust binary). | -| S3-compatible store | any | Stores the cold data; SeaweedFS, MinIO, AWS S3, and GCS all work. | -| Azure ADLS Gen2 | any | Stores the cold data instead of an S3 store, through `set_storage_secret_azure` and an `adls` warehouse. | - -Building from source needs the Go toolchain (the version is pinned in -[go.mod](go.mod)). The Go module dependencies are the source of truth in -[go.mod](go.mod) / [go.sum](go.sum); the archiver and partitioner build as -static, CGO-free binaries on `pgx/v5`, and the compactor is a separate module -([cmd/compactor/go.mod](cmd/compactor/go.mod)) built on `apache/iceberg-go`. - -## Versioning - -ColdFront has two independent version numbers, each following its own -convention: - -- Release tags use three-part [Semantic Versioning](https://semver.org) - (`vMAJOR.MINOR.PATCH`, for example `v1.0.0`); Git tags, GitHub releases, - container image tags, and the changelog all use this form. Three parts are - required because ColdFront is a Go module, and the toolchain recognizes only - full `vX.Y.Z` tags as releases. The patch field keeps a bugfix-only release - (`v1.0.1`) distinct from a feature release (`v1.1.0`), which matters for a - data-writing extension, where a release that keeps the same behavior and adds - one safety fix must be easy to recognize. -- The PostgreSQL extension uses the conventional two-part version in its - control file (`default_version = '1.0'`) and upgrade-script filenames - (`coldfront--1.0--1.1.sql`), as is standard for PostgreSQL extensions. - -The two map cleanly: extension `1.0` ships inside release `v1.0.0`, and a patch -release may keep the same extension version or bump it with an upgrade script -when the SQL changes. - -## Support & Resources - -For more information about pgEdge products, visit -[docs.pgedge.com](https://docs.pgedge.com). - -To report an issue with the software, visit -[GitHub Issues](https://github.com/pgEdge/ColdFront/issues). - -## Contributing - -We welcome your project contributions; for more information, see -[CONTRIBUTING.md](CONTRIBUTING.md). - -## Author - -Jimmy Angelakos created ColdFront. - ## License -This project is licensed under the [PostgreSQL License](LICENSE.md). The -redistributed third-party components and their notices are listed in -[THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md). +ColdFront is open source, licensed under the +[PostgreSQL License](LICENSE.md). The redistributed third-party components and +their notices are listed in [THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md). From 2acd940832218cd0c31967547c290e4ecce4e7d3 Mon Sep 17 00:00:00 2001 From: Susan Douglas Date: Thu, 15 Oct 2026 07:20:56 -0400 Subject: [PATCH 03/33] docs: update README descriptions for the moved Using ColdFront sections --- README.md | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index 95c926f9..70a5a6ba 100644 --- a/README.md +++ b/README.md @@ -32,17 +32,16 @@ The ColdFront documentation consists of the following guides: Documents running a multi-writer decoupled-mode Spock mesh against one shared lake. - [Using ColdFront](docs/using_coldfront/index.md) - Compares the three - operating modes, documents the shared SQL surface, and links to each mode's - guide. + operating modes, documents the shared SQL surface, and covers dropping an + Iceberg table and the partition management CLI. - [Using ColdFront in Tiered Mode](docs/usage_tiered.md) - Documents archiving aging partitions to Iceberg while recent rows stay in the PostgreSQL heap. - [Using ColdFront in Decoupled Mode](docs/usage_decoupled.md) - Documents - tables that live entirely in Iceberg from the first row, and dropping or - releasing a registered table. + tables that live entirely in Iceberg from the first row, including + adopting a table from the catalog and handing it back. - [Using ColdFront in Standalone Partitioned Mode](docs/usage_partitioner.md) - - Documents managing PostgreSQL partitions with no cold tier at all, and - the partition CLI shared with tiered mode. + Documents managing PostgreSQL partitions with no cold tier at all. - [Working with Embeddings](docs/usage_vectors.md) - Documents storing and searching embeddings with the pgvector interface. - [Exploring ColdFront Modes](docs/exploring_coldfront/index.md) - Sets up the From 0efc90a99427a3f20f789eb78a70356d44b5e644 Mon Sep 17 00:00:00 2001 From: Susan Douglas Date: Thu, 15 Oct 2026 07:31:48 -0400 Subject: [PATCH 04/33] docs: say the object store credential is set in each database --- README.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 70a5a6ba..e06fc3a8 100644 --- a/README.md +++ b/README.md @@ -369,7 +369,9 @@ tier in an object store, through a Lakekeeper warehouse. ColdFront can store its Iceberg tables in AWS S3, in an S3-compatible store such as SeaweedFS or MinIO, in Google Cloud Storage, or in Azure ADLS Gen2. You -will need to connect ColdFront to a store once for each deployment. +set up the object store and the Lakekeeper warehouse once for each deployment. +The store's credential is kept in each database, so you repeat the credential +step in every database that uses ColdFront. #### Connecting a Package or Bare-Metal Installation to Lakekeeper and an Object Store @@ -469,7 +471,7 @@ the host. `storage-profile` and `storage-credential` as shown in the store's section of [Configuring your Object Store](docs/object_store.md#configuring-lakekeeper). -7. Store the SeaweedFS credential in the database: +7. Store the SeaweedFS credential in each database that uses ColdFront: ```sql SELECT coldfront.set_storage_secret('admin', 'adminsecret', @@ -521,7 +523,7 @@ to evaluate ColdFront without a cloud account. -H "Content-Type: application/json" -d '{"namespace":["public"]}' ``` -4. Store the SeaweedFS credential in the database: +4. Store the SeaweedFS credential in each database that uses ColdFront: ```sql SELECT coldfront.set_storage_secret('admin', 'adminsecret', From deed31b0751f74b98b63e72942753c3f89294a47 Mon Sep 17 00:00:00 2001 From: Susan Douglas Date: Thu, 15 Oct 2026 07:55:38 -0400 Subject: [PATCH 05/33] docs: use the placeholder in the README grant step --- README.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index e06fc3a8..7c8d1432 100644 --- a/README.md +++ b/README.md @@ -533,10 +533,11 @@ to evaluate ColdFront without a cloud account. #### Granting Access to Application Roles After connecting ColdFront to Lakekeeper and an object store, you must grant -access to each application role that connects as a non-superuser: +access to each application role that connects as a non-superuser. Replace +`` with the name of the role: ```sql -SELECT coldfront.grant_app_access(''); +SELECT coldfront.grant_app_access(''); ``` Run this command again after you create, adopt, or first tier a table. The From 5831c68b340ea54117e5da2cc2fe4f437389e5ce Mon Sep 17 00:00:00 2001 From: Susan Douglas Date: Thu, 15 Oct 2026 08:31:09 -0400 Subject: [PATCH 06/33] docs: run the Lakekeeper migration as the lakekeeper user in the README --- README.md | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index 7c8d1432..28d9672a 100644 --- a/README.md +++ b/README.md @@ -408,8 +408,8 @@ the host. repository: ```bash - sudo install -D -m 0640 packaging/lakekeeper/common/lakekeeper.env \ - /etc/lakekeeper/lakekeeper.env + sudo install -D -m 0640 -g lakekeeper \ + packaging/lakekeeper/common/lakekeeper.env /etc/lakekeeper/lakekeeper.env sudo install -m 0644 packaging/lakekeeper/common/lakekeeper.service \ /etc/systemd/system/lakekeeper.service sudo systemctl daemon-reload @@ -430,11 +430,13 @@ the host. LAKEKEEPER__PG_ENCRYPTION_KEY= ``` -5. Run the one-time migration, then start Lakekeeper: +5. Run the one-time migration as the `lakekeeper` user, then start + Lakekeeper. The migration reads the environment file as that user, because + the file is not readable by other accounts: ```bash - set -a; . /etc/lakekeeper/lakekeeper.env; set +a - sudo -E -u lakekeeper /usr/bin/lakekeeper migrate + sudo -u lakekeeper sh -c \ + 'set -a; . /etc/lakekeeper/lakekeeper.env; set +a; exec /usr/bin/lakekeeper migrate' sudo systemctl enable --now lakekeeper ``` From 89d4b469065d677eca5e4fa399c503b6e71ce614 Mon Sep 17 00:00:00 2001 From: Susan Douglas Date: Thu, 15 Oct 2026 08:33:26 -0400 Subject: [PATCH 07/33] docs: take the warehouse ID from the create response in the README --- README.md | 17 ++++++++--------- 1 file changed, 8 insertions(+), 9 deletions(-) diff --git a/README.md b/README.md index 28d9672a..737bdae4 100644 --- a/README.md +++ b/README.md @@ -448,7 +448,7 @@ the host. curl -X POST http://localhost:8181/management/v1/bootstrap \ -H "Content-Type: application/json" -d '{"accept-terms-of-use":true}' - curl -X POST http://localhost:8181/management/v1/warehouse \ + WID=$(curl -s -X POST http://localhost:8181/management/v1/warehouse \ -H "Content-Type: application/json" -d '{ "warehouse-name": "wh", "storage-profile": { @@ -461,10 +461,9 @@ the host. "type": "s3", "credential-type": "access-key", "aws-access-key-id": "admin", "aws-secret-access-key": "adminsecret" } - }' + }' \ + | grep -oE '"warehouse-id":"[^"]+"' | cut -d'"' -f4) - WID=$(curl -s http://localhost:8181/management/v1/warehouse \ - | grep -oE '"warehouse-id":"[^"]+"' | head -1 | cut -d'"' -f4) curl -X POST "http://localhost:8181/catalog/v1/$WID/namespaces" \ -H "Content-Type: application/json" -d '{"namespace":["public"]}' ``` @@ -496,10 +495,11 @@ to evaluate ColdFront without a cloud account. -H "Content-Type: application/json" -d '{"accept-terms-of-use":true}' ``` -2. Create the `wh` warehouse on the SeaweedFS store: +2. Create the `wh` warehouse on the SeaweedFS store, and save its ID for the + next step: ```bash - curl -X POST http://localhost:8181/management/v1/warehouse \ + WID=$(curl -s -X POST http://localhost:8181/management/v1/warehouse \ -H "Content-Type: application/json" -d '{ "warehouse-name": "wh", "storage-profile": { @@ -512,15 +512,14 @@ to evaluate ColdFront without a cloud account. "type": "s3", "credential-type": "access-key", "aws-access-key-id": "admin", "aws-secret-access-key": "adminsecret" } - }' + }' \ + | grep -oE '"warehouse-id":"[^"]+"' | cut -d'"' -f4) ``` 3. Create the `public` namespace in the warehouse; decoupled mode requires this namespace: ```bash - WID=$(curl -s http://localhost:8181/management/v1/warehouse \ - | grep -oE '"warehouse-id":"[^"]+"' | head -1 | cut -d'"' -f4) curl -X POST "http://localhost:8181/catalog/v1/$WID/namespaces" \ -H "Content-Type: application/json" -d '{"namespace":["public"]}' ``` From 2390d6fa49add3c5a172f7efcd86478f36401caa Mon Sep 17 00:00:00 2001 From: Susan Douglas Date: Thu, 15 Oct 2026 08:57:34 -0400 Subject: [PATCH 08/33] docs: warn when the warehouse ID is empty in the README --- README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/README.md b/README.md index 737bdae4..256f7da5 100644 --- a/README.md +++ b/README.md @@ -464,6 +464,7 @@ the host. }' \ | grep -oE '"warehouse-id":"[^"]+"' | cut -d'"' -f4) + [ -n "$WID" ] || echo "No warehouse ID - check that the warehouse create succeeded." curl -X POST "http://localhost:8181/catalog/v1/$WID/namespaces" \ -H "Content-Type: application/json" -d '{"namespace":["public"]}' ``` @@ -520,6 +521,7 @@ to evaluate ColdFront without a cloud account. this namespace: ```bash + [ -n "$WID" ] || echo "No warehouse ID - check that the warehouse create succeeded." curl -X POST "http://localhost:8181/catalog/v1/$WID/namespaces" \ -H "Content-Type: application/json" -d '{"namespace":["public"]}' ``` From 30d7fc9f829a0f3785aa75a62a183634c6b5a4cd Mon Sep 17 00:00:00 2001 From: Susan Douglas Date: Thu, 15 Oct 2026 09:49:39 -0400 Subject: [PATCH 09/33] docs: bind Lakekeeper to loopback in the README evaluation steps --- README.md | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 256f7da5..b7a44842 100644 --- a/README.md +++ b/README.md @@ -423,13 +423,21 @@ the host. ``` 4. In `/etc/lakekeeper/lakekeeper.env`, set the catalog's connection string - and an encryption key. Keep the key stable and backed up: + and an encryption key, and bind Lakekeeper to the loopback address. Keep + the key stable and backed up: ```bash LAKEKEEPER__PG_DATABASE_URL_WRITE=postgresql://lakekeeper:change-me@127.0.0.1:5432/lakekeeper LAKEKEEPER__PG_ENCRYPTION_KEY= + LAKEKEEPER__BIND_IP=127.0.0.1 ``` + By default, Lakekeeper listens on every address and accepts every request. + The loopback bind keeps it reachable only from this host. To accept clients + from other hosts, configure authentication and authorization first, as + described in + [Preparing and Starting Lakekeeper](docs/configure_lakekeeper.md#preparing-and-starting-lakekeeper). + 5. Run the one-time migration as the `lakekeeper` user, then start Lakekeeper. The migration reads the environment file as that user, because the file is not readable by other accounts: From 44c1e314300582d7ee661d53f81208f8c20a4d8a Mon Sep 17 00:00:00 2001 From: Susan Douglas Date: Thu, 15 Oct 2026 09:51:47 -0400 Subject: [PATCH 10/33] docs: name the README Table of Contents as the index in CLAUDE.md --- CLAUDE.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CLAUDE.md b/CLAUDE.md index 0eaf4df7..a7a7cc88 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -109,7 +109,7 @@ Snapshots of upstream state for the next major moves. Re-verify before acting on ## Documentation - User docs live in `docs/` as a MkDocs site (pgEdge MkDocs structure); `README.md` is the GitHub landing/TOC. Build with `mkdocs build --strict` — it must pass with no warnings. - Keep `README.md` and `docs/index.md` near-identical, EXCEPT `docs/index.md` is user-facing and omits the developer-specific content that belongs only in a GitHub README: CI badges, build-from-source internals, the project-structure tree, dependency/build tables, and author/contributor info. -- Three indexes point at the pages in `docs/`, each with its own focus: the `mkdocs.yml` `nav:` (everything, including changelog, license and `formal/`), the `README.md` Documentation table (one terse row per guide), and the `docs/index.md` Next Steps list (prose, user-facing, top-level guides only; the per-mode deep dives are reached through `docs/architecture_guides/index.md`, which lists them). Adding, renaming, or removing a guide under `docs/` means updating all three in the same commit; adding a deep dive means the nav, the README table, and the list in `docs/architecture_guides/index.md`. `mkdocs build --strict` passes on a page no index links, so it will not catch this: check by listing `docs/*.md` against each index. +- Three indexes point at the pages in `docs/`, each with its own focus: the `mkdocs.yml` `nav:` (everything, including changelog, license and `formal/`), the `README.md` Table of Contents (mirrors the nav's titles, order and nesting, with a short description after each link), and the `docs/index.md` Next Steps list (prose, user-facing, top-level guides only; the per-mode deep dives are reached through `docs/architecture_guides/index.md`, which lists them). Adding, renaming, or removing a guide under `docs/` means updating all three in the same commit; adding a deep dive means the nav, the README Table of Contents, and the list in `docs/architecture_guides/index.md`. `mkdocs build --strict` passes on a page no index links, so it will not catch this: check by listing `docs/*.md` against each index. - Internal/scratch docs (bench, experimental, hardening, comparison, backlog, scale-test, E2E) are gitignored and NEVER pushed to GitHub; never reference them from published docs. - ARCHITECTURE docs are reference material, not work logs: brief, present-tense, no history ("we used to…", version churn), no rejected-alternative narration. Keep the *why* (rationale that explains the design); cut the journey. - One home per topic, no duplication across docs: shared day-to-day usage and the partition CLI in `docs/using_coldfront/index.md`; each mode's own guide in `docs/usage_tiered.md`, `docs/usage_decoupled.md`, and `docs/usage_partitioner.md`; PostgreSQL/ColdFront config in `docs/configuration.md`; Lakekeeper install/config/bootstrap in `docs/configure_lakekeeper.md`; build-from-source and testing/CI in `docs/installation.md`; storage-backend setup in `docs/object_store.md`; deep dives in `docs/architecture_guides/index.md` and `docs/architecture_tiered.md`/`docs/architecture_decoupled.md`/`docs/architecture_vectors.md`. When moving a section, fix every cross-anchor that pointed at it. From 0d2a3ab75d0502d8ce1bcf5741d0f66e5af68004 Mon Sep 17 00:00:00 2001 From: Susan Douglas Date: Thu, 15 Oct 2026 09:52:10 -0400 Subject: [PATCH 11/33] docs: describe how README.md and docs/index.md relate in CLAUDE.md --- CLAUDE.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CLAUDE.md b/CLAUDE.md index a7a7cc88..e7de8223 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -108,7 +108,7 @@ Snapshots of upstream state for the next major moves. Re-verify before acting on ## Documentation - User docs live in `docs/` as a MkDocs site (pgEdge MkDocs structure); `README.md` is the GitHub landing/TOC. Build with `mkdocs build --strict` — it must pass with no warnings. -- Keep `README.md` and `docs/index.md` near-identical, EXCEPT `docs/index.md` is user-facing and omits the developer-specific content that belongs only in a GitHub README: CI badges, build-from-source internals, the project-structure tree, dependency/build tables, and author/contributor info. +- `README.md` and `docs/index.md` open with the same introduction (what ColdFront is, its three operating modes, and the bakery protocol), differing only in the warning-box syntax and link paths each format needs; change both together. After the introduction they serve different readers: `README.md` is the GitHub landing page, with the install-and-configure walkthrough, support links, and developer content (CI badge, dependencies, versioning, project structure, author); `docs/index.md` is the docs-site landing page and points readers to the walkthroughs and guides. - Three indexes point at the pages in `docs/`, each with its own focus: the `mkdocs.yml` `nav:` (everything, including changelog, license and `formal/`), the `README.md` Table of Contents (mirrors the nav's titles, order and nesting, with a short description after each link), and the `docs/index.md` Next Steps list (prose, user-facing, top-level guides only; the per-mode deep dives are reached through `docs/architecture_guides/index.md`, which lists them). Adding, renaming, or removing a guide under `docs/` means updating all three in the same commit; adding a deep dive means the nav, the README Table of Contents, and the list in `docs/architecture_guides/index.md`. `mkdocs build --strict` passes on a page no index links, so it will not catch this: check by listing `docs/*.md` against each index. - Internal/scratch docs (bench, experimental, hardening, comparison, backlog, scale-test, E2E) are gitignored and NEVER pushed to GitHub; never reference them from published docs. - ARCHITECTURE docs are reference material, not work logs: brief, present-tense, no history ("we used to…", version churn), no rejected-alternative narration. Keep the *why* (rationale that explains the design); cut the journey. From 854c5fdd129147ca02b62b716dfbb9f3c2a357a7 Mon Sep 17 00:00:00 2001 From: Susan Douglas Date: Thu, 15 Oct 2026 10:32:37 -0400 Subject: [PATCH 12/33] docs: say the Iceberg cold tier applies to tiered and decoupled modes in the README --- README.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index b7a44842..e24ba180 100644 --- a/README.md +++ b/README.md @@ -73,9 +73,10 @@ The ColdFront documentation consists of the following guides: - [Release Notes - Changelog](docs/changelog.md) - [License](docs/LICENSE.md) -ColdFront keeps tables in PostgreSQL and cold data in Apache Iceberg (Parquet -on S3-compatible, Azure, or GCS storage). The cold tier is both readable and -writable through the same SQL, with no application changes. The application +ColdFront keeps tables in PostgreSQL and, in its tiered and decoupled modes, +keeps cold data in Apache Iceberg (Parquet on S3-compatible, Azure, or GCS +storage). The cold tier is both readable and writable through the same SQL, +with no application changes. The application queries every table as an ordinary PostgreSQL relation, with standard `SELECT`, `INSERT`, `UPDATE`, and `DELETE`, and `MERGE` on PostgreSQL 17 and later. From 4ce297c95bc616a696447e020e52ec2013e7411b Mon Sep 17 00:00:00 2001 From: Susan Douglas Date: Thu, 15 Oct 2026 10:32:45 -0400 Subject: [PATCH 13/33] docs: skip the namespace request when the warehouse ID is empty in the README --- README.md | 18 ++++++++++++------ 1 file changed, 12 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index e24ba180..16d4931c 100644 --- a/README.md +++ b/README.md @@ -473,9 +473,12 @@ the host. }' \ | grep -oE '"warehouse-id":"[^"]+"' | cut -d'"' -f4) - [ -n "$WID" ] || echo "No warehouse ID - check that the warehouse create succeeded." - curl -X POST "http://localhost:8181/catalog/v1/$WID/namespaces" \ - -H "Content-Type: application/json" -d '{"namespace":["public"]}' + if [ -z "$WID" ]; then + echo "No warehouse ID - check that the warehouse create succeeded." + else + curl -X POST "http://localhost:8181/catalog/v1/$WID/namespaces" \ + -H "Content-Type: application/json" -d '{"namespace":["public"]}' + fi ``` If you're using a different type of object store, change the @@ -530,9 +533,12 @@ to evaluate ColdFront without a cloud account. this namespace: ```bash - [ -n "$WID" ] || echo "No warehouse ID - check that the warehouse create succeeded." - curl -X POST "http://localhost:8181/catalog/v1/$WID/namespaces" \ - -H "Content-Type: application/json" -d '{"namespace":["public"]}' + if [ -z "$WID" ]; then + echo "No warehouse ID - check that the warehouse create succeeded." + else + curl -X POST "http://localhost:8181/catalog/v1/$WID/namespaces" \ + -H "Content-Type: application/json" -d '{"namespace":["public"]}' + fi ``` 4. Store the SeaweedFS credential in each database that uses ColdFront: From 8f68c9f0bb528783802ba6f13c2d74ae4b3b5bb5 Mon Sep 17 00:00:00 2001 From: Susan Douglas Date: Thu, 15 Oct 2026 10:33:00 -0400 Subject: [PATCH 14/33] docs: set duckdb.postgres_role and create its role in the README install steps --- README.md | 22 ++++++++++++++++------ 1 file changed, 16 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index 16d4931c..18dcae00 100644 --- a/README.md +++ b/README.md @@ -225,20 +225,25 @@ and create the extensions: duckdb.extension_directory = '/usr/lib/pgedge/coldfront/duckdb-extensions' duckdb.allow_unsigned_extensions = true duckdb.autoinstall_known_extensions = false + duckdb.postgres_role = 'coldfront_duckdb' coldfront.warehouse = '' coldfront.lakekeeper_endpoint = 'http://:8181/catalog' ``` - The three `duckdb.*` settings make pg_duckdb load ColdFront's patched - DuckDB extensions. Without them, pg_duckdb can fetch the unpatched - upstream extensions instead. + `duckdb.extension_directory`, `duckdb.allow_unsigned_extensions`, and + `duckdb.autoinstall_known_extensions` make pg_duckdb load ColdFront's + patched DuckDB extensions. Without them, pg_duckdb can fetch the unpatched + upstream extensions instead. `duckdb.postgres_role` lets the application + roles you grant access to run DuckDB; without it, only superusers can. 2. Restart PostgreSQL. `CREATE EXTENSION coldfront` fails on a server that does not preload the extension. -3. Connect to the database and create the extensions: +3. Connect to the database, create the role that `duckdb.postgres_role` + names, and create the extensions: ```sql + CREATE ROLE coldfront_duckdb NOLOGIN; CREATE EXTENSION pg_duckdb; CREATE EXTENSION coldfront; ``` @@ -342,19 +347,24 @@ golangci-lint. ```ini shared_preload_libraries = 'pg_duckdb,coldfront' duckdb.allow_unsigned_extensions = true + duckdb.postgres_role = 'coldfront_duckdb' coldfront.warehouse = '' coldfront.lakekeeper_endpoint = 'http://:8181/catalog' ``` The locally built `iceberg` extension is unsigned, so - `duckdb.allow_unsigned_extensions` must be on. + `duckdb.allow_unsigned_extensions` must be on. `duckdb.postgres_role` lets + the application roles you grant access to run DuckDB; without it, only + superusers can. 6. Restart PostgreSQL. `CREATE EXTENSION coldfront` fails on a server that does not preload the extension. -7. Connect to the database and create the extensions: +7. Connect to the database, create the role that `duckdb.postgres_role` + names, and create the extensions: ```sql + CREATE ROLE coldfront_duckdb NOLOGIN; CREATE EXTENSION pg_duckdb; CREATE EXTENSION coldfront; ``` From 9d73c504b905194e5ff3d7fb9887a68808196bbe Mon Sep 17 00:00:00 2001 From: Susan Douglas Date: Thu, 15 Oct 2026 10:33:10 -0400 Subject: [PATCH 15/33] docs: set coldfront.warehouse to wh in the README install steps --- README.md | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 18dcae00..405a54da 100644 --- a/README.md +++ b/README.md @@ -226,7 +226,7 @@ and create the extensions: duckdb.allow_unsigned_extensions = true duckdb.autoinstall_known_extensions = false duckdb.postgres_role = 'coldfront_duckdb' - coldfront.warehouse = '' + coldfront.warehouse = 'wh' coldfront.lakekeeper_endpoint = 'http://:8181/catalog' ``` @@ -235,6 +235,9 @@ and create the extensions: patched DuckDB extensions. Without them, pg_duckdb can fetch the unpatched upstream extensions instead. `duckdb.postgres_role` lets the application roles you grant access to run DuckDB; without it, only superusers can. + `coldfront.warehouse` names the `wh` warehouse that + [Connecting ColdFront to Lakekeeper and Object Storage](#connecting-coldfront-to-lakekeeper-and-object-storage) + creates. 2. Restart PostgreSQL. `CREATE EXTENSION coldfront` fails on a server that does not preload the extension. @@ -348,14 +351,16 @@ golangci-lint. shared_preload_libraries = 'pg_duckdb,coldfront' duckdb.allow_unsigned_extensions = true duckdb.postgres_role = 'coldfront_duckdb' - coldfront.warehouse = '' + coldfront.warehouse = 'wh' coldfront.lakekeeper_endpoint = 'http://:8181/catalog' ``` The locally built `iceberg` extension is unsigned, so `duckdb.allow_unsigned_extensions` must be on. `duckdb.postgres_role` lets the application roles you grant access to run DuckDB; without it, only - superusers can. + superusers can. `coldfront.warehouse` names the `wh` warehouse that + [Connecting ColdFront to Lakekeeper and Object Storage](#connecting-coldfront-to-lakekeeper-and-object-storage) + creates. 6. Restart PostgreSQL. `CREATE EXTENSION coldfront` fails on a server that does not preload the extension. From bc2fa76308122acb31b89550eec0dc39a63bddfe Mon Sep 17 00:00:00 2001 From: Susan Douglas Date: Thu, 15 Oct 2026 10:33:16 -0400 Subject: [PATCH 16/33] docs: say when the Next Steps list needs updating in CLAUDE.md --- CLAUDE.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CLAUDE.md b/CLAUDE.md index e7de8223..6e7896af 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -109,7 +109,7 @@ Snapshots of upstream state for the next major moves. Re-verify before acting on ## Documentation - User docs live in `docs/` as a MkDocs site (pgEdge MkDocs structure); `README.md` is the GitHub landing/TOC. Build with `mkdocs build --strict` — it must pass with no warnings. - `README.md` and `docs/index.md` open with the same introduction (what ColdFront is, its three operating modes, and the bakery protocol), differing only in the warning-box syntax and link paths each format needs; change both together. After the introduction they serve different readers: `README.md` is the GitHub landing page, with the install-and-configure walkthrough, support links, and developer content (CI badge, dependencies, versioning, project structure, author); `docs/index.md` is the docs-site landing page and points readers to the walkthroughs and guides. -- Three indexes point at the pages in `docs/`, each with its own focus: the `mkdocs.yml` `nav:` (everything, including changelog, license and `formal/`), the `README.md` Table of Contents (mirrors the nav's titles, order and nesting, with a short description after each link), and the `docs/index.md` Next Steps list (prose, user-facing, top-level guides only; the per-mode deep dives are reached through `docs/architecture_guides/index.md`, which lists them). Adding, renaming, or removing a guide under `docs/` means updating all three in the same commit; adding a deep dive means the nav, the README Table of Contents, and the list in `docs/architecture_guides/index.md`. `mkdocs build --strict` passes on a page no index links, so it will not catch this: check by listing `docs/*.md` against each index. +- Three indexes point at the pages in `docs/`, each with its own focus: the `mkdocs.yml` `nav:` (everything, including changelog, license and `formal/`), the `README.md` Table of Contents (mirrors the nav's titles, order and nesting, with a short description after each link), and the `docs/index.md` Next Steps list (prose, user-facing, top-level guides only; the per-mode deep dives are reached through `docs/architecture_guides/index.md`, which lists them). Adding, renaming, or removing a guide under `docs/` means updating the nav and the README Table of Contents in the same commit, plus the `docs/index.md` Next Steps list when the guide is a top-level user guide; adding a deep dive means the nav, the README Table of Contents, and the list in `docs/architecture_guides/index.md`. `mkdocs build --strict` passes on a page no index links, so it will not catch this: check by listing `docs/*.md` against each index. - Internal/scratch docs (bench, experimental, hardening, comparison, backlog, scale-test, E2E) are gitignored and NEVER pushed to GitHub; never reference them from published docs. - ARCHITECTURE docs are reference material, not work logs: brief, present-tense, no history ("we used to…", version churn), no rejected-alternative narration. Keep the *why* (rationale that explains the design); cut the journey. - One home per topic, no duplication across docs: shared day-to-day usage and the partition CLI in `docs/using_coldfront/index.md`; each mode's own guide in `docs/usage_tiered.md`, `docs/usage_decoupled.md`, and `docs/usage_partitioner.md`; PostgreSQL/ColdFront config in `docs/configuration.md`; Lakekeeper install/config/bootstrap in `docs/configure_lakekeeper.md`; build-from-source and testing/CI in `docs/installation.md`; storage-backend setup in `docs/object_store.md`; deep dives in `docs/architecture_guides/index.md` and `docs/architecture_tiered.md`/`docs/architecture_decoupled.md`/`docs/architecture_vectors.md`. When moving a section, fix every cross-anchor that pointed at it. From f515164372fef3cf6c623fda0fb4cdd3564b9b4a Mon Sep 17 00:00:00 2001 From: Susan Douglas Date: Thu, 15 Oct 2026 11:13:08 -0400 Subject: [PATCH 17/33] docs: initialize, configure, and restart PostgreSQL in the README package steps --- README.md | 32 +++++++++++++++++++++++++------- 1 file changed, 25 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index 405a54da..b3be9f85 100644 --- a/README.md +++ b/README.md @@ -215,10 +215,23 @@ Install `pgedge-coldfront` and `pgedge-lakekeeper` on the hosts that need them. For repository setup and every package name, see [Using a Package to Install ColdFront](docs/installation.md#using-a-package-to-install-coldfront). -After installing the packages, use the following steps to configure PostgreSQL -and create the extensions: +After installing the packages, use the following steps to start PostgreSQL, +configure it, and create the extensions: -1. Add the following settings to `postgresql.conf`: +1. Initialize and start PostgreSQL. On RHEL, Rocky Linux, or AlmaLinux, the + package does not create a database cluster, so create and start one: + + ```bash + sudo /usr/pgsql-18/bin/postgresql-18-setup initdb + sudo systemctl enable --now postgresql-18 + ``` + + On Ubuntu or Debian, the package creates and starts the `18/main` cluster; + `sudo pg_lsclusters` shows it. + +2. Add the following settings to `postgresql.conf`. The file is in + `/var/lib/pgsql/18/data/` on RHEL-family systems, and in + `/etc/postgresql/18/main/` on Ubuntu and Debian: ```ini shared_preload_libraries = 'pg_duckdb,coldfront' @@ -239,11 +252,16 @@ and create the extensions: [Connecting ColdFront to Lakekeeper and Object Storage](#connecting-coldfront-to-lakekeeper-and-object-storage) creates. -2. Restart PostgreSQL. `CREATE EXTENSION coldfront` fails on a server that - does not preload the extension. +3. Restart PostgreSQL. `CREATE EXTENSION coldfront` fails on a server that + does not preload the extension: -3. Connect to the database, create the role that `duckdb.postgres_role` - names, and create the extensions: + ```bash + sudo systemctl restart postgresql-18 # RHEL, Rocky Linux, AlmaLinux + sudo systemctl restart postgresql@18-main # Ubuntu, Debian + ``` + +4. Connect as the `postgres` superuser with `sudo -u postgres psql`, create + the role that `duckdb.postgres_role` names, and create the extensions: ```sql CREATE ROLE coldfront_duckdb NOLOGIN; From 0ddb19c5d4cd4d60b893a4885fcab99b1a1165d1 Mon Sep 17 00:00:00 2001 From: Susan Douglas Date: Thu, 15 Oct 2026 11:13:20 -0400 Subject: [PATCH 18/33] docs: list a PostgreSQL server in the README bare-metal requirements --- README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index b3be9f85..1daa7b75 100644 --- a/README.md +++ b/README.md @@ -330,9 +330,9 @@ Before building ColdFront on bare metal, install the pg_duckdb PostgreSQL extension (built on DuckDB 1.5.4) and a patched duckdb-iceberg DuckDB extension. For the prerequisites and the patches, see [Building ColdFront from Source](docs/installation.md#building-coldfront-from-source). -You also need `pg_config`, the PostgreSQL server development headers, the -libpq headers and library, `make`, `gcc`, Go 1.26.5 or later, and -golangci-lint. +You also need a running PostgreSQL 16, 17, or 18 server, `pg_config`, the +PostgreSQL server development headers, the libpq headers and library, `make`, +`gcc`, Go 1.26.5 or later, and golangci-lint. 1. Clone the repository: From 4644c5f5cfc79b02e569b57cf681335d3c727f00 Mon Sep 17 00:00:00 2001 From: Susan Douglas Date: Thu, 15 Oct 2026 11:13:20 -0400 Subject: [PATCH 19/33] docs: say the role is created once and the extensions in each database --- README.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/README.md b/README.md index 1daa7b75..18072865 100644 --- a/README.md +++ b/README.md @@ -269,6 +269,9 @@ configure it, and create the extensions: CREATE EXTENSION coldfront; ``` + You create the role once for the PostgreSQL server. Run the + `CREATE EXTENSION` commands in each database that uses ColdFront. + ### Installing and Configuring ColdFront in a Docker Container The Docker build adds a thin ColdFront layer to a published base image. The @@ -392,6 +395,9 @@ PostgreSQL server development headers, the libpq headers and library, `make`, CREATE EXTENSION coldfront; ``` + You create the role once for the PostgreSQL server. Run the + `CREATE EXTENSION` commands in each database that uses ColdFront. + For the full steps, see [Building ColdFront on Bare Metal](docs/installation.md#building-coldfront-on-bare-metal). From 18edfd1ce89e5760a6d154832237625ffcb49a88 Mon Sep 17 00:00:00 2001 From: Susan Douglas Date: Thu, 15 Oct 2026 11:13:20 -0400 Subject: [PATCH 20/33] docs: turn off DuckDB extension autoinstall in the README bare-metal steps --- README.md | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 18072865..fb839843 100644 --- a/README.md +++ b/README.md @@ -371,15 +371,19 @@ PostgreSQL server development headers, the libpq headers and library, `make`, ```ini shared_preload_libraries = 'pg_duckdb,coldfront' duckdb.allow_unsigned_extensions = true + duckdb.autoinstall_known_extensions = false duckdb.postgres_role = 'coldfront_duckdb' coldfront.warehouse = 'wh' coldfront.lakekeeper_endpoint = 'http://:8181/catalog' ``` The locally built `iceberg` extension is unsigned, so - `duckdb.allow_unsigned_extensions` must be on. `duckdb.postgres_role` lets - the application roles you grant access to run DuckDB; without it, only - superusers can. `coldfront.warehouse` names the `wh` warehouse that + `duckdb.allow_unsigned_extensions` must be on. + `duckdb.autoinstall_known_extensions` stays off so that pg_duckdb never + fetches the unpatched upstream extensions in place of yours. + `duckdb.postgres_role` lets the application roles you grant access to run + DuckDB; without it, only superusers can. `coldfront.warehouse` names the + `wh` warehouse that [Connecting ColdFront to Lakekeeper and Object Storage](#connecting-coldfront-to-lakekeeper-and-object-storage) creates. From 694a72f6d4a7e3c44e602f89bc740e8ee541178b Mon Sep 17 00:00:00 2001 From: Susan Douglas Date: Thu, 15 Oct 2026 11:17:16 -0400 Subject: [PATCH 21/33] docs: install and start PostgreSQL before ColdFront in the README package steps --- README.md | 49 ++++++++++++++++++++++++++++--------------------- 1 file changed, 28 insertions(+), 21 deletions(-) diff --git a/README.md b/README.md index fb839843..8d6b3f34 100644 --- a/README.md +++ b/README.md @@ -192,10 +192,29 @@ build it from source in a Docker container or on bare metal. ColdFront packages come from the pgEdge package repository. First, configure the repository for your platform, as described in the -[pgEdge Enterprise Postgres guide](https://docs.pgedge.com/enterprise/). Then, -install the ColdFront extension package for your PostgreSQL major version. The -package manager also installs the pgEdge PostgreSQL server, pg_duckdb, and -ColdFront's DuckDB extensions. +[pgEdge Enterprise Postgres guide](https://docs.pgedge.com/enterprise/). + +Next, install pgEdge Enterprise Postgres and start the server. On RHEL, Rocky +Linux, or AlmaLinux, the package does not create a database cluster, so create +and start one: + +```bash +sudo dnf install -y pgedge-enterprise-postgres_18 +sudo /usr/pgsql-18/bin/postgresql-18-setup initdb +sudo systemctl enable --now postgresql-18 +``` + +On Ubuntu or Debian, the package creates and starts the `18/main` cluster; +`sudo pg_lsclusters` shows it: + +```bash +sudo apt update +sudo apt install -y pgedge-enterprise-postgres-18 +``` + +Then, install the ColdFront extension package for your PostgreSQL major +version. The package manager also installs pg_duckdb and ColdFront's DuckDB +extensions. For PostgreSQL 18 on RHEL, Rocky Linux, or AlmaLinux: @@ -206,7 +225,6 @@ sudo dnf install -y pgedge-coldfront_18 For PostgreSQL 18 on Ubuntu or Debian: ```bash -sudo apt update sudo apt install -y pgedge-postgresql-18-coldfront ``` @@ -215,21 +233,10 @@ Install `pgedge-coldfront` and `pgedge-lakekeeper` on the hosts that need them. For repository setup and every package name, see [Using a Package to Install ColdFront](docs/installation.md#using-a-package-to-install-coldfront). -After installing the packages, use the following steps to start PostgreSQL, -configure it, and create the extensions: - -1. Initialize and start PostgreSQL. On RHEL, Rocky Linux, or AlmaLinux, the - package does not create a database cluster, so create and start one: - - ```bash - sudo /usr/pgsql-18/bin/postgresql-18-setup initdb - sudo systemctl enable --now postgresql-18 - ``` - - On Ubuntu or Debian, the package creates and starts the `18/main` cluster; - `sudo pg_lsclusters` shows it. +After installing the packages, use the following steps to configure +PostgreSQL and create the extensions: -2. Add the following settings to `postgresql.conf`. The file is in +1. Add the following settings to `postgresql.conf`. The file is in `/var/lib/pgsql/18/data/` on RHEL-family systems, and in `/etc/postgresql/18/main/` on Ubuntu and Debian: @@ -252,7 +259,7 @@ configure it, and create the extensions: [Connecting ColdFront to Lakekeeper and Object Storage](#connecting-coldfront-to-lakekeeper-and-object-storage) creates. -3. Restart PostgreSQL. `CREATE EXTENSION coldfront` fails on a server that +2. Restart PostgreSQL. `CREATE EXTENSION coldfront` fails on a server that does not preload the extension: ```bash @@ -260,7 +267,7 @@ configure it, and create the extensions: sudo systemctl restart postgresql@18-main # Ubuntu, Debian ``` -4. Connect as the `postgres` superuser with `sudo -u postgres psql`, create +3. Connect as the `postgres` superuser with `sudo -u postgres psql`, create the role that `duckdb.postgres_role` names, and create the extensions: ```sql From 682433a5e587dd50102fd010033c76829f86691e Mon Sep 17 00:00:00 2001 From: Susan Douglas Date: Thu, 15 Oct 2026 11:25:12 -0400 Subject: [PATCH 22/33] docs: create the Debian cluster and check the server in the README package steps --- README.md | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 8d6b3f34..dd56cced 100644 --- a/README.md +++ b/README.md @@ -204,14 +204,26 @@ sudo /usr/pgsql-18/bin/postgresql-18-setup initdb sudo systemctl enable --now postgresql-18 ``` -On Ubuntu or Debian, the package creates and starts the `18/main` cluster; -`sudo pg_lsclusters` shows it: +On Ubuntu or Debian, install the package, then check for a cluster with +`sudo pg_lsclusters`. If it lists no `18/main` cluster, create and start one: ```bash sudo apt update sudo apt install -y pgedge-enterprise-postgres-18 +sudo pg_lsclusters +sudo pg_createcluster 18 main --start # only if pg_lsclusters lists none ``` +On either platform, confirm that the server is running before you continue: + +```bash +sudo -u postgres psql -c "SELECT version();" +``` + +The command prints the PostgreSQL version. A connection error means the server +is not running; check `sudo systemctl status postgresql-18` on RHEL-family +systems or `sudo pg_lsclusters` on Ubuntu and Debian. + Then, install the ColdFront extension package for your PostgreSQL major version. The package manager also installs pg_duckdb and ColdFront's DuckDB extensions. From e2bc6bea2c0339a57075713cd54679b5bb7d4904 Mon Sep 17 00:00:00 2001 From: Susan Douglas Date: Thu, 15 Oct 2026 11:46:49 -0400 Subject: [PATCH 23/33] docs: split the README package steps and install the tools and Lakekeeper with ColdFront --- README.md | 49 +++++++++++++++++++++++++++++++++---------------- 1 file changed, 33 insertions(+), 16 deletions(-) diff --git a/README.md b/README.md index dd56cced..5ce1aba3 100644 --- a/README.md +++ b/README.md @@ -190,13 +190,22 @@ build it from source in a Docker container or on bare metal. ### Installing and Configuring ColdFront with Packages -ColdFront packages come from the pgEdge package repository. First, configure -the repository for your platform, as described in the -[pgEdge Enterprise Postgres guide](https://docs.pgedge.com/enterprise/). +In the sections that follow, we'll walk you through installing PostgreSQL and +ColdFront using packages. ColdFront packages come from the pgEdge package +repository, so first configure the repository for your platform, as described +in the [pgEdge Enterprise Postgres guide](https://docs.pgedge.com/enterprise/). -Next, install pgEdge Enterprise Postgres and start the server. On RHEL, Rocky -Linux, or AlmaLinux, the package does not create a database cluster, so create -and start one: +If you already have a pgEdge Enterprise Postgres server installed that you +would like to use, configure the pgEdge repository and install only ColdFront, +continuing with +[Installing the ColdFront Packages](#installing-the-coldfront-packages). +ColdFront packages from pgEdge require pgEdge's PostgreSQL build. + +#### Installing PostgreSQL with Packages + +Before installing ColdFront, you will need to install pgEdge Enterprise Postgres +and start the server. On RHEL, Rocky Linux, or AlmaLinux, the package does not +create a database cluster, so create and start one: ```bash sudo dnf install -y pgedge-enterprise-postgres_18 @@ -224,27 +233,34 @@ The command prints the PostgreSQL version. A connection error means the server is not running; check `sudo systemctl status postgresql-18` on RHEL-family systems or `sudo pg_lsclusters` on Ubuntu and Debian. -Then, install the ColdFront extension package for your PostgreSQL major -version. The package manager also installs pg_duckdb and ColdFront's DuckDB -extensions. +#### Installing the ColdFront Packages + +Install the ColdFront extension package for your PostgreSQL major version, the +ColdFront command-line tools (the archiver, the partitioner, and the +compactor), and Lakekeeper (the Iceberg catalog). The package manager also +installs pg_duckdb and ColdFront's DuckDB extensions. For PostgreSQL 18 on RHEL, Rocky Linux, or AlmaLinux: ```bash -sudo dnf install -y pgedge-coldfront_18 +sudo dnf install -y pgedge-coldfront_18 pgedge-coldfront pgedge-lakekeeper ``` For PostgreSQL 18 on Ubuntu or Debian: ```bash -sudo apt install -y pgedge-postgresql-18-coldfront +sudo apt install -y pgedge-postgresql-18-coldfront pgedge-coldfront \ + pgedge-lakekeeper ``` -The extension package does not include the command-line tools or Lakekeeper. -Install `pgedge-coldfront` and `pgedge-lakekeeper` on the hosts that need -them. For repository setup and every package name, see +The `pgedge-lakekeeper` package installs Lakekeeper but does not start it; you +configure and start it later, in +[Connecting ColdFront to Lakekeeper and Object Storage](#connecting-coldfront-to-lakekeeper-and-object-storage). +For repository setup and every package name, see [Using a Package to Install ColdFront](docs/installation.md#using-a-package-to-install-coldfront). +#### Configuring PostgreSQL and Creating the Extensions + After installing the packages, use the following steps to configure PostgreSQL and create the extensions: @@ -456,8 +472,9 @@ the host. [Configuring your Object Store](docs/object_store.md) guide covers configuration of each one, including production stores. -2. Install Lakekeeper. With the pgEdge repository configured, install the - `pgedge-lakekeeper` package: +2. Install Lakekeeper. If you installed ColdFront from packages, Lakekeeper is + already installed; continue with step 3. On bare metal with the pgEdge + repository configured, install the `pgedge-lakekeeper` package: ```bash sudo dnf install -y pgedge-lakekeeper # RHEL, Rocky Linux, AlmaLinux From 63291212dd56d67267d880814891dc5dd231733c Mon Sep 17 00:00:00 2001 From: Susan Douglas Date: Thu, 15 Oct 2026 11:57:27 -0400 Subject: [PATCH 24/33] docs: bullet the settings notes and add next-step links in the README install sections --- README.md | 46 ++++++++++++++++++++++++++++------------------ 1 file changed, 28 insertions(+), 18 deletions(-) diff --git a/README.md b/README.md index 5ce1aba3..4d544b4b 100644 --- a/README.md +++ b/README.md @@ -278,14 +278,15 @@ PostgreSQL and create the extensions: coldfront.lakekeeper_endpoint = 'http://:8181/catalog' ``` - `duckdb.extension_directory`, `duckdb.allow_unsigned_extensions`, and - `duckdb.autoinstall_known_extensions` make pg_duckdb load ColdFront's - patched DuckDB extensions. Without them, pg_duckdb can fetch the unpatched - upstream extensions instead. `duckdb.postgres_role` lets the application - roles you grant access to run DuckDB; without it, only superusers can. - `coldfront.warehouse` names the `wh` warehouse that - [Connecting ColdFront to Lakekeeper and Object Storage](#connecting-coldfront-to-lakekeeper-and-object-storage) - creates. + - `duckdb.extension_directory`, `duckdb.allow_unsigned_extensions`, and + `duckdb.autoinstall_known_extensions` make pg_duckdb load ColdFront's + patched DuckDB extensions. Without them, pg_duckdb can fetch the + unpatched upstream extensions instead. + - `duckdb.postgres_role` lets the application roles you grant access to run + DuckDB; without it, only superusers can. + - `coldfront.warehouse` names the `wh` warehouse that + [Connecting ColdFront to Lakekeeper and Object Storage](#connecting-coldfront-to-lakekeeper-and-object-storage) + creates. 2. Restart PostgreSQL. `CREATE EXTENSION coldfront` fails on a server that does not preload the extension: @@ -307,6 +308,9 @@ PostgreSQL and create the extensions: You create the role once for the PostgreSQL server. Run the `CREATE EXTENSION` commands in each database that uses ColdFront. +When the extensions are created, continue with +[Connecting a Package or Bare-Metal Installation to Lakekeeper and an Object Store](#connecting-a-package-or-bare-metal-installation-to-lakekeeper-and-an-object-store). + ### Installing and Configuring ColdFront in a Docker Container The Docker build adds a thin ColdFront layer to a published base image. The @@ -362,6 +366,9 @@ without a password. For information about the image's environment variables and a smoke test, see [Building the Image on Docker](docs/installation.md#building-the-image-on-docker). +When the extensions are created, continue with +[Connecting a Docker Installation to Lakekeeper and an Object Store](#connecting-a-docker-installation-to-lakekeeper-and-an-object-store). + ### Installing and Configuring ColdFront on Bare Metal Before building ColdFront on bare metal, install the pg_duckdb PostgreSQL @@ -412,15 +419,15 @@ PostgreSQL server development headers, the libpq headers and library, `make`, coldfront.lakekeeper_endpoint = 'http://:8181/catalog' ``` - The locally built `iceberg` extension is unsigned, so - `duckdb.allow_unsigned_extensions` must be on. - `duckdb.autoinstall_known_extensions` stays off so that pg_duckdb never - fetches the unpatched upstream extensions in place of yours. - `duckdb.postgres_role` lets the application roles you grant access to run - DuckDB; without it, only superusers can. `coldfront.warehouse` names the - `wh` warehouse that - [Connecting ColdFront to Lakekeeper and Object Storage](#connecting-coldfront-to-lakekeeper-and-object-storage) - creates. + - `duckdb.allow_unsigned_extensions` must be on, because the locally built + `iceberg` extension is unsigned. + - `duckdb.autoinstall_known_extensions` stays off so that pg_duckdb never + fetches the unpatched upstream extensions in place of yours. + - `duckdb.postgres_role` lets the application roles you grant access to run + DuckDB; without it, only superusers can. + - `coldfront.warehouse` names the `wh` warehouse that + [Connecting ColdFront to Lakekeeper and Object Storage](#connecting-coldfront-to-lakekeeper-and-object-storage) + creates. 6. Restart PostgreSQL. `CREATE EXTENSION coldfront` fails on a server that does not preload the extension. @@ -437,9 +444,12 @@ PostgreSQL server development headers, the libpq headers and library, `make`, You create the role once for the PostgreSQL server. Run the `CREATE EXTENSION` commands in each database that uses ColdFront. -For the full steps, see +For more information, see [Building ColdFront on Bare Metal](docs/installation.md#building-coldfront-on-bare-metal). +When the extensions are created, continue with +[Connecting a Package or Bare-Metal Installation to Lakekeeper and an Object Store](#connecting-a-package-or-bare-metal-installation-to-lakekeeper-and-an-object-store). + ### Connecting ColdFront to Lakekeeper and Object Storage After configuring the server and creating the extensions, you're ready to From 344414919caa6a598d0bd13e5ea0c39ffc284d4c Mon Sep 17 00:00:00 2001 From: Susan Douglas Date: Thu, 15 Oct 2026 12:00:50 -0400 Subject: [PATCH 25/33] docs: link SeaweedFS's docs for the host object store step in the README --- README.md | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 4d544b4b..e8ed81ff 100644 --- a/README.md +++ b/README.md @@ -469,14 +469,23 @@ the host. 1. Start an object store. If you're walking through this installation to evaluate ColdFront, you can use SeaweedFS, a free S3-compatible store, on - the host. The repository's `seaweedfs-s3.json` sets its credentials to - `admin` and `adminsecret`: + the host. Install SeaweedFS as described in its + [Getting Started](https://github.com/seaweedfs/seaweedfs/wiki/Getting-Started) + guide; the [releases page](https://github.com/seaweedfs/seaweedfs/releases) + has prebuilt `weed` binaries. Then create an S3 credentials file with the + access key `admin` and the secret key `adminsecret`, in the format shown in + [S3 Configuration](https://github.com/seaweedfs/seaweedfs/wiki/Amazon-S3-API#s3-configuration), + and start SeaweedFS with its S3 gateway: ```bash - weed server -s3 -dir=/var/lib/seaweedfs \ - -s3.config=docker/seaweedfs-s3.json + sudo mkdir -p /var/lib/seaweedfs + sudo weed server -s3 -dir=/var/lib/seaweedfs -s3.config=/etc/seaweedfs/s3.json ``` + The ColdFront repository's + [`docker/seaweedfs-s3.json`](docker/seaweedfs-s3.json) is a ready-made + credentials file you can copy to `/etc/seaweedfs/s3.json`. + If you would like to evaluate ColdFront on AWS S3, Google Cloud Storage, or Azure ADLS Gen2, create a bucket or container there instead. The [Configuring your Object Store](docs/object_store.md) guide covers From 5414ab5c770b926da2e0633ced97cadfb62aa765 Mon Sep 17 00:00:00 2001 From: Susan Douglas Date: Tue, 20 Oct 2026 07:19:57 -0400 Subject: [PATCH 26/33] docs: connect as a superuser in the README bare-metal steps --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index e8ed81ff..de3345d8 100644 --- a/README.md +++ b/README.md @@ -432,8 +432,8 @@ PostgreSQL server development headers, the libpq headers and library, `make`, 6. Restart PostgreSQL. `CREATE EXTENSION coldfront` fails on a server that does not preload the extension. -7. Connect to the database, create the role that `duckdb.postgres_role` - names, and create the extensions: +7. Connect as a PostgreSQL superuser, such as `postgres`, create the role + that `duckdb.postgres_role` names, and create the extensions: ```sql CREATE ROLE coldfront_duckdb NOLOGIN; From b737ad4096faa2133e17adf3058473a0ee3c2276 Mon Sep 17 00:00:00 2001 From: Susan Douglas Date: Tue, 20 Oct 2026 07:20:01 -0400 Subject: [PATCH 27/33] docs: run make install with sudo in the README bare-metal steps --- README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index de3345d8..545a4430 100644 --- a/README.md +++ b/README.md @@ -391,7 +391,8 @@ PostgreSQL server development headers, the libpq headers and library, `make`, ```bash cd extension/coldfront - make && make install + make + sudo make install PG_CONFIG="$(command -v pg_config)" ``` 3. Build the command-line tools. `make build` builds the archiver and the From 7a58bd6fdc8f3847bbd2ccbf84c31ab7478556ee Mon Sep 17 00:00:00 2001 From: Susan Douglas Date: Tue, 20 Oct 2026 07:20:35 -0400 Subject: [PATCH 28/33] docs: say how to recover the warehouse ID in the README --- README.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/README.md b/README.md index 545a4430..958d6d98 100644 --- a/README.md +++ b/README.md @@ -580,6 +580,12 @@ the host. fi ``` + If you run these commands again after the warehouse exists, the create + request fails and `WID` is empty. Set `WID` to the `warehouse-id` of the + `wh` warehouse in the output of + `curl -s http://localhost:8181/management/v1/warehouse`, then rerun the + namespace request. + If you're using a different type of object store, change the `storage-profile` and `storage-credential` as shown in the store's section of [Configuring your Object Store](docs/object_store.md#configuring-lakekeeper). @@ -640,6 +646,12 @@ to evaluate ColdFront without a cloud account. fi ``` + If you run these commands again after the warehouse exists, the create + request fails and `WID` is empty. Set `WID` to the `warehouse-id` of the + `wh` warehouse in the output of + `curl -s http://localhost:8181/management/v1/warehouse`, then rerun the + namespace request. + 4. Store the SeaweedFS credential in each database that uses ColdFront: ```sql From aacbe2d724c91bf086909734d2a73e43bdb34c27 Mon Sep 17 00:00:00 2001 From: Susan Douglas Date: Tue, 20 Oct 2026 07:48:19 -0400 Subject: [PATCH 29/33] docs: say the pgEdge repository requires a certificate in the README --- README.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 958d6d98..62d0c7df 100644 --- a/README.md +++ b/README.md @@ -192,8 +192,10 @@ build it from source in a Docker container or on bare metal. In the sections that follow, we'll walk you through installing PostgreSQL and ColdFront using packages. ColdFront packages come from the pgEdge package -repository, so first configure the repository for your platform, as described -in the [pgEdge Enterprise Postgres guide](https://docs.pgedge.com/enterprise/). +repository, which requires a certificate from pgEdge. First, request a +certificate and install it, then configure the repository for your platform, +as described in the +[pgEdge Enterprise Postgres guide](https://docs.pgedge.com/enterprise/). If you already have a pgEdge Enterprise Postgres server installed that you would like to use, configure the pgEdge repository and install only ColdFront, From 763111c87b90f2a0c324a8dd27dcfe277f1e6abc Mon Sep 17 00:00:00 2001 From: Susan Douglas Date: Tue, 20 Oct 2026 08:26:41 -0400 Subject: [PATCH 30/33] docs: create the Lakekeeper catalog role as a superuser in the README --- README.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 62d0c7df..ff92715c 100644 --- a/README.md +++ b/README.md @@ -517,7 +517,9 @@ the host. sudo systemctl daemon-reload ``` -3. Create a role and a database for the Lakekeeper catalog: +3. Connect as a PostgreSQL superuser, such as `postgres` (for example, with + `sudo -u postgres psql`), and create a role and a database for the + Lakekeeper catalog: ```sql CREATE ROLE lakekeeper LOGIN PASSWORD 'change-me'; From 1ec85d3e7076248210d504ba03ccdaba93b529c2 Mon Sep 17 00:00:00 2001 From: Susan Douglas Date: Tue, 20 Oct 2026 08:59:29 -0400 Subject: [PATCH 31/33] docs: publish the Docker stack's ports on loopback only in the README --- README.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index ff92715c..2eaa302b 100644 --- a/README.md +++ b/README.md @@ -344,10 +344,14 @@ creates. 3. Build the ColdFront layer and start the stack. The `local-store` profile adds a SeaweedFS object store to the stack. The stack uses PostgreSQL 18 - unless you set `PG_MAJOR`: + unless you set `PG_MAJOR`. The command publishes each port on the loopback + address only, because PostgreSQL trusts every connection and Lakekeeper + runs without authentication: ```bash - docker compose --profile local-store up -d --build + COLDFRONT_PG_PORT=127.0.0.1:5432 COLDFRONT_LK_PORT=127.0.0.1:8181 \ + COLDFRONT_S3_PORT=127.0.0.1:8333 \ + docker compose --profile local-store up -d --build ``` 4. Connect to the database: From 6ab7a2446b4bf63799bb0e90fccf3520f778876d Mon Sep 17 00:00:00 2001 From: Susan Douglas Date: Tue, 20 Oct 2026 10:20:50 -0400 Subject: [PATCH 32/33] docs: use a generated password for the Lakekeeper catalog role in the README --- README.md | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index 2eaa302b..85224eec 100644 --- a/README.md +++ b/README.md @@ -523,19 +523,21 @@ the host. 3. Connect as a PostgreSQL superuser, such as `postgres` (for example, with `sudo -u postgres psql`), and create a role and a database for the - Lakekeeper catalog: + Lakekeeper catalog. Replace `` with a generated password, such + as the output of `openssl rand -hex 32`; a hex value needs no escaping in + the connection string in step 4: ```sql - CREATE ROLE lakekeeper LOGIN PASSWORD 'change-me'; + CREATE ROLE lakekeeper LOGIN PASSWORD ''; CREATE DATABASE lakekeeper OWNER lakekeeper; ``` -4. In `/etc/lakekeeper/lakekeeper.env`, set the catalog's connection string - and an encryption key, and bind Lakekeeper to the loopback address. Keep - the key stable and backed up: +4. In `/etc/lakekeeper/lakekeeper.env`, set the catalog's connection string, + with the same password, and an encryption key, and bind Lakekeeper to the + loopback address. Keep the key stable and backed up: ```bash - LAKEKEEPER__PG_DATABASE_URL_WRITE=postgresql://lakekeeper:change-me@127.0.0.1:5432/lakekeeper + LAKEKEEPER__PG_DATABASE_URL_WRITE=postgresql://lakekeeper:@127.0.0.1:5432/lakekeeper LAKEKEEPER__PG_ENCRYPTION_KEY= LAKEKEEPER__BIND_IP=127.0.0.1 ``` From daa20473a02a585fb8ca9a8799ca05ade2376dd5 Mon Sep 17 00:00:00 2001 From: Susan Douglas Date: Tue, 20 Oct 2026 10:20:58 -0400 Subject: [PATCH 33/33] docs: say the deployment YAML is an import input, not the settings source --- README.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 85224eec..f00663c8 100644 --- a/README.md +++ b/README.md @@ -170,10 +170,12 @@ Settings live in two places: - `coldfront.storage_secret` holds the cold-store credential. The archiver, partitioner, and compactor connect through the libpq environment -or `--dsn`, and read the settings from the server. To load a deployment YAML -into the server, run `archiver import --config ` once. Model the file on -[config.example.yaml](config.example.yaml). A later run checks any YAML you -pass against the server and uses only its `postgres.dsn`. +or `--dsn`, and read the settings from the server. A deployment YAML file is +only an input for loading settings: run `archiver import --config ` once +to write it into the server, and model the file on +[config.example.yaml](config.example.yaml). After that, a run that you pass a +YAML file checks it against the server and takes only its `postgres.dsn`, +which supplies the connection when you don't pass `--dsn`. To review a list of available settings, see: