From 2db85db3994a268f572060cd5e954a31ceecbe66 Mon Sep 17 00:00:00 2001 From: Susan Douglas Date: Tue, 6 Oct 2026 12:08:08 -0400 Subject: [PATCH 01/11] docs: document package-based install and sync nav titles to page headings --- docs/index.md | 4 +- docs/installation.md | 116 ++++++++++++++++++++++++++++++++++--- docs/object_store.md | 2 +- docs/walkthrough.md | 4 +- docs/walkthrough_tiered.md | 4 +- mkdocs.yml | 30 +++++----- 6 files changed, 129 insertions(+), 31 deletions(-) diff --git a/docs/index.md b/docs/index.md index b60f72d..d56ca39 100644 --- a/docs/index.md +++ b/docs/index.md @@ -159,8 +159,8 @@ To go further with ColdFront, consult the following guides: partition manager, supported types, and tuning. - The [Embeddings](usage_vectors.md) guide covers storing and searching embeddings through the pgvector interface. -- The [Object Store Setup](object_store.md) guide takes you from an empty - bucket to a working cold tier on cloud S3. +- The [Configuring your Object Store](object_store.md) guide takes you from an + empty bucket to a working cold tier on cloud S3. - The [Compaction](compaction.md) guide covers cold-tier maintenance: compaction, snapshot expiry, and orphan-file removal. - The [Architecture](architecture.md) overview explains the shared mechanics diff --git a/docs/installation.md b/docs/installation.md index 5e044ee..35564b2 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -1,13 +1,111 @@ -# Building ColdFront from Source +# Installing ColdFront -This guide builds ColdFront from source, either in Docker on top of the -published base image or on bare metal. +This guide covers installing ColdFront from the published package and +building it from source. + +## Using a Package to Install ColdFront + +Before using a package to install ColdFront, you'll need to add the pgEdge +package repository to your system. + +For RHEL, Rocky Linux, or AlmaLinux 10, first install EPEL: + +```bash +sudo dnf -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-10.noarch.rpm +``` + +On RHEL 10, also enable the CodeReady Builder repository: + +```bash +sudo subscription-manager repos --enable codeready-builder-for-rhel-10-$(arch)-rpms +``` + +On Rocky Linux 10 or AlmaLinux 10, enable the equivalent CRB repository instead: + +```bash +sudo dnf config-manager --set-enabled crb +``` + +Version-specific commands for RHEL, OEL, Alma, and Rocky are available in the +[pgEdge Enterprise Postgres documentation](https://docs.pgedge.com/enterprise/el/configure-repo/). +On every EL variant, disable any PGDG repository so it does not shadow +pgEdge's packages, then install the pgEdge repository itself: + +```bash +sudo dnf config-manager --set-disabled 'pgdg*' +sudo dnf install -y https://dnf.pgedge.com/reporpm/pgedge-release-latest.noarch.rpm +``` + +For Ubuntu or Debian, first install the repository prerequisites: + +```bash +sudo apt-get update +sudo apt-get install -y curl gnupg2 lsb-release +``` -!!! note "Most users should install from packages" +Then add the repository: - See [Installation](https://github.com/pgEdge/ColdFront/blob/main/README.md#installation) - in the README. This document is the **build-from-source** workflow: build - the patched DuckDB-1.5.x stack yourself, in Docker or bare-metal. +```bash +sudo curl -sSL https://apt.pgedge.com/repodeb/pgedge-release_latest_all.deb -o /tmp/pgedge-release.deb +sudo dpkg -i /tmp/pgedge-release.deb && rm -f /tmp/pgedge-release.deb || true +``` + +With the repository in place, install the meta-package for your PostgreSQL +major version; it pulls in every other package ColdFront needs. The +following table shows each package's purpose and its name on RHEL-family and +Debian-family systems: + +| Package | RHEL, Rocky Linux, AlmaLinux | Ubuntu, Debian | +|---|---|---| +| Meta-package (installs the following packages) | `pgedge-coldfront_` | `pgedge-postgresql--coldfront` | +| ColdFront extension | `pgedge-coldfront` | `pgedge-coldfront` | +| ColdFront's DuckDB extensions | `pgedge-coldfront-duckdb-extensions` | `pgedge-coldfront-duckdb-extensions` | +| pg_duckdb | `pgedge-pg-duckdb_` | `pgedge-postgresql--pg-duckdb` | +| Lakekeeper | `pgedge-lakekeeper` | `pgedge-lakekeeper` | + +For PostgreSQL 18 on RHEL, Rocky Linux, or AlmaLinux: + +```bash +sudo dnf install -y pgedge-coldfront_18 +``` + +On Ubuntu or Debian: + +```bash +sudo apt install -y pgedge-postgresql-18-coldfront +``` + +Install the individual packages from the table instead of the meta-package +only if you need a different combination than it provides. + +## Configuring ColdFront + +ColdFront reads its settings from two places. The server settings live in +`postgresql.conf`: + +- `shared_preload_libraries = 'pg_duckdb,coldfront'`. +- `coldfront.warehouse` and `coldfront.lakekeeper_endpoint`. +- `snowflake.node` and `coldfront.loopback_dsn` on every node of a mesh. + +A package install does not set these for you - the Docker image is what +writes them on first start - so add them to `postgresql.conf` yourself and +restart PostgreSQL before creating the extensions. + +The archiver, partitioner, and compactor connect from the libpq environment +or `--dsn` and read everything else from the server: each table's lifecycle +in `coldfront.partition_config`, the cold-store credential in +`coldfront.storage_secret`, and the catalog settings above. The `import` +command takes a deployment YAML, modeled on +[config.example.yaml](https://github.com/pgEdge/ColdFront/blob/main/config.example.yaml), +and writes it into the server once. + +For every setting, see the [One-Time Setup](usage.md#one-time-setup) and +[Tuning Knobs](usage.md#tuning-knobs) sections of the Using ColdFront guide. + +## Building ColdFront from Source + +This guide builds ColdFront from source, either in Docker on top of the +published base image or on bare metal. ColdFront runs on a **DuckDB 1.5.x** stack: PostgreSQL + pg_duckdb (DuckDB 1.5.4) and a **patched** duckdb-iceberg that includes ColdFront's five @@ -312,5 +410,5 @@ To go further with ColdFront, consult the following guides: - The [Walkthrough](walkthrough.md) guide runs the demo stack hands-on. - The [Using ColdFront](usage.md) guide covers the one-time setup and both modes. -- The [Object Store Setup](object_store.md) guide connects the cold tier to AWS - S3. +- The [Configuring your Object Store](object_store.md) guide connects the cold + tier to AWS S3. diff --git a/docs/object_store.md b/docs/object_store.md index fbf1ecd..dc46723 100644 --- a/docs/object_store.md +++ b/docs/object_store.md @@ -1,4 +1,4 @@ -# Get ColdFront Running on S3 +# Configuring your Object Store This walkthrough takes you from an empty S3 bucket to a working ColdFront cold tier in one sitting. You stand up the ColdFront stack (PostgreSQL + the diff --git a/docs/walkthrough.md b/docs/walkthrough.md index 73f64c7..2b4cba8 100644 --- a/docs/walkthrough.md +++ b/docs/walkthrough.md @@ -165,8 +165,8 @@ store: | GCS (HMAC) | `SELECT coldfront.set_storage_secret(p_key_id => '', p_secret => '', p_endpoint => 'storage.googleapis.com', p_region => 'us-east-1', p_url_style => 'path', p_use_ssl => true);` | | Azure ADLS Gen2 | `SELECT coldfront.set_storage_secret_azure('AccountName=;AccountKey=;EndpointSuffix=core.windows.net');` | -The [Object Store Setup](object_store.md) guide shows the matching warehouse -JSON for AWS S3; for GCS and Azure, see the +The [Configuring your Object Store](object_store.md) guide shows the matching +warehouse JSON for AWS S3; for GCS and Azure, see the [Storage Backends](usage.md#storage-backends) section of the Using ColdFront guide. diff --git a/docs/walkthrough_tiered.md b/docs/walkthrough_tiered.md index 6276431..62bc626 100644 --- a/docs/walkthrough_tiered.md +++ b/docs/walkthrough_tiered.md @@ -595,8 +595,8 @@ To go further with ColdFront, consult the following guides: - The [Decoupled Mode Demo](walkthrough_decoupled.md) stores a table in Iceberg from the first row and adopts a table that another engine wrote. -- The [Object Store Setup](object_store.md) guide takes you from an empty - bucket to a working cold tier on AWS S3. +- The [Configuring your Object Store](object_store.md) guide takes you from an + empty bucket to a working cold tier on AWS S3. - The [Architecture](architecture.md) overview explains the shared mechanics and links to the per-mode deep dives. - The [Tearing Down the Stack](walkthrough.md#tearing-down-the-stack) section diff --git a/mkdocs.yml b/mkdocs.yml index e5fd719..305fef2 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -56,28 +56,28 @@ copyright: Copyright © 2026 pgEdge, Inc repo_url: https://github.com/pgEdge/ColdFront nav: + - pgEdge ColdFront: index.md + - Installing ColdFront: installation.md - Getting Started: - - Introduction: index.md - - Walkthrough: walkthrough.md - - Tiered Storage Demo: walkthrough_tiered.md - - Decoupled Mode Demo: walkthrough_decoupled.md - - Partitioner Demo: walkthrough_partitioner.md - - Distributed Demo: walkthrough_distributed.md - - Installation: installation.md - - Object Store Setup: object_store.md + - Getting Started with ColdFront: walkthrough.md + - Exploring Tiered Storage: walkthrough_tiered.md + - Exploring Decoupled Mode: walkthrough_decoupled.md + - Exploring the Standalone Partitioner: walkthrough_partitioner.md + - Exploring Distributed Mode: walkthrough_distributed.md + - Configuring your Object Store: object_store.md - Architecture: - - Overview: architecture.md - - Tiered Mode: architecture_tiered.md - - Decoupled Mode: architecture_decoupled.md - - Vector Storage: architecture_vectors.md + - pgEdge ColdFront - Architecture: architecture.md + - ColdFront - Tiered Operating Mode: architecture_tiered.md + - Decoupled (Iceberg-Only) Operating Mode: architecture_decoupled.md + - Vector Storage Architecture: architecture_vectors.md - Using ColdFront: usage.md - - Embeddings: usage_vectors.md - - Compaction: compaction.md + - Working with Embeddings: usage_vectors.md + - COMPACTOR - Cold-Tier Table Maintenance: compaction.md - Developer Resources: - - Formal Verification: formal/README.md + - Formal Model - ColdFront Decoupled-Mode Bakery (TLA+/PlusCal): formal/README.md - Changelog: changelog.md - License: LICENSE.md From d15f6e7f312ff752ee253d1689267fdb17f014c0 Mon Sep 17 00:00:00 2001 From: Susan Douglas Date: Tue, 6 Oct 2026 12:28:42 -0400 Subject: [PATCH 02/11] docs: reorganize installation.md headings and fix a stale anchor --- docs/installation.md | 59 +++++++++++++++++++++++++++----------------- docs/usage.md | 3 ++- 2 files changed, 39 insertions(+), 23 deletions(-) diff --git a/docs/installation.md b/docs/installation.md index 35564b2..bb2a1ce 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -87,14 +87,14 @@ ColdFront reads its settings from two places. The server settings live in - `coldfront.warehouse` and `coldfront.lakekeeper_endpoint`. - `snowflake.node` and `coldfront.loopback_dsn` on every node of a mesh. -A package install does not set these for you - the Docker image is what -writes them on first start - so add them to `postgresql.conf` yourself and -restart PostgreSQL before creating the extensions. +A package installation does not set these parameters for you; you must add +them to the `postgresql.conf` file and restart PostgreSQL before creating +the extensions. The archiver, partitioner, and compactor connect from the libpq environment or `--dsn` and read everything else from the server: each table's lifecycle in `coldfront.partition_config`, the cold-store credential in -`coldfront.storage_secret`, and the catalog settings above. The `import` +`coldfront.storage_secret`, and the catalog settings. The `import` command takes a deployment YAML, modeled on [config.example.yaml](https://github.com/pgEdge/ColdFront/blob/main/config.example.yaml), and writes it into the server once. @@ -102,10 +102,25 @@ and writes it into the server once. For every setting, see the [One-Time Setup](usage.md#one-time-setup) and [Tuning Knobs](usage.md#tuning-knobs) sections of the Using ColdFront guide. +After PostgreSQL restarts with `shared_preload_libraries` set, create both +extensions in your database: + +```sql +CREATE EXTENSION IF NOT EXISTS pg_duckdb; +CREATE EXTENSION IF NOT EXISTS coldfront; +``` + +!!! warning "CREATE EXTENSION coldfront is required and easy to miss" + + Preloading the library is not the same as creating the extension. + Skipping this step leaves `coldfront`'s schema and functions missing from + your database, and the next call fails with + `schema "coldfront" does not exist`. + ## Building ColdFront from Source -This guide builds ColdFront from source, either in Docker on top of the -published base image or on bare metal. +This section walks you through building ColdFront from source, either in +Docker on top of the published base image or on bare metal. ColdFront runs on a **DuckDB 1.5.x** stack: PostgreSQL + pg_duckdb (DuckDB 1.5.4) and a **patched** duckdb-iceberg that includes ColdFront's five @@ -118,7 +133,18 @@ a new table schema above the highest schema id. No released pg_duckdb tag includes DuckDB 1.5.x yet, so the stack is built from a pinned upstream PR plus ColdFront's patches - all from sources you can fetch. -## What the Build Produces +### Prerequisites + +The following table lists the prerequisites for each build path: + +| For | You need | +|---|---| +| Docker build (below) | Docker, network access (GitHub, ghcr.io, quay.io, curl.se, and the distribution's RPM repositories), a few GB of disk and RAM, and 30-60 minutes for the base compile. | +| The archiver and partitioner (all paths) | Go 1.26.5+ (pinned in [go.mod](https://github.com/pgEdge/ColdFront/blob/main/go.mod)) and `make` (`make build` produces `./bin/archiver` and `./bin/partitioner`). | +| The compactor and the CI gate | golangci-lint for `make compactor`; `./run-ci-local.sh` also needs Docker and mkdocs with mkdocs-material. | +| Bare metal (near the end of this section) | `pg_config`, PostgreSQL server dev headers, libpq client headers and library (libpq-dev / libpq-devel), `make`, and `gcc`. | + +### What the Build Produces `docker/Dockerfile.duckdb15-base` is the recipe; it fetches the requirements, applies ColdFront's patches, and compiles a set of components. The following @@ -163,7 +189,7 @@ canonical recipe - every source pin and compile step - is [`docker/Dockerfile.duckdb15-base`](https://github.com/pgEdge/ColdFront/blob/main/docker/Dockerfile.duckdb15-base) itself. -## Build the Image (Docker) +## Building the Image on Docker Build the stack in two stages, the prebuilt base and the thin app layer: @@ -216,7 +242,7 @@ ColdFront guide: bootstrap Lakekeeper, create a table, tier it, and verify. need pull access to that image (or substitute an equivalent PostgreSQL base with the same layout). -### Image Environment Variables +### Using Environment Variables The entrypoint reads the following variables when the container starts. It writes the server settings they control into `postgresql.conf` only when the @@ -239,7 +265,7 @@ describes each variable: The image is built for development and testing: `pg_hba.conf` trusts every connection from any address without a password. -## Verify the Build +### Verifying the Build A self-contained smoke test confirms the freshly built stack works end to end: pg_duckdb, the patched duckdb-iceberg, Lakekeeper, and the object store. The @@ -296,18 +322,7 @@ cloud store, drop the `local-store` profile, point the warehouse at your own bucket, and follow the [One-Time Setup](usage.md#one-time-setup) section of the Using ColdFront guide for the full tier-and-verify journey. -## Build Prerequisites - -The following table lists the prerequisites for each build path: - -| For | You need | -|---|---| -| Docker build (above) | Docker, network access (GitHub, ghcr.io, quay.io, curl.se, and the distribution's RPM repositories), a few GB of disk and RAM, and 30-60 minutes for the base compile. | -| The archiver and partitioner (all paths) | Go 1.26.5+ (pinned in [go.mod](https://github.com/pgEdge/ColdFront/blob/main/go.mod)) and `make` (`make build` produces `./bin/archiver` and `./bin/partitioner`). | -| The compactor and the CI gate | golangci-lint for `make compactor`; `./run-ci-local.sh` also needs Docker and mkdocs with mkdocs-material. | -| Bare metal (below) | `pg_config`, PostgreSQL server dev headers, libpq client headers and library (libpq-dev / libpq-devel), `make`, and `gcc`. | - -## Bare Metal (No Docker) +## Building ColdFront on Bare Metal The coldfront extension is a standard PGXS C extension: diff --git a/docs/usage.md b/docs/usage.md index af8453e..33606b9 100644 --- a/docs/usage.md +++ b/docs/usage.md @@ -96,7 +96,8 @@ and sets `p_region` to the bucket's Region. The image writes the server settings ColdFront needs into `postgresql.conf` when it initializes a new data directory. A server built another way sets them -itself, as [installation.md](installation.md#bare-metal-no-docker) shows: +itself, as [installation.md](installation.md#building-coldfront-on-bare-metal) +shows: - `shared_preload_libraries = 'pg_duckdb,coldfront'` loads both extensions at server start. coldfront refuses to load any other way: `CREATE EXTENSION From c04cdf0386313483206d0228f1df8a5c9e0119e2 Mon Sep 17 00:00:00 2001 From: "coderabbitai[bot]" <136622811+coderabbitai[bot]@users.noreply.github.com> Date: Tue, 6 Oct 2026 18:37:51 +0000 Subject: [PATCH 03/11] docs: fix Debian install commands, link to shared setup guidance, and align the object store guide title --- README.md | 2 +- docs/installation.md | 47 +++++++------------------------------------- 2 files changed, 8 insertions(+), 41 deletions(-) diff --git a/README.md b/README.md index 5c9c3cc..717edb1 100644 --- a/README.md +++ b/README.md @@ -186,7 +186,7 @@ The following table lists the ColdFront guides and what each one covers: | [Embeddings](docs/usage_vectors.md) | Covers storing and searching embeddings with the pgvector interface. | | [Usage](docs/usage.md) | Covers day-to-day use: both modes plus the standalone partition manager, one-time setup, reading and writing, supported types, the partition CLI, storage backends, distributed (mesh) setup, and tuning. | | [Installation](docs/installation.md) | Covers building from source (Docker or bare-metal), and testing and CI. | -| [Object store setup](docs/object_store.md) | Gets ColdFront running on cloud S3 (virtual-hosted), end to end. | +| [Configuring your Object Store](docs/object_store.md) | Gets ColdFront running on cloud S3 (virtual-hosted), end to end. | | [Compaction](docs/compaction.md) | Covers cold-tier table maintenance: compaction, snapshot expiry, and orphan-file removal. | | [Architecture](docs/architecture.md) | Describes the shared architecture and core mechanics. | | [Architecture: tiered](docs/architecture_tiered.md) | Describes tiered mode (hot PG plus cold Iceberg) in depth. | diff --git a/docs/installation.md b/docs/installation.md index bb2a1ce..725ef89 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -46,8 +46,9 @@ sudo apt-get install -y curl gnupg2 lsb-release Then add the repository: ```bash -sudo curl -sSL https://apt.pgedge.com/repodeb/pgedge-release_latest_all.deb -o /tmp/pgedge-release.deb -sudo dpkg -i /tmp/pgedge-release.deb && rm -f /tmp/pgedge-release.deb || true +sudo curl -sSL https://apt.pgedge.com/repodeb/pgedge-release_latest_all.deb -o /tmp/pgedge-release.deb && \ + sudo dpkg -i /tmp/pgedge-release.deb && \ + sudo rm -f /tmp/pgedge-release.deb ``` With the repository in place, install the meta-package for your PostgreSQL @@ -72,50 +73,16 @@ sudo dnf install -y pgedge-coldfront_18 On Ubuntu or Debian: ```bash +sudo apt update sudo apt install -y pgedge-postgresql-18-coldfront ``` Install the individual packages from the table instead of the meta-package only if you need a different combination than it provides. -## Configuring ColdFront - -ColdFront reads its settings from two places. The server settings live in -`postgresql.conf`: - -- `shared_preload_libraries = 'pg_duckdb,coldfront'`. -- `coldfront.warehouse` and `coldfront.lakekeeper_endpoint`. -- `snowflake.node` and `coldfront.loopback_dsn` on every node of a mesh. - -A package installation does not set these parameters for you; you must add -them to the `postgresql.conf` file and restart PostgreSQL before creating -the extensions. - -The archiver, partitioner, and compactor connect from the libpq environment -or `--dsn` and read everything else from the server: each table's lifecycle -in `coldfront.partition_config`, the cold-store credential in -`coldfront.storage_secret`, and the catalog settings. The `import` -command takes a deployment YAML, modeled on -[config.example.yaml](https://github.com/pgEdge/ColdFront/blob/main/config.example.yaml), -and writes it into the server once. - -For every setting, see the [One-Time Setup](usage.md#one-time-setup) and -[Tuning Knobs](usage.md#tuning-knobs) sections of the Using ColdFront guide. - -After PostgreSQL restarts with `shared_preload_libraries` set, create both -extensions in your database: - -```sql -CREATE EXTENSION IF NOT EXISTS pg_duckdb; -CREATE EXTENSION IF NOT EXISTS coldfront; -``` - -!!! warning "CREATE EXTENSION coldfront is required and easy to miss" - - Preloading the library is not the same as creating the extension. - Skipping this step leaves `coldfront`'s schema and functions missing from - your database, and the next call fails with - `schema "coldfront" does not exist`. +A package installation does not configure PostgreSQL. Follow the +[One-Time Setup](usage.md#one-time-setup) section of the Using ColdFront guide +to configure PostgreSQL and create the extensions. ## Building ColdFront from Source From 2b913b8f11740cd1c038c323f3915f48f8f3645b Mon Sep 17 00:00:00 2001 From: Jimmy Angelakos Date: Tue, 6 Oct 2026 19:49:54 +0100 Subject: [PATCH 04/11] docs: describe the ColdFront packages by their actual dependencies --- docs/installation.md | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-) diff --git a/docs/installation.md b/docs/installation.md index 725ef89..d1ad61c 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -51,17 +51,18 @@ sudo curl -sSL https://apt.pgedge.com/repodeb/pgedge-release_latest_all.deb -o / sudo rm -f /tmp/pgedge-release.deb ``` -With the repository in place, install the meta-package for your PostgreSQL -major version; it pulls in every other package ColdFront needs. The -following table shows each package's purpose and its name on RHEL-family and -Debian-family systems: +With the repository in place, install the ColdFront extension package for +your PostgreSQL major version. It depends on the pgEdge PostgreSQL server, +pg_duckdb, and ColdFront's DuckDB extensions, so the package manager installs +those as well. The following table shows each package's purpose and its name +on RHEL-family and Debian-family systems: | Package | RHEL, Rocky Linux, AlmaLinux | Ubuntu, Debian | |---|---|---| -| Meta-package (installs the following packages) | `pgedge-coldfront_` | `pgedge-postgresql--coldfront` | -| ColdFront extension | `pgedge-coldfront` | `pgedge-coldfront` | +| ColdFront extension | `pgedge-coldfront_` | `pgedge-postgresql--coldfront` | | ColdFront's DuckDB extensions | `pgedge-coldfront-duckdb-extensions` | `pgedge-coldfront-duckdb-extensions` | | pg_duckdb | `pgedge-pg-duckdb_` | `pgedge-postgresql--pg-duckdb` | +| ColdFront command-line tools (archiver, partitioner, compactor) | `pgedge-coldfront` | `pgedge-coldfront` | | Lakekeeper | `pgedge-lakekeeper` | `pgedge-lakekeeper` | For PostgreSQL 18 on RHEL, Rocky Linux, or AlmaLinux: @@ -77,8 +78,9 @@ sudo apt update sudo apt install -y pgedge-postgresql-18-coldfront ``` -Install the individual packages from the table instead of the meta-package -only if you need a different combination than it provides. +The extension package does not install the command-line tools or Lakekeeper. +Install `pgedge-coldfront` and `pgedge-lakekeeper` separately on the hosts +that need them. A package installation does not configure PostgreSQL. Follow the [One-Time Setup](usage.md#one-time-setup) section of the Using ColdFront guide From 7a0fd8f919343cf7739621cdee1989c862d75952 Mon Sep 17 00:00:00 2001 From: Jimmy Angelakos Date: Wed, 7 Oct 2026 15:53:42 +0100 Subject: [PATCH 05/11] docs: link the Enterprise Postgres pages for configuring the package repository --- docs/installation.md | 52 ++++++++------------------------------------ 1 file changed, 9 insertions(+), 43 deletions(-) diff --git a/docs/installation.md b/docs/installation.md index d1ad61c..f25e726 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -5,51 +5,17 @@ building it from source. ## Using a Package to Install ColdFront -Before using a package to install ColdFront, you'll need to add the pgEdge -package repository to your system. +ColdFront's packages come from the pgEdge package repository. Before you +install them, configure the repository by following the page for your platform +in the Enterprise Postgres documentation: -For RHEL, Rocky Linux, or AlmaLinux 10, first install EPEL: +- For RHEL, Oracle Linux, AlmaLinux, and Rocky Linux, see + [Configuring the Repository on Enterprise Linux](https://docs.pgedge.com/enterprise/el/configure-repo/). +- For Debian and Ubuntu, see + [Configuring the Repository on Debian and Ubuntu](https://docs.pgedge.com/enterprise/debian/configure-repo/). -```bash -sudo dnf -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-10.noarch.rpm -``` - -On RHEL 10, also enable the CodeReady Builder repository: - -```bash -sudo subscription-manager repos --enable codeready-builder-for-rhel-10-$(arch)-rpms -``` - -On Rocky Linux 10 or AlmaLinux 10, enable the equivalent CRB repository instead: - -```bash -sudo dnf config-manager --set-enabled crb -``` - -Version-specific commands for RHEL, OEL, Alma, and Rocky are available in the -[pgEdge Enterprise Postgres documentation](https://docs.pgedge.com/enterprise/el/configure-repo/). -On every EL variant, disable any PGDG repository so it does not shadow -pgEdge's packages, then install the pgEdge repository itself: - -```bash -sudo dnf config-manager --set-disabled 'pgdg*' -sudo dnf install -y https://dnf.pgedge.com/reporpm/pgedge-release-latest.noarch.rpm -``` - -For Ubuntu or Debian, first install the repository prerequisites: - -```bash -sudo apt-get update -sudo apt-get install -y curl gnupg2 lsb-release -``` - -Then add the repository: - -```bash -sudo curl -sSL https://apt.pgedge.com/repodeb/pgedge-release_latest_all.deb -o /tmp/pgedge-release.deb && \ - sudo dpkg -i /tmp/pgedge-release.deb && \ - sudo rm -f /tmp/pgedge-release.deb -``` +Those pages cover the platform prerequisites, disabling any PGDG repository, +and creating the pgEdge repository. With the repository in place, install the ColdFront extension package for your PostgreSQL major version. It depends on the pgEdge PostgreSQL server, From c0f83652ba9d17889686706c51659001e0154cf6 Mon Sep 17 00:00:00 2001 From: Jimmy Angelakos Date: Wed, 7 Oct 2026 15:53:42 +0100 Subject: [PATCH 06/11] docs: list the postgresql.conf settings a package install needs --- docs/installation.md | 39 ++++++++++++++++++++++++++++++++++++--- docs/usage.md | 8 +++++--- 2 files changed, 41 insertions(+), 6 deletions(-) diff --git a/docs/installation.md b/docs/installation.md index f25e726..b01106d 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -48,9 +48,42 @@ The extension package does not install the command-line tools or Lakekeeper. Install `pgedge-coldfront` and `pgedge-lakekeeper` separately on the hosts that need them. -A package installation does not configure PostgreSQL. Follow the -[One-Time Setup](usage.md#one-time-setup) section of the Using ColdFront guide -to configure PostgreSQL and create the extensions. +### Configuring PostgreSQL + +A package installation does not configure PostgreSQL. Add the following +settings to `postgresql.conf`, then restart PostgreSQL: + +```ini +shared_preload_libraries = 'pg_duckdb,coldfront' +duckdb.extension_directory = '/usr/lib/pgedge/coldfront/duckdb-extensions' +duckdb.allow_unsigned_extensions = true +duckdb.autoinstall_known_extensions = false +coldfront.iceberg_async_parquet = on +coldfront.iceberg_bakery_patch = on +coldfront.warehouse = '' +coldfront.lakekeeper_endpoint = 'http://:8181/catalog' +coldfront.local_pg_dsn = 'host=/var/run/postgresql dbname= user=' +``` + +The three `duckdb.*` settings make pg_duckdb load ColdFront's patched DuckDB +extensions from the directory where `pgedge-coldfront-duckdb-extensions` +installs them. Without these settings, pg_duckdb downloads the unpatched +upstream extensions, and concurrent cold writes can then fail with HTTP 409. +The patched extensions are unsigned, so `duckdb.allow_unsigned_extensions` must +be on. Keeping `duckdb.autoinstall_known_extensions` off stops DuckDB from +downloading an unpatched upstream copy when an extension file is missing. + +The two `coldfront.iceberg_*` settings take effect only on a Spock mesh, where +a node then uploads Parquet files outside the bakery claim and serializes only +the catalog commit. The packaged duckdb-iceberg includes the patch that this +ordering requires, and the +[Distributed Setup](usage.md#distributed-setup-3-node-mesh-decoupled-mode) +section of the Using ColdFront guide describes the mesh settings. + +The [One-Time Setup](usage.md#one-time-setup) section of the Using ColdFront +guide describes the `coldfront.*` settings. Follow that section from the +Lakekeeper bootstrap onward to create the warehouse, the extensions, and the +cold-store credential. ## Building ColdFront from Source diff --git a/docs/usage.md b/docs/usage.md index 33606b9..41459a9 100644 --- a/docs/usage.md +++ b/docs/usage.md @@ -95,9 +95,11 @@ endpoint such as GCS's needs `p_use_ssl => true`, and AWS S3 omits the endpoint and sets `p_region` to the bucket's Region. The image writes the server settings ColdFront needs into `postgresql.conf` -when it initializes a new data directory. A server built another way sets them -itself, as [installation.md](installation.md#building-coldfront-on-bare-metal) -shows: +when it initializes a new data directory. A server installed from packages or +built another way sets them itself, as the +[package](installation.md#configuring-postgresql) and +[bare-metal](installation.md#building-coldfront-on-bare-metal) sections of +installation.md show: - `shared_preload_libraries = 'pg_duckdb,coldfront'` loads both extensions at server start. coldfront refuses to load any other way: `CREATE EXTENSION From 408fe044260260cc55a39e6e23298c2860f41304 Mon Sep 17 00:00:00 2001 From: Jimmy Angelakos Date: Wed, 7 Oct 2026 15:53:42 +0100 Subject: [PATCH 07/11] docs: document setting up the packaged Lakekeeper service --- docs/installation.md | 44 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 44 insertions(+) diff --git a/docs/installation.md b/docs/installation.md index b01106d..735a4bd 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -85,6 +85,50 @@ guide describes the `coldfront.*` settings. Follow that section from the Lakekeeper bootstrap onward to create the warehouse, the extensions, and the cold-store credential. +### Setting Up Lakekeeper + +The `pgedge-lakekeeper` package installs the `lakekeeper` binary and a +`lakekeeper` systemd service, but it does not enable or start the service. +Lakekeeper stores its catalog in a PostgreSQL 15 or later database and reads +its settings from `/etc/lakekeeper/lakekeeper.env`. The following steps +prepare and start the service: + +1. Create a role and a database for the catalog: + + ```sql + CREATE ROLE lakekeeper LOGIN PASSWORD 'change-me'; + CREATE DATABASE lakekeeper OWNER lakekeeper; + ``` + + The migration in step 3 creates the `uuid-ossp`, `pgcrypto`, `pg_trgm`, + `btree_gin`, and `btree_gist` extensions, so either the role must be + allowed to run `CREATE EXTENSION` or a superuser must create them first. + +2. In `/etc/lakekeeper/lakekeeper.env`, set + `LAKEKEEPER__PG_DATABASE_URL_WRITE` to the database's connection string and + `LAKEKEEPER__PG_ENCRYPTION_KEY` to a random secret, such as the output of + `openssl rand -base64 32`. Lakekeeper encrypts stored credentials with the + key, so keep the key stable and backed up. Every node that shares the + catalog needs the same key. + +3. Run the one-time database migration as the `lakekeeper` user: + + ```bash + set -a; . /etc/lakekeeper/lakekeeper.env; set +a + sudo -E -u lakekeeper /usr/bin/lakekeeper migrate + ``` + +4. Enable and start the service: + + ```bash + sudo systemctl enable --now lakekeeper + ``` + +Lakekeeper listens on port 8181 on every address by default. Without an +authorization backend in `lakekeeper.env`, the catalog accepts every request, +so configure authentication before you expose the service beyond a trusted +network. + ## Building ColdFront from Source This section walks you through building ColdFront from source, either in From 39732842bd9b1edabf0434ec952f52b720582683 Mon Sep 17 00:00:00 2001 From: Jimmy Angelakos Date: Wed, 7 Oct 2026 15:53:42 +0100 Subject: [PATCH 08/11] docs: state that ColdFront keeps its configuration in the database --- README.md | 2 ++ docs/installation.md | 28 ++++++++++++++++++++++++++++ 2 files changed, 30 insertions(+) diff --git a/README.md b/README.md index 717edb1..388f33f 100644 --- a/README.md +++ b/README.md @@ -119,6 +119,8 @@ else from the server: each table's lifecycle in `coldfront.partition_config`, the cold-store credential in `coldfront.storage_secret` and the catalog settings above. The `import` command takes a deployment YAML, modeled on [config.example.yaml](config.example.yaml), and writes it into the server once. +ColdFront has no configuration file. A YAML passed to any later run is checked +against the server, and the run takes only `postgres.dsn` from the file. For every setting, see the [One-Time Setup](docs/usage.md#one-time-setup) and [Tuning Knobs](docs/usage.md#tuning-knobs) sections of the Using ColdFront guide. diff --git a/docs/installation.md b/docs/installation.md index 735a4bd..371ccf4 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -129,6 +129,34 @@ authorization backend in `lakekeeper.env`, the catalog accepts every request, so configure authentication before you expose the service beyond a trusted network. +### Configuring ColdFront + +ColdFront has no configuration file, because the database holds its +configuration. The server settings are the `postgresql.conf` lines above, each +managed table is a row in `coldfront.partition_config`, and the cold-store +credential is in `coldfront.storage_secret`. The archiver, partitioner, and +compactor connect the way psql does, from the libpq environment (`PGHOST`, +`PGDATABASE`, `PGUSER`, `PGPASSWORD`, `PGSERVICE`) or `--dsn`, and read every +other setting from the server. + +You write that configuration with `coldfront.set_storage_secret()` and the +`register` command, or in one step by importing a deployment YAML: + +```bash +archiver import --config deploy.yaml +``` + +The `pgedge-coldfront` package installs an example deployment YAML at +`/etc/pgedge/coldfront/config.yaml`. That file is only an example to edit and +pass to `import`, and no ColdFront tool reads it unless `--config` names it. +After an import, the server holds the configuration. A later run that is given +a YAML checks the file against the server and refuses to run if any value +differs. The only value such a run takes from the file is `postgres.dsn`, which +connects when `--dsn` is unset. The +[Managing Partitioned Tables (CLI)](usage.md#managing-partitioned-tables-cli) +section of the Using ColdFront guide describes `register`, `import`, and +`export`. + ## Building ColdFront from Source This section walks you through building ColdFront from source, either in From fb30d0596b0bb94b60bbe6ccbfa0a8049ab378ec Mon Sep 17 00:00:00 2001 From: Jimmy Angelakos Date: Wed, 7 Oct 2026 16:22:41 +0100 Subject: [PATCH 09/11] docs: require Lakekeeper authorization as well as authentication before exposing it --- docs/installation.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/installation.md b/docs/installation.md index 371ccf4..f8c6167 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -126,8 +126,8 @@ prepare and start the service: Lakekeeper listens on port 8181 on every address by default. Without an authorization backend in `lakekeeper.env`, the catalog accepts every request, -so configure authentication before you expose the service beyond a trusted -network. +so configure authentication and authorization before you expose the service +beyond a trusted network. ### Configuring ColdFront From 1d1a1927cf7546306f4e290305ba7dc9dc06ba65 Mon Sep 17 00:00:00 2001 From: Susan Douglas Date: Wed, 7 Oct 2026 11:30:48 -0400 Subject: [PATCH 10/11] docs: rename Installing ColdFront to Installing and Configuring ColdFront --- docs/installation.md | 6 +++--- mkdocs.yml | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/installation.md b/docs/installation.md index f8c6167..72d721b 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -1,7 +1,7 @@ -# Installing ColdFront +# Installing and Configuring ColdFront -This guide covers installing ColdFront from the published package and -building it from source. +This guide covers installing ColdFront from the published package or +building it from source, and configuring it once installed. ## Using a Package to Install ColdFront diff --git a/mkdocs.yml b/mkdocs.yml index 305fef2..162a029 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -57,7 +57,7 @@ repo_url: https://github.com/pgEdge/ColdFront nav: - pgEdge ColdFront: index.md - - Installing ColdFront: installation.md + - Installing and Configuring ColdFront: installation.md - Getting Started: - Getting Started with ColdFront: walkthrough.md - Exploring Tiered Storage: walkthrough_tiered.md From 148bae854b953d6de037b47e5a1b8e728563e34a Mon Sep 17 00:00:00 2001 From: Susan Douglas Date: Thu, 8 Oct 2026 07:16:20 -0400 Subject: [PATCH 11/11] docs: nudge PR sync with an empty commit Co-Authored-By: Claude Sonnet 5