feat(sites): add PostgreSQL as a selectable database engine by mihir-kandoi · Pull Request #119 · frappe/pilot

mihir-kandoi · 2026-06-26T04:52:39Z

Overview

Frappe and ERPNext support PostgreSQL, but bench could only create MariaDB sites. This adds PostgreSQL as a selectable database engine end to end — installation, provisioning, site creation, restore/reinstall, and the admin UI — while keeping MariaDB the default.

The engine is a per-bench choice (every site on a bench uses it), selected at bench new or in the setup wizard — matching how a bench already owns one database. A PostgreSQL bench can run a dedicated per-bench cluster (its own port, systemd Linux) or share the system server.

What changed

Config

bench.db_type (mariadb | postgres) selects the engine for the whole bench. New [postgres] section (PostgresConfig: host, port, root_password, admin_user, version, instance) parsed/serialized in bench.toml and validated.

Provisioning

PostgresManager mirrors MariaDBManager. bench init installs PostgreSQL (apt / Homebrew / apk) and provisions it: ensures the superuser role exists and sets its password.
Dedicated clusters (systemd Linux): a postgres bench gets its own cluster via pg_createcluster on its own (collision-checked) port; bench drop tears it down with pg_dropcluster. Alpine/macOS use the shared system server. The cluster version is detected from the installed server, not hardcoded.
frappe imports mysqlclient in its __init__.py for every engine, so the MariaDB client headers are always installed; libpq headers are added only for postgres benches.

Site lifecycle (engine-aware)

Site.create() passes --db-type postgres + the bench's postgres root connection. restore / reinstall / drop-site use the right root credentials per the bench's engine (frappe reads the engine from site_config.json, so no --db-type flag is needed there).

Admin UI

Setup wizard picks the engine at bench creation; PostgreSQL offers a dedicated/shared choice (systemd Linux). The engine is shown read-only in Settings → Bench; PostgreSQL connection fields are read-only (edit via the config file). The New Bench dialog no longer asks for the engine — the new bench's own wizard does.

Tooling

bench build-admin fails with a clear message on Node < 20.11 instead of an opaque unplugin stack trace.

Review feedback addressed

@tanmoysrt

Engine is per-bench, not per-site; the New Bench dialog no longer takes the engine/details (the bench's setup wizard does).
PostgreSQL now offers a dedicated/shared choice like MariaDB.
Removed the explanatory line under the engine selector; moved the engine from a sidebar logo to a read-only field in Settings; made the PostgreSQL settings fields read-only.
Kept the MariaDB system dependencies unconditional — fixes the mysqlclient==2.2.8 build failure on postgres benches.

Greptile

Versioned Alpine PostgreSQL package names (server, client, and -dev headers).
Reject a blank PostgreSQL superuser password in the setup wizard (it would otherwise only fail at first site creation).

Tests

Unit coverage across config parse/roundtrip/validation, Site argument building, PostgresManager (shared + dedicated install/secure/provision/remove), the setup-wizard validation + port assignment, engine-aware bench drop, SiteReader.db_type, and the admin endpoints.
e2e: a new postgres-dedicated CI variant runs the full create → wizard → site → drop lifecycle (alongside postgres shared and the MariaDB variants).
Full unit suite: 583 passing.

Reviewer notes

Dedicated PostgreSQL is not yet exercised on a real machine — implemented, unit-tested, and code-traced; the new postgres-dedicated e2e variant is its first real validation on a clean Ubuntu runner.
Built admin assets (admin/backend/static/dist/) are gitignored; rebuild the frontend (Node ≥ 20.11) or download prebuilt.

🤖 Generated with Claude Code

mihir-kandoi · 2026-06-26T04:54:41Z

Can someone pull and test this? Works on my machine

tanmoysrt · 2026-06-26T05:06:01Z

 # data_dir = "/var/lib/mysql-my-bench"              # per-instance datadir (auto-derived; used as bind-mount target with ZFS)

+# ── PostgreSQL (opt-in per-site engine; installed & provisioned by init) ──────
+# Shared across benches (one server, port 5432); `bench new` generates the password.


DB can be dedicated per bench also.
Not sure, if we should give choice of db per site.

It will be simple to keep it at bench level only. We can ask them the choice in bench's setup wizard.

The current bench supports postgres and mariadb on the same bench so I just copied that. Shall I make it per bench?

in first version, we can make it per bench, as bench is responsible for managing the db also.

For SQLite it's fine to give the choices, as bench doesn't need to take care of the db much.

We have only one DB support per-bench, either dedicated or shared.

@Aradhya-Tripathi is this a new bench thing? shifting to per bench anyways. will update.

There is an option to users whether they want shared or dedicated per bench db in bench's setup wizard.
Create a new bench and check the setup wizard. That has been updated in last few days.

Aradhya-Tripathi · 2026-06-26T05:12:21Z

Can someone pull and test this? Works on my machine

Can you add playwright with this integration as well, we recently added for e2e setup.

mihir-kandoi · 2026-06-26T07:23:09Z

@Aradhya-Tripathi @tanmoysrt done. let me know if you need me to test anything. Works fine in dev mode. Will rebase on your green light (did it twice, kept getting new conflicts on each push 😢 )

tanmoysrt · 2026-06-26T07:29:16Z

@mihir-kandoi the repo renamed, so you have to modify the code ig.

mihir-kandoi · 2026-06-26T07:30:23Z

@tanmoysrt will do, anything else pending? code look right?

Frappe/ERPNext support PostgreSQL, but bench could only create MariaDB sites. This adds PostgreSQL end-to-end while keeping MariaDB the default. What's included: - New [postgres] config section (PostgresConfig) parsed/serialized in bench.toml and editable on the admin Settings page (write-only password). - PostgresManager: bench init installs and provisions a shared PostgreSQL server (apt/Homebrew/apk), starts/enables the service, and sets the superuser password. bench new generates the password. - Engine choice (db_type) plumbed through the admin Create Site dialog, bench new-site, the task runner, and Site.create(). Empty Postgres password falls back to a placeholder so frappe never hangs on getpass. - Restore/reinstall are engine-aware: restore-from-backup, create-from- upload, and reinstall use the right root credentials per engine (engine inferred from the source site / existing site, or chosen for uploads). - Per-site engine logos (MariaDB/PostgreSQL) in the sites list + detail, with db_type exposed by SiteReader. - build-admin now fails with a clear message on Node < 20.11 instead of an opaque toolchain stack trace. MariaDB remains the default engine; PostgreSQL is opt-in per site. Tests added across config, core, tasks, managers, and admin endpoints. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

A bench now runs exactly one engine, selected at creation via `bench new --database mariadb|postgres` (default mariadb) and stored as bench.db_type. Previously every bench co-provisioned a shared PostgreSQL server alongside MariaDB and the engine was chosen per-site. - init installs/provisions only the bench's engine and its client/build headers (libmariadb-dev/mariadb-dev vs libpq-dev/postgresql-dev); a dedicated MariaDB instance is only created for MariaDB benches. - Drop the per-site db_type: Site reads the engine from the bench config, so new-site/reinstall/restore no longer thread it through. - Admin: wizard picks the engine at bench creation, adds a /validate-postgres endpoint, and surfaces db_type in settings. - Add the postgres DB lifecycle e2e spec and update unit tests.

DropSiteCommand only ever passed MariaDB credentials, so dropping a site on a PostgreSQL bench connected as `postgres` with no password and failed under password auth — it passed on trust-auth setups (e.g. Homebrew) which masked it locally and only surfaced in CI. Move the engine-keyed root args onto Bench.db_root_args() so drop/restore/reinstall share one source of truth. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

The engine is a bench-wide property, so drop the per-site engine badges in the site list and detail views and surface it once in the sidebar footer, fed by a new db_type field on /api/status. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

In a source checkout, `bench start` served whatever was already in dist, so local admin UI edits never appeared without a manual `build-admin --force`. Rebuild from source when the frontend is newer than the built bundle; best-effort so a build failure (e.g. old Node) still serves the existing dist. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

The bench_cli -> pilot rename moved the package dir and pyproject, but feature code added afterward still imported the old bench_cli package, breaking `pip install -e .` and the unit-test CI run. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

GitHub runners preconfigure packages.microsoft.com / azure-cli apt repos that intermittently 403, failing `apt-get update` for the whole step even though none of the installed packages come from them. Remove them first. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

tanmoysrt · 2026-06-26T16:24:23Z

There are few issues on the UI -

It added a unnecessary explanation line under db selector
For postgres, not showing up option to opt for dedicated / shared instance like mariadb one.

During postgres setup, it's throwing this error

× Failed to build `mysqlclient==2.2.8`
├─▶ The build backend returned an error
╰─▶ Call to `setuptools.build_meta.build_wheel` failed (exit status: 1)

[stdout]
Trying pkg-config --exists mysqlclient
Command 'pkg-config --exists mysqlclient' returned non-zero exit status 1.

Trying pkg-config --exists mariadb
Command 'pkg-config --exists mariadb' returned non-zero exit status 1.

Trying pkg-config --exists libmariadb
Command 'pkg-config --exists libmariadb' returned non-zero exit status 1.

Trying pkg-config --exists perconaserverclient
Command 'pkg-config --exists perconaserverclient' returned non-zero exit status 1.

[stderr]
Exception: Can not find valid pkg-config name.

Specify MYSQLCLIENT_CFLAGS and MYSQLCLIENT_LDFLAGS env vars manually.

Regardless of the db, we need to keep the system dependencies, as frappe import mysqlclient in its init.py

We don't need to take database type and details in Bench Manager as new bench's setup wizard takes care of it.
The db logo in sidebar is weird. We can show the db type in the Settings Modal instead.

7. In settings modal under postgres section, the fields are editable. Ideally those need to be read-only and can be modified from config file directly.

Codwise looks fine. Maybe need to reduce the verbosity , explanations little bit.

Create a blank server and try setting up the bench there to know if it's raising any issue during first setup.

Also, it will be nice to add Integration test for postgres also to ensure this flow keeps working. As of now, most integration test use MariaDB only.

tanmoysrt · 2026-06-26T16:32:40Z

@greptile review

greptile-apps · 2026-06-26T16:35:32Z

Confidence Score: 4/5

PostgreSQL init is broken on Alpine until the package names are corrected.

Alpine PostgreSQL benches can fail during package installation.
A blank PostgreSQL password can let setup finish while the first site creation fails on password-auth servers.
The rest of the changed engine plumbing looks consistent with the bench-level database model.

pilot/managers/postgres_manager.py, pilot/commands/init.py, admin/backend/views/setup.py

Important Files Changed

Filename	Overview
pilot/managers/postgres_manager.py	Adds PostgreSQL install, service control, provisioning, and credential checks; Alpine server package naming needs a fix.
pilot/commands/init.py	Switches init to install and provision only the selected database engine; Alpine PostgreSQL build package naming needs a fix.
admin/backend/views/setup.py	Adds PostgreSQL setup validation and relaxes password requirements for PostgreSQL benches.
pilot/core/site.py	Builds database-specific arguments for site creation, restore, and reinstall.

Prompt To Fix All With AI

Fix the following 3 code review issues. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 3
pilot/managers/postgres_manager.py:47
**Alpine Server Package Missing**

When a PostgreSQL bench is initialized on Alpine, this passes the unversioned `postgresql` package name to `apk`. Alpine PostgreSQL packages are versioned, so the install step fails before provisioning can start.

### Issue 2 of 3
pilot/commands/init.py:202
**Alpine Header Package Missing**

The PostgreSQL dev package is versioned on Alpine too, so `apk add postgresql-dev` fails for PostgreSQL benches. That aborts init before the Frappe environment can install its PostgreSQL client dependencies.

### Issue 3 of 3
admin/backend/views/setup.py:190
**Blank Password Reaches Site Creation**

The setup API allows PostgreSQL benches to save an empty superuser password. On a password-auth PostgreSQL server, init skips setting the role password and later `new-site` sends the `trust_auth` placeholder, so the wizard can complete setup but fail on the first site creation.

_{Reviews (1): Last reviewed commit: "ci(e2e): drop flaky Microsoft apt repos ..." | Re-trigger Greptile}

greptile-apps · 2026-06-26T16:35:36Z

+        if is_macos():
+            get_package_manager().install(self._brew_package())
+            return
+        get_package_manager().install("postgresql", "postgresql-client")


Alpine Server Package Missing

When a PostgreSQL bench is initialized on Alpine, this passes the unversioned postgresql package name to apk. Alpine PostgreSQL packages are versioned, so the install step fails before provisioning can start.

Prompt To Fix With AI

This is a comment left during a code review. Path: pilot/managers/postgres_manager.py Line: 47 Comment: **Alpine Server Package Missing** When a PostgreSQL bench is initialized on Alpine, this passes the unversioned `postgresql` package name to `apk`. Alpine PostgreSQL packages are versioned, so the install step fails before provisioning can start. How can I resolve this? If you propose a fix, please make it concise.

Fixed in 171cdd2. Verified against the Alpine package index — there's no unversioned postgresql/postgresql-client, only versioned postgresql<N> (e.g. postgresql17). install() now resolves the major (the configured postgres.version, else the newest apk offers) and installs postgresql<N> + postgresql<N>-client.

greptile-apps · 2026-06-26T16:35:37Z

+    # Engine-specific client/build headers — only the bench's engine is installed.
+    _DB_BUILD_PACKAGES = {
+        "mariadb": {"debian": "libmariadb-dev", "alpine": "mariadb-dev"},
+        "postgres": {"debian": "libpq-dev", "alpine": "postgresql-dev"},


Alpine Header Package Missing

The PostgreSQL dev package is versioned on Alpine too, so apk add postgresql-dev fails for PostgreSQL benches. That aborts init before the Frappe environment can install its PostgreSQL client dependencies.

Prompt To Fix With AI

This is a comment left during a code review. Path: pilot/commands/init.py Line: 202 Comment: **Alpine Header Package Missing** The PostgreSQL dev package is versioned on Alpine too, so `apk add postgresql-dev` fails for PostgreSQL benches. That aborts init before the Frappe environment can install its PostgreSQL client dependencies. How can I resolve this? If you propose a fix, please make it concise.

Fixed in 171cdd2. The dev headers are versioned on Alpine too, so init now installs postgresql<N>-dev via PostgresManager.alpine_dev_package(), and only for postgres benches. (Related: the MariaDB client headers are now installed for every bench, since frappe imports mysqlclient in its __init__.py regardless of engine.)

greptile-apps · 2026-06-26T16:35:38Z

+    # A MariaDB bench needs a root password; PostgreSQL benches don't (init sets
+    # the superuser password, falling back to a placeholder for trust/peer auth).
+    if data.get("db_type", "mariadb") == "mariadb" and not data.get("mariadb_password"):
+        return "mariadb_password is required"


Blank Password Reaches Site Creation

The setup API allows PostgreSQL benches to save an empty superuser password. On a password-auth PostgreSQL server, init skips setting the role password and later new-site sends the trust_auth placeholder, so the wizard can complete setup but fail on the first site creation.

Prompt To Fix With AI

This is a comment left during a code review. Path: admin/backend/views/setup.py Line: 190 Comment: **Blank Password Reaches Site Creation** The setup API allows PostgreSQL benches to save an empty superuser password. On a password-auth PostgreSQL server, init skips setting the role password and later `new-site` sends the `trust_auth` placeholder, so the wizard can complete setup but fail on the first site creation. How can I resolve this? If you propose a fix, please make it concise.

Fixed in 171cdd2. _validate now rejects a blank PostgreSQL superuser password (and the wizard's Database step does too), so a password-less postgres bench can no longer reach site creation. On a fresh/dedicated server, init sets this password during provisioning.

mihir-kandoi · 2026-06-26T16:45:50Z

@tanmoysrt the UI was mostly me not Claude, I really do have shit UI taste 🤣

Will update the PR in a while

Add dedicated PostgreSQL clusters and apply the review feedback from Tanmoy and Greptile. Dedicated Postgres: - A postgres bench can now run its own cluster on its own port (pg_createcluster/pg_dropcluster/pg_ctlcluster) on systemd Linux, or share the system server (Alpine/macOS always shared). Plumbed through postgres config, `bench new`, the setup wizard, init, and `bench drop` (engine-aware teardown). The cluster version is detected, not hardcoded. UI / review fixes: - Wizard: PostgreSQL gets a dedicated/shared selector (systemd Linux); dropped the explanatory line under the engine selector. - Bench Manager (New Bench dialog) no longer asks for the engine — the new bench's own setup wizard does. - Show the engine in Settings (read-only) instead of a sidebar logo, and make the PostgreSQL settings fields read-only. Provisioning fixes: - Always install the MariaDB client headers (frappe imports mysqlclient in its __init__.py for every engine); add libpq only for postgres. - Greptile: use versioned Alpine postgres package names; reject a blank postgres superuser password in the setup wizard. Tests: - Dedicated-cluster unit coverage (manager, config, new, drop, setup) and a `postgres-dedicated` e2e CI variant. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

pg_createcluster --start starts the cluster via `systemctl start postgresql@<ver>-<cluster>`, whose templated unit fails with "Assertion failed on job" on the GitHub Actions runner (and in containers). Create the cluster, then start it directly with `pg_ctlcluster --skip-systemctl-redirect`. Boot autostart still works via the enabled postgresql meta-service (the cluster's start.conf stays auto). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

mihir-kandoi · 2026-06-27T06:02:01Z

@tanmoysrt @Aradhya-Tripathi done.

I couldnt check the new bench manager UI as production is Linux only. Please test on my behalf...

tanmoysrt · 2026-06-27T06:19:59Z

@mihir-kandoi ping me if you don't have hetzner or digitalocean access.
You can create a server and run the bench setup there and drop the server thereafter. It's better to test on a blank system to caught first time setup issues.

mihir-kandoi · 2026-06-27T08:08:01Z

@tanmoysrt works on my VPS

mihir-kandoi force-pushed the feat/postgres-database-engine branch from df15212 to 2223146 Compare June 26, 2026 04:56

mihir-kandoi requested a review from Aradhya-Tripathi June 26, 2026 05:00

tanmoysrt reviewed Jun 26, 2026

View reviewed changes

mihir-kandoi marked this pull request as draft June 26, 2026 05:53

mihir-kandoi force-pushed the feat/postgres-database-engine branch from d526055 to f636739 Compare June 26, 2026 07:17

mihir-kandoi and others added 5 commits June 26, 2026 13:08

mihir-kandoi force-pushed the feat/postgres-database-engine branch from f636739 to ffa4e6b Compare June 26, 2026 07:43

mihir-kandoi and others added 2 commits June 26, 2026 13:19

mihir-kandoi marked this pull request as ready for review June 26, 2026 10:19

greptile-apps Bot reviewed Jun 26, 2026

View reviewed changes

mihir-kandoi and others added 2 commits June 27, 2026 11:08

Uh oh!

Conversation

mihir-kandoi commented Jun 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

What changed

Review feedback addressed

Tests

Reviewer notes

Uh oh!

mihir-kandoi commented Jun 26, 2026

Uh oh!

tanmoysrt Jun 26, 2026

Choose a reason for hiding this comment

Uh oh!

mihir-kandoi Jun 26, 2026

Choose a reason for hiding this comment

Uh oh!

tanmoysrt Jun 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Aradhya-Tripathi Jun 26, 2026

Choose a reason for hiding this comment

Uh oh!

mihir-kandoi Jun 26, 2026

Choose a reason for hiding this comment

Uh oh!

tanmoysrt Jun 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Aradhya-Tripathi commented Jun 26, 2026

Uh oh!

mihir-kandoi commented Jun 26, 2026

Uh oh!

tanmoysrt commented Jun 26, 2026

Uh oh!

mihir-kandoi commented Jun 26, 2026

Uh oh!

tanmoysrt commented Jun 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tanmoysrt commented Jun 26, 2026

Uh oh!

greptile-apps Bot commented Jun 26, 2026

Confidence Score: 4/5

Important Files Changed

Uh oh!

greptile-apps Bot Jun 26, 2026

Choose a reason for hiding this comment

Uh oh!

mihir-kandoi Jun 27, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Jun 26, 2026

Choose a reason for hiding this comment

Uh oh!

mihir-kandoi Jun 27, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Jun 26, 2026

Choose a reason for hiding this comment

Uh oh!

mihir-kandoi Jun 27, 2026

Choose a reason for hiding this comment

Uh oh!

mihir-kandoi commented Jun 26, 2026

Uh oh!

mihir-kandoi commented Jun 27, 2026

Uh oh!

tanmoysrt commented Jun 27, 2026

Uh oh!

mihir-kandoi commented Jun 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

mihir-kandoi commented Jun 26, 2026 •

edited

Loading

tanmoysrt Jun 26, 2026 •

edited

Loading

tanmoysrt Jun 26, 2026 •

edited

Loading

tanmoysrt commented Jun 26, 2026 •

edited

Loading