feat(pre1): four-VM deployment and clean lifecycle tooling

.github/workflows/fast.yml

@@ -160,6 +160,9 @@ jobs:
       - name: Check MVP release evidence contract
         run: python3 scripts/ci/test_mvp_ci.py
 
+      - name: Check deployment preflight safety contract
+        run: python3 -B -m unittest discover -s scripts/deploy/pre1/tests -v
+
       - name: Lint commit message
         if: github.event_name == 'pull_request'
         run: |

PGRAC_VERSION

@@ -1 +1 @@
-0.131.0
+0.132.0-pre1.1

README.md

@@ -75,6 +75,7 @@ User-facing manual:
 | Topic | File |
 |---|---|
 | Installation | [docs/user-guide/install.md](docs/user-guide/install.md) |
+| Four-VM GFS2 lab milestone (separate qualification) | [Operator-assisted installation](docs/deployment/pre1-quickstart.md), [scope and release requirements](docs/release-notes/v0.132.0-pre1.1.md) |
 | Bootstrap a node | [docs/user-guide/bootstrap.md](docs/user-guide/bootstrap.md) |
 | Configuration (`cluster.*` GUCs + `pgrac.conf`) | [docs/user-guide/configuration.md](docs/user-guide/configuration.md) |
 | System views reference | [docs/reference/system-views.md](docs/reference/system-views.md) |

docs/deployment/pre1-cold-snapshots.md

@@ -0,0 +1,116 @@
+# PRE1: complete cold snapshots
+
+Author: SqlRush <sqlrush@gmail.com>
+
+This tool preserves a **normally stopped** four-node dataset. It does not
+implement crash recovery, hot backup, PITR, or automatic restoration into an
+installed cluster. The restore command creates an independent offline copy;
+it never writes a voting device or overwrites an existing PGDATA.
+
+## Prerequisites
+
+- All four instances have passed the native clean-stop checks: clean controls,
+  exact process absence, this shutdown's protocol closure, and cleared ALIVE
+  slots with no remaining required debt.
+- `clean_restart.py prepare` has produced an immutable
+  `CLEAN_RESTART_PREPARED` JSON artifact from that closure. Keep the request,
+  native logs, observations and their hashes. Do not invent a PASS document.
+- The same deployment-tool tree is present on the controller and each guest.
+- No operator, service manager or automation may start an instance during
+  copying. Leave voting, GFS2, DLM and the underlying storage available.
+- Destination parents already exist, are trusted and are outside the protected
+  data/install/shared roots. Destination directories themselves must not exist.
+- Budget space for all four private PGDATAs, one shared-data tree, and three
+  full voting images. Files containing credentials remain private.
+
+## Create four pieces
+
+Transfer the prepared artifact to every guest without modifying it. On each
+guest, create an input JSON using its actual node ID and that file's SHA-256:
+
+```json
+{
+  "prepared": {
+    "path": "/srv/pgrac/evidence/restart-prepared.json",
+    "sha256": "REPLACE_WITH_ACTUAL_SHA256"
+  },
+  "node_id": 0
+}
+```
+
+Run locally as the designated administrative operator. Use a new destination
+for each attempt; never delete a partial result to reuse its name.
+
+```sh
+sudo python3 scripts/deploy/pre1/snapshot.py create-piece \
+  --request snapshot-node0.json --out /srv/pgrac/snapshots/run001-node0
+```
+
+Repeat on nodes 1, 2 and 3 with their own input and destination. Each piece
+contains that member's complete PGDATA, including its own WAL and control.
+Node 0's piece additionally contains the shared-data tree and all three raw
+voting images. Member pieces alone are **not** a complete snapshot.
+
+The command checks the native stopped state and identity before and after
+copying, uses exclusive no-follow destinations, and records file hashes.
+Copy failure leaves an incomplete directory, not a usable snapshot.
+
+## Seal the complete set
+
+Transfer all four piece directories intact to protected controller storage.
+Do not include newly started or independently initialized members. Prepare:
+
+```json
+{
+  "prepared": {
+    "path": "/srv/pgrac/evidence/restart-prepared.json",
+    "sha256": "REPLACE_WITH_ACTUAL_SHA256"
+  },
+  "pieces": [
+    "/srv/pgrac/cold/run001/node0",
+    "/srv/pgrac/cold/run001/node1",
+    "/srv/pgrac/cold/run001/node2",
+    "/srv/pgrac/cold/run001/node3"
+  ]
+}
+```
+
+```sh
+python3 scripts/deploy/pre1/snapshot.py seal \
+  --request snapshot-set.json --out /srv/pgrac/cold/run001/set.json
+sha256sum /srv/pgrac/cold/run001/set.json
+```
+
+Sealing verifies every copied file against its source capture and re-observes
+all four stopped instances through pinned SSH. It embeds the configuration and
+closure provenance. Missing members, changed WAL, wrong voting images, links,
+source changes or inconsistent identities are failures. Only the sealed
+`COLD_SET_VERIFIED` artifact represents a complete set.
+
+## Verify restoration to an independent directory
+
+Create a request with the sealed manifest's actual path and SHA-256:
+
+```json
+{
+  "collection": {
+    "path": "/srv/pgrac/cold/run001/set.json",
+    "sha256": "REPLACE_WITH_ACTUAL_SHA256"
+  }
+}
+```
+
+```sh
+python3 scripts/deploy/pre1/snapshot.py restore \
+  --request restore-set.json --out /srv/pgrac/restore-check/run001
+```
+
+The result is `COLD_SET_RESTORED_NOT_STARTED`: all four members, shared bytes,
+voting images and provenance have been copied and verified as one generation.
+The output is deliberately **not** connected to a running database. Startup
+permission remains false. Never copy one member's WAL/control into another
+member, restore just the shared table files, or write these images to live
+voting devices. Keep the original dataset until the independent check succeeds.
+
+Any unclean control or missing shutdown closure remains outside this workflow;
+preserve it for the separately qualified recovery procedure.

docs/deployment/pre1-prerequisites.md

@@ -0,0 +1,120 @@
+# Four-VM deployment: preliminary checks
+
+Author: SqlRush <sqlrush@gmail.com>
+
+Status: development tooling, **not a certified four-VM installation procedure**.
+The current commands validate planned inputs and collect guest identity. Storage,
+fencing, installed binary, effective configuration and seed verification are still
+separate, mandatory checks. Do not format or mount shared disks based on a profile
+validation result.
+
+## Requirements
+
+- Python 3.9 or newer and OpenSSH on the designated Linux/macOS controller.
+- Four independent KVM/libvirt guests, not four containers sharing one kernel.
+- Dedicated shared block storage for GFS2 and three distinct voting devices.
+- Separate negative-test voting devices and directories; never reuse live data.
+- Exact WWIDs and capacities, not `/dev/sdX` names or wildcard device permission.
+- Known SSH server public keys obtained through a trusted management channel.
+- A protected SSH key file per administrative endpoint; no passwords or private
+  key contents in JSON. The controller disables inherited SSH configuration and
+  agent forwarding. It never automatically accepts a new host key.
+- Non-root database UID/GID, explicit local data/install/log paths and a shared
+  mountpoint. Lexical path checks do not replace subsequent guest realpath checks.
+- A frozen source commit/tree, binary hash, build information, configuration
+  hashes, seed identity and workload/judge identity.
+
+The original deployment profile is `pre1-gfs2-v1` (RHEL 9 x86_64). The separate
+`pre1-gfs2-arm64-lab-v1` is an ARM64 laboratory candidate, not RHEL/x86 certification.
+Accepting a profile name in JSON does not certify that platform. Shared cloud disks,
+other filesystems and failure-domain HA also require their own qualification.
+
+## Prepare the profile
+
+Use [profile.schema.json](../../scripts/deploy/pre1/profile.schema.json) as the
+closed input contract. All fields are required; unknown fields are rejected.
+Fill values from the actual build, guests and storage inventory. Do not copy
+synthetic unit-test identities into a deployment manifest.
+
+Each `nodes` entry declares node ID 0–3, VM UUID, machine ID, boot ID, administrative
+SSH endpoint, pinned `ssh-ed25519` host public key, SQL/control/data addresses,
+number of data workers, local paths and database UID/GID. `admin_endpoint` contains
+`host`, `port`, `user` and `identity_file`. The last is a local private-key **path**,
+not the key itself. IPv4 addresses must be concrete and non-loopback.
+
+`votes` contains three records (`index`, `wwid`, `size` in bytes,
+`logical_sector` = 512). `authorization.device_allowlist` separately identifies
+each authorized device's WWID, byte size, purpose and fresh-media declaration.
+`fixture_inventory.MAIN` and `.NEGATIVE` have separate roots and voting WWIDs.
+Declaring `fresh: true` does not cause a write and is not proof a disk is empty.
+
+Protect manifests and raw inventory as operational information. Public reports
+must not include credentials, private keys or customer infrastructure identities.
+
+## Check inputs without contacting guests
+
+```sh
+python3 scripts/deploy/pre1/preflight.py check-profile --profile /secure/pre1.json
+```
+
+`PASS` here means **PROFILE_ONLY**: syntax, required fields and internal consistency.
+The result always includes `deployment_qualified: false`. It is not a database,
+storage, fencing or four-node test result.
+
+## Collect read-only guest identity
+
+Create a private evidence directory first, then choose a new output filename:
+
+```sh
+install -d -m 700 /secure/pre1-evidence
+python3 scripts/deploy/pre1/preflight.py inventory \
+  --profile /secure/pre1.json \
+  --out /secure/pre1-evidence/identity-001.json
+python3 scripts/deploy/pre1/preflight.py verify \
+  --profile /secure/pre1.json \
+  --inventory /secure/pre1-evidence/identity-001.json
+```
+
+The guest probe reads machine/boot/domain UUIDs and asks `systemd-detect-virt` for
+the virtualization type. Reading the DMI UUID may require passwordless permission
+for the exact read-only `sudo -n cat /sys/class/dmi/id/product_uuid` command.
+It does not install packages, mount disks, alter PostgreSQL or execute SQL writes.
+
+At this implementation stage, a successful identity collection still returns
+`BLOCKED / QUALIFICATION_PENDING` and lists checks not yet performed. In particular,
+four different boot IDs are not sufficient proof of four independent libvirt
+domains. `verify` cannot remove those pending obligations.
+
+## Results and preservation
+
+| Exit | Meaning |
+|---:|---|
+| 0 | Requested scoped check passed; inspect `scope`, not just exit status |
+| 2 | Invalid/missing inputs, identity mismatch, or qualification pending |
+| 3 | SSH, evidence, parsing or filesystem operation failed |
+
+Standard output is one JSON result. Existing evidence is never intentionally
+replaced. Each output uses a private temporary file, local controller lock, file
+sync and directory sync. A publication failure is not PASS; an artifact left by a
+directory-sync failure must be reconciled, not overwritten. Only one controller
+may operate a campaign. A local lock does not coordinate two separate hosts.
+
+The identity artifact is bound to the canonical profile hash. Changing the binary,
+configuration or other profile fields invalidates that binding. Preserve old
+artifacts and collect a new observation under a new filename.
+
+## Scope limitations
+
+The deployment work does not enable shared native catalogs/control files/WAL,
+crash takeover, online membership changes or database raw-device storage. Normal
+shutdown and same-data restart must be validated separately. A normal-restart
+result never authorizes reusing an unclean crash image as if it were clean.
+
+## Tool tests
+
+```sh
+python3 -B -m unittest discover -s scripts/deploy/pre1/tests -v
+```
+
+These tests use synthetic profiles and temporary local files. Their PASS is not
+GFS2, fencing or live database certification.