feat: add postgres-wal-growth troubleshooting lab

labs/postgres-wal-growth/cloud-init.yaml

@@ -0,0 +1,18 @@
+#cloud-config
+packages:
+  - postgresql
+  - postgresql-contrib
+
+runcmd:
+  - |
+    # Start PostgreSQL
+    systemctl start postgresql
+
+    # Create a stale physical replication slot
+    # This prevents WAL segments from being removed until the slot is used or dropped
+    sudo -u postgres psql -c "SELECT pg_create_physical_replication_slot('stale_replica');"
+
+    # Generate some 'fake' WAL traffic to fill things up (simulated)
+    # We'll just create a table and put some data in it
+    sudo -u postgres psql -c "CREATE TABLE filler AS SELECT generate_series(1, 100000) AS id;"
+    sudo -u postgres psql -c "CHECKPOINT;"

labs/postgres-wal-growth/lab.yaml

@@ -0,0 +1,25 @@
+id: postgres-wal-growth
+name: PostgreSQL WAL Growth Fills Disk
+category: database
+difficulty: advanced
+vm:
+  name: postgres-wal-lab
+  image: ubuntu-24.04-base.qcow2
+  memory: 2048
+  cpu: 2
+  disk: 10G
+cloud_init: cloud-init.yaml
+verify_script: verify.sh
+
+description:
+  summary: "The database partition is rapidly filling up due to excessive PostgreSQL WAL (Write Ahead Log) accumulation."
+  story: "Monitoring has alerted us that the database server is running out of disk space. Investigation shows that /var/lib/postgresql is being consumed by hundreds of WAL segments. You need to identify why PostgreSQL is not recycling these logs and fix the root cause."
+  objectives:
+    - Identify the growth of pg_wal directory
+    - Check for stale replication slots or long-running transactions
+    - Remove the stale replication slot to allow WAL recycling
+tags:
+  - postgresql
+  - database
+  - linux
+  - storage

labs/postgres-wal-growth/question.md

@@ -0,0 +1,12 @@
+### Scenario
+The monitoring system has alerted us that the PostgreSQL database server is running out of disk space on the data partition. After investigation, you find that the `pg_wal` (or `pg_xlog`) directory contains a large number of WAL segments that aren't being cleaned up or recycled.
+
+### Objective
+Identify why PostgreSQL is failing to recycle WAL segments and fix the issue to free up disk space.
+
+### Useful Commands
+- `df -h`
+- `sudo du -sh /var/lib/postgresql/*/main/pg_wal`
+- `sudo -u postgres psql -c "SELECT * FROM pg_replication_slots;"`
+- `sudo -u postgres psql -c "SELECT pg_drop_replication_slot('slot_name');"`
+- `sudo -u postgres psql -c "CHECKPOINT;"`

labs/postgres-wal-growth/solution.md

@@ -0,0 +1,35 @@
+### The Issue
+PostgreSQL uses Write Ahead Logs (WAL) to ensure data integrity. These logs are normally recycled after a checkpoint. However, if there is a **stale replication slot** (a slot created for a replica that is no longer connected), PostgreSQL will keep all WAL segments from the time the replica last connected to ensure it can catch up. This can quickly fill up the disk.
+
+### Step-by-Step Fix
+
+1. **Verify disk usage**:
+   ```bash
+   df -h
+   sudo du -sh /var/lib/postgresql/*/main/pg_wal
+   ```
+
+2. **Check for replication slots**:
+   Connect to PostgreSQL and check the status of replication slots.
+   ```bash
+   sudo -u postgres psql -c "SELECT * FROM pg_replication_slots;"
+   ```
+   You will see a slot named `stale_replica` that is `active = f`.
+
+3. **Drop the stale slot**:
+   Since no replica is using this slot, it can be safely removed.
+   ```bash
+   sudo -u postgres psql -c "SELECT pg_drop_replication_slot('stale_replica');"
+   ```
+
+4. **Force a checkpoint**:
+   Encourage PostgreSQL to clean up the now-unneeded WAL files immediately.
+   ```bash
+   sudo -u postgres psql -c "CHECKPOINT;"
+   ```
+
+5. **Verify the fix**:
+   Check the `pg_wal` directory size again. It should be significantly smaller.
+   ```bash
+   sudo du -sh /var/lib/postgresql/*/main/pg_wal
+   ```

labs/postgres-wal-growth/solution.sh

@@ -0,0 +1,7 @@
+#!/bin/bash
+
+# Drop the stale replication slot
+sudo -u postgres psql -c "SELECT pg_drop_replication_slot('stale_replica');"
+
+# Force a checkpoint to encourage WAL recycling
+sudo -u postgres psql -c "CHECKPOINT;"

labs/postgres-wal-growth/verify.sh

@@ -0,0 +1,12 @@
+#!/bin/bash
+
+# Check if the stale replication slot still exists
+SLOT_EXISTS=$(sudo -u postgres psql -t -c "SELECT count(*) FROM pg_replication_slots WHERE slot_name = 'stale_replica';" | xargs)
+
+if [ "$SLOT_EXISTS" -ne 0 ]; then
+    echo "FAILURE: Stale replication slot 'stale_replica' still exists, preventing WAL cleanup."
+    exit 1
+fi
+
+echo "SUCCESS: Stale replication slot removed."
+exit 0