Infrastructure

This repository manages the infrastructure for multiple hosts using Ansible and Docker Compose.

Architecture

The project follows a host-based isolation strategy where each top-level directory corresponds to a specific host (e.g., mx1, web1).

Most host directories contain:

• ansible/: Playbooks and roles for system configuration.
• containers/: Docker Compose definitions for services running on that host.

Some hosts are provisioned with OpenTofu instead of Ansible. These contain a terraform/ directory with .tf files and a node-specific docs/ directory.

Deployment Pattern

1. Ansible: Configures the host system (OS hardening, users, firewall, etc.).
2. Containers: The system/containers role copies the local containers/ directory to /opt/containers/ on the remote host.
3. Services: Services are deployed via docker compose by iterating over the defined compose files.
4. Network: All containers typically attach to an external Docker network named app-infra.

Hosts

The repository currently manages the following hosts:

Host	Tool	Purpose
mx1	Ansible + Docker	Mail server
web1	Ansible + Docker	Web/app services
web2	Ansible + Docker	Web/app services
web3	Ansible + Docker	Web/app services
tower	OpenTofu	ARM VM on OCI — WireGuard VPN endpoint and SSH bastion

See tower/docs/setup.md for the one-time bootstrap steps for the tower node.

Usage

Prerequisites

• Ansible: Must be installed on the machine running the playbooks.
• SSH Access: You need SSH access to the target hosts.

Running Playbooks

Navigate to the host's ansible directory and run the playbook:

cd <host>/ansible
ansible-playbook playbook.yml

Environment Variables & Secrets

Secrets are injected via environment variables using lookup('env', 'VAR_NAME'). Ensure these are set in your environment before running playbooks.

Common required variables include:

• BORG_PASSPHRASE
• BORG_REPOSITORY
• MYSQL_PASSWORD
• BORG_HEARTBEAT_URL

Scheduled vs. Quick Runs

The scheduled_run variable (controlled by the SCHEDULED_RUN environment variable) determines the scope of the execution:

• SCHEDULED_RUN=true: Runs "heavy" tasks like OS hardening, APT updates, and Docker installation. This is typically used for maintenance runs.
• SCHEDULED_RUN=false (default): Skips heavy tasks for quick app deployments.

Migrations

This infrastructure uses a migration system similar to Laravel's database migrations, but for system configuration. Migrations are one-time Ansible tasks that run only once per host.

Migration Features

• One-time execution: Each migration runs only once, tracked in /opt/ansible/migrations.db
• Per-host state: Each host maintains its own migration history
• Error handling: Playbook execution stops if a migration fails
• Versioned: Migrations use timestamped filenames (e.g., 20240121_0001_setup_database.yml)

Creating Migrations

1. Create a new .yml file in <host>/ansible/migrations/
2. Use timestamp format: YYYYMMDD_NNNN_description.yml
3. Write standard Ansible tasks in the file

Example Migration

---
# Migration: 20240121_0001_create_app_user
- name: Create application user
  user:
    name: myapp
    system: yes
    shell: /bin/bash

- name: Create application directory
  file:
    path: /opt/myapp
    state: directory
    owner: myapp
    group: myapp
    mode: '0755'

Migration Execution

Migrations run automatically as part of the playbook execution via the system/migrations role. They execute before container deployment to ensure system prerequisites are met.

If a migration fails, the entire playbook stops to prevent inconsistent states.

Validation

Every pull request runs .github/workflows/validate.yml, which performs static checks only. It never connects to a host, reads a state file, or touches a cloud provider, so a broken change is caught before it can reach production rather than during a deploy.

Check	Tool	Catches
YAML	yamllint	Syntax errors, duplicate keys, style drift
Ansible	ansible-lint	Broken syntax, missing FQCN, non-idempotent commands, unset file modes
OpenTofu	tofu validate / tofu fmt	Invalid or unformatted configuration for tower
Actions	zizmor	Workflow-level privilege footguns

Running the checks locally

ci/validate.sh

Needs ansible-lint, yamllint, zizmor and tofu on PATH. The script runs the same checks as CI, so failures reproduce locally.

The container security contract is not enforced

The rules in CLAUDE.md (drop all capabilities, add no-new-privileges, bind ports to loopback, pin image tags) are review-time guidance, not a gate. There is no automated check, so a new service that omits cap_drop will pass CI and rely on the reviewer noticing.

This was a deliberate trade-off. An earlier version enforced it with a script keyed by an exemption list, but the contract is prose in CLAUDE.md and would have had two places to drift apart. It was removed rather than maintained.

If you want it enforced, the exemption list is the part that needs writing first — a service name alone is ambiguous, since worker is the Docker-socket-holding authentik worker in one stack and an ordinary hardened twenty.crm worker in another.

Why the validation workflow holds no secrets

validate.yml runs on pull_request, so it is configured to hold no credentials at all — the checks are entirely static analysis:

• It references no secrets.* and no environment:, so the job runs with no access to repository or environment secrets regardless of who opened the pull request.
• permissions: contents: read, so the automatic GITHUB_TOKEN cannot write.
• Every actions/checkout sets persist-credentials: false, so the token is not left in .git/config where later steps could read it.
• Third-party actions are pinned to a full commit SHA, fixing the code that runs with this job's token.

All four checks are required status checks on main, so a pull request cannot merge until they pass. If a check is retired or renamed, remove it from the required_status_checks.contexts list in the same pull request — otherwise the merge blocks waiting on a check that no longer reports.

The deploy workflows do use secrets — they have to. They are triggered by push to main, schedule and manual dispatch, never by pull_request, so they never run code from a pull request.

Dependency Management

This repository uses Renovate to keep dependencies up-to-date:

• Docker Images: Updates Docker images in docker-compose.yml files across all host containers/ directories.
• Ansible Requirements: Manages Ansible collections and roles in requirements.yml files for each host.

To enable Renovate:

• Install the Renovate GitHub app on the repository.

For private registries, configure hostRules in renovate.json with appropriate credentials.

Development

Adding a Service

1. Create a new directory in <host>/containers/<service_name>/.
2. Add a docker-compose.yml file.
   • Ensure it uses the app-infra network:

     networks:
       app-infra:
         external: true

   • Note: Do not include the top-level version property in docker-compose.yml files.
3. Add the service name to the compose_files list in the host's Ansible variables (usually in defaults/main.yml) if it is not dynamically discovered.

Role Responsibilities

• system/containers: Core deployment role. Copies containers/ to /opt/containers/ and runs docker compose up.
• system/config: System-level config including UFW rules and systemd-resolved.
• system/backup: Configures Borgmatic backups.
• system/apt: Handles package updates and installation (usually conditional on scheduled_run).