feat(nextcloud): standalone Docker stack + split provisioner pattern (misp-style)#190
Closed
t0kubetsu wants to merge 76 commits into
Closed
feat(nextcloud): standalone Docker stack + split provisioner pattern (misp-style)#190t0kubetsu wants to merge 76 commits into
t0kubetsu wants to merge 76 commits into
Conversation
…gs (#153) Critical fixes: - C1: Replace host-wide `docker system prune` + `docker ps -aq` clean target with scoped `docker compose down --rmi all --volumes --remove-orphans` - C2: Fix wrong SERVICE in CVE-2024-6387 (was openssh-cve-2018-15473) - C3: Fix wrong SERVICE in CVE-2019-11043 (was apache-cve-2021-42013) High fixes: - H1: Eliminate $(shell docker ps -aq) parse-time expansion in all files - H2: Fix stop-debug-build hardcoded 'debug' literal → $(DEBUG_SERVICE) - H3: Scope docker compose build/up to $(SERVICE) in all build targets - H4: Moot — resolved by C1 fix Medium fixes: - M1: Add .PHONY declarations to all 18 Makefiles - M2: Add `print: help` alias (was advertised but missing) - M3: Update vite CVE-2023-34092 SERVICE + compose.yml service name (stale vite-cve-2022-44615 from pre-rename) - M4: Remove dead 5-line commented block from term-debug-build - M5: Fix stop-debug-build comment ("stop only" → "stop and remove") - M6: Set blank_template SERVICE = REPLACE_ME (was tomcat service name) - M7: Fix lpe-01 duplicate help entry; expose make term-user properly - M8: Moot — resolved by C1 fix Low fixes: - L1: Normalize ISSUE header formats to `# ISSUE N` across all files - L2: Standardize term shell to /bin/bash (vite CVEs used /bin/sh) - L3: Remove trailing whitespace on target definition lines
New symbolic, parametric templates consumed by the r42topo compiler
(range42-playbooks) to author + compile topology.json into deployable
scenarios. Directory-per-version layout: <id>/v<MAJOR.MINOR.PATCH>/template.yml.
- subnet_layouts/default-3zone — admin/student/ctf bridge plan
- box_templates/{admin-wazuh,deployer,vuln-box,student-box} — VM archetypes
(role, inventory group, spec, default catalog role attachments)
- network_policies/air-gap-ctf — symbolic generalization of demo_lab_network's
hardcoded 05_network_isolation (zones + services + allow/deny matrix + air-gap)
- network_policies/flat-open — no-isolation baseline
- manifest.json: register the topology layer + categories
Box templates can now declare their base OS, so a box's clone image is chosen at authoring time while the warmup roles keep self-detecting the OS at runtime via ansible_facts.distribution (feat/local-apt-mirror). - os: ubuntu added explicitly to admin-wazuh, deployer, student-box, vuln-box (optional, defaults to ubuntu in the loader — no behaviour change). - New debian-jump box: os: debian, Debian 13 (trixie), 2cpu/4gb/32gb student jump box — the reference example of a non-ubuntu box. Values match ansible_facts.distribution lower-cased (ubuntu/debian/fedora). See the generator's docs/box-template-os-field.md for the full rationale + the template-image gate (debian images exist as trixie; fedora not yet).
…ename>) os: ubuntu/debian -> image: ubuntu_noble/debian_trixie. The field now carries the version (image-set name = the 01_init_proxmox templates/<image>/ dir), so a box is never ambiguous about which Debian. debian-jump -> image: debian_trixie.
The renderer now emits attachment params into stage_01, so the box catalog is the parametric source of a VM's config: software.configure.firewalls carries its firewall_rules (22/80/8080 tcp), and a container attachment (cve/crypto/openssl/ CVE-2014-0160) wires a docker-compose play. Replaces the concrete copy in the create-vms-vuln bundle.
…ry box All box templates now own their host firewall rules via the software.configure.firewalls attachment params (emitted into stage_01 by the generator), matching the demo_lab rules: - admin-wazuh : 22, 443, 1514, 1515 (ssh, dashboard, wazuh agent events/enroll) - deployer : 22, 80, 443 (ssh, http, https) - student-box : 22 (ssh) - debian-jump : 22 (ssh) - vuln-box : 22, 80, 8080 + docker CTF stack (prior commit) IP-dependent role config (wazuh indexer IPs, etc.) intentionally stays at the role defaults / deploy time — not catalog-appropriate. The catalog box_template is now the parametric source of a VM's host config, replacing the concrete copies in the create-vms-* bundles.
… debian_trixie Each image now carries the cloud_image.url + cloud_image.filename needed by the generator to render stage_00-download_cloudinit_files/<image>.yml. Only the image actually used by templates is listed (noble-minimal, not noble-server).
… debian_trixie (2 VMs) Each image now carries the full Proxmox template VM specs (vm_id, vm_name, spec, ip, bridge) needed by the generator to render stage_01-create_templates without any hardcoded data in the playbooks repo.
… convention
- box_templates: replace spec/image with template_vm (direct ProxmoxTemplateSpec ref)
- proxmox_templates: replace ip/bridge with ip_octet; remove bridge (both now
come from the subnet layout's template_subnet at generation time)
- vm_names renamed to template-vm-{distro}-{codename}-* for global uniqueness
(ubuntu: template-vm-ubuntu-noble-*, debian: template-vm-debian-trixie-*)
- default-3zone: add template_subnet: {cidr, bridge} — keeps "3-zone" accurate
(templates subnet is infrastructure, not a lab zone)
…sed/, update when conditions and docker repo URLs - software.configure.firewalls: ubuntu/ → debian-based/, when includes Debian - software.install.warmup.basic_packages: all ubuntu/ subdirs → debian-based/, all dispatcher _main.yaml when conditions updated, docker GPG/repo URLs now use ansible_facts.distribution | lower for Ubuntu and Debian
…ta to layout Box templates no longer carry role. Subnet layouts gain base_octet, section, and label per subnet so the generator derives placement purely from the layout.
Group is now derived from subnet name at generation time (r42_{subnet}_group).
…l-lan layout Section directory names and Ansible task labels are now derived automatically from each subnet's list index and name — no need to declare them explicitly in template YAML. Also adds dual-lan layout (lan1/lan2, vmbr150/vmbr151).
Mutual DROP isolation between lan1 and lan2 for the dual-lan layout. Zone names match subnet names so compile_network_policy_from_alloc resolves concrete CIDRs (192.168.150.0/24 <-> 192.168.151.0/24).
default_action: drop was blocking all internet/NAT traffic from VMs (cloud-init apt updates, internet reachability). The isolation intent is only to block lan1<->lan2 cross traffic, not all forwarded packets. Matrix DROP rules at w=500 fire before the default ACCEPT at w=900, so the lan1/lan2 mutual block still holds.
Revert default_action to drop (deny-by-default posture).
Add explicit {src: lan1/lan2, dst: wan, action: accept} rows for
internet egress. These compile at W_EGRESS_ACCEPT=700, AFTER the
cross-LAN DROP rules (500), so isolation is preserved.
Adds a clean apt-mirror/cache implementation scoped to the catalog layer: - software.install.apt_cacher_ng: installs, configures, and starts apt-cacher-ng (port 3142, passthrough enabled, ufw rule opened) - software.configure.apt_mirror_client: deploys /etc/apt/apt.conf.d/01proxy, requires apt_proxy_url to be set, triggers apt cache refresh via handler - apt-mirror box_template: debian-trixie-small base, firewall (22+3142), basic_packages warmup, apt_cacher_ng server role attached Rebased from feat/topology-layer-templates; replaces old airgapped implementation with a simpler proxy-cache-only approach.
…full mirror Extends the apt-mirror box template to support both operating modes: - Proxy-cache mode (default): apt-cacher-ng on port 3142, transparent caching - Airgapped full-mirror mode (apt_mirror_airgapped: true): apt-mirror + nginx on port 80 Server role (software.install.apt_cacher_ng): - defaults: add airgapped vars (apt_mirror_root, apt_mirror_threads, suite flags) - tasks/main.yml: conditional dispatch between proxy and airgapped task sets - tasks/install_mirror.yml: install apt-mirror + nginx - tasks/configure_mirror.yml: create mirror dirs, deploy mirror.list - tasks/prewarm_mirror.yml: optional async fire-and-forget apt-mirror run - tasks/serve_mirror.yml: deploy nginx config, enable service, open ufw port - templates/mirror.list.j2: apt-mirror mirror.list with Debian trixie + Ubuntu suite conditionals - templates/nginx-aptmirror.conf.j2: serve apt_mirror_root/mirror on apt_mirror_http_port + /healthz - handlers/main.yml: add restart nginx handler Client role (software.configure.apt_mirror_client): - defaults: replace old proxy-only vars with apt_mirror_enabled, apt_mirror_airgapped, apt_mirror_vm_ip, apt_mirror_http_port - tasks/main.yml: URL-rewriting approach — computes _apt_effective_url and rewrites /etc/apt/mirrors/debian.list, debian-security.list (Debian) and ubuntu.sources URIs (Ubuntu) Box template: update description, add port 80 to firewall_rules
…gap is a network policy concern
…her_ng and software.install.apt_mirror - software.install.apt_cacher_ng: proxy-cache only, no mode flag - software.install.apt_mirror: new role, full mirror (apt-mirror + nginx) - apt-mirror box template now references software.install.apt_mirror with no params - task/template files moved via git mv to preserve history
…e-based names; add resolute
- Remove ip_octet from all proxmox_templates in debian_trixie and ubuntu_noble
- Add ubuntu_resolute image layer (Ubuntu 26.04 LTS, 12 VM sizes)
- Rename apt_mirror suite variables to codename-based names:
apt_mirror_debian_stable → apt_mirror_debian_trixie
apt_mirror_ubuntu_2204 → apt_mirror_ubuntu_jammy
apt_mirror_ubuntu_2404 → apt_mirror_ubuntu_noble
- Add apt_mirror_debian_bookworm and apt_mirror_ubuntu_resolute flags
- Add Ubuntu resolute section to mirror.list.j2; fix duplicate Ubuntu clean entries
- Add ubuntu-jump box template (Ubuntu 26.04 Resolute example with apt integration docs)
Add an admin-rocketchat box template (Debian, template-vm-debian-trixie-medium) that bootstraps the existing 03_container_layer/docker/admin/rocketchat compose stack: firewall (22/3000) -> Docker baseline -> software.install.rocketchat. - New role software.install.rocketchat: rsyncs the catalog Rocket.Chat stack onto the box and brings it up, delegating to software.configure.docker-compose (no stack duplication; env-lookup path lives in role defaults, not in the injection-guarded box-template params). - software.configure.docker-compose: was Ubuntu-only (silent no-op on Debian); broadened the deploy gate to ['Ubuntu','Debian'] and renamed tasks/ubuntu -> tasks/debian-based to match the basic_packages convention. Tasks are apt/rsync based, so behaviour on Ubuntu is unchanged.
# Conflicts: # .gitignore
…_packages In an egress-filtered / air-gapped range udp/123 is blocked (even the Proxmox node can't NTP-sync), so software.install.warmup.basic_packages' hard "wait for NTPSynchronized == yes" never succeeds and fails stage_01. Don't enable NTP sync for this box; VMs take host (kvm-clock) time.
…er reqs software-properties-common is absent from Debian trixie 'main' (Unable to locate package), so the docker prerequisites install failed on Debian hosts. It's an Ubuntu-ism and unused here anyway — the Docker CE repo is added via the apt_repository/apt_key modules, not add-apt-repository. Removing it unblocks docker install on Debian; Ubuntu is unaffected (repo-add path is identical).
…r repo (Debian) apt-key is gone on Debian 12+/trixie (and deprecated on Ubuntu), so the apt_key module failed: "Failed to find required executable apt-key". Switch to the modern keyring method: fetch the armored Docker GPG key into /etc/apt/keyrings/docker.asc (get_url) and pin the repo with signed-by=. Also derive arch (amd64/arm64) instead of hardcoding amd64. Works on both Debian and Ubuntu.
…ompose path) The compose install path had three problems that would break the Rocket.Chat stack deploy (community.docker.docker_compose_v2 / docker_container_info): - listed software-properties-common (absent on Debian trixie) → apt failure; - downloaded the standalone docker-compose binary from a v1-era asset name (docker-compose-Linux-x86_64) that 404s for v2, and that binary does not provide the `docker compose` plugin docker_compose_v2 invokes; - no python Docker SDK for the community.docker modules. Now: install docker-compose-plugin from the docker-ce repo (provides `docker compose`), add python3-docker, drop software-properties-common, and check via `docker compose version`. Works on Debian and Ubuntu.
…ires >=8.0)
rocketchat/rocket.chat:latest is now 8.4.3, which refuses to start against
MongoDB 6.0 ("YOUR CURRENT MONGODB VERSION IS NOT SUPPORTED ... UPGRADE TO 8.0
OR LATER") — the container crash-loops and the compose deploy fails with
"container rocketchat is unhealthy". Pin mongo:8.0 for both the mongodb and
mongo-init-replica services. Fresh deploys only (no in-place 6.0->8.0 data
migration); the lab recreates the volume each deploy.
…meouts On egress-filtered ranges, image pulls intermittently time out against CDN anycast endpoints (auth.docker.io / registry-1.docker.io, debian-security, etc.) from the VM's NAT path, failing the whole deploy even though a retry would connect to a working PoP. `docker compose up` is idempotent, so wrap the run with until/retries (5x, 20s) so transient pull failures self-heal; a persistent failure still surfaces after retries are exhausted.
…nt in image) The rocket.chat image (Alpine) ships neither curl nor wget, which broke two things: - the rocketchat healthcheck (curl -f .../api/v1/info) never passed → container stuck "health: starting" forever → provisioner (depends_on service_healthy) never started → `docker compose up` hung indefinitely; - the provisioner init.sh drives the REST API with curl → would fail on "curl: not found" once it ran. Fixes: - healthcheck now uses Node's built-in fetch (Node 18+; the image runs node main.js); - provisioner Dockerfile installs curl (apk add --no-cache curl) in the runtime stage. Rocket.Chat 8.4.3 itself boots fine (SERVER RUNNING, MongoDB 8.0.23).
Rocket.Chat 8.4.3 removed the unauthenticated /api/v1/info endpoint (returns 404), which broke two readiness checks against a server that is actually up: - compose healthcheck -> hit /health (200) instead; - provisioner init.sh readiness wait -> hit /api/info (200) instead. Confirmed live: /health and /api/info both return 200; /api/v1/info returns 404.
… BoxTemplate, catalog_try.yml
# Conflicts: # 03_container_layer/docker/admin/nextcloud/catalog_try.yml
…ttern Drop the two-stage Dockerfile (php:8.3 builder + nextcloud runtime) and static users.yml in favour of the misp-standalone provisioner pattern: - Alpine single-stage image (bash/curl/jq/openssl); scripts volume-mounted at runtime rather than baked in - provision.sh: thin orchestrator (provision-users → provision-tokens) - provision-users.sh: OCS API user creation driven by NC_TEAMS / NC_INSTRUCTOR_COUNT env vars; auto-generated passwords; writes /tokens/nc-credentials.json + idempotency stamp - provision-tokens.sh: reads nc-credentials.json; generates an app password per user via /ocs/v2.php/core/apppassword; writes tokens.txt - .env.example: adds NC_TEAMS, NC_INSTRUCTOR_ORG, NC_INSTRUCTOR_COUNT, NC_USERS_PER_TEAM, NC_USER_DOMAIN - compose.yml: provisioner now uses env_file + three :ro script mounts - Makefile: adds `keys` target (nc-credentials.json) - README: documents new env-driven user model Closes #146
t0kubetsu
changed the title
Contributor
Author
|
Superseded by #192 (feat/nextcloud-docker-stack) which is scoped to 03_container_layer/docker/admin/nextcloud/ only. The mixed topology/Ansible/Docker history on this branch made a scoped PR impossible without a dedicated branch. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Nextcloud standalone Docker stack with automated team-aware provisioning, mirroring the misp-standalone provisioner pattern.
What's in the PR
Provisioner refactor (this commit)
:ro), not baked inprovision.sh— thin orchestrator:provision-users.sh→provision-tokens.shprovision-users.sh— OCS API user creation driven byNC_TEAMS/NC_INSTRUCTOR_COUNT/NC_USERS_PER_TEAMenv vars; auto-generated passwords viaopenssl; writes/tokens/nc-credentials.json+ idempotency stamp/tokens/.provisionedprovision-tokens.sh— readsnc-credentials.json; generates a Nextcloud app password per user viaPOST /ocs/v2.php/core/apppassword; writes/tokens/tokens.txt.env.example— addsNC_TEAMS,NC_INSTRUCTOR_ORG,NC_INSTRUCTOR_COUNT,NC_USERS_PER_TEAM,NC_USER_DOMAINcompose.yml— provisioner usesenv_file: .env+ three:roscript volume mountsMakefile— addskeystarget to printnc-credentials.jsonprovisioning/init.shandprovisioning/users.ymlTopology and Ansible (earlier commits)
05_topology_layer) — Debian Trixie medium: firewall (22/8080) → Docker baseline →software.install.nextcloudsoftware.install.nextcloudrole — rsyncs the catalog03_container_layer/docker/admin/nextcloudstack onto the box and starts it viasoftware.configure.docker-composecatalog_try.yml—catalog_try_mode: service, port 8080, endpoint/status.php,init_timeout: 180Dependency note
Builds on the topology layer + admin-role structure from the other open dev PRs (#180, #181, #188, #189). Deploys cleanly where
dev_adacoexists; account for merge ordering to dev.Test plan
test_nextcloudscenario; Nextcloud healthy on :8080,/status.phpreturns{"installed":true,...}make build-up→ provisioner runs →make tokensshows app passwords for all team usersmake keysshowsnc-credentials.jsonwith roles, teams, and passwordsmake reprovisionclears volume and re-runs cleanly.envbefore non-throwaway use (defaults:nc-admin/Admin1234!)