feat(quality): untimed output quality pass with Lighthouse and SEO signals (spec 009)

[matbrgz]

Implements spec 009 (closes #22). --quality adds an untimed pass that audits what each generator ships. It runs Lighthouse, collects SEO signals, validates the HTML, checks accessibility and broken links, and measures weight and JS. It does not change how builds are timed.

How it works

• Export: after a successful build, DockerHelper.benchmark copies <WorkingDir>/<output_folder> out of the generator container before removing it.

• Collection:

  • A pinned image, quality/, runs quality/run.mjs with network_mode="none" and no mounts.
  • The site goes in with put_archive, and raw.json comes out with get_archive. The toolset itself runs in a container, so bind mounts would need host paths.
  • It audits three pages: the index, the first post and 404.html. It runs Lighthouse with the mobile and desktop presets, 3 runs each, plus axe-core, html-validate, lychee (offline) and gzip/brotli sizes.

• Reduction: pure Python in toolset/quality/ (site, seo, reduce, runner) turns raw.json into quality.json. That covers the medians and ranges, 12 per-page SEO signals plus site-wide ones, HTML/axe/link counts, weight, jsClass, worksWithoutJs and extra output.

• Results:

  • per generator: results/<ts>/quality/<fw>/quality.json;
  • in results.json: quality.<fw> and environment.quality (the tool versions and the image id);
  • a "Qualidade" section in summary.md and a quality-summary.csv.

  There is no combined score and no ranking. --parse keeps quality.

• Isolation: the pass runs only after an ok build. With --suite it runs only in the nf50-cs5 cell. A failing check is recorded as status: error and never changes a timing, a status or a ranking. Tests pin this.

Pinned versions: lighthouse 13.5.0, chrome-launcher 1.2.2, playwright 1.64.0 (Chromium 156.0.8078.4), axe-core 4.14.0, html-validate 11.16.2, lychee 0.24.2, Node 24.21.0.

Spec changes found while planning

• The collector container takes the site through put_archive instead of a bind mount.
• Chromium comes from Playwright, because Chrome for Testing has no Linux arm64 build.

Testing

• pytest: 341 passed, 1 skipped. ruff is clean.
• Locally, the collector image passed quality/test/check-shape.mjs on the fixture site, and quality/test/bad-url.mjs passed: a malformed URL does not crash the collector.
• New CI jobs:
  • quality-image runs the collector on the fixture for amd64 and arm64.
  • quality runs ./ssgberk --test hugo -nf 50 -cs 5 -mr 1 --quality and checks every section, the 6 Lighthouse entries, scores in [0, 1], environment.quality and the summaries.
• benchmark-round.yml gets a quality input.

Each of the 8 tasks had its own review, followed by a whole-branch review. Its findings were fixed in 5a712be:

• the collector's output directory collided with the site of generators whose output folder is out;
• the pass's I/O errors could escape;
• the bad-url check was not in CI.

🤖 Generated with Claude Code