[matbrgz]

Refs #1

Implements spec 001 plan §3 (repo tooling) and §4 (CI). Docs (AGENTS.md runbook, README) stay for the docs PR (tasks 6.x).

What's in it

• tools/catalog.py (stdlib only, Python 3.9+, written from scratch, no upstream toolset code), with --root to point at another catalog:
  • check: parses every */*/benchmark_config.json, resolves each variant's Dockerfile (<framework>.dockerfile, <framework>-<key>.dockerfile or the dockerfile override), requires a port and a known *_url, normalizes database, reports test types per stage. Exits 1 on errors.
  • list [--lang L] [--json]
  • changed --since <ref> [--json]: changed framework folders. Prints ALL_TOOLING when tools/, tests/ or .github/workflows/ change. --json adds the smoke sample to matrix.
  • lint-docker --since <ref> (or --all): fails on :latest or untagged FROM. Stage aliases, scratch, ARG images and digests are allowed.
  • verify <Language[/framework[:key]]> [--stage A]: docker build (one retry, recorded in the summary), run on a private network, wait up to 120 s for HTTP on the published port, run the stage-A verifiers, then remove the container and network (--rmi also removes the image). Broken variants → skipped-broken. Stage B/C types → declared-not-verified.
  • summarize: merges shard summaries into one JSON plus a Markdown job summary.
• tools/verify/: json, plaintext and common header rules (spec R5) behind a single function, verify_variant(). Where the spec and the pinned Python verifier differ, I followed the verifier: a malformed Date only warns, UTC is accepted, and the minimum routes are /json (5) and /plaintext (10).
• tests/: 60 unittest cases: fixture catalogs (valid, missing Dockerfile, no URLs, bad JSON, override, broken), temporary git repos for changed/lint-docker, a local stub HTTP server for each pass/warn/fail rule, and a fake Docker for the verify orchestration.
• .github/workflows/ci.yml: the existing docs job is unchanged. New jobs:
  • metadata: Python 3.9, check plus unit tests.
  • changed: lint plus matrix.
  • verify: stage A, fail-fast: false.
  • full-plan / full / full-summary: weekly plus workflow_dispatch with a folders input, one shard per language, max-parallel: 8, JSON summary artifact.

Smoke sample

The plan lists e.g. Go/gin, Java/spring, ... as the smoke sample. Their default variants connect to a database at startup, so they cannot pass stage A without one. Go/gin:default exits with lookup tfb-database ... no such host. The sample is therefore one variant per language that needs no database:

C/libreactor:default, Go/go-std:default, Java/javalin:default, JavaScript/express:default, Python/fastpysgi:default, Rust/axum:default.

Local validation

• python3 -m unittest discover -s tests: 60 tests OK.
• python3 tools/catalog.py check on this repo:
  • 41 languages, 407 frameworks, 965 variants, 160 broken, 1016 Dockerfiles, 0 errors.
  • Databases: postgres 507, mysql 209, none 203, mongodb 46.
  • json 674 (574 non-broken), plaintext 664 (563), db 669, query 652, fortune 655, update 606, cached-query 140.
  • All of these match the spec.
• lint-docker --all: 39 existing problems (14 :latest, 25 untagged). The spec's "about 45 untagged" counted stage aliases. CI lints only changed Dockerfiles.
• verify end to end (Docker Desktop, arm64, native builds): all 6 smoke variants pass json and plaintext. No containers or networks are left behind. Go/gin:default fails as described above.

Follow-up for spec 002

Every variant whose application connects to its database at startup will fail stage A on hosted CI until the dataset/database service exists. That covers most database != none variants, such as Go/gin:default. The weekly full run will show how many.

🤖 Generated with Claude Code