diff --git a/docs/specs/010-catalog-contract/plan.md b/docs/specs/010-catalog-contract/plan.md new file mode 100644 index 0000000..a0b89c0 --- /dev/null +++ b/docs/specs/010-catalog-contract/plan.md @@ -0,0 +1,182 @@ +# 010 Catalog Contract: Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. The steps are in `tasks.md` and use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** write the catalog contract down as JSON schemas and a checker that catalogs run in CI, and make BT select a catalog, load its manifest, accept generic `BENCH_*` markers (with `SSGBERK_*` aliases) and dispatch to per-catalog test types. CS output parses exactly as before. + +**Architecture:** +1. **Schemas** in `schema/` (new, repo root): `catalog.schema.json`, `benchmark_config.schema.json`, `implementation.base.schema.json`. Catalogs reference them by raw URL at a BT tag. +2. **Checker** `tools/check_catalog.py`: stdlib only, no Docker, run by catalog CI. +3. **Catalog loader** `toolset/utils/catalog.py`: resolves `--catalog`/`--catalog-dir`, loads and validates `catalog.json` (or the built-in CS default), lists implementations from the glob. `metadata.py` uses it instead of the hard-coded `frameworks/*/*`. +4. **Marker parser:** `results.py` normalizes `SSGBERK_` to `BENCH_` once, then runs the existing BT 002/008 rules unchanged. +5. **Test types:** `algorithm_type.py` next to `datarate_type.py`; `http` is declared (parameters and config keys only) and refused at run time until its own spec. + +**Tech stack:** Python 3.12 stdlib and docker-py 7.1.0, as today. JSON Schema draft 2020-12 for the schema files; validation in BT uses the small stdlib validator subset already used by CS `build_index.py` (`type`, `enum`, `required`, `properties`, `additionalProperties`, `items`, `pattern`, `allOf`, `$ref` within the file), so no new dependency. + +**Spec:** `docs/specs/010-catalog-contract/spec.md`. Requirement ids (R-n) refer to it. + +## Global Constraints + +- CS output parses to identical `rawData`, `succeeded`, `failed`, `unsupported`, `failureReasons` and rankings, apart from the new `markerPrefix` and `catalog` keys (goal 1, 2). +- `results.json` stays `schemaVersion: 2`; every new key is optional for readers (R-19, BT 008 R-28). +- No new runtime dependency in the toolset image. +- No change to any catalog repo from this spec. +- `SSGBERK_*` aliases are accepted for all of `contractVersion: 1` (R-16). +- Commits: author and committer `Matheus Breguêz `, GPG-signed (key `B6FA8458D5176E83`), trailer `Co-Authored-By: Claude Opus 5.5 (1M context) `. Never use `--no-gpg-sign` or `--author`. + +## Contracts + +### `catalog.json` (R-1) + +CS (to be added by CS #203; also BT's built-in default while `frameworks/` exists): + +```json +{ + "$schema": "https://raw.githubusercontent.com/BenchCommons/benchmark-tool//schema/catalog.schema.json", + "contractVersion": 1, + "name": "ssg", + "testTypes": ["datarate"], + "implementations": "*/*", + "runner": {"file": "build.sh", "canonical": "Go/hugo/build.sh"}, + "metadata": {"file": "generator.json", "schema": "schema/generator.schema.json"}, + "index": {"file": "generators.json", "key": "generators", "check": "python3 tools/build_index.py --check"} +} +``` + +CA (illustrative; CA 011 decides the real paths): + +```json +{ + "contractVersion": 1, + "name": "algorithms", + "testTypes": ["algorithm"], + "implementations": "algorithms/*/*", + "runner": {"file": "run.sh", "canonical": "tools/run.sh"}, + "metadata": {"file": "implementation.json", "schema": "schema/implementation.schema.json"}, + "index": {"file": "implementations.json", "key": "implementations", "check": "python3 tools/build_index.py --check"} +} +``` + +CW: same shape with `"testTypes": ["http"]`, `"implementations": "frameworks/*/*"` and no `runner`. + +`runner.canonical` may sit outside the implementation glob (CA `tools/run.sh`); then every implementation's copy is compared with it and none is the reference. + +### Runner environment (R-7) + +| Variable | Test types | Meaning | Legacy name (still set for `datarate`) | +|---|---|---|---| +| `BENCH_TEST_TYPE` | all | `datarate`, `algorithm` | — | +| `BENCH_RUNS` | all | timed runs | `min_runs` | +| `BENCH_WARMUP` | all | untimed warmup runs before timing | — | +| `BENCH_VERBOSE` | all | `true`/`false` | `verbose_build` | +| `BENCH_NUMBER_OF_FILES` | `datarate` | cell `numberOfFiles` | `number_of_files` | +| `BENCH_CONTENT_SIZE` | `datarate` | cell `contentSize` | `content_size` | +| `BENCH_ALGORITHM` | `algorithm` | algorithm id | — | +| `BENCH_N` | `algorithm` | input size | — | +| `BENCH_DATASET` | `algorithm` | `@` or empty | — | + +Profiles (`profile`, BT 008) and `KEEP_CONTENT`/`SSGBERK_GENERATE_ONLY` stay CS-specific and unchanged. + +### `benchmark_config.json` test entry (R-8, R-12) + +`algorithm` (CA): + +```json +{ + "framework": "nbody-python", + "tests": [{ + "default": { + "test_type": "algorithm", + "approach": "Realistic", "classification": "Micro", "framework": "nbody-python", + "language": "Python", "display_name": "n-body (CPython)", "notes": "", "versus": "c", + "algorithm": "nbody", + "build_command": "", + "run_command": "python3 nbody.py ${BENCH_N}" + } + }] +} +``` + +`http` (CW, TechEmpower keys kept as they are so imported configs need no rewrite): + +```json +{ + "framework": "actix", + "tests": [{ + "default": { + "test_type": "http", + "approach": "Realistic", "classification": "Micro", "framework": "actix", + "language": "Rust", "display_name": "actix", "notes": "", "versus": "", + "port": 8080, "json_url": "/json", "plaintext_url": "/plaintext" + } + }] +} +``` + +### Markers (R-15, R-16) + +Parser normalization, applied once to the raw text before every existing regex: + +```python +_PREFIX = re.compile(r'^([ \t]*)(?:BENCH|SSGBERK)_', re.M) # same line start as _BOL +text = _PREFIX.sub(r'\1BENCH_', raw) +text = re.sub(r'^([ \t]*)BENCH_PROFILE_UNSUPPORTED\b', r'\1BENCH_UNSUPPORTED', text, flags=re.M) +``` + +`markerPrefix` is `SSGBERK` if any line starts with `SSGBERK_`, else `BENCH` (mixed output is legal and counts as `SSGBERK`). The `BENCH_RESULT_BEGIN` line's optional `engine=`/`version=` pairs are stored as `engine` and `engineVersion` in the result; missing means `hyperfine` and `null`. + +### `suites.json` (R-18) + +New optional keys per suite: `"catalog": "ssg"`, `"testType": "datarate"`. An `algorithm` suite looks like: + +```json +"algo-smoke": { + "catalog": "algorithms", "testType": "algorithm", + "version": 1, "runs": 3, "cooldownSeconds": 0, "timeoutSeconds": 600, + "cells": [{"algorithm": "nbody", "n": 1000}, {"algorithm": "nbody", "n": 50000}] +} +``` + +Cell keys are validated against the test type's declared parameters. + +### `results.json` additions (R-19) + +```json +{ + "schemaVersion": 2, + "catalog": {"name": "ssg", "commit": "327bd5e", "contractVersion": 1}, + "rawData": {"datarate": {"hugo": [{"...": "...", "markerPrefix": "SSGBERK", "engine": "hyperfine", "engineVersion": null}]}} +} +``` + +`rawData.algorithm.[0]` has the same timing keys as `datarate` (`mean`, `stddev`, `median`, `min`, `max`, `times`, `user`, `system`, `memoryUsageBytes`, `peakRssBytes`) plus `algorithm` and `n` instead of `numberOfFiles` and `contentSize`. + +## Decisions + +1. **A manifest file, not a convention.** Layouts already differ (`*/*`, `algorithms/*/*`, `frameworks/*/*`) and the runner and metadata file names differ. One small `catalog.json` states them; BT keeps a built-in CS manifest only until CS adds its own (spec, Migration step 5). +2. **Byte-identical runner copies, not a shared file.** The Docker build context is the implementation directory (R-3), so a shared script cannot be `COPY`ed without widening every context. CS already works this way and checks it with `cmp`. +3. **Prefix `BENCH_`.** Short, catalog-neutral, and not an org name that may change again. `SSGBERK_` stays as an alias for the whole `contractVersion: 1`. +4. **`http` has no runner.** A server cannot time itself; the load generator must run in another container, as in TechEmpower. Verification for `http` is BT's job, so the same `VERIFY_*` markers are written by BT and parsed by the same code. +5. **One catalog per run.** Results, suites and rankings are per catalog; mixing them in one `results.json` adds keys for no current use. `merge_results` refuses inputs from different catalogs. +6. **`schemaVersion` stays 2.** The additions are optional keys. Engine-driven changes (energy, custom metrics) are the reason for v3 and belong to BT #30. +7. **Submodules under `catalogs//`.** One directory per catalog, pinned like `frameworks/` is today. `--catalog-dir` covers local development (it replaces mounting a worktree at `frameworks/`). + +## Review Focus + +1. **Alias normalization changes a CS result.** The rewrite must touch only line-initial `SSGBERK_`, never JSON content between the result markers (a generator name or path containing `SSGBERK_` inside the hyperfine JSON command string). Pinned in Task 3 (`test_alias_inside_json_untouched`). +2. **`FWROOT/frameworks` is assumed in several places** (`benchmark_config.py` `lang_root`, `metadata.py`, `audit.py`, `github_actions_diff.py`, `scaffolding.py`, `run-tests.py`). Task 4 replaces every path use with `catalog.root`; afterwards `grep -rn "frameworks" toolset` shows no path built from it except the built-in default manifest (the word may remain in log text and results keys such as `frameworks` lists). +3. **Glob depth differs per catalog.** `metadata.gather_tests` and `--test-lang` assume `/`. For `algorithms/*/*` the language is the metadata `language` field, not a path segment. Pinned in Task 4 (`test_gather_tests_algorithms_layout`). +4. **Checker false positives on CS today.** `check_catalog.py` must pass on the current CS default branch with only `catalog.json` added (goal 1). Task 2 runs it against a CS checkout in CI. + +## File structure + +| File | Responsibility | Task | +|---|---|---| +| `schema/catalog.schema.json`, `schema/benchmark_config.schema.json`, `schema/implementation.base.schema.json` | the contract as schemas (R-1, R-8, R-10) | 1 | +| `toolset/utils/jsonschema_lite.py` | stdlib validator subset | 1 | +| `tools/check_catalog.py`, `tests/test_check_catalog.py`, `tests/fixtures/catalogs/**` | conformance checker (R-20) | 2 | +| `toolset/utils/results.py`, `tests/test_results.py` | alias normalization, `markerPrefix`, `engine` (R-15, R-16) | 3 | +| `toolset/utils/catalog.py`, `toolset/utils/benchmark_config.py`, `toolset/utils/metadata.py`, `toolset/utils/docker_helper.py`, `toolset/run-tests.py`, `tests/test_catalog.py` | `--catalog`, `--catalog-dir`, manifest loading, implementation discovery (R-2, R-11, R-17) | 4 | +| `toolset/benchmark/test_types/algorithm_type.py`, `toolset/benchmark/test_types/http_type.py`, `toolset/benchmark/suites.py`, `toolset/benchmark/suites.json`, `tests/test_algorithm_type.py` | test types and suites (R-12–R-14, R-18) | 5 | +| `toolset/utils/results.py`, `toolset/utils/merge_results.py` | `catalog` key, merge refusal across catalogs (R-19) | 6 | +| `.gitmodules`, `catalogs/ssg`, `README.md`, `.github/workflows/ci.yml` | submodule, docs, CI (Migration step 2) | 7 | diff --git a/docs/specs/010-catalog-contract/spec.md b/docs/specs/010-catalog-contract/spec.md new file mode 100644 index 0000000..9006a05 --- /dev/null +++ b/docs/specs/010-catalog-contract/spec.md @@ -0,0 +1,152 @@ +# 010 Catalog Contract + +- **Date:** 2026-10-12 +- **Status:** proposed, awaiting review +- **Repo:** `BenchCommons/benchmark-tool` (BT). This repo owns the contract; catalogs reference it and never redefine it (`.product` `CONTRACTS.md`). +- **Issue:** #29 +- **Consumers:** `BenchCommons/catalog-ssg` (CS), `BenchCommons/catalog-algorithms` (CA), `BenchCommons/catalog-web-frameworks` (CW). +- **Related:** BT #30 and `perfratio#240` (measurement engine and its output schema, owned by perfratio), `dataset-tool#1` (input datasets), `catalog-algorithms#11` (tracking), `.product` ADR 0001 (naming) and ADR 0004 (TechEmpower continuation), BT 002 (markers, toolset side), BT 008 (suites, `results.json` v2), CS 001 (canonical `build.sh`), CS 004 (`generator.json`, `generators.json`). + +## Context + +BT measures one catalog, and assumes SSG semantics everywhere: + +| Item | Today | Problem for other catalogs | +|---|---|---| +| Catalog source | one submodule, `frameworks/` → `ssg-frameworks` (now CS) | no way to point at CA or CW | +| Layout | `//` with `.dockerfile`, `build.sh`, `benchmark_config.json`, `generator.json` | CA plans `algorithms///`; CW inherits TechEmpower's `frameworks///` | +| Runner | `build.sh`, byte-identical in every generator, checked by CS `tools/check-build-sh.sh` | the rule is only written down in CS | +| Metadata | `generator.json` + `schema/generator.schema.json`; index `generators.json` built by CS `tools/build_index.py --check` | CA has `meta.json`; no shared base fields | +| Test type | only `datarate` (`numberOfFiles`, `contentSize`) | CA needs input size `n`; CW needs HTTP endpoints | +| Correctness | `SSGBERK_VERIFY_OK/FAIL` (output file count) and `SSGBERK_CONFORMANCE_*`, untimed, before `STARTTIME` (BT 008 R-15) | named after one catalog; HTTP needs the toolset to verify, as TechEmpower's verify stage did | +| Result | hyperfine JSON between `SSGBERK_RESULT_BEGIN/END`, into `results.json` `schemaVersion: 2` `rawData.datarate.` | marker names; engine fixed to hyperfine | + +TechEmpower solved the same problem for web frameworks: a per-implementation `benchmark_config.json` with one entry per test, a Dockerfile per test, and a verify stage that checks every endpoint before any load is applied. BT is derived from that toolset. This spec keeps that shape and writes down, once, what every catalog must provide and what BT does with it. + +## Terms + +- **Catalog:** a repo of implementations measured by BT (`catalog-*`, ADR 0001). +- **Implementation:** one directory in a catalog that BT builds into one image and measures. Its **id** is unique within the catalog. +- **Test type:** what is measured and how it is checked: `datarate`, `algorithm`, `http`. A test type defines its parameters, its verification and its result shape. +- **Cell:** one combination of test type parameters (e.g. `numberOfFiles=1000, contentSize=500`, or `algorithm=nbody, n=5000000`). +- **Engine:** the program that times the runs inside the container: hyperfine today, perfratio after BT #30. + +## Requirements + +### Catalog manifest + +- **R-1** A catalog has `catalog.json` at its root, valid against BT `schema/catalog.schema.json`. Fields: + - `contractVersion` (integer, this spec defines `1`); + - `name` (`ssg`, `algorithms`, `web-frameworks`); + - `testTypes`: the test types the catalog uses, from R-12; + - `implementations`: a glob, relative to the root, that matches every implementation directory (`*/*` for CS, `algorithms/*/*` for CA, `frameworks/*/*` for CW); + - `runner`: `{ "file": "", "canonical": "" }`, the per-implementation runner file name and the reference copy (CS: `build.sh`, `Go/hugo/build.sh`). `file` is omitted for `http`-only catalogs (R-6); + - `metadata`: `{ "file": "", "schema": "" }` (CS: `generator.json`, `schema/generator.schema.json`); + - `index`: `{ "file": "", "key": "", "check": "" }` (CS: `generators.json`, `generators`, `python3 tools/build_index.py --check`). +- **R-2** BT supports a closed range of `contractVersion` values and refuses a catalog outside it, with an error that names both versions. The range starts at `[1, 1]`. + +### Implementation directory + +- **R-3** Every directory matched by `implementations` contains: + - `.dockerfile` (R-4); + - `benchmark_config.json` (R-8); + - the metadata file (R-10); + - the runner file, unless the catalog is `http`-only (R-5, R-6); + - `src/` or other inputs the Dockerfile copies. No file outside the directory is read at build time; the build context is the directory. + + `` is the directory name. It matches `^[a-z0-9][a-z0-9._-]*$` and is unique in the catalog. CA's `algorithms///` uses the id `-` through the metadata `id` field; R-11 checks the two agree. +- **R-4** The Dockerfile is **pinned**: + - the base image names an exact tag (`ubuntu:24.04`, `node:24.11.0-bookworm-slim`), never `latest` or a bare name; a digest (`@sha256:`) is recommended and, when absent, BT records the resolved digest in `results.json` (BT 008, unchanged); + - every downloaded tool, toolchain and engine is installed at an exact version, set by an `ARG _VERSION=` line (CS `hugo.dockerfile`: `HYPERFINE_VERSION`, `HUGO_VERSION`); + - package-manager installs use a lockfile or exact versions; + - the engine (hyperfine 1.20.0 today; perfratio after BT #30) is installed in the image, so timing happens inside the container and never wraps `docker run`; + - the image works on `linux/amd64` and `linux/arm64`, or the metadata says which one it supports (R-10 `platforms`). + +### Runner + +- **R-5** The runner is the only entry point BT calls in a measured container. Every implementation in a catalog carries a **byte-identical** copy of `runner.canonical`; implementation-specific behaviour comes from `benchmark_config.json`, never from editing the runner. The check is the one CS already runs (`cmp` against the canonical copy, `tools/check-build-sh.sh`), generalized in R-20. +- **R-6** `http` implementations have no runner: the image's `CMD` starts the server and BT drives it from outside (R-15). This is the TechEmpower model and the only exception to R-5. +- **R-7** The runner reads its inputs from environment variables set by BT (names in `plan.md`, Contracts): the generic `BENCH_TEST_TYPE`, `BENCH_RUNS`, `BENCH_WARMUP`, `BENCH_VERBOSE`, plus the cell parameters of its test type (R-12). For compatibility, `datarate` also receives today's names (`number_of_files`, `content_size`, `min_runs`, `verbose_build`). + +### Metadata + +- **R-8** `benchmark_config.json` keeps the TechEmpower shape that BT already reads (`toolset/utils/metadata.py`): top-level `framework` (= id) and `tests`, a list of one-key objects `{ "": { ... } }` with `default` required. Each test entry carries: + - the display fields BT already validates (`approach`, `classification`, `framework`, `language`, `display_name`, `notes`, `versus`); + - `test_type` (from the catalog's `testTypes`; absent means `datarate`, for compatibility); + - the test-type keys from R-12. + + CS's top-level `content` and `config` lists stay valid as `datarate` keys (R-12). +- **R-9** BT publishes `schema/benchmark_config.schema.json` and validates every `benchmark_config.json` on load. An invalid file is reported with its path and skipped, not a crash. +- **R-10** The metadata file describes the implementation for people and the website; BT reads only the base fields. BT publishes `schema/implementation.base.schema.json` with the base fields: `id`, `name`, `language`, `version` (string, the measured version), `license`, `homepage`, `repository`, `benchmark.status` (`active`, `wip`, `removed`), optional `platforms` (`["linux/amd64", "linux/arm64"]` by default). A catalog's metadata schema references the base with `allOf` and adds its own fields (CS: `templateEngines`, `contentFormats`, …). CS `generator.json` already has every base field except `version`, which CS writes into `generators.json`; R-11 accepts the version from either place. +- **R-11** The catalog index (`index.file`) lists every implementation: `schemaVersion`, `generatedFrom` (catalog commit) and, under `index.key`, one entry per implementation with at least `id`, `path` (relative to the root), `version` and `benchmark.status`. It is generated by the catalog's own tool, and `index.check` exits non-zero when the index is stale or invalid (CS `build_index.py --check` already does this). BT reads versions from the index (as BT 008 does with `generators.json`) and skips implementations whose status is not `active` unless named with `--test`. + +### Test types + +- **R-12** A test type is a BT plug-in under `toolset/benchmark/test_types/` that declares its cell parameters, the `benchmark_config.json` keys it reads, its verification and its result shape. This spec defines three: + + | Test type | Catalog | Cell parameters | Config keys | Verification (untimed, before any timed run) | Timed work | + |---|---|---|---|---|---| + | `datarate` | CS | `numberOfFiles`, `contentSize` | `content[]`, `config[]` (`build_command`, `output_folder`, `output_glob`, `cache_folders`, …) | one build; runner prints `VERIFY_OK` when the output count equals `numberOfFiles`, plus `CONFORMANCE_*` (CS 006) | cold full rebuild per run (BT 008 R-15) | + | `algorithm` | CA | `algorithm` (id), `n` (input size), optional `dataset` (`@`, dataset-tool#1) | `run_command`, optional `build_command` (compile step, untimed) | one run with the cell's input; runner prints `VERIFY_OK` when the SHA-256 of stdout equals the expected checksum for `(algorithm, n)` from the catalog's algorithm definition (CA 010) | the run command with the same input | + | `http` | CW | `endpoint` (`json`, `plaintext`, `db`, `query`, `fortune`, `update`, `cached-query`), `concurrency`, `durationSeconds` | `port`, `_url` per endpoint (TechEmpower keys) | BT requests each URL and applies TechEmpower's verifier rules (status, headers, body); `VERIFY_OK/FAIL` are written by BT | load generator in a separate container, after a warmup | + +- **R-13** The verification is mandatory and untimed. A cell is timed only after `VERIFY_OK` (and, where the test type has one, `CONFORMANCE_OK` subject to `--require-conformance`, BT 008 decision 5). A missing or failed verification gives status `failed` with reason `verify: ` or `protocol: verification markers missing before STARTTIME` (BT 008 R-15, unchanged). +- **R-14** A test type may declare that an implementation does not support a cell (e.g. an algorithm not implemented in that language, a profile not supported). The runner prints `UNSUPPORTED ` and exits 3; BT records `unsupported.` (BT 008 R-7, generalized). + +### Result markers + +- **R-15** The process that times the work writes the markers to stdout: the runner for `datarate` and `algorithm`; BT's load-generator step for `http`. Markers start at the beginning of a line. The generic set, in order: + + ``` + BENCH_INPUT files= bytes= (optional) + BENCH_VERIFY_OK
| BENCH_VERIFY_FAIL + BENCH_CONFORMANCE_OK|FAIL|PENDING
(optional, test-type defined) + BENCH_OUTPUT files= bytes= (optional) + BENCH_UNSUPPORTED (instead of everything after it; exit 3) + STARTTIME + BENCH_RESULT_BEGIN engine= version= + + BENCH_RESULT_END + ENDTIME + ``` + + `engine=` and `version=` are optional; absent means `hyperfine`. The JSON between the result markers is the engine's export, unchanged (hyperfine `--export-json`; perfratio's versioned schema from perfratio#240). +- **R-16** **Compatibility:** every `SSGBERK_` marker is accepted as an alias of `BENCH_`, and `SSGBERK_PROFILE_UNSUPPORTED` as an alias of `BENCH_UNSUPPORTED`. The parser normalizes both prefixes before applying BT 002 and BT 008 rules, so CS output today parses to the same result as before. Each result records `markerPrefix` (`BENCH` or `SSGBERK`) so the alias's use can be measured before it is removed. Removing the aliases needs a new `contractVersion`. + +### Toolset + +- **R-17** BT runs one catalog per run, chosen by `--catalog ` (default `ssg`) from submodules under `catalogs//`, or by `--catalog-dir ` for a local checkout. `--test`, `--exclude`, `--list-tests` and the suite options work on the selected catalog. +- **R-18** `toolset/benchmark/suites.json` entries gain `catalog` and `testType`, defaulting to `ssg` and `datarate`, and their cells hold the test type's parameters. Existing suites (`smoke`, `standard`, `stress`, `legacy-2019`, `P`, `M`, `G`, `GG`) are unchanged and keep their versions. `--suite` refuses a suite whose `catalog` differs from the selected one. +- **R-19** `results.json` stays `schemaVersion: 2` with one additive key, `catalog: {"name", "commit", "contractVersion"}`, and per-result `markerPrefix` (R-16). `rawData`, `succeeded`, `failed` and `unsupported` are already keyed by test type, so `algorithm` and `http` results go under their own keys. Engine fields (energy, custom metrics) and `schemaVersion: 3` belong to BT #30. +- **R-20** BT ships `tools/check_catalog.py ` (stdlib only), which a catalog's CI runs against a pinned BT tag. It checks R-1 to R-11: manifest valid, every implementation has its files, runner copies are byte-identical, Dockerfiles have no unpinned base image or `latest`, every `benchmark_config.json` and metadata file validates, ids are unique and match the index, and `index.check` passes. It exits non-zero with one line per violation. + +## Goals and success criteria + +1. **CS conforms with only a manifest.** *Measured by:* with a `catalog.json` added to a CS checkout and no other change, `tools/check_catalog.py` exits 0, and `./ssgberk --test hugo -nf 50 -cs 5` produces a `results.json` whose `rawData`, `succeeded`, `failed` and rankings equal those of the same run before this spec (fixture comparison of `parse_build_output`). +2. **Aliases are equivalent.** *Measured by:* a unit test parses each existing `tests/fixtures/raw_*.txt` fixture as is and with `SSGBERK_` replaced by `BENCH_`; both give the same result except `markerPrefix`. +3. **A second catalog runs end to end.** *Measured by:* a fixture catalog in `tests/fixtures/catalogs/algorithms/` (two implementations, one algorithm, two values of `n`) passes `check_catalog.py`, and a unit test drives `algorithm` parsing from recorded runner output into `rawData.algorithm.`, including one `VERIFY_FAIL` (checksum mismatch) recorded as `failed` and one `UNSUPPORTED` recorded under `unsupported.algorithm`. +4. **Violations are caught.** *Measured by:* `check_catalog.py` unit tests, one per rule in R-20, each on a fixture with exactly that violation. +5. **Contract is referenced, not copied.** *Measured by:* CS, CA and CW each have an issue linking to this spec (CS #203, CA #11, CW #1), and `.product` `CONTRACTS.md` changes the row's status from "to specify" to this spec's path. + +## Out of scope + +- The perfratio switch, its JSON parsing and `results.json` `schemaVersion: 3` (BT #30, perfratio#240). +- Renames: the `ssgberk` wrapper and image names, `SSGBERK_HOST_*` launcher variables, docs (issue #29, rename checkbox; separate spec). +- The `http` load generator and verifier implementation (forking TFBVerifier, ADR 0004): this spec fixes only the `http` test type's interface. Its BT implementation is a separate spec after CW 001. +- CA's algorithm set, reference inputs and expected checksums (CA 010) and the dataset input contract (dataset-tool#1). +- Changes inside any catalog; each catalog adopts the contract in its own spec. +- Running several catalogs in one invocation, and cross-catalog rankings. +- The website and published results layout (website#5). + +## Migration and compatibility + +| Step | Where | Change | Breaks anything? | +|---|---|---|---| +| 1 | BT | Schemas, `check_catalog.py`, alias-tolerant parser, `catalog` key in `results.json` | No. With no `catalog.json`, BT treats `frameworks/` as `ssg` with the manifest in `plan.md` (built-in default). | +| 2 | BT | `catalogs/ssg` submodule pointing at `BenchCommons/catalog-ssg`; `frameworks/` stays a submodule at the same commit until step 5 | No. `--catalog-dir frameworks` still works. | +| 3 | CS | Add `catalog.json`, add `version` to `generator.json` or keep it in the index (R-10), run `check_catalog.py` in CI (CS #203) | No. Runner, markers and layout unchanged. | +| 4 | CS | `build.sh` prints `BENCH_*` instead of `SSGBERK_*` (one PR, canonical copy and all copies) | No. BT accepts both (R-16). | +| 5 | BT | Remove `frameworks/` and the built-in default manifest once CS has `catalog.json` on its default branch | Local scripts that mount `frameworks/` must use `--catalog ssg`. | +| 6 | CA, CW | Adopt the contract in their own specs (CA 011–013, CW 001) | — | + +The `SSGBERK_*` aliases stay for all of `contractVersion: 1`. `results.json` readers need no change: the new keys are optional (BT 008 R-28). diff --git a/docs/specs/010-catalog-contract/tasks.md b/docs/specs/010-catalog-contract/tasks.md new file mode 100644 index 0000000..67c36d5 --- /dev/null +++ b/docs/specs/010-catalog-contract/tasks.md @@ -0,0 +1,112 @@ +# 010 Catalog Contract: Tasks + +Read `spec.md` and `plan.md` (Global Constraints, Contracts, Decisions, Review Focus) first. Every task's requirements include the Global Constraints. Run commands from the repository root. Commit with `git commit -S` as configured; never pass `--no-gpg-sign` or `--author`. End every commit message with: + +``` +Co-Authored-By: Claude Opus 5.5 (1M context) +``` + +Test command (no Docker): `python3 -m pytest -q` and `ruff check toolset tests tools`. + +Each task is one PR with green CI. Tasks 1 to 3 are independent; 4 needs 1; 5 needs 3 and 4; 6 needs 4; 7 needs 4. + +--- + +### Task 1: Schemas and the stdlib validator + +**Files:** +- Create: `schema/catalog.schema.json`, `schema/benchmark_config.schema.json`, `schema/implementation.base.schema.json`, `toolset/utils/jsonschema_lite.py` +- Test: `tests/test_schemas.py` + +**Interfaces:** +- Produces: `jsonschema_lite.validate(instance, schema) -> list[str]` (empty when valid; each item `": "`). + +- [ ] **Step 1:** Write failing tests: the CS manifest from `plan.md` validates; a manifest with `contractVersion: "1"`, a missing `implementations` or an unknown test type fails with a pointer to the field; `tests/fixtures/generators.json` entries plus a `version` validate against the base schema; CS `Go/hugo/benchmark_config.json` (copied to `tests/fixtures/catalogs/ssg/Go/hugo/`) validates; the `algorithm` and `http` examples from `plan.md` validate; an entry with `test_type: "algorithm"` and no `run_command` fails. +- [ ] **Step 2:** Implement the validator subset listed in `plan.md` (Tech stack) and the three schemas. `implementation.base.schema.json` leaves `additionalProperties` open so catalog schemas can extend it with `allOf`. +- [ ] **Step 3:** `python3 -m pytest -q tests/test_schemas.py`, ruff, commit `feat(schema): catalog contract schemas (spec 010 R-1, R-8, R-10)`. + +### Task 2: `tools/check_catalog.py` + +**Files:** +- Create: `tools/check_catalog.py`, `tests/test_check_catalog.py`, `tests/fixtures/catalogs/ssg/**` (`catalog.json`, `generators.json`, `schema/generator.schema.json`, `Go/hugo/{hugo.dockerfile,build.sh,benchmark_config.json,generator.json}`, `Go/zola/…`), `tests/fixtures/catalogs/algorithms/**` (two implementations of one algorithm) +- Modify: `.github/workflows/ci.yml` (job `catalog-ssg-contract`: check out `BenchCommons/catalog-ssg` default branch, write the CS manifest from `plan.md` into it, run the checker) + +**Interfaces:** +- Consumes: Task 1 schemas (loaded from `schema/` relative to the script). +- Produces: CLI `python3 tools/check_catalog.py [--skip-index-check]`; exit 0 when clean, 1 with one `: : ` line per violation. Rules are named `manifest`, `files`, `runner`, `dockerfile`, `config`, `metadata`, `ids`, `index` (R-20). + +- [ ] **Step 1:** Write failing tests, one fixture per rule: missing `catalog.json`; implementation without its Dockerfile; runner copy differing by one byte; `FROM ubuntu` (no tag) and `FROM node:latest`; a download without an `ARG *_VERSION`; an invalid `benchmark_config.json`; metadata `id` differing from the directory name (and, for the algorithms layout, from the index entry); a duplicated id; `index.check` exiting 1. Plus: both clean fixtures exit 0. +- [ ] **Step 2:** Implement. `dockerfile` rule: every `FROM` has an explicit tag other than `latest` (or a digest); every `curl`/`wget` URL that contains a version-like segment uses `${…_VERSION}`. `index` rule runs `index.check` with `cwd=` and a 300 s timeout. +- [ ] **Step 3:** Add the CI job (Review Focus 4). Tests, ruff, commit `feat(tools): catalog conformance checker (spec 010 R-20)`. + +### Task 3: Generic markers with `SSGBERK_*` aliases + +**Files:** +- Modify: `toolset/utils/results.py` +- Test: `tests/test_results.py` (append) + +**Interfaces:** +- Produces: `results.normalize_markers(raw: str) -> tuple[str, str]` returning `(text, marker_prefix)`; `parse_build_output` calls it first and adds `markerPrefix`, `engine`, `engineVersion` to every result dict. Constants move to `BENCH_*` names; `PROFILE_UNSUPPORTED` becomes `BENCH_UNSUPPORTED`. + +- [ ] **Step 1:** Write failing tests: + - each of `raw_ok.txt`, `raw_v2_ok.txt`, `raw_conformance_fail.txt`, `raw_unsupported.txt` parses to the same dicts as on `master`, plus `markerPrefix: "SSGBERK"`, `engine: "hyperfine"`, `engineVersion: null` (goal 2); + - the same fixtures with `SSGBERK_` replaced by `BENCH_` give the same dicts with `markerPrefix: "BENCH"`; + - `BENCH_RESULT_BEGIN engine=perfratio version=1.3.0` sets `engine` and `engineVersion`; + - `test_alias_inside_json_untouched`: a hyperfine `command` string containing `SSGBERK_GENERATE_ONLY` keeps it verbatim (Review Focus 1). +- [ ] **Step 2:** Implement with the normalization in `plan.md` (Markers). Every existing regex keeps `_BOL`. +- [ ] **Step 3:** Tests, ruff, commit `feat(results): generic BENCH_ markers, SSGBERK_ aliases (spec 010 R-15, R-16)`. + +### Task 4: Catalog selection and discovery + +**Files:** +- Create: `toolset/utils/catalog.py`, `tests/test_catalog.py` +- Modify: `toolset/run-tests.py` (`--catalog`, `--catalog-dir`), `toolset/utils/benchmark_config.py` (`lang_root` → `catalog.root`), `toolset/utils/metadata.py` (discovery by `catalog.implementations`; language from metadata), `toolset/utils/audit.py`, `toolset/utils/scaffolding.py`, `toolset/github_actions/github_actions_diff.py` + +**Interfaces:** +- Produces: `catalog.load(fw_root: str, name: str | None, directory: str | None) -> Catalog`; `Catalog` has `name`, `root`, `contract_version`, `test_types`, `runner_file`, `metadata_file`, `index_path`, `index_key`, `implementation_dirs() -> list[str]`, `index_entry(id) -> dict | None`, `commit() -> str | None`. `SUPPORTED_CONTRACT = (1, 1)`. + +- [ ] **Step 1:** Write failing tests: + - no flags and no `catalog.json` in `frameworks/` → built-in CS manifest, root `frameworks/`; + - `--catalog ssg` with `catalogs/ssg/catalog.json` present → that manifest; + - `--catalog-dir tests/fixtures/catalogs/algorithms` → algorithms manifest; + - `contractVersion: 2` → error naming `2` and the supported range (R-2); + - `test_gather_tests_algorithms_layout`: `metadata.gather_tests` finds both fixture implementations and `--test-lang Python` filters by the metadata `language` (Review Focus 3); + - status `removed` in the index is skipped unless named with `--test` (R-11). +- [ ] **Step 2:** Implement; then `grep -rn "frameworks" toolset` per Review Focus 2. +- [ ] **Step 3:** Existing suite passes unchanged (`python3 -m pytest -q`). Commit `feat(toolset): select a catalog with --catalog/--catalog-dir (spec 010 R-17)`. + +### Task 5: `algorithm` test type, `http` declaration, suites per catalog + +**Files:** +- Create: `toolset/benchmark/test_types/algorithm_type.py`, `toolset/benchmark/test_types/http_type.py`, `tests/test_algorithm_type.py`, `tests/fixtures/raw_algorithm_ok.txt`, `tests/fixtures/raw_algorithm_verify_fail.txt`, `tests/fixtures/raw_algorithm_unsupported.txt` +- Modify: `toolset/benchmark/test_types/__init__.py`, `toolset/benchmark/suites.py`, `toolset/benchmark/suites.json` (add `algo-smoke` from `plan.md`), `toolset/utils/docker_helper.py` (pass the runner environment from `plan.md`) + +**Interfaces:** +- Produces: each test type class exposes `PARAMETERS` (cell keys with types), `CONFIG_KEYS`, `env(cell) -> dict[str, str]` and `parse(raw) -> list[dict]`. `DataRateType.env` returns both `BENCH_*` and legacy names. `HttpType` raises `NotImplementedError("http: see spec 010, Out of scope")` from `env` and `parse`. + +- [ ] **Step 1:** Write failing tests: + - `DataRateType.env({"numberOfFiles": 100, "contentSize": "0.500"})` contains `number_of_files=100`, `BENCH_NUMBER_OF_FILES=100`, `content_size=0.500`, `BENCH_CONTENT_SIZE=0.500`; + - `AlgorithmType.parse` of the three fixtures gives one result with `algorithm`, `n` and the timing keys; `failed` with reason `verify: BENCH_VERIFY_FAIL sha256 …`; `unsupported` (goal 3); + - `suites.load` accepts `algo-smoke`, rejects a cell with `numberOfFiles` in an `algorithm` suite, and defaults old suites to `ssg`/`datarate`; + - `--suite algo-smoke` with `--catalog ssg` is refused (R-18); + - selecting a catalog whose `testTypes` is `["http"]` exits with the `NotImplementedError` message, not a traceback. +- [ ] **Step 2:** Implement. +- [ ] **Step 3:** Tests, ruff, commit `feat(test-types): algorithm test type and per-catalog suites (spec 010 R-12–R-14, R-18)`. + +### Task 6: `catalog` in `results.json` + +**Files:** +- Modify: `toolset/utils/results.py`, `toolset/utils/merge_results.py` +- Test: `tests/test_results.py`, `tests/test_merge_results.py` (append) + +- [ ] **Step 1:** Write failing tests: a run writes `catalog: {"name", "commit", "contractVersion"}`; `--parse` of an old run without it keeps it absent; `merge()` of two `ssg` inputs keeps `catalog`; `merge()` of `ssg` and `algorithms` inputs raises with both names (Decision 5); `schemaVersion` stays `2`. +- [ ] **Step 2:** Implement. Commit `feat(results): record the catalog (spec 010 R-19)`. + +### Task 7: Submodule, README, CONTRACTS link + +**Files:** +- Modify: `.gitmodules` (add `catalogs/ssg` → `https://github.com/BenchCommons/catalog-ssg`, same commit as `frameworks/`), `README.md` (catalog selection, link to this spec as the contract), `ssgberk` (no flag change; document `--catalog-dir`) + +- [ ] **Step 1:** `git submodule add https://github.com/BenchCommons/catalog-ssg catalogs/ssg` and check out the commit `frameworks/` points at. +- [ ] **Step 2:** README section "Catalogs" with the `catalog.json` example and the marker table from `spec.md`. +- [ ] **Step 3:** Commit `chore: add catalogs/ssg submodule and document catalogs (spec 010)`. After merge, open a PR in `BenchCommons/.product` changing the `CONTRACTS.md` catalog row's status to `docs/specs/010-catalog-contract` (goal 5), and comment on CS #203, CA #11 and CW #1 with the link.