From 52f4913feafaa1a7ec08ce870edb84faba81b912 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Matheus=20Bregu=C3=AAz?= Date: Mon, 12 Oct 2026 15:10:11 -0300 Subject: [PATCH] docs(spec): catalog-algorithms uses languages/// The contract examples assumed algorithms///, which CA spec 011 replaced: language first, one folder per algorithm. Updates the glob, id rule (.) and language derivation. Refs #29 Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/specs/010-catalog-contract/plan.md | 8 ++++---- docs/specs/010-catalog-contract/spec.md | 6 +++--- 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/docs/specs/010-catalog-contract/plan.md b/docs/specs/010-catalog-contract/plan.md index a0b89c0..ae6bfd5 100644 --- a/docs/specs/010-catalog-contract/plan.md +++ b/docs/specs/010-catalog-contract/plan.md @@ -50,9 +50,9 @@ CA (illustrative; CA 011 decides the real paths): "contractVersion": 1, "name": "algorithms", "testTypes": ["algorithm"], - "implementations": "algorithms/*/*", + "implementations": "languages/*/*", "runner": {"file": "run.sh", "canonical": "tools/run.sh"}, - "metadata": {"file": "implementation.json", "schema": "schema/implementation.schema.json"}, + "metadata": {"file": "meta.json", "schema": "schema/implementation.schema.json"}, "index": {"file": "implementations.json", "key": "implementations", "check": "python3 tools/build_index.py --check"} } ``` @@ -153,7 +153,7 @@ Cell keys are validated against the test type's declared parameters. ## Decisions -1. **A manifest file, not a convention.** Layouts already differ (`*/*`, `algorithms/*/*`, `frameworks/*/*`) and the runner and metadata file names differ. One small `catalog.json` states them; BT keeps a built-in CS manifest only until CS adds its own (spec, Migration step 5). +1. **A manifest file, not a convention.** Layouts already differ (`*/*`, `languages/*/*`, `frameworks/*/*`) and the runner and metadata file names differ. One small `catalog.json` states them; BT keeps a built-in CS manifest only until CS adds its own (spec, Migration step 5). 2. **Byte-identical runner copies, not a shared file.** The Docker build context is the implementation directory (R-3), so a shared script cannot be `COPY`ed without widening every context. CS already works this way and checks it with `cmp`. 3. **Prefix `BENCH_`.** Short, catalog-neutral, and not an org name that may change again. `SSGBERK_` stays as an alias for the whole `contractVersion: 1`. 4. **`http` has no runner.** A server cannot time itself; the load generator must run in another container, as in TechEmpower. Verification for `http` is BT's job, so the same `VERIFY_*` markers are written by BT and parsed by the same code. @@ -165,7 +165,7 @@ Cell keys are validated against the test type's declared parameters. 1. **Alias normalization changes a CS result.** The rewrite must touch only line-initial `SSGBERK_`, never JSON content between the result markers (a generator name or path containing `SSGBERK_` inside the hyperfine JSON command string). Pinned in Task 3 (`test_alias_inside_json_untouched`). 2. **`FWROOT/frameworks` is assumed in several places** (`benchmark_config.py` `lang_root`, `metadata.py`, `audit.py`, `github_actions_diff.py`, `scaffolding.py`, `run-tests.py`). Task 4 replaces every path use with `catalog.root`; afterwards `grep -rn "frameworks" toolset` shows no path built from it except the built-in default manifest (the word may remain in log text and results keys such as `frameworks` lists). -3. **Glob depth differs per catalog.** `metadata.gather_tests` and `--test-lang` assume `/`. For `algorithms/*/*` the language is the metadata `language` field, not a path segment. Pinned in Task 4 (`test_gather_tests_algorithms_layout`). +3. **Glob depth differs per catalog.** `metadata.gather_tests` and `--test-lang` assume `/`. For `languages/*/*` the language is the parent path segment and the algorithm the leaf (CA spec 011); language-level metadata lives in `languages//language.json`. Pinned in Task 4 (`test_gather_tests_algorithms_layout`). 4. **Checker false positives on CS today.** `check_catalog.py` must pass on the current CS default branch with only `catalog.json` added (goal 1). Task 2 runs it against a CS checkout in CI. ## File structure diff --git a/docs/specs/010-catalog-contract/spec.md b/docs/specs/010-catalog-contract/spec.md index 9006a05..fd5747c 100644 --- a/docs/specs/010-catalog-contract/spec.md +++ b/docs/specs/010-catalog-contract/spec.md @@ -14,7 +14,7 @@ BT measures one catalog, and assumes SSG semantics everywhere: | Item | Today | Problem for other catalogs | |---|---|---| | Catalog source | one submodule, `frameworks/` → `ssg-frameworks` (now CS) | no way to point at CA or CW | -| Layout | `//` with `.dockerfile`, `build.sh`, `benchmark_config.json`, `generator.json` | CA plans `algorithms///`; CW inherits TechEmpower's `frameworks///` | +| Layout | `//` with `.dockerfile`, `build.sh`, `benchmark_config.json`, `generator.json` | CA uses `languages///` (CA spec 011); CW inherits TechEmpower's `frameworks///` | | Runner | `build.sh`, byte-identical in every generator, checked by CS `tools/check-build-sh.sh` | the rule is only written down in CS | | Metadata | `generator.json` + `schema/generator.schema.json`; index `generators.json` built by CS `tools/build_index.py --check` | CA has `meta.json`; no shared base fields | | Test type | only `datarate` (`numberOfFiles`, `contentSize`) | CA needs input size `n`; CW needs HTTP endpoints | @@ -39,7 +39,7 @@ TechEmpower solved the same problem for web frameworks: a per-implementation `be - `contractVersion` (integer, this spec defines `1`); - `name` (`ssg`, `algorithms`, `web-frameworks`); - `testTypes`: the test types the catalog uses, from R-12; - - `implementations`: a glob, relative to the root, that matches every implementation directory (`*/*` for CS, `algorithms/*/*` for CA, `frameworks/*/*` for CW); + - `implementations`: a glob, relative to the root, that matches every implementation directory (`*/*` for CS, `languages/*/*` for CA, `frameworks/*/*` for CW); - `runner`: `{ "file": "", "canonical": "" }`, the per-implementation runner file name and the reference copy (CS: `build.sh`, `Go/hugo/build.sh`). `file` is omitted for `http`-only catalogs (R-6); - `metadata`: `{ "file": "", "schema": "" }` (CS: `generator.json`, `schema/generator.schema.json`); - `index`: `{ "file": "", "key": "", "check": "" }` (CS: `generators.json`, `generators`, `python3 tools/build_index.py --check`). @@ -54,7 +54,7 @@ TechEmpower solved the same problem for web frameworks: a per-implementation `be - the runner file, unless the catalog is `http`-only (R-5, R-6); - `src/` or other inputs the Dockerfile copies. No file outside the directory is read at build time; the build context is the directory. - `` is the directory name. It matches `^[a-z0-9][a-z0-9._-]*$` and is unique in the catalog. CA's `algorithms///` uses the id `-` through the metadata `id` field; R-11 checks the two agree. + `` is the directory name. It matches `^[a-z0-9][a-z0-9._-]*$` and is unique in the catalog. In CA's `languages///` the leaf name repeats across languages (every language has `hello-world`), so CA's id is `.` (e.g. `python.hello-world`) through the metadata `id` field; R-11 checks it equals the parent and leaf directory names joined by a dot. - **R-4** The Dockerfile is **pinned**: - the base image names an exact tag (`ubuntu:24.04`, `node:24.11.0-bookworm-slim`), never `latest` or a bare name; a digest (`@sha256:`) is recommended and, when absent, BT records the resolved digest in `results.json` (BT 008, unchanged); - every downloaded tool, toolchain and engine is installed at an exact version, set by an `ARG _VERSION=` line (CS `hugo.dockerfile`: `HYPERFINE_VERSION`, `HUGO_VERSION`);