Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions docs/specs/010-catalog-contract/plan.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,9 +50,9 @@ CA (illustrative; CA 011 decides the real paths):
"contractVersion": 1,
"name": "algorithms",
"testTypes": ["algorithm"],
"implementations": "algorithms/*/*",
"implementations": "languages/*/*",
"runner": {"file": "run.sh", "canonical": "tools/run.sh"},
"metadata": {"file": "implementation.json", "schema": "schema/implementation.schema.json"},
"metadata": {"file": "meta.json", "schema": "schema/implementation.schema.json"},
"index": {"file": "implementations.json", "key": "implementations", "check": "python3 tools/build_index.py --check"}
}
```
Expand Down Expand Up @@ -153,7 +153,7 @@ Cell keys are validated against the test type's declared parameters.

## Decisions

1. **A manifest file, not a convention.** Layouts already differ (`*/*`, `algorithms/*/*`, `frameworks/*/*`) and the runner and metadata file names differ. One small `catalog.json` states them; BT keeps a built-in CS manifest only until CS adds its own (spec, Migration step 5).
1. **A manifest file, not a convention.** Layouts already differ (`*/*`, `languages/*/*`, `frameworks/*/*`) and the runner and metadata file names differ. One small `catalog.json` states them; BT keeps a built-in CS manifest only until CS adds its own (spec, Migration step 5).
2. **Byte-identical runner copies, not a shared file.** The Docker build context is the implementation directory (R-3), so a shared script cannot be `COPY`ed without widening every context. CS already works this way and checks it with `cmp`.
3. **Prefix `BENCH_`.** Short, catalog-neutral, and not an org name that may change again. `SSGBERK_` stays as an alias for the whole `contractVersion: 1`.
4. **`http` has no runner.** A server cannot time itself; the load generator must run in another container, as in TechEmpower. Verification for `http` is BT's job, so the same `VERIFY_*` markers are written by BT and parsed by the same code.
Expand All @@ -165,7 +165,7 @@ Cell keys are validated against the test type's declared parameters.

1. **Alias normalization changes a CS result.** The rewrite must touch only line-initial `SSGBERK_`, never JSON content between the result markers (a generator name or path containing `SSGBERK_` inside the hyperfine JSON command string). Pinned in Task 3 (`test_alias_inside_json_untouched`).
2. **`FWROOT/frameworks` is assumed in several places** (`benchmark_config.py` `lang_root`, `metadata.py`, `audit.py`, `github_actions_diff.py`, `scaffolding.py`, `run-tests.py`). Task 4 replaces every path use with `catalog.root`; afterwards `grep -rn "frameworks" toolset` shows no path built from it except the built-in default manifest (the word may remain in log text and results keys such as `frameworks` lists).
3. **Glob depth differs per catalog.** `metadata.gather_tests` and `--test-lang` assume `<Lang>/<id>`. For `algorithms/*/*` the language is the metadata `language` field, not a path segment. Pinned in Task 4 (`test_gather_tests_algorithms_layout`).
3. **Glob depth differs per catalog.** `metadata.gather_tests` and `--test-lang` assume `<Lang>/<id>`. For `languages/*/*` the language is the parent path segment and the algorithm the leaf (CA spec 011); language-level metadata lives in `languages/<lang>/language.json`. Pinned in Task 4 (`test_gather_tests_algorithms_layout`).
4. **Checker false positives on CS today.** `check_catalog.py` must pass on the current CS default branch with only `catalog.json` added (goal 1). Task 2 runs it against a CS checkout in CI.

## File structure
Expand Down
6 changes: 3 additions & 3 deletions docs/specs/010-catalog-contract/spec.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ BT measures one catalog, and assumes SSG semantics everywhere:
| Item | Today | Problem for other catalogs |
|---|---|---|
| Catalog source | one submodule, `frameworks/` → `ssg-frameworks` (now CS) | no way to point at CA or CW |
| Layout | `<Lang>/<id>/` with `<id>.dockerfile`, `build.sh`, `benchmark_config.json`, `generator.json` | CA plans `algorithms/<algo>/<lang>/`; CW inherits TechEmpower's `frameworks/<Lang>/<id>/` |
| Layout | `<Lang>/<id>/` with `<id>.dockerfile`, `build.sh`, `benchmark_config.json`, `generator.json` | CA uses `languages/<lang>/<algorithm>/` (CA spec 011); CW inherits TechEmpower's `frameworks/<Lang>/<id>/` |
| Runner | `build.sh`, byte-identical in every generator, checked by CS `tools/check-build-sh.sh` | the rule is only written down in CS |
| Metadata | `generator.json` + `schema/generator.schema.json`; index `generators.json` built by CS `tools/build_index.py --check` | CA has `meta.json`; no shared base fields |
| Test type | only `datarate` (`numberOfFiles`, `contentSize`) | CA needs input size `n`; CW needs HTTP endpoints |
Expand All @@ -39,7 +39,7 @@ TechEmpower solved the same problem for web frameworks: a per-implementation `be
- `contractVersion` (integer, this spec defines `1`);
- `name` (`ssg`, `algorithms`, `web-frameworks`);
- `testTypes`: the test types the catalog uses, from R-12;
- `implementations`: a glob, relative to the root, that matches every implementation directory (`*/*` for CS, `algorithms/*/*` for CA, `frameworks/*/*` for CW);
- `implementations`: a glob, relative to the root, that matches every implementation directory (`*/*` for CS, `languages/*/*` for CA, `frameworks/*/*` for CW);
- `runner`: `{ "file": "<name>", "canonical": "<path>" }`, the per-implementation runner file name and the reference copy (CS: `build.sh`, `Go/hugo/build.sh`). `file` is omitted for `http`-only catalogs (R-6);
- `metadata`: `{ "file": "<name>", "schema": "<path>" }` (CS: `generator.json`, `schema/generator.schema.json`);
- `index`: `{ "file": "<path>", "key": "<array key>", "check": "<command>" }` (CS: `generators.json`, `generators`, `python3 tools/build_index.py --check`).
Expand All @@ -54,7 +54,7 @@ TechEmpower solved the same problem for web frameworks: a per-implementation `be
- the runner file, unless the catalog is `http`-only (R-5, R-6);
- `src/` or other inputs the Dockerfile copies. No file outside the directory is read at build time; the build context is the directory.

`<id>` is the directory name. It matches `^[a-z0-9][a-z0-9._-]*$` and is unique in the catalog. CA's `algorithms/<algo>/<lang>/` uses the id `<algo>-<lang>` through the metadata `id` field; R-11 checks the two agree.
`<id>` is the directory name. It matches `^[a-z0-9][a-z0-9._-]*$` and is unique in the catalog. In CA's `languages/<lang>/<algorithm>/` the leaf name repeats across languages (every language has `hello-world`), so CA's id is `<lang>.<algorithm>` (e.g. `python.hello-world`) through the metadata `id` field; R-11 checks it equals the parent and leaf directory names joined by a dot.
- **R-4** The Dockerfile is **pinned**:
- the base image names an exact tag (`ubuntu:24.04`, `node:24.11.0-bookworm-slim`), never `latest` or a bare name; a digest (`@sha256:`) is recommended and, when absent, BT records the resolved digest in `results.json` (BT 008, unchanged);
- every downloaded tool, toolchain and engine is installed at an exact version, set by an `ARG <NAME>_VERSION=` line (CS `hugo.dockerfile`: `HYPERFINE_VERSION`, `HUGO_VERSION`);
Expand Down
Loading