[matbrgz]

What changed

Spec 004, the source toolkit: the generic parts of dataset-names that download, read and register upstream sources move here as the installable package benchcommons-tools-dataset (import datasettool), per the owner decision of 2026-10-12.

• Spec, plan and tasks: docs/specs/004-source-toolkit/; roadmap row 4 (no dependency on 001).
• ADRs: 0003 package name, 0004 dataset repos depend on a uv git source pinned to a tag, 0005 runtime dependencies (pyyaml==6.0.3, xlrd==2.0.2), 0006 accepted licences per dataset (links .product ADR 0003), 0007 the toolkit lives in tools-dataset rather than tools-commons. 0001 and 0002 stay reserved for spec 001's T1.
• datasettool.sources.cache, .spreadsheet, .adapters, .registry and datasettool.files, each ported with its tests (origin in each commit message: ported from BenchCommons/dataset-names@e7e8405 ...; the spreadsheet DTD and encoding guards from dataset-names@eaaa4d1, PR #39).
• Apache-2.0 LICENSE, pyproject.toml, uv.lock, AGENTS.md and README.md with stack and commands.
• CI keeps the org docs job and adds tests (uv sync --locked, uv run pytest -q) and install-from-git (fresh uv project installs the commit under test from this repo's git URL and imports every module).
• Spec 001 T2 (skeleton) now points to spec 004; repo URLs follow the rename to tools-dataset.

Acceptance criteria and evidence

1. Ported tests pass with unchanged vectors: 360 tests pass locally (uv run pytest -q). Port map in plan section 3; not ported (names-specific) is listed there: the INE sample test, RawRecord parse, the names default adapter package, pipeline tests.
2. No network: an autouse fixture in tests/conftest.py refuses every non-loopback connect and name lookup in every test (tests/test_no_network.py proves it).
3. A registry schema built only from the public API (tests/names_registry.py) passes every test of dataset-names' test_sources.py and test_sources_extensoes.py.
4. Installable from git: in an empty uv init --bare project, uv add "benchcommons-tools-dataset @ git+https://github.com/BenchCommons/tools-dataset@41e563c" installs and every module imports; the ADR 0004 form ([tool.uv.sources] git source) also syncs (checked with the branch, since the tag comes after merge). The CI job repeats this on the PR head.
5. CI: see the checks on this PR.
6. Adoption by dataset-names is tracked in dataset-names adopts the source toolkit (spec 004) #5.

Review fix round

• Path traversal: cached_path/fetch_resource now refuse a source id outside the registry id format, an unsafe file name, and any path resolving outside the cache (ValueError), with tests (.., /etc/passwd, symlinked source directory).
• A str given for accepted_licenses, fields, checks or optional is refused (TypeError); collections are coerced to frozenset/tuples (a str licence list made "BY" pass for "CC-BY-4.0").
• Damaged ZIP members (bad CRC, bad deflate data, truncation) become FormatError, with tests.
• The default Source is hashable (extra left out of the hash).
• hatchling==1.32.4 pinned; CI runs on v* tags and fails when the tag differs from the package version.
• Plan 2.4 states the real registry error order and its drift from dataset-names (noted in dataset-names adopts the source toolkit (spec 004) #5); placement decision split into ADR 0007; spec 001's plan uses a placeholder tag (v0.1.0 has no CLI); the test-only example adapter is in English; the spreadsheet docstring names both origins (e7e8405 and eaaa4d1).
• Left open as issues: cache hardening Cache hardening: size limit, total timeout, redirect rules, concurrent .part files #8; the CONTRACTS.md row in CONTRACTS.md: add the source toolkit contract (tools-dataset spec 004) .product#4. Comment on tools-commons#1 points to the existing cache/retry code.

Release step (maintainer, after merge)

git tag -s v0.1.0 <merge sha> -m "benchcommons-tools-dataset 0.1.0 (spec 004)"
git push origin v0.1.0
uv init --bare --python 3.12 && uv add "benchcommons-tools-dataset @ git+https://github.com/BenchCommons/tools-dataset@v0.1.0"
uv run python -c "import datasettool.sources.cache, datasettool.sources.spreadsheet, datasettool.sources.adapters, datasettool.sources.registry, datasettool.files"

Decisions taken alone (cost if wrong)

• Distribution benchcommons-tools-dataset after the repo rename, import datasettool kept from spec 001's plan. Cost: renaming after v0.1.0 breaks every consumer's imports and lock file; before the tag it is a find-and-replace.
• The toolkit stays in tools-dataset, not tools-commons (ADR 0007): its consumers are the dataset repos. tools-commons#1 lists "retry wrappers for flaky downloads" as a candidate, which overlaps with the cache. Cost: if tools-benchmark needs the cache, it moves to tools-commons by a spec there and this package depends on it.
• Accepted licences are per dataset, no implicit default; ATTRIBUTION_ONLY_LICENSES is a shared starting set without statcan-open-licence (dataset-names adds it). Cost: if .product ADR 0003 sets an org-wide list, it becomes one more constant.
• Messages and identifiers translated to English (repo convention); vectors unchanged. Cost: dataset-names' assertions on Portuguese messages change on adoption (dataset-names adopts the source toolkit (spec 004) #5).
• Registry extension by Field list, cross-entry checks and a factory rather than subclassing or JSON Schema. Cost: an API change in a 0.x minor release.
• describe_files refuses files with no registered row counter instead of counting them as CSV (dataset-names' fallback). Cost: none for dataset-names, which only has CSV and Parquet.
• INE spreadsheet samples (CC BY 4.0 data) stay in dataset-names; only the synthetic types.xls moved. Cost: one real-file test less here.
• Apache-2.0 LICENSE added; the owner confirmed tools-dataset keeps Apache-2.0.
• ADR numbers 0003 to 0006 leave 0001 and 0002 to spec 001's T1.

Closes #4

🤖 Generated with Claude Code