From 5084800411222be065fe7a5d327b0075fef8e50d Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Fri, 5 Jun 2026 13:45:48 -0700 Subject: [PATCH 01/57] Add test-suite-and-validation ADR suggestion --- .../suggestions/test-suite-and-validation.md | 437 ++++++++++++++++++ 1 file changed, 437 insertions(+) create mode 100644 docs/dev/adrs/suggestions/test-suite-and-validation.md diff --git a/docs/dev/adrs/suggestions/test-suite-and-validation.md b/docs/dev/adrs/suggestions/test-suite-and-validation.md new file mode 100644 index 000000000..173d4c63f --- /dev/null +++ b/docs/dev/adrs/suggestions/test-suite-and-validation.md @@ -0,0 +1,437 @@ +# ADR: Test Suite and Validation Strategy + +## Status + +Proposed. + +## Date + +2026-06-05 + +## Group + +Quality. + +## Context + +EasyDiffraction now has five test layers — unit, functional, +integration, script, and notebook — plus a tutorial-output regression +check. The layers were established by +[Test Strategy](../accepted/test-strategy.md), which defines each in a +single line and states that the unit tree mirrors the source tree +"where practical." + +That high-level statement is no longer enough. Concrete problems have +accumulated: + +- **Placement is under-specified and already violated.** The only + discriminator between functional and integration is "without heavy + external dependencies" vs "real calculation engines and data," yet + functional tests perform real network `download_data()`. Some unit + tests are slow (parametrised sampler/plotting/display cases) and some + call `download_data()` (mocked, but undocumented). There is no written + rule an author can apply to a borderline test. +- **Codecov patch status is always red.** `.codecov.yml` runs a + blocking `patch: target: auto` against a **unit-only** coverage + baseline (`coverage.yml` uploads only `coverage-unit.xml`). Any pull + request that touches code exercised mainly by functional, integration, + or script tests scores near-zero patch coverage and fails. `test.yml` + uploads no coverage at all, so most feature-branch pull requests grade + against a stale develop baseline. See + [discussion #69](https://github.com/orgs/easyscience/discussions/69). +- **Coverage is line-only and unenforced.** `fail_under = 65` is checked + locally but never gated in CI, and line coverage says nothing about + input-domain coverage (negative/zero/non-numeric inputs to numeric + code, out-of-range crystallographic values, etc.). +- **The mirrored structure is enforced by a script that CI never runs.** + `tools/test_structure_check.py` validates the `src` ↔ `tests/unit` + mirror (209/209 modules today) but is not wired into any workflow, so + drift can land. +- **No cross-engine numerical validation and no place to show it.** The + documentation has no section comparing calculated patterns or refined + parameters across calculation engines (`cryspy`, `crysfml`, `pdffit`) + or against external software (FullProf, GSAS-II). Engines are keyed + only by `scattering_type`, so there is no declared matrix of which + `beam_mode × radiation_probe` each engine supports. +- **No performance-regression control.** `tools/benchmark_tutorials.py` + records local-only, whole-tutorial wall-clock CSVs with no baseline, + no per-experiment granularity, and no gate. +- **No broad robustness check against real-world files.** Nothing + exercises EasyDiffraction against a large, varied corpus of CIF files + to catch parsing/recognition failures before users hit them. +- **Documentation drift is not caught on every push.** `docs.yml` + executes all tutorials and then builds and deploys the site; it is slow + and therefore runs on pull requests only. There is no fast, + every-push check that the site builds strictly, links resolve, and + prose is clean. This overlaps the unimplemented + [Documentation CI and Build Verification](documentation-ci-build.md) + suggestion. + +This ADR amends [Test Strategy](../accepted/test-strategy.md): the +five-layer decomposition stands, but its definitions become strict and +testable, and the strategy is extended to cover test cost tiers, +coverage policy, codecov configuration, cross-engine verification +documentation, performance benchmarks, a nightly validation harness, and +a fast documentation-build gate. It deliberately combines these into one +document because they are one coherent quality story with shared +infrastructure (markers, the data repository, CI triggers); large +sub-areas are explicitly phased and several are documented now but +implemented in follow-up pull requests. + +## Decision + +### 1. Strict layer definitions and placement criteria + +Replace the one-line definitions with observable, testable rules. A test +belongs to the **lowest** layer whose constraints it can satisfy. + +| Layer | May use | Must NOT use | Speed | +| --- | --- | --- | --- | +| **unit** | one module under test; in-process logic; `tmp_path` | real calculation engine; network / `download_data()`; filesystem outside `tmp_path`; `sleep`; subprocess | sub-second | +| **functional** | several modules / a workflow; small bundled fixtures | real calculation engine; **network / `download_data()`** | seconds | +| **integration** | real engines, real fits, real downloaded data; the only layer allowed network and real backends | — | slow; xdist | +| **script** | full tutorial `.py` executed subprocess-isolated | (already correct) | slow; xdist | +| **notebook** | generated `.ipynb` executed via `nbmake` | — | slow | + +Mocking a forbidden dependency (for example a mocked `download_data()`) +keeps a test in a lower layer **only when the mock is explicit**; an +implicit or accidental real call is a layer violation. + +Consequences for the current suite: + +- The functional tests that call real `download_data()` move to + integration (the hard line: functional is in-process with bundled + fixtures, never network). +- A one-time relocation pass moves slow, engine-adjacent, or + network-touching "unit" tests to their correct layer, and tags the + remainder per §2. +- The ADR ships a short "where does this test go?" decision list so + authors do not re-derive the boundary. + +### 2. Test cost tiers via opt-in escalation markers + +Cost tiers are **orthogonal** to layers. The default is fast; expensive +tests opt *into* a heavier tier, so only the minority are tagged. + +- **default (unmarked):** fast. Runs on every push, every pull request, + and nightly. +- **`@pytest.mark.pr`:** heavier. Runs on pull requests and on + `develop`/`master`, skipped on intermediate feature-branch pushes. +- **`@pytest.mark.nightly`:** very expensive (the §8 corpus harness, + generative fuzzing, full cross-engine sweeps, full benchmarks). Runs + on the scheduled nightly job and on demand; never on ordinary pushes. + +CI marker selection: + +```text +feature-branch push: -m "not pr and not nightly" +pull request + main: -m "not nightly" +nightly schedule: (all markers, including -m nightly) +``` + +The current `fast` marker (which today selects a *cheap subset* and is +applied to six integration files) is **retired**; its intent is inverted +into the scheme above. Markers are registered in +`[tool.pytest.ini_options].markers`. + +### 3. Mirrored unit structure as a CI gate + +`tools/test_structure_check.py` remains the canonical enforcer of the +`src/easydiffraction//.py` → `tests/unit/easydiffraction//test_.py` +mirror, including its three match strategies (direct mirror, known +aliases such as `singleton → singletons` and `variable → parameters`, +and parent-level roll-up for `default.py`/`factory.py` category +packages). It is added to CI (the lint/format or test workflow) as a +fast, static gate so structural drift fails before merge. + +The check should be driven by a **single source-of-truth enumeration of +the `src/` tree**. `tools/generate_package_docs.py` already walks that +tree (`build_tree()`) to generate `docs/dev/package-structure/short.md` +and `full.md` (regenerated by `pixi run fix`). Today +`test_structure_check.py` walks `src/` independently, so the two can +drift. Reuse or adapt the existing tree-walk so structure generation and +the mirror check share one enumeration; the check fails CI and local +runs on any discrepancy between `src/` and `tests/unit/`. The related +scaffold generator `tools/gen_tests_scaffold.py` stays the way authors +create the mirrored test file for a new module. + +### 4. Coverage policy: line/branch and input-domain + +Two distinct bars, because line coverage and case coverage are different +guarantees. + +- **Line/branch coverage.** Raise `fail_under` from 65 to **80 now**, + with a documented ramp toward **90–95** as the suite fills, and gate it + in CI through the codecov project status (§5) rather than only locally. +- **Validators are the input boundary.** All user input is verified at + runtime through the project's custom validator framework in + `src/easydiffraction/core/validation.py`: an `AttributeSpec` pairs a + `TypeValidator` (data type) with a content `ValidatorBase` subclass + (membership, range, and similar), and parameter (`core/variable.py`) + and category (`core/category.py`) classes route writes through it. + Input-domain tests therefore target the **validators directly** — both + that they accept the full valid domain and that they reject (or fall + back, per their contract) on invalid values — rather than re-checking + the same boundaries at every call site. This matches the project + principle of explicit handling at the boundary and no defensive padding + past it. +- **Input-domain coverage.** Adopt **property-based testing with + `hypothesis`** for the validator-guarded numeric and crystallographic + inputs: cell lengths (> 0), cell angles (valid ranges and lattice + constraints), fractional coordinates, site occupancies ∈ [0, 1], + ADP/`Biso` positivity and ranges, space-group numbers (1–230), + wavelengths (> 0), plus rejection of wrong-typed input + (int/float/str). `hypothesis` runs in a **deterministic profile** + (`derandomize`, fixed seed, no committed `.hypothesis` database) to + honour the no-flakiness and no-ordering-dependence rules. + Known-critical boundary cases are also written as **explicit + parametrised tables** so they are visible and named; `hypothesis` adds + generative exploration on top. +- **Numeric tolerance convention.** Replace the scattered mix of + `pytest.approx`, `np.testing.assert_allclose`, and + `assert_almost_equal(decimal=...)` with one documented intra-engine + `rtol`/`atol` pair and one cross-engine pair, defined once (a root + `tests/conftest.py` fixture) and referenced everywhere. + +### 5. Codecov policy + +Adopt the recommendation from +[discussion #69](https://github.com/orgs/easyscience/discussions/69): + +- **Upload unit-test coverage only** (keep the single, fast, reliable + source; functional/integration coverage stays out of codecov). +- **`project` status: target 80%, blocking** (`informational: false`) — + this becomes the real coverage gate. +- **`patch` status: `informational: true`** (non-blocking) — stops the + always-red patch failures, which were an artefact of grading + diff lines against a unit-only baseline. +- **Upload unit coverage from `test.yml` on every pull request** (not + only from `coverage.yml` on develop) so patch/project grade against a + current baseline instead of a stale one. + +### 6. Verification documentation (cross-engine pattern comparison) + +Add a new top-level **Verification** section to the documentation nav +(between Tutorials and Command-Line), generated like tutorials +(`.py` source → notebook via `pixi run notebook-prepare`, built with +`execute: false`). + +- **Calculation-only comparisons (no minimisation).** Feed identical + input parameters to each supported engine, compute patterns, and + compare them pairwise (`ed-cryspy`, `ed-crysfml`, … and later + `fullprof`). This is far faster than fitting, so the same pages double + as **fast regression scripts** under `script-tests`. +- **Metrics.** Report clear, documented closeness metrics per pair — a + profile-difference metric (Rwp-style), maximum point-wise deviation, + and an integrated-intensity ratio — with explicit tolerances. +- **Overlay plots.** Plot all engines on one chart with distinct colours + and line styles (solid/dotted/…) for visual comparison. +- **Coverage of conditions.** Include **every valid experiment × + instrument-parameter combination at least once** (powder/single + crystal × constant-wavelength/time-of-flight × neutron/x-ray × + bragg/total, per the support matrix below). +- **External software, incrementally.** External tools (FullProf first, + then GSAS-II/TOPAS) are compared by loading a **pre-calculated profile + from a zipped project** stored in the `diffraction` data repository + (§8), so EasyDiffraction need not run them. The page structure ships + now with an external placeholder; data is added incrementally. +- **Prerequisite — engine support matrix.** Declare which engine + supports which `beam_mode × radiation_probe` (and `scattering_type`) + via the existing `CalculatorSupport`/`Compatibility` metadata, so + "every valid combination" is well-defined. Today engines are keyed + only by `scattering_type` at the factory level. The precise metadata + wiring is a scoped sub-task (see Deferred Work). + +### 7. Performance-regression benchmarks + +Replace the ad-hoc `tools/benchmark_tutorials.py` CSV tool with +**`pytest-benchmark`**, matching the prior art in +`deps-pycrysfml`: + +- Benchmark **per experiment type** (one benchmark per + `beam_mode × radiation_probe × engine`) rather than whole-tutorial + wall-clock. +- Store baseline JSON (full machine info + per-benchmark statistics) in + the `diffraction` data repository (§8), written by a CI job. +- **Informational now** (no gating), to avoid false failures from noisy + CI timing. A regression gate (`--benchmark-compare-fail`) is added + later once variance is characterised, ideally on a dedicated runner + (see Deferred Work). Benchmarks run in the `nightly` tier. + +### 8. Nightly validation harness (CIF corpus and generative fuzzing) + +A robustness harness exercising EasyDiffraction against many real and +synthetic structures. + +- **Code vs data split.** Harness *code* lives in `diffraction-lib` + (`tests/nightly/`, `@pytest.mark.nightly`) so it versions with the + code it checks. The *corpus*, *results database*, FullProf profiles, + and benchmark baselines live in the **`diffraction` data repository** + (consistent with `download_data()`), fetched at runtime. +- **Acceptance-style run.** A scheduled nightly CI job installs + EasyDiffraction **from PyPI**, runs the harness, and writes results + back to the data repository via a bot commit (the `deps-pycrysfml` + "auto-push" pattern). The same harness runs locally on demand. +- **CIF corpus check.** Download ~100–200 CIF files from the + Crystallography Open Database (COD), load each, and record per-file + status: + - `ok` — parsed, all recognised; + - `partial` — parsed, some information missing (EasyDiffraction + applied defaults), with a comment naming what was not recognised; + - `fail` — could not be parsed, with the error. + The status lets the harness (a) skip re-downloading already-`ok` + files on later nights and (b) flag genuine EasyDiffraction recognition + bugs vs malformed files; problematic files convert to issues. +- **Results database — CSV.** A git-diffable manifest, **one row per CIF + keyed by COD id and ordered by id** (so new files insert in order): + `id, parse_status, missing_fields, calc_status_per_engine, comment, last_checked`. + CSV keeps diffs reviewable and issue-friendly. +- **Re-check flag.** The harness script exposes a flag to **re-run only + the failed/partial entries already in the database** (rather than + drawing new random files from COD), so fixes in EasyDiffraction can be + re-validated against the exact files that previously failed. +- **Cross-engine calculation on the corpus.** For loadable structures, + call each supported calculator, compare patterns, and store the + per-engine result (with comments) in the same database. +- **Generative fuzzing (documented now, implemented later).** Randomly + generate ~100–200 structures (random space group, cell parameters, + 1–10 atoms with random coordinates/ADP/occupancy), compute patterns + across engines, and record disagreements in the same database. This + reuses the corpus harness and database; its implementation is a + follow-up pull request (see Deferred Work). + +### 9. Fast documentation-build gate in the test workflow + +Add a **fast, every-push job to `test.yml`** that does **not execute +tutorials**: + +- `mkdocs build --strict` (catches missing nav entries and broken + internal references; tutorials build with `execute: false`), +- link checking (`lychee` or equivalent, with an allowlist), and +- spelling/grammar (`codespell` first; `Vale` later, see Deferred Work). + +This is expected to take about a minute and runs on every push, giving +prompt drift feedback. It is deliberately **separate from `docs.yml`**, +which executes all tutorials and then builds and deploys — slow, and +therefore pull-request-only. The detailed catalogue of documentation +checks is owned by +[Documentation CI and Build Verification](documentation-ci-build.md), +which this ADR coordinates with: that ADR defines *what* the checks are; +this ADR's decision is that the cheap, deterministic subset runs as part +of the every-push test workflow. Promoting that ADR is part of this +work. + +Known limitation: links that appear **only inside executed notebook +output cells** (for example a generated table linking to parameter +definitions) are invisible to the non-executing strict-build job. That +output-cell-link feature does not exist yet; the limitation is recorded +for the future rather than solved now. + +### Implementation phasing + +1. **Quick wins:** §5 codecov policy, §3 structure-check gate, §9 fast + docs gate, §2 markers + §1 placement rules and the relocation pass. +2. **Coverage and cases:** §4 `fail_under` 80 + `hypothesis` + tolerance + convention. +3. **Verification + benchmarks:** §6 engine support matrix and + calculation-only comparison pages; §7 `pytest-benchmark`. +4. **Nightly harness:** §8 corpus check and results database; generative + fuzzing in a later pull request. + +## Consequences + +### Positive + +- A test's correct layer and cost tier are decidable from written rules, + not judgement, so the suite stops drifting. +- Codecov patch stops failing spuriously and the project status becomes + a meaningful, enforced 80% gate. +- Coverage gains a case-quality dimension (input domains, boundaries, + wrong types), not just line counts. +- Cross-engine and (later) external agreement is visible to scientists + in the documentation and regression-checked cheaply. +- Performance and real-world-file robustness gain dedicated, low-noise + signals without slowing ordinary development. +- Documentation drift is caught on every push in about a minute. + +### Trade-offs + +- Relocating functional/unit tests and retiring `fast` touches many + existing test files in one pass. +- New dependencies (`hypothesis`, `pytest-benchmark`, plus `codespell` + and a link checker for the docs job) add configuration and maintenance. +- The nightly harness and data-repository round-trip add CI and + cross-repository coordination. +- Raising `fail_under` to 80 and gating it can block merges until + coverage catches up; the ramp is deliberate. + +## Alternatives Considered + +- **One combined ADR vs several focused ADRs.** A split (taxonomy / + codecov / benchmarks / verification) was considered. Chosen: one + combined ADR, because the goals share infrastructure (markers, the + data repository, CI triggers) and read as one quality story; + large sub-areas are phased instead. +- **Upload combined coverage to codecov.** Rejected for now: slower, + flakier (engine-dependent), and needs per-flag setup. Unit-only upload + with a non-blocking patch status is simpler and fixes the reported + pain directly. +- **Keep the current `fast` marker semantics.** Rejected: marking the + cheap majority is more error-prone than opt-in escalation of the + expensive minority. +- **`asv` for benchmarking.** Rejected vs `pytest-benchmark`: `asv` + wants a dedicated dashboard/runner; `pytest-benchmark` reuses pytest, + matches `deps-pycrysfml`, and supports committed JSON baselines. +- **SQLite results database.** Rejected as the source of truth: opaque + in diffs and harder to convert to issues. CSV keyed and ordered by id + is reviewable; a derived cache can be added later if querying demands + it. +- **`syrupy` snapshot testing and `mutmut` mutation testing.** Deferred, + not adopted now (see Deferred Work): the tutorial `baseline.json` + already covers fit-result regression, and mutation testing is only + meaningful once line coverage is solid. + +## Deferred Work + +- Generative random-structure fuzzing implementation (§8) — follow-up + pull request. +- External-software reference data and comparisons (FullProf, then + GSAS-II/TOPAS) for the Verification section (§6). +- Benchmark regression gating threshold and a dedicated, low-noise + runner (§7). +- Precise `CalculatorSupport`/`Compatibility` wiring for the engine + support matrix (§6 prerequisite); may warrant its own short ADR. +- `Vale` prose linting after `codespell` has a baseline and a + crystallography/CIF-tag vocabulary (§9). +- Link-checking of URLs that appear only in executed notebook output + cells (§9 limitation); revisit if/when that output feature exists. +- Mutation testing (`mutmut`) once line coverage reaches ≥ 80%. +- Snapshot testing (`syrupy`) for CIF/report output — reconsider if + explicit assertions prove insufficient. +- The exact coverage ramp schedule from 80% toward 90–95%. + +## Dependencies + +New dependencies introduced by this ADR (approval recorded in the +drafting conversation, per the dependency-approval rule): + +- `hypothesis` — property-based / input-domain testing (§4). +- `pytest-benchmark` — performance-regression benchmarks (§7). + +Coordinated with [Documentation CI and Build Verification](documentation-ci-build.md), +which carries the documentation-check tools (`codespell`, a link +checker such as `lychee`, and later `Vale`) used by §9. + +## Related ADRs + +- [Test Strategy](../accepted/test-strategy.md) — amended by this ADR. +- [Documentation CI and Build Verification](documentation-ci-build.md) — + coordinated with §9. +- [Lint Complexity Thresholds](../accepted/lint-complexity-thresholds.md) + — sibling Quality guardrail. +- [Notebook Generation Source of Truth](../accepted/notebook-generation.md) + — the `.py` → notebook pipeline reused by §6. +- [Factory Contracts and Metadata](../accepted/factory-contracts.md) — + the `CalculatorSupport`/`Compatibility` metadata used by §6. +- [Enum-Backed Closed Value Sets](../accepted/enum-backed-closed-values.md) + — any new closed set (engine tags, experiment axes) stays `(str, Enum)`. From 472a0dd3fd807e3d555175f4a514f704fdd6d5b1 Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Fri, 5 Jun 2026 14:46:51 -0700 Subject: [PATCH 02/57] Update test-suite-and-validation ADR and index from review --- docs/dev/adrs/index.md | 1 + .../suggestions/test-suite-and-validation.md | 32 ++++++++++++------- 2 files changed, 21 insertions(+), 12 deletions(-) diff --git a/docs/dev/adrs/index.md b/docs/dev/adrs/index.md index 6b9f3379a..1e2b39bdc 100644 --- a/docs/dev/adrs/index.md +++ b/docs/dev/adrs/index.md @@ -45,6 +45,7 @@ folders. | Persistence | Accepted | Python and CIF Category Correspondence | Compares current Python paths and CIF tags, then records scoped one-to-one mapping for project-level categories. | [`python-cif-category-correspondence.md`](accepted/python-cif-category-correspondence.md) | | Quality | Accepted | Lint Complexity Thresholds | Treats ruff PLR complexity limits as design guardrails that should not be bypassed. | [`lint-complexity-thresholds.md`](accepted/lint-complexity-thresholds.md) | | Quality | Accepted | Test Strategy | Defines layered unit, functional, integration, script, and notebook testing. | [`test-strategy.md`](accepted/test-strategy.md) | +| Quality | Suggestion | Test Suite and Validation Strategy | Strict test layers, cost tiers, coverage/codecov policy, cross-engine verification docs, and a nightly validation harness. | [`test-suite-and-validation.md`](suggestions/test-suite-and-validation.md) | | Structure model | Accepted | Type-Neutral ADP Parameters | Keeps ADP parameter object identities stable across B/U and iso/ani switches. | [`type-neutral-adp-parameters.md`](accepted/type-neutral-adp-parameters.md) | | Structure model | Accepted | Automatic Wyckoff Position Detection | Detects Wyckoff letter, multiplicity, and site symmetry from space group and coordinates; calculators consume them. | [`wyckoff-letter-detection.md`](accepted/wyckoff-letter-detection.md) | | Structure model | Accepted | Complete Space-Group Reference Database | One-time build of a complete space_groups.json.gz (all 230 groups) from cctbx, verified against multiple sources. | [`space-group-database.md`](accepted/space-group-database.md) | diff --git a/docs/dev/adrs/suggestions/test-suite-and-validation.md b/docs/dev/adrs/suggestions/test-suite-and-validation.md index 173d4c63f..b4ad68047 100644 --- a/docs/dev/adrs/suggestions/test-suite-and-validation.md +++ b/docs/dev/adrs/suggestions/test-suite-and-validation.md @@ -32,12 +32,14 @@ accumulated: call `download_data()` (mocked, but undocumented). There is no written rule an author can apply to a borderline test. - **Codecov patch status is always red.** `.codecov.yml` runs a - blocking `patch: target: auto` against a **unit-only** coverage - baseline (`coverage.yml` uploads only `coverage-unit.xml`). Any pull - request that touches code exercised mainly by functional, integration, - or script tests scores near-zero patch coverage and fails. `test.yml` - uploads no coverage at all, so most feature-branch pull requests grade - against a stale develop baseline. See + blocking `patch: target: auto` against a **unit-only** coverage upload + (`coverage.yml` uploads only `coverage-unit.xml`). Any pull request + that touches code exercised mainly by functional, integration, or + script tests scores near-zero patch coverage and fails the blocking + patch check. `coverage.yml` already runs on pull requests (not only on + push to `develop`), so the baseline is current — the failure is purely + a blocking patch status graded against unit-only data, not a stale + baseline. See [discussion #69](https://github.com/orgs/easyscience/discussions/69). - **Coverage is line-only and unenforced.** `fail_under = 65` is checked locally but never gated in CI, and line coverage says nothing about @@ -195,19 +197,25 @@ guarantees. ### 5. Codecov policy +The always-red patch status has a single cause: a **blocking `patch` +status (`target: auto`) graded against a unit-only coverage upload**. +Diff lines exercised mainly by functional, integration, or script tests +show near-zero unit coverage and fail the patch check. `coverage.yml` +already uploads unit coverage on pull requests and on push to `develop`, +so the baseline is current; only the status configuration needs to +change — **no new coverage-upload path is introduced**. + Adopt the recommendation from [discussion #69](https://github.com/orgs/easyscience/discussions/69): - **Upload unit-test coverage only** (keep the single, fast, reliable - source; functional/integration coverage stays out of codecov). + source already produced by `coverage.yml`; functional/integration + coverage stays out of codecov). - **`project` status: target 80%, blocking** (`informational: false`) — this becomes the real coverage gate. - **`patch` status: `informational: true`** (non-blocking) — stops the - always-red patch failures, which were an artefact of grading - diff lines against a unit-only baseline. -- **Upload unit coverage from `test.yml` on every pull request** (not - only from `coverage.yml` on develop) so patch/project grade against a - current baseline instead of a stale one. + always-red patch failures, which were an artefact of grading diff + lines against a unit-only baseline. ### 6. Verification documentation (cross-engine pattern comparison) From c4d0532437dd57faf3ed81ebf52700c4025fdeb1 Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Fri, 5 Jun 2026 14:46:51 -0700 Subject: [PATCH 03/57] Add test-suite-and-validation implementation plan --- docs/dev/plans/test-suite-and-validation.md | 373 ++++++++++++++++++++ 1 file changed, 373 insertions(+) create mode 100644 docs/dev/plans/test-suite-and-validation.md diff --git a/docs/dev/plans/test-suite-and-validation.md b/docs/dev/plans/test-suite-and-validation.md new file mode 100644 index 000000000..d5ecaa689 --- /dev/null +++ b/docs/dev/plans/test-suite-and-validation.md @@ -0,0 +1,373 @@ +# Plan: Test Suite and Validation Strategy + +This plan follows [`AGENTS.md`](../../../AGENTS.md) with one **declared +exception** to the two-phase workflow. [`AGENTS.md`](../../../AGENTS.md) +§Workflow keeps test creation in Phase 2 ("Phase 1 — Code and docs +updates only ... Do not create or run tests unless the user explicitly +asks"). Because this ADR's subject *is* the test suite, Phase 1 here +necessarily includes test relocation, shared fixtures, new unit/property +tests, and benchmark tests as implementation work — deferring them to +verification would leave Phase 1 empty of its actual deliverable. The +implementer running `/draft-impl-1` will therefore edit and add files +under `tests/**` during Phase 1. Phase 2 remains the standard +verification gate (`pixi run fix`/`check`/`unit-tests`/ +`integration-tests`/`script-tests`) and does not author new test +suites. + +A scope decision is also recorded below (§Scope): the cross-repository +work is documented for the future rather than implemented in this +branch, per the author's instruction. + +## ADR + +Implements the suggestion ADR +[`test-suite-and-validation.md`](../adrs/suggestions/test-suite-and-validation.md) +(drafted via `/draft-adr`; review cycle closed). The plan owns this ADR. +Per [`AGENTS.md`](../../../AGENTS.md) §Change Discipline, the ADR is +promoted from `suggestions/` to `accepted/` as part of this change, +before the PR is opened (step P1.15). + +Coordinated, not implemented here: +[`documentation-ci-build.md`](../adrs/suggestions/documentation-ci-build.md) +(stays a suggestion; this plan implements only its strict-build, link, +and spelling subset for the every-push test workflow — see §Open +questions). + +Amends the accepted +[`test-strategy.md`](../adrs/accepted/test-strategy.md) (sharper layer +definitions). + +## Branch and PR + +- Branch: `test-suite-and-validation` (already checked out, created off + `develop`). +- PR target: `develop`. +- Do not push the branch until asked. + +## Scope + +**In scope (all in-repository ADR work):** §1 strict layers + +relocation, §2 cost-tier markers, §3 structure-check CI gate, §4 +coverage target + `hypothesis`, §5 codecov policy, §6 engine support +matrix + cross-engine (cryspy ↔ crysfml) calculation-only comparison +pages, §7 `pytest-benchmark` suite, §9 fast docs build gate. + +**Documented for the future (cross-repository, NOT implemented here):** +- §8 nightly COD corpus harness, its results database, and the + pip-install acceptance CI job that writes results back to the + `diffraction` data repository. +- §8 generative random-structure fuzzing (already ADR-deferred). +- §7 benchmark baseline history committed to the `diffraction` data + repository, and any performance regression gate. +- §6 external-software comparison (FullProf/GSAS-II) via zipped projects + stored in the `diffraction` data repository. + +These are captured in step P1.14 (a future-work record in +`docs/dev/issues/open.md`) and remain in the ADR's Deferred Work. + +## Decisions (already made) + +- **Layers (§1):** functional is in-process with bundled fixtures only — + **no real calculation engine and no network/`download_data()`**; those + move to integration. A test goes to the lowest layer whose constraints + it can satisfy. +- **Markers (§2):** opt-in escalation. Default (unmarked) = fast. + Add `@pytest.mark.pr` (PR + `develop`/`master`) and + `@pytest.mark.nightly` (scheduled only). **Retire the current `fast` + marker** (6 integration files). CI selection: + `not pr and not nightly` (feature push) / `not nightly` (PR + main) / + all (nightly schedule). Because markers are **orthogonal to layers**, + the selected expression is applied to **every** pytest invocation under + the policy — unit, functional, and integration, in both the source and + package CI jobs — not to integration alone as today. +- **Structure gate (§3):** unify on a single `src/` tree walk shared by + `tools/generate_package_docs.py` and `tools/test_structure_check.py`; + run the check in CI as a gate. +- **Coverage (§4):** raise `fail_under` 65 → 80 (ramp to 90–95 later). + Adopt `hypothesis` (approved) in a deterministic profile; input-domain + tests target the validators in `core/validation.py`. One documented + numeric-tolerance convention via a root `tests/conftest.py`. +- **Codecov (§5):** unit-only upload (unchanged), `patch` → + `informational: true`, `project` → target 80% blocking. No new upload + path. +- **Verification (§6):** calculation-only (no minimisation) cross-engine + comparison pages with profile-difference / max-deviation / + integrated-intensity metrics and overlay plots; new top-level + `Verification` nav node; pages double as `script-tests`. Engine + support matrix declared first. +- **Benchmarks (§7):** `pytest-benchmark` (approved), per + `beam_mode × radiation_probe × engine`, `nightly`-marked, output as a + CI artifact; informational. +- **Docs gate (§9):** fast every-push job in `test.yml` — strict + `mkdocs build` (no tutorial execution), link check (`lychee`), spell + check (`codespell`) — separate from the slow `docs.yml`. +- **Dependencies named for pre-approval** (per + [`AGENTS.md`](../../../AGENTS.md) §Architecture): `hypothesis`, + `pytest-benchmark`, `codespell` (dev dependencies); `lychee` (CI + link-checker, GitHub Action or binary — no Python dependency). + +## Open questions + +1. **Engine support matrix — extend existing metadata (§6, P1.11).** The + `TypeInfo`/`Compatibility`/`CalculatorSupport` dataclasses already + exist in `src/easydiffraction/core/metadata.py` (lines 19, 40, 88) + and are populated per instrument category (e.g. + `datablocks/experiment/categories/instrument/cwl.py` declares + `compatibility` and `calculator_support`). P1.11 therefore *applies + and extends* this model — notably adding `radiation_probe` to + `Compatibility` if missing, and exposing a query to enumerate + comparable engine × condition combinations — rather than introducing + new classes. **Stop and ask** about a dedicated ADR only if extending + the metadata shape proves structural (a broad change to + `Compatibility`). +2. **`fail_under = 80` feasibility (§4, P1.9).** Current unit-only + coverage is unverified against 80. If Phase 2 shows it below 80 after + the new tests, either add more unit tests or set a documented + intermediate value and ramp — decide in Phase 2, do not silence the + gate. +3. **`documentation-ci-build` promotion (§9).** The ADR text says + promoting it "is part of this work," but this plan implements only a + subset (strict build + link + spell). Recommendation: keep it a + suggestion and cross-reference; revisit promotion when its remaining + items (mkdocstrings, snippet smoke tests, notebook-freshness) land. +4. **`lychee` packaging.** GitHub Action vs pinned binary in the pixi + environment — pick during P1.10. +5. **Location of the strict-criteria testing guide and the + tolerance-convention text (§1/§4).** A new `docs/dev/` testing guide + vs extending the amended `test-strategy.md` — decide in P1.5. + +## Concrete files likely to change + +- `.codecov.yml` (status config) +- `pyproject.toml` (`[tool.pytest.ini_options].markers`, + `[tool.coverage.report].fail_under`, `hypothesis`/`codespell` config, + dev dependencies) +- `pixi.toml` (new tasks: `docs-build-strict`, `link-check`, + `spell-check`, `benchmarks`; structure-check wiring; deps) +- `.github/workflows/test.yml` (marker selection, nightly scheduled job, + strict-docs job, structure-check gate) +- `tools/test_structure_check.py`, `tools/generate_package_docs.py` + (shared tree walk) +- `tests/conftest.py` (new: seeded RNG + tolerance fixtures) +- `tests/functional/**`, `tests/integration/**`, `tests/unit/**` + (relocation; remove `fast` marks; new property tests) +- `tests/integration/fitting/*.py` (6 files: remove `@pytest.mark.fast`) +- `tests/benchmarks/**` (new: `nightly` benchmarks) +- `src/easydiffraction/core/metadata.py`, + `src/easydiffraction/datablocks/experiment/categories/instrument/**`, + `src/easydiffraction/analysis/calculators/**` (engine support matrix — + extend existing `Compatibility`/`CalculatorSupport`) +- `src/easydiffraction/core/validation.py` (property-test target; expose + domains if needed) +- `docs/mkdocs.yml` (Verification nav node) +- `docs/docs/verification/*.py` (new comparison tutorials) +- `docs/dev/adrs/suggestions/test-suite-and-validation.md` → `accepted/` + (promotion); `docs/dev/adrs/index.md` (status flip) +- `docs/dev/issues/open.md` (cross-repo future-work record) +- new config: `.codespellrc` (or `[tool.codespell]`), `lychee` config + +## Implementation discipline + +When an AI agent follows this plan, **every completed Phase 1 step must +be staged with explicit paths and committed locally before moving to the +next step or the Phase 1 review gate**, per +[`AGENTS.md`](../../../AGENTS.md) §Commits. Keep commits atomic, +single-purpose, and aligned to the step. Do not stage unrelated dirty +files or generated artifacts. Do not run Phase 2 commands during Phase 1. + +## Implementation steps (Phase 1) + +- [ ] **P1.1 — Codecov status policy (§5)** + Edit `.codecov.yml`: add `informational: true` to `patch.default`; set + `project.default` to `target: 80%`, `informational: false`. Leave the + unit-only upload untouched. + Files: `.codecov.yml`. + Commit: `Make codecov patch informational and gate project at 80%` + +- [ ] **P1.2 — Cost-tier markers and test retagging (§2)** + Register `pr` and `nightly` markers in + `[tool.pytest.ini_options].markers`; remove the `fast` marker + definition. Remove `@pytest.mark.fast` from the 6 + `tests/integration/fitting/*.py` files (retag the genuinely heavy ones + with `pr` where appropriate). + Files: `pyproject.toml`, `tests/integration/fitting/*.py`. + Commit: `Replace fast marker with pr and nightly test tiers` + +- [ ] **P1.3 — CI marker selection across all layers and nightly job (§2)** + Update `.github/workflows/test.yml` mark logic to + `-m "not pr and not nightly"` (feature push) and `-m "not nightly"` + (PR + `develop`/`master`), and apply the selected expression to + **every** pytest invocation in both the source-test and package-test + jobs — unit, functional, and integration (today `-m` reaches only the + integration runs at lines 132 and 311; unit/functional run unfiltered). + Thread a marker passthrough into the `unit-tests` and `functional-tests` + pixi tasks (the `integration-tests` task already accepts an appended + expression). Add a `schedule:` trigger and a nightly job running + `-m nightly`. + Files: `.github/workflows/test.yml`, `pixi.toml`. + Commit: `Select test tiers per trigger across all test layers` + +- [ ] **P1.4 — Unify src-tree walk and gate structure check (§3)** + Extract the `src/` enumeration so `tools/test_structure_check.py` and + `tools/generate_package_docs.py` share one walker; add the check to CI + (lint/format or test workflow) as a blocking gate. + Files: `tools/test_structure_check.py`, + `tools/generate_package_docs.py`, `.github/workflows/*.yml`, + `pixi.toml`. + Commit: `Gate unit-test structure check on shared src tree walk` + +- [ ] **P1.5 — Strict layer-criteria testing guide (§1)** + Write the may/must-not criteria and the "where does this test go?" + decision list (location per Open question 5), and tighten the layer + wording referenced by the amended `test-strategy.md`. + Files: new `docs/dev/` testing guide (or `test-strategy.md` update). + Commit: `Document strict test layer placement criteria` + +- [ ] **P1.6 — Test relocation pass (§1)** + Move functional tests that call real `download_data()` into + integration; relocate or correctly mark slow/engine/network-touching + unit tests (the 16 `download_data()` unit call sites must be explicit + mocks or move out). Keep `test-structure-check` green. + Files: `tests/functional/**`, `tests/integration/**`, + `tests/unit/**`. + Commit: `Relocate network and engine tests to correct layers` + +- [ ] **P1.7 — Shared fixtures, hypothesis profile, tolerance convention (§4)** + Add `hypothesis` (dev dep) and a deterministic profile + (`derandomize`, fixed seed, no committed `.hypothesis` DB). Add a root + `tests/conftest.py` with seeded-RNG and one documented + `rtol`/`atol` pair (intra-engine) and one cross-engine pair. + Files: `pyproject.toml`, `pixi.toml`, `tests/conftest.py`, testing + guide. + Commit: `Add hypothesis deterministic profile and shared test fixtures` + +- [ ] **P1.8 — Input-domain property tests on validators (§4)** + Property-based + explicit boundary-table tests against + `core/validation.py` (`TypeValidator`, content `ValidatorBase` + subclasses) through parameter (`core/variable.py`) and category + (`core/category.py`): valid-domain acceptance and invalid/ wrong-type + rejection or fallback per contract. + Files: `tests/unit/easydiffraction/core/**` (validator/variable/ + category tests). + Commit: `Add property-based input-domain tests for validators` + +- [ ] **P1.9 — Raise coverage gate to 80% (§4)** + Set `[tool.coverage.report] fail_under = 80`. (Resolve Open question 2 + in Phase 2 if unit coverage is below 80 after P1.8.) + Files: `pyproject.toml`. + Commit: `Raise coverage fail_under to 80 percent` + +- [ ] **P1.10 — Fast docs build gate (§9)** + Add `docs-build-strict` (`mkdocs build --strict`, tutorials not + executed), `link-check` (`lychee`), and `spell-check` (`codespell`) + pixi tasks with config and ignore lists; add a fast every-push job to + `test.yml`. Add `codespell` dev dep; wire `lychee` (Open question 4). + Files: `pixi.toml`, `pyproject.toml`, `.github/workflows/test.yml`, + `.codespellrc`, `lychee` config. + Commit: `Add strict docs build, link, and spell checks on every push` + +- [ ] **P1.11 — Apply/extend calculator support metadata (§6 prerequisite)** + Build on the existing `Compatibility`/`CalculatorSupport` model in + `core/metadata.py` (already declared per instrument category): add + `radiation_probe` to `Compatibility` if missing, and add a small query + helper to enumerate comparable engine × experiment-condition + combinations for the verification pages. Prefer this declared metadata + over the ad-hoc per-calculator `if beam_mode == …` checks. **Stop and + ask** only if extending the metadata shape proves structural (Open + question 1). + Files: `src/easydiffraction/core/metadata.py`, + `src/easydiffraction/datablocks/experiment/categories/instrument/**`, + `src/easydiffraction/analysis/calculators/**`. + Commit: `Extend calculator support metadata with radiation probe` + +- [ ] **P1.12 — Cross-engine verification pages + script wiring (§6)** + Add the `Verification` nav node (between Tutorials and Command-Line) + and calculation-only `.py` comparison pages (cryspy ↔ crysfml) across + the supported experiment combinations, with closeness metrics, overlay + plots, and metric-tolerance assertions. **Wire the new + `docs/docs/verification/` directory into the script-test runner** — + `tools/test_scripts.py` discovers only `docs/docs/tutorials/*.py` + today (lines 24-27) — and into the notebook pipeline (`notebook-prepare` + / `notebook-convert` / `notebook-tests`, which target the tutorials + dir), so the pages are generated and exercised as regressions. Run + `pixi run notebook-prepare`. + Files: `docs/mkdocs.yml`, `docs/docs/verification/*.py` (+ generated + `*.ipynb`), `tools/test_scripts.py`, `pixi.toml`. + Commit: `Add cross-engine verification comparison pages and script wiring` + +- [ ] **P1.13 — Per-experiment performance benchmarks (§7)** + Add `pytest-benchmark` (dev dep), `nightly`-marked benchmarks keyed by + `beam_mode × radiation_probe × engine`, and a `benchmarks` pixi task + emitting JSON as a CI artifact (data-repo history deferred). + Files: `pyproject.toml`, `pixi.toml`, `tests/benchmarks/**`. + Commit: `Add per-experiment performance benchmarks (nightly)` + +- [ ] **P1.14 — Record cross-repository future work (§8 + deferred)** + Add prioritised entries to `docs/dev/issues/open.md` for the nightly + COD harness + results DB + pip-install acceptance job, generative + fuzzing, data-repo benchmark history, and external-software + comparison data. Confirm the ADR Deferred Work covers them. + Files: `docs/dev/issues/open.md`. + Commit: `Record cross-repo nightly harness and benchmarks as future work` + +- [ ] **P1.15 — Promote ADR to accepted (§Change Discipline)** + `git mv docs/dev/adrs/suggestions/test-suite-and-validation.md + docs/dev/adrs/accepted/`; set its `## Status` to `Accepted.`; flip the + `docs/dev/adrs/index.md` row to `Accepted` with the `accepted/...` + link; fix any links that pointed at the `suggestions/` path + (`git grep -n`). + Files: ADR file (moved), `docs/dev/adrs/index.md`. + Commit: `Promote test-suite-and-validation ADR to accepted` + +- [ ] **P1.16 — Phase 1 review gate (no code)** + Confirm every box above is `[x]`. Mark this step and commit the + checklist update alone. + Commit: `Reach Phase 1 review gate` + +## Phase 2 verification + +Run after the Phase 1 review gate closes. Use the zsh-safe log-capture +pattern where output is needed. + +```shell +pixi run fix +pixi run check > /tmp/easydiffraction-check.log 2>&1; check_exit_code=$?; tail -n 200 /tmp/easydiffraction-check.log; exit $check_exit_code +pixi run unit-tests > /tmp/easydiffraction-unit.log 2>&1; unit_tests_exit_code=$?; tail -n 200 /tmp/easydiffraction-unit.log; exit $unit_tests_exit_code +pixi run integration-tests > /tmp/easydiffraction-integration.log 2>&1; integration_tests_exit_code=$?; tail -n 200 /tmp/easydiffraction-integration.log; exit $integration_tests_exit_code +pixi run script-tests > /tmp/easydiffraction-script.log 2>&1; script_tests_exit_code=$?; tail -n 200 /tmp/easydiffraction-script.log; exit $script_tests_exit_code +``` + +Phase 2 also: confirm `pixi run test-structure-check` passes after the +relocation; confirm the new `docs-build-strict`/`link-check`/ +`spell-check` tasks pass; resolve Open question 2 if `fail_under = 80` +fails. `pixi run fix` regenerates +`docs/dev/package-structure/{full,short}.md` — accept those. Leave +generated `docs/dev/benchmarking/*.csv` and tutorial project outputs +untracked unless explicitly asked. + +## Status checklist + +- [ ] Phase 1 complete (P1.1–P1.16) and reviewed via `/review-impl-1`. +- [ ] Phase 2 verification complete and reviewed via `/review-impl-2`. +- [ ] ADR promoted to `accepted/`. +- [ ] Cross-repository follow-ups recorded in `docs/dev/issues/open.md`. + +## Suggested Pull Request + +**Title:** Stronger, clearer test suite with cross-engine verification + +**Description:** This change makes EasyDiffraction's tests easier to +trust and easier to contribute to. It defines exactly which kind of test +belongs where (fast unit checks vs. slower engine tests), so the suite +stays quick day to day and runs the heavy checks on pull requests and +overnight. It fixes the long-standing red "patch" mark on pull requests +by correcting how coverage is reported, and raises the coverage target +while adding smarter tests that probe edge cases (negative, zero, and +out-of-range inputs), not just lines of code. It adds a new +**Verification** section to the documentation that calculates the same +diffraction pattern with each supported engine and shows, with clear +metrics and overlaid plots, how closely they agree. It also adds an +every-push documentation check (strict build, working links, spelling) +and a performance-benchmark suite. Larger overnight checks against many +real-world crystal files, and comparisons with external software such as +FullProf, are documented as planned follow-up work. From 36609daf245c161baf62d55206023512d2d6123f Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Fri, 5 Jun 2026 14:47:41 -0700 Subject: [PATCH 04/57] Make codecov patch informational and gate project at 80% --- .codecov.yml | 12 ++++++++---- docs/dev/plans/test-suite-and-validation.md | 2 +- 2 files changed, 9 insertions(+), 5 deletions(-) diff --git a/.codecov.yml b/.codecov.yml index af816b1a2..97a636020 100644 --- a/.codecov.yml +++ b/.codecov.yml @@ -20,12 +20,16 @@ coverage: status: project: default: - target: auto + target: 80% threshold: 1% - # Make project coverage informational (won't block PR) - informational: true + # Project coverage is the real gate (see ADR + # test-suite-and-validation). + informational: false patch: default: target: auto - # Require patch coverage but with threshold threshold: 1% + # Non-blocking: patch grades diff lines against unit-only + # coverage, so engine/integration-tested code would otherwise + # always fail. See ADR test-suite-and-validation. + informational: true diff --git a/docs/dev/plans/test-suite-and-validation.md b/docs/dev/plans/test-suite-and-validation.md index d5ecaa689..774657be2 100644 --- a/docs/dev/plans/test-suite-and-validation.md +++ b/docs/dev/plans/test-suite-and-validation.md @@ -177,7 +177,7 @@ files or generated artifacts. Do not run Phase 2 commands during Phase 1. ## Implementation steps (Phase 1) -- [ ] **P1.1 — Codecov status policy (§5)** +- [x] **P1.1 — Codecov status policy (§5)** Edit `.codecov.yml`: add `informational: true` to `patch.default`; set `project.default` to `target: 80%`, `informational: false`. Leave the unit-only upload untouched. From 4db5ee9b507ed1cc85d3422f7da124e74cf70936 Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Fri, 5 Jun 2026 14:56:06 -0700 Subject: [PATCH 05/57] Replace fast marker with pr and nightly test tiers --- docs/dev/plans/test-suite-and-validation.md | 2 +- pyproject.toml | 5 ++- tests/integration/conftest.py | 34 +++++++++++++++++++ .../fitting/test_aniso_adp_fitting.py | 3 -- .../test_pair-distribution-function.py | 2 -- ..._powder-diffraction_constant-wavelength.py | 2 -- .../test_powder-diffraction_joint-fit.py | 3 -- .../test_single-crystal-diffraction.py | 2 -- .../fitting/test_switch-calculator.py | 2 -- 9 files changed, 39 insertions(+), 16 deletions(-) create mode 100644 tests/integration/conftest.py diff --git a/docs/dev/plans/test-suite-and-validation.md b/docs/dev/plans/test-suite-and-validation.md index 774657be2..90255cc97 100644 --- a/docs/dev/plans/test-suite-and-validation.md +++ b/docs/dev/plans/test-suite-and-validation.md @@ -184,7 +184,7 @@ files or generated artifacts. Do not run Phase 2 commands during Phase 1. Files: `.codecov.yml`. Commit: `Make codecov patch informational and gate project at 80%` -- [ ] **P1.2 — Cost-tier markers and test retagging (§2)** +- [x] **P1.2 — Cost-tier markers and test retagging (§2)** Register `pr` and `nightly` markers in `[tool.pytest.ini_options].markers`; remove the `fast` marker definition. Remove `@pytest.mark.fast` from the 6 diff --git a/pyproject.toml b/pyproject.toml index 4ef0ad73f..c2e2563a4 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -194,7 +194,10 @@ fail_under = 65 # Minimum coverage percentage to pass [tool.pytest.ini_options] addopts = '--import-mode=importlib' -markers = ['fast: mark test as fast (should be run on every push)'] +markers = [ + 'pr: heavier test; runs on pull requests and develop/master, not feature-branch pushes', + 'nightly: very expensive test; runs only on the scheduled nightly job', +] testpaths = ['tests'] filterwarnings = [ # TEMPRORARY: Suppress some warnings diff --git a/tests/integration/conftest.py b/tests/integration/conftest.py new file mode 100644 index 000000000..7036bbd97 --- /dev/null +++ b/tests/integration/conftest.py @@ -0,0 +1,34 @@ +# SPDX-FileCopyrightText: 2026 EasyScience contributors +# SPDX-License-Identifier: BSD-3-Clause +"""Shared pytest configuration for the integration test layer. + +Integration tests exercise real calculation engines and downloaded data, +so they belong to the ``pr`` cost tier: they run on pull requests and on +``develop``/``master``, but not on every feature-branch push. Rather than +decorate each integration test individually, they are marked here at +collection time. A test may still opt up to the ``nightly`` tier with +``@pytest.mark.nightly``; such tests are left untouched. + +This auto-marking is scoped to ``tests/integration/`` by path, so it does +not affect unit or functional tests collected in the same session. +""" + +from __future__ import annotations + +import pytest + + +@pytest.hookimpl(tryfirst=True) +def pytest_collection_modifyitems(items: list[pytest.Item]) -> None: + """Mark every integration test as ``pr`` unless it is ``nightly``. + + ``tryfirst`` ensures this runs before pytest's own ``-m`` deselection, + so the added marker participates in marker-expression filtering. + """ + for item in items: + path = str(getattr(item, 'fspath', '')).replace('\\', '/') + if '/tests/integration/' not in path: + continue + if item.get_closest_marker('nightly'): + continue + item.add_marker(pytest.mark.pr) diff --git a/tests/integration/fitting/test_aniso_adp_fitting.py b/tests/integration/fitting/test_aniso_adp_fitting.py index ccfbef066..fd3bb0542 100644 --- a/tests/integration/fitting/test_aniso_adp_fitting.py +++ b/tests/integration/fitting/test_aniso_adp_fitting.py @@ -4,8 +4,6 @@ import tempfile -import pytest - import easydiffraction as ed TEMP_DIR = tempfile.gettempdir() @@ -36,7 +34,6 @@ def _setup_tbti_project(): return project -@pytest.mark.fast def test_iso_then_aniso_fit() -> None: """Fit Uiso first, then switch to Uani and fit again.""" project = _setup_tbti_project() diff --git a/tests/integration/fitting/test_pair-distribution-function.py b/tests/integration/fitting/test_pair-distribution-function.py index 58910532c..d6551d3d5 100644 --- a/tests/integration/fitting/test_pair-distribution-function.py +++ b/tests/integration/fitting/test_pair-distribution-function.py @@ -3,7 +3,6 @@ import tempfile -import pytest from numpy.testing import assert_almost_equal import easydiffraction as ed @@ -75,7 +74,6 @@ def test_single_fit_pdf_xray_pd_cw_nacl() -> None: assert_almost_equal(chi2, desired=1.48, decimal=2) -@pytest.mark.fast def test_single_fit_pdf_neutron_pd_cw_ni(): project = ed.Project() diff --git a/tests/integration/fitting/test_powder-diffraction_constant-wavelength.py b/tests/integration/fitting/test_powder-diffraction_constant-wavelength.py index 17f84be3d..a689c950a 100644 --- a/tests/integration/fitting/test_powder-diffraction_constant-wavelength.py +++ b/tests/integration/fitting/test_powder-diffraction_constant-wavelength.py @@ -3,7 +3,6 @@ import tempfile -import pytest from numpy.testing import assert_almost_equal from easydiffraction import ExperimentFactory @@ -144,7 +143,6 @@ def test_single_fit_neutron_pd_cwl_lbco() -> None: ) -@pytest.mark.fast def test_single_fit_neutron_pd_cwl_lbco_with_constraints() -> None: # Set structure model = StructureFactory.from_scratch(name='lbco') diff --git a/tests/integration/fitting/test_powder-diffraction_joint-fit.py b/tests/integration/fitting/test_powder-diffraction_joint-fit.py index b3b224b6f..9ce18c48e 100644 --- a/tests/integration/fitting/test_powder-diffraction_joint-fit.py +++ b/tests/integration/fitting/test_powder-diffraction_joint-fit.py @@ -3,7 +3,6 @@ import tempfile -import pytest from numpy.testing import assert_almost_equal from easydiffraction import ExperimentFactory @@ -14,7 +13,6 @@ TEMP_DIR = tempfile.gettempdir() -@pytest.mark.fast def test_joint_fit_split_dataset_neutron_pd_cwl_pbso4() -> None: # Set structure model = StructureFactory.from_scratch(name='pbso4') @@ -141,7 +139,6 @@ def test_joint_fit_split_dataset_neutron_pd_cwl_pbso4() -> None: ) -@pytest.mark.fast def test_joint_fit_neutron_xray_pd_cwl_pbso4() -> None: # Set structure model = StructureFactory.from_scratch(name='pbso4') diff --git a/tests/integration/fitting/test_single-crystal-diffraction.py b/tests/integration/fitting/test_single-crystal-diffraction.py index df3818a76..6879f2e9f 100644 --- a/tests/integration/fitting/test_single-crystal-diffraction.py +++ b/tests/integration/fitting/test_single-crystal-diffraction.py @@ -10,7 +10,6 @@ TEMP_DIR = tempfile.gettempdir() -@pytest.mark.fast def test_single_fit_neut_sc_cwl_tbti() -> None: project = ed.Project() @@ -48,7 +47,6 @@ def test_single_fit_neut_sc_cwl_tbti() -> None: assert chi2 == pytest.approx(expected=12.9, abs=0.1) -@pytest.mark.fast def test_single_fit_neut_sc_tof_taurine() -> None: project = ed.Project() diff --git a/tests/integration/fitting/test_switch-calculator.py b/tests/integration/fitting/test_switch-calculator.py index 0d853442b..31c9d91ed 100644 --- a/tests/integration/fitting/test_switch-calculator.py +++ b/tests/integration/fitting/test_switch-calculator.py @@ -3,13 +3,11 @@ import tempfile -import pytest from numpy.testing import assert_almost_equal TEMP_DIR = tempfile.gettempdir() -@pytest.mark.fast def test_neutron_pd_cwl_lbco_crysfml(tmp_path) -> None: import easydiffraction as ed from easydiffraction.analysis.calculators.crysfml import CrysfmlCalculator From f1710a70b10188a35b3be7d33350cf2a23ef3dda Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Fri, 5 Jun 2026 14:59:49 -0700 Subject: [PATCH 06/57] Select test tiers per trigger across all test layers --- .github/workflows/nightly.yml | 34 +++++++++++++++++++++ .github/workflows/test.yml | 23 ++++++++------ docs/dev/plans/test-suite-and-validation.md | 2 +- pixi.toml | 2 ++ 4 files changed, 50 insertions(+), 11 deletions(-) create mode 100644 .github/workflows/nightly.yml diff --git a/.github/workflows/nightly.yml b/.github/workflows/nightly.yml new file mode 100644 index 000000000..9645adb2a --- /dev/null +++ b/.github/workflows/nightly.yml @@ -0,0 +1,34 @@ +# Nightly run of the most expensive `nightly`-tier tests (for example the +# performance benchmarks). These are excluded from the per-push and +# per-PR runs in test.yml; see ADR test-suite-and-validation. + +name: Nightly tests + +on: + # Run the nightly-tier tests on a daily schedule + schedule: + - cron: '0 3 * * *' # 03:00 UTC daily + # Allow manual runs from the Actions tab + workflow_dispatch: + +permissions: + contents: read + +# Allow only one concurrent nightly run per ref. +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + +jobs: + nightly-tests: + runs-on: ubuntu-latest + + steps: + - name: Check-out repository + uses: actions/checkout@v6 + + - name: Set up pixi + uses: ./.github/actions/setup-pixi + + - name: Run nightly-tier tests + run: pixi run nightly-tests diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index 86bfa6063..ee7320f35 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -56,15 +56,17 @@ jobs: pytest-marks: ${{ steps.set-mark.outputs.pytest_marks }} steps: - # Determine if integration tests should be run fully or only the fast ones - # (to save time on branches other than master and develop) - - name: Set mark for integration tests + # Select the test cost tier by branch (see ADR test-suite-and-validation): + # the pr tier runs on develop/master and pull requests; feature-branch + # pushes run only the default (fast) tier. The nightly tier runs + # separately in nightly.yml. + - name: Set test tier marker expression id: set-mark run: | if [[ "${{ env.CI_BRANCH }}" == "master" || "${{ env.CI_BRANCH }}" == "develop" ]]; then - echo "pytest_marks=" >> $GITHUB_OUTPUT + echo 'pytest_marks=-m "not nightly"' >> "$GITHUB_OUTPUT" else - echo "pytest_marks=-m fast" >> $GITHUB_OUTPUT + echo 'pytest_marks=-m "not pr and not nightly"' >> "$GITHUB_OUTPUT" fi # Job 2: Test code @@ -99,7 +101,7 @@ jobs: env="py-$(echo $py_ver | tr -d .)-env" # Converts 3.XX -> py-3XX-env echo "Running tests in environment: $env" - pixi run --environment $env unit-tests + pixi run --environment $env unit-tests ${{ needs.env-prepare.outputs.pytest-marks }} done - name: Run functional tests @@ -114,7 +116,7 @@ jobs: env="py-$(echo $py_ver | tr -d .)-env" # Converts 3.XX -> py-3XX-env echo "Running tests in environment: $env" - pixi run --environment $env functional-tests + pixi run --environment $env functional-tests ${{ needs.env-prepare.outputs.pytest-marks }} done - name: Run integration tests ${{ needs.env-prepare.outputs.pytest-marks }} @@ -173,7 +175,8 @@ jobs: # Job 3: Test the package package-test: - needs: source-test # depend on previous job + # env-prepare provides the tier marker expression; source-test the wheel + needs: [env-prepare, source-test] strategy: fail-fast: false @@ -270,7 +273,7 @@ jobs: cd easydiffraction_py$py_ver echo "Running tests" - pixi run python -m pytest ../tests/unit/ --color=yes -v + pixi run python -m pytest ../tests/unit/ --color=yes -v ${{ needs.env-prepare.outputs.pytest-marks }} echo "Exiting pixi project directory" cd .. @@ -289,7 +292,7 @@ jobs: cd easydiffraction_py$py_ver echo "Running tests" - pixi run python -m pytest ../tests/functional/ --color=yes -v + pixi run python -m pytest ../tests/functional/ --color=yes -v ${{ needs.env-prepare.outputs.pytest-marks }} echo "Exiting pixi project directory" cd .. diff --git a/docs/dev/plans/test-suite-and-validation.md b/docs/dev/plans/test-suite-and-validation.md index 90255cc97..1f3f10d8e 100644 --- a/docs/dev/plans/test-suite-and-validation.md +++ b/docs/dev/plans/test-suite-and-validation.md @@ -193,7 +193,7 @@ files or generated artifacts. Do not run Phase 2 commands during Phase 1. Files: `pyproject.toml`, `tests/integration/fitting/*.py`. Commit: `Replace fast marker with pr and nightly test tiers` -- [ ] **P1.3 — CI marker selection across all layers and nightly job (§2)** +- [x] **P1.3 — CI marker selection across all layers and nightly job (§2)** Update `.github/workflows/test.yml` mark logic to `-m "not pr and not nightly"` (feature push) and `-m "not nightly"` (PR + `develop`/`master`), and apply the selected expression to diff --git a/pixi.toml b/pixi.toml index 928c8bb22..6bba5c773 100644 --- a/pixi.toml +++ b/pixi.toml @@ -106,6 +106,8 @@ user = { features = ['py-max', 'user'] } unit-tests = 'python -m pytest tests/unit/ --color=yes -v' functional-tests = 'python -m pytest tests/functional/ --color=yes -v' integration-tests = 'python -m pytest tests/integration/ --color=yes -n auto -v' +# Run only nightly-tier tests (e.g. performance benchmarks) across the suite. +nightly-tests = 'python -m pytest tests/ -m nightly --color=yes -n auto -v' # Remove previously saved tutorial output projects (projects/ed_*) so the # tutorial-output checks cannot pass against a stale artifact from an earlier From 29701b732546381e1ed615deb2a884090b22a512 Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Fri, 5 Jun 2026 15:04:09 -0700 Subject: [PATCH 07/57] Gate unit-test structure check on shared src tree walk --- .github/workflows/lint-format.yml | 8 ++++ docs/dev/plans/test-suite-and-validation.md | 2 +- tools/_src_tree.py | 44 +++++++++++++++++++++ tools/generate_package_docs.py | 5 ++- tools/test_structure_check.py | 43 ++------------------ 5 files changed, 59 insertions(+), 43 deletions(-) create mode 100644 tools/_src_tree.py diff --git a/.github/workflows/lint-format.yml b/.github/workflows/lint-format.yml index 16dd95c95..0f45cefb2 100644 --- a/.github/workflows/lint-format.yml +++ b/.github/workflows/lint-format.yml @@ -91,6 +91,12 @@ jobs: shell: bash run: pixi run notebook-lint-check + - name: Check unit-test directory mirrors src/ structure + id: test_structure + continue-on-error: true + shell: bash + run: pixi run test-structure-check + # Add summary - name: Add quality checks summary if: always() @@ -108,6 +114,7 @@ jobs: echo "| docstring lint | ${{ steps.docstring_lint.outcome == 'success' && '✅' || '❌' }} |" echo "| nonpy format | ${{ steps.nonpy_format.outcome == 'success' && '✅' || '❌' }} |" echo "| notebooks lint | ${{ steps.notebook_lint.outcome == 'success' && '✅' || '❌' }} |" + echo "| test structure | ${{ steps.test_structure.outcome == 'success' && '✅' || '❌' }} |" } >> "$GITHUB_STEP_SUMMARY" # Fail job if any check failed @@ -120,5 +127,6 @@ jobs: || steps.docstring_lint.outcome == 'failure' || steps.nonpy_format.outcome == 'failure' || steps.notebook_lint.outcome == 'failure' + || steps.test_structure.outcome == 'failure' shell: bash run: exit 1 diff --git a/docs/dev/plans/test-suite-and-validation.md b/docs/dev/plans/test-suite-and-validation.md index 1f3f10d8e..80774765c 100644 --- a/docs/dev/plans/test-suite-and-validation.md +++ b/docs/dev/plans/test-suite-and-validation.md @@ -207,7 +207,7 @@ files or generated artifacts. Do not run Phase 2 commands during Phase 1. Files: `.github/workflows/test.yml`, `pixi.toml`. Commit: `Select test tiers per trigger across all test layers` -- [ ] **P1.4 — Unify src-tree walk and gate structure check (§3)** +- [x] **P1.4 — Unify src-tree walk and gate structure check (§3)** Extract the `src/` enumeration so `tools/test_structure_check.py` and `tools/generate_package_docs.py` share one walker; add the check to CI (lint/format or test workflow) as a blocking gate. diff --git a/tools/_src_tree.py b/tools/_src_tree.py new file mode 100644 index 000000000..f57618226 --- /dev/null +++ b/tools/_src_tree.py @@ -0,0 +1,44 @@ +# SPDX-FileCopyrightText: 2026 EasyScience contributors +# SPDX-License-Identifier: BSD-3-Clause +"""Single source of truth for walking the easydiffraction source tree. + +Shared by ``tools/test_structure_check.py`` (the unit-test mirror check) +and ``tools/generate_package_docs.py`` (the package-structure docs), so +the two tools cannot drift on where the source tree lives or which +modules count as source. +""" + +from __future__ import annotations + +from pathlib import Path + +REPO_ROOT = Path(__file__).resolve().parents[1] +SRC_ROOT = REPO_ROOT / 'src' / 'easydiffraction' +TEST_ROOT = REPO_ROOT / 'tests' / 'unit' / 'easydiffraction' + +# Source directories whose contents are excluded entirely (vendored code +# and tooling caches). +EXCLUDED_DIRS: set[str] = { + '_vendored', + '__pycache__', + 'vendor', +} + +# Source module stems that do not need a dedicated unit-test file. +EXCLUDED_MODULES: set[str] = { + '__init__', + '__main__', +} + + +def iter_source_modules() -> list[Path]: + """Return non-excluded source modules as paths relative to ``SRC_ROOT``.""" + modules: list[Path] = [] + for py in sorted(SRC_ROOT.rglob('*.py')): + rel = py.relative_to(SRC_ROOT) + if any(part in EXCLUDED_DIRS for part in rel.parts): + continue + if py.stem in EXCLUDED_MODULES: + continue + modules.append(rel) + return modules diff --git a/tools/generate_package_docs.py b/tools/generate_package_docs.py index 78915c2f7..c30cc5c3e 100644 --- a/tools/generate_package_docs.py +++ b/tools/generate_package_docs.py @@ -19,8 +19,9 @@ from pathlib import Path from typing import List -REPO_ROOT = Path(__file__).resolve().parents[1] -SRC_ROOT = REPO_ROOT / 'src' / 'easydiffraction' +from _src_tree import REPO_ROOT +from _src_tree import SRC_ROOT + DOCS_OUT_DIR = REPO_ROOT / 'docs' / 'dev' / 'package-structure' diff --git a/tools/test_structure_check.py b/tools/test_structure_check.py index fdacc2392..8046ce4e2 100644 --- a/tools/test_structure_check.py +++ b/tools/test_structure_check.py @@ -31,30 +31,8 @@ import argparse from pathlib import Path -# --------------------------------------------------------------------------- -# Paths -# --------------------------------------------------------------------------- - -ROOT = Path(__file__).resolve().parents[1] -SRC_ROOT = ROOT / 'src' / 'easydiffraction' -TEST_ROOT = ROOT / 'tests' / 'unit' / 'easydiffraction' - -# --------------------------------------------------------------------------- -# Exclusions -# --------------------------------------------------------------------------- - -# Source modules that do not need a dedicated unit-test file. -EXCLUDED_MODULES: set[str] = { - '__init__', - '__main__', -} - -# Source directories whose contents are excluded entirely. -EXCLUDED_DIRS: set[str] = { - '_vendored', - '__pycache__', - 'vendor', -} +from _src_tree import TEST_ROOT +from _src_tree import iter_source_modules # --------------------------------------------------------------------------- # Known aliases: src module stem → accepted test stem(s) @@ -73,21 +51,6 @@ # --------------------------------------------------------------------------- -def _source_modules() -> list[Path]: - """Return all non-excluded source modules as paths relative to SRC_ROOT.""" - modules: list[Path] = [] - for py in sorted(SRC_ROOT.rglob('*.py')): - rel = py.relative_to(SRC_ROOT) - # Skip excluded directories - if any(part in EXCLUDED_DIRS for part in rel.parts): - continue - # Skip excluded module names - if py.stem in EXCLUDED_MODULES: - continue - modules.append(rel) - return modules - - def _find_existing_tests(src_rel: Path) -> list[Path]: """Return existing test files that cover a source module. @@ -159,7 +122,7 @@ def main() -> int: ) args = parser.parse_args() - modules = _source_modules() + modules = iter_source_modules() missing: list[tuple[Path, Path]] = [] covered: list[tuple[Path, list[Path]]] = [] From 8f68124decbb50cc44ef1673b825e63248501ef7 Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Fri, 5 Jun 2026 15:05:57 -0700 Subject: [PATCH 08/57] Document strict test layer placement criteria --- docs/dev/adrs/accepted/test-strategy.md | 9 +++ docs/dev/plans/test-suite-and-validation.md | 2 +- docs/dev/testing-guide.md | 72 +++++++++++++++++++++ 3 files changed, 82 insertions(+), 1 deletion(-) create mode 100644 docs/dev/testing-guide.md diff --git a/docs/dev/adrs/accepted/test-strategy.md b/docs/dev/adrs/accepted/test-strategy.md index 17194c51c..a4c23f5df 100644 --- a/docs/dev/adrs/accepted/test-strategy.md +++ b/docs/dev/adrs/accepted/test-strategy.md @@ -39,3 +39,12 @@ aliases. New features should add focused tests at the lowest useful layer and broader tests when behavior crosses module boundaries. The mirrored structure makes missing coverage easier to spot. + +## Amendments + +[Test Suite and Validation Strategy](../suggestions/test-suite-and-validation.md) +sharpens these layer definitions into strict, testable placement +criteria and adds test cost tiers, coverage policy, codecov +configuration, cross-engine verification documentation, and a nightly +validation harness. The practical placement rules live in the +[Testing Guide](../../testing-guide.md). diff --git a/docs/dev/plans/test-suite-and-validation.md b/docs/dev/plans/test-suite-and-validation.md index 80774765c..b249a34ba 100644 --- a/docs/dev/plans/test-suite-and-validation.md +++ b/docs/dev/plans/test-suite-and-validation.md @@ -216,7 +216,7 @@ files or generated artifacts. Do not run Phase 2 commands during Phase 1. `pixi.toml`. Commit: `Gate unit-test structure check on shared src tree walk` -- [ ] **P1.5 — Strict layer-criteria testing guide (§1)** +- [x] **P1.5 — Strict layer-criteria testing guide (§1)** Write the may/must-not criteria and the "where does this test go?" decision list (location per Open question 5), and tighten the layer wording referenced by the amended `test-strategy.md`. diff --git a/docs/dev/testing-guide.md b/docs/dev/testing-guide.md new file mode 100644 index 000000000..443c8ca7a --- /dev/null +++ b/docs/dev/testing-guide.md @@ -0,0 +1,72 @@ +# Testing Guide + +Practical placement rules for the easydiffraction test suite. The +rationale is recorded in the ADRs +[Test Strategy](adrs/accepted/test-strategy.md) and +[Test Suite and Validation Strategy](adrs/suggestions/test-suite-and-validation.md). + +## Layers — what goes where + +A test belongs to the **lowest** layer whose constraints it can satisfy. + +| Layer | May use | Must NOT use | Speed | +| --------------- | ------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------- | ----------- | +| **unit** | one module under test; in-process logic; `tmp_path` | real calculation engine; network / `download_data()`; filesystem outside `tmp_path`; `sleep`; subprocess | sub-second | +| **functional** | several modules / a workflow; small bundled fixtures | real calculation engine; network / `download_data()` | seconds | +| **integration** | real engines, real fits, real downloaded data (the only layer allowed network and real backends) | — | slow | +| **script** | a full tutorial `.py` executed subprocess-isolated | — | slow | +| **notebook** | a generated `.ipynb` executed via `nbmake` | — | slow | + +Mocking a forbidden dependency (for example a mocked `download_data()`) +keeps a test in a lower layer **only when the mock is explicit**; an +accidental real call is a layer violation. + +The unit tree mirrors `src/` one-to-one. `tools/test_structure_check.py` +enforces this and is gated in CI (`lint-format.yml`); create the mirror +test file for a new module with `tools/gen_tests_scaffold.py`. + +## Where does this test go? + +1. Does it call a real calculation engine (cryspy / crysfml / pdffit) or + download data? → **integration**. +2. Does it run a whole tutorial `.py`? → **script** (and **notebook** + for the generated `.ipynb`). +3. Does it exercise several modules together, in-process, with bundled + fixtures only? → **functional**. +4. Otherwise — one module, in-process, fast? → **unit**, mirrored next + to its source module. + +## Cost tiers (orthogonal to layers) + +Tiers select *when* a test runs in CI; they are independent of the layer. + +| Tier | Marker | Runs on | +| ----------- | ----------------------- | ------------------------------------------------ | +| **fast** | (none — the default) | every push, every pull request, and nightly | +| **pr** | `@pytest.mark.pr` | pull requests and `develop`/`master` | +| **nightly** | `@pytest.mark.nightly` | the scheduled nightly job only (`nightly.yml`) | + +Integration tests are `pr`-tier by default — they are auto-marked in +`tests/integration/conftest.py` because they use real engines. Escalate +an individual test to the heaviest tier with `@pytest.mark.nightly`. + +CI marker selection: + +- feature-branch push: `-m "not pr and not nightly"` +- pull request + `develop`/`master`: `-m "not nightly"` +- nightly schedule: `-m nightly` + +## Numeric tolerances + +Prefer the shared comparison fixtures in `tests/conftest.py` over ad-hoc +per-test tolerances: one documented `rtol`/`atol` pair for intra-engine +numerics, and one (looser) pair for cross-engine comparison. + +## Input-domain coverage + +User input is validated at runtime through `core/validation.py` +(`AttributeSpec` pairs a `TypeValidator` with a content `ValidatorBase`). +Aim input-domain tests at the validators directly — both that they accept +the full valid domain and that they reject (or fall back on) invalid +values. Use `hypothesis` (deterministic profile) for generative coverage +and explicit parametrised tables for the known-critical boundaries. From 906f20d8f93c0d2112d672d60c5d9492988769ca Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Fri, 5 Jun 2026 15:08:12 -0700 Subject: [PATCH 09/57] Relocate network and engine tests to correct layers --- docs/dev/plans/test-suite-and-validation.md | 2 +- .../workflows}/test_background_auto_estimate_corpus.py | 0 .../workflows}/test_experiment_workflow.py | 0 .../workflows}/test_fitting_workflow.py | 0 .../workflows}/test_switchable_categories.py | 0 5 files changed, 1 insertion(+), 1 deletion(-) rename tests/{functional => integration/workflows}/test_background_auto_estimate_corpus.py (100%) rename tests/{functional => integration/workflows}/test_experiment_workflow.py (100%) rename tests/{functional => integration/workflows}/test_fitting_workflow.py (100%) rename tests/{functional => integration/workflows}/test_switchable_categories.py (100%) diff --git a/docs/dev/plans/test-suite-and-validation.md b/docs/dev/plans/test-suite-and-validation.md index b249a34ba..f1ee79503 100644 --- a/docs/dev/plans/test-suite-and-validation.md +++ b/docs/dev/plans/test-suite-and-validation.md @@ -223,7 +223,7 @@ files or generated artifacts. Do not run Phase 2 commands during Phase 1. Files: new `docs/dev/` testing guide (or `test-strategy.md` update). Commit: `Document strict test layer placement criteria` -- [ ] **P1.6 — Test relocation pass (§1)** +- [x] **P1.6 — Test relocation pass (§1)** Move functional tests that call real `download_data()` into integration; relocate or correctly mark slow/engine/network-touching unit tests (the 16 `download_data()` unit call sites must be explicit diff --git a/tests/functional/test_background_auto_estimate_corpus.py b/tests/integration/workflows/test_background_auto_estimate_corpus.py similarity index 100% rename from tests/functional/test_background_auto_estimate_corpus.py rename to tests/integration/workflows/test_background_auto_estimate_corpus.py diff --git a/tests/functional/test_experiment_workflow.py b/tests/integration/workflows/test_experiment_workflow.py similarity index 100% rename from tests/functional/test_experiment_workflow.py rename to tests/integration/workflows/test_experiment_workflow.py diff --git a/tests/functional/test_fitting_workflow.py b/tests/integration/workflows/test_fitting_workflow.py similarity index 100% rename from tests/functional/test_fitting_workflow.py rename to tests/integration/workflows/test_fitting_workflow.py diff --git a/tests/functional/test_switchable_categories.py b/tests/integration/workflows/test_switchable_categories.py similarity index 100% rename from tests/functional/test_switchable_categories.py rename to tests/integration/workflows/test_switchable_categories.py From 6bfc73dcaf031016a0cc00260ee55fed3bf14c5b Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Fri, 5 Jun 2026 15:11:46 -0700 Subject: [PATCH 10/57] Document integration layer defaults to pr tier --- docs/dev/adrs/suggestions/test-suite-and-validation.md | 10 ++++++++++ docs/dev/plans/test-suite-and-validation.md | 5 ++++- 2 files changed, 14 insertions(+), 1 deletion(-) diff --git a/docs/dev/adrs/suggestions/test-suite-and-validation.md b/docs/dev/adrs/suggestions/test-suite-and-validation.md index b4ad68047..5abeca8aa 100644 --- a/docs/dev/adrs/suggestions/test-suite-and-validation.md +++ b/docs/dev/adrs/suggestions/test-suite-and-validation.md @@ -123,6 +123,16 @@ tests opt *into* a heavier tier, so only the minority are tagged. generative fuzzing, full cross-engine sweeps, full benchmarks). Runs on the scheduled nightly job and on demand; never on ordinary pushes. +Orthogonality holds at the **unit and functional** layers: those default +to fast, and an individual test opts into `pr`/`nightly`. The +**integration** layer is the one principled exception — *every* +integration test uses a real engine and/or downloaded data, so the layer +**defaults to the `pr` tier**, applied once in +`tests/integration/conftest.py` rather than by tagging each of ~150 +tests. An integration test may still escalate to `nightly`. This keeps +feature-branch pushes fast (unit + functional only) without scattering +`@pytest.mark.pr` across the whole integration suite. + CI marker selection: ```text diff --git a/docs/dev/plans/test-suite-and-validation.md b/docs/dev/plans/test-suite-and-validation.md index f1ee79503..bbcb85653 100644 --- a/docs/dev/plans/test-suite-and-validation.md +++ b/docs/dev/plans/test-suite-and-validation.md @@ -79,7 +79,10 @@ These are captured in step P1.14 (a future-work record in all (nightly schedule). Because markers are **orthogonal to layers**, the selected expression is applied to **every** pytest invocation under the policy — unit, functional, and integration, in both the source and - package CI jobs — not to integration alone as today. + package CI jobs — not to integration alone as today. The **integration + layer defaults to the `pr` tier** (auto-marked once in + `tests/integration/conftest.py`) because every integration test uses a + real engine; unit/functional default to fast and escalate individually. - **Structure gate (§3):** unify on a single `src/` tree walk shared by `tools/generate_package_docs.py` and `tools/test_structure_check.py`; run the check in CI as a gate. From 31f88823de789121ec6bdcbedfb2939afb7abae4 Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Fri, 5 Jun 2026 15:14:43 -0700 Subject: [PATCH 11/57] Add hypothesis deterministic profile and shared test fixtures --- docs/dev/plans/test-suite-and-validation.md | 2 +- pixi.lock | 66 +++++++++++++++++++++ pyproject.toml | 1 + tests/conftest.py | 56 +++++++++++++++++ 4 files changed, 124 insertions(+), 1 deletion(-) create mode 100644 tests/conftest.py diff --git a/docs/dev/plans/test-suite-and-validation.md b/docs/dev/plans/test-suite-and-validation.md index bbcb85653..cec4cacf0 100644 --- a/docs/dev/plans/test-suite-and-validation.md +++ b/docs/dev/plans/test-suite-and-validation.md @@ -235,7 +235,7 @@ files or generated artifacts. Do not run Phase 2 commands during Phase 1. `tests/unit/**`. Commit: `Relocate network and engine tests to correct layers` -- [ ] **P1.7 — Shared fixtures, hypothesis profile, tolerance convention (§4)** +- [x] **P1.7 — Shared fixtures, hypothesis profile, tolerance convention (§4)** Add `hypothesis` (dev dep) and a deterministic profile (`derandomize`, fixed seed, no committed `.hypothesis` DB). Add a root `tests/conftest.py` with seeded-RNG and one documented diff --git a/pixi.lock b/pixi.lock index 59f1ad7ed..07c98e205 100644 --- a/pixi.lock +++ b/pixi.lock @@ -231,6 +231,7 @@ environments: - pypi: https://files.pythonhosted.org/packages/2f/40/dc34d1a8d5f1e51fc64640b62b191684da52ca469da9cd74e84936ffa4a6/msgpack-1.1.2-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl - pypi: https://files.pythonhosted.org/packages/31/26/ef168b184a25701f9995e8fb7e503fafd7a99c1c77cda1bc1a26ea2ed486/sqlalchemy-2.0.50-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl - pypi: https://files.pythonhosted.org/packages/32/28/79f0f8de97cce916d5ae88a7bee1ad724855e83e6019c0b4d5b3fabc80f3/mkdocstrings_python-2.0.3-py3-none-any.whl + - pypi: https://files.pythonhosted.org/packages/32/46/9cb0e58b2deb7f82b84065f37f3bffeb12413f947f9388e4cac22c4621ce/sortedcontainers-2.4.0-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/32/5d/f7e914f7d9325abff4057cee62c0fa70263683189f774473cbfb534cd13b/matplotlib-3.10.9-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl - pypi: https://files.pythonhosted.org/packages/33/6b/e0547afaf41bf2c42e52430072fa5658766e3d65bd4b03a563d1b6336f57/distlib-0.4.0-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/33/b2/986d1220f6ee931e338d272bc1f3ec02cfe5f9b5fad84e95afdad57f1ebc/format_docstring-0.2.7-py3-none-any.whl @@ -331,6 +332,7 @@ environments: - pypi: https://files.pythonhosted.org/packages/e7/05/c19819d5e3d95294a6f5947fb9b9629efb316b96de511b418c53d245aae6/cycler-0.12.1-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/e7/f9/b06c934a6aa8bc91f566bd2a214fd04c30506c2d9e2b6b171953216a65b6/kiwisolver-1.5.0-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl - pypi: https://files.pythonhosted.org/packages/eb/be/b257e12f9710819fde40adc972578bee6b72c5992da1bc8369bef2597756/nbmake-1.5.5-py3-none-any.whl + - pypi: https://files.pythonhosted.org/packages/ec/6e/e735f27ac1a530a4cd0a31cd970ec495a3a11830fdc5d281cc292593b330/hypothesis-6.155.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/ee/8c/83087ebc47ab0396ce092363001fa37c17153119ee282700c0713a195853/prettytable-3.17.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/f1/16/d905e7f53e661ce2c24686c38048d8e2b750ffc4350009d41c4e6c6c9826/h5py-3.16.0-cp314-cp314-manylinux_2_28_x86_64.whl - pypi: https://files.pythonhosted.org/packages/f1/d9/7fb5aa316bc299258e68c73ba3bddbc499654a07f151cba08f6153988714/pathspec-1.1.1-py3-none-any.whl @@ -554,6 +556,7 @@ environments: - pypi: https://files.pythonhosted.org/packages/2c/01/bc663630c510822c95c47a66af9fa7a443c295b47d5f041e5e6ae62ef659/mkdocs_material-9.7.6-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/2c/19/04f9b178c2d8a15b076c8b5140708fa6ffc5601fb6f1e975537072df5b2a/mergedeep-1.3.4-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/32/28/79f0f8de97cce916d5ae88a7bee1ad724855e83e6019c0b4d5b3fabc80f3/mkdocstrings_python-2.0.3-py3-none-any.whl + - pypi: https://files.pythonhosted.org/packages/32/46/9cb0e58b2deb7f82b84065f37f3bffeb12413f947f9388e4cac22c4621ce/sortedcontainers-2.4.0-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/33/6b/e0547afaf41bf2c42e52430072fa5658766e3d65bd4b03a563d1b6336f57/distlib-0.4.0-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/33/b2/986d1220f6ee931e338d272bc1f3ec02cfe5f9b5fad84e95afdad57f1ebc/format_docstring-0.2.7-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/33/f0/3fe8c6e69135a845f4106f2ff8b6805638d4e85c264e70114e8126689587/tokenize_rt-6.2.0-py2.py3-none-any.whl @@ -655,6 +658,7 @@ environments: - pypi: https://files.pythonhosted.org/packages/e0/f9/0595336914c5619e5f28a1fb793285925a8cd4b432c9da0a987836c7f822/shellingham-1.5.4-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/e7/05/c19819d5e3d95294a6f5947fb9b9629efb316b96de511b418c53d245aae6/cycler-0.12.1-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/eb/be/b257e12f9710819fde40adc972578bee6b72c5992da1bc8369bef2597756/nbmake-1.5.5-py3-none-any.whl + - pypi: https://files.pythonhosted.org/packages/ec/6e/e735f27ac1a530a4cd0a31cd970ec495a3a11830fdc5d281cc292593b330/hypothesis-6.155.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/ee/8c/83087ebc47ab0396ce092363001fa37c17153119ee282700c0713a195853/prettytable-3.17.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/f1/4f/733c48f270565d78b4544f2baddc2fb2a245e5a8640254b12c36ac7ac68e/multidict-6.7.1-cp314-cp314-macosx_11_0_arm64.whl - pypi: https://files.pythonhosted.org/packages/f1/d9/7fb5aa316bc299258e68c73ba3bddbc499654a07f151cba08f6153988714/pathspec-1.1.1-py3-none-any.whl @@ -869,6 +873,7 @@ environments: - pypi: https://files.pythonhosted.org/packages/2c/01/bc663630c510822c95c47a66af9fa7a443c295b47d5f041e5e6ae62ef659/mkdocs_material-9.7.6-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/2c/19/04f9b178c2d8a15b076c8b5140708fa6ffc5601fb6f1e975537072df5b2a/mergedeep-1.3.4-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/32/28/79f0f8de97cce916d5ae88a7bee1ad724855e83e6019c0b4d5b3fabc80f3/mkdocstrings_python-2.0.3-py3-none-any.whl + - pypi: https://files.pythonhosted.org/packages/32/46/9cb0e58b2deb7f82b84065f37f3bffeb12413f947f9388e4cac22c4621ce/sortedcontainers-2.4.0-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/33/6b/e0547afaf41bf2c42e52430072fa5658766e3d65bd4b03a563d1b6336f57/distlib-0.4.0-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/33/b2/986d1220f6ee931e338d272bc1f3ec02cfe5f9b5fad84e95afdad57f1ebc/format_docstring-0.2.7-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/33/f0/3fe8c6e69135a845f4106f2ff8b6805638d4e85c264e70114e8126689587/tokenize_rt-6.2.0-py2.py3-none-any.whl @@ -972,6 +977,7 @@ environments: - pypi: https://files.pythonhosted.org/packages/e0/f9/0595336914c5619e5f28a1fb793285925a8cd4b432c9da0a987836c7f822/shellingham-1.5.4-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/e7/05/c19819d5e3d95294a6f5947fb9b9629efb316b96de511b418c53d245aae6/cycler-0.12.1-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/eb/be/b257e12f9710819fde40adc972578bee6b72c5992da1bc8369bef2597756/nbmake-1.5.5-py3-none-any.whl + - pypi: https://files.pythonhosted.org/packages/ec/6e/e735f27ac1a530a4cd0a31cd970ec495a3a11830fdc5d281cc292593b330/hypothesis-6.155.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/ee/8c/83087ebc47ab0396ce092363001fa37c17153119ee282700c0713a195853/prettytable-3.17.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/f1/d9/7fb5aa316bc299258e68c73ba3bddbc499654a07f151cba08f6153988714/pathspec-1.1.1-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/f2/f2/728f041460f1b9739b85ee23b45fa5a505962ea11fd85bdbe2a02b021373/darkdetect-0.8.0-py3-none-any.whl @@ -1211,6 +1217,7 @@ environments: - pypi: https://files.pythonhosted.org/packages/2c/01/bc663630c510822c95c47a66af9fa7a443c295b47d5f041e5e6ae62ef659/mkdocs_material-9.7.6-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/2c/19/04f9b178c2d8a15b076c8b5140708fa6ffc5601fb6f1e975537072df5b2a/mergedeep-1.3.4-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/32/28/79f0f8de97cce916d5ae88a7bee1ad724855e83e6019c0b4d5b3fabc80f3/mkdocstrings_python-2.0.3-py3-none-any.whl + - pypi: https://files.pythonhosted.org/packages/32/46/9cb0e58b2deb7f82b84065f37f3bffeb12413f947f9388e4cac22c4621ce/sortedcontainers-2.4.0-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/32/91/d024616abdba99e83120e07a20658976f6a343646710760c4a51df126029/matplotlib-3.10.9-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl - pypi: https://files.pythonhosted.org/packages/33/6b/e0547afaf41bf2c42e52430072fa5658766e3d65bd4b03a563d1b6336f57/distlib-0.4.0-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/33/b2/986d1220f6ee931e338d272bc1f3ec02cfe5f9b5fad84e95afdad57f1ebc/format_docstring-0.2.7-py3-none-any.whl @@ -1317,6 +1324,7 @@ environments: - pypi: https://files.pythonhosted.org/packages/e7/05/c19819d5e3d95294a6f5947fb9b9629efb316b96de511b418c53d245aae6/cycler-0.12.1-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/e8/88/5a431cd1ea7587408a66947384b39beb2ab2bcc1c87b7c4082f05036719f/gemmi-0.7.5-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl - pypi: https://files.pythonhosted.org/packages/eb/be/b257e12f9710819fde40adc972578bee6b72c5992da1bc8369bef2597756/nbmake-1.5.5-py3-none-any.whl + - pypi: https://files.pythonhosted.org/packages/ec/6e/e735f27ac1a530a4cd0a31cd970ec495a3a11830fdc5d281cc292593b330/hypothesis-6.155.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/ee/8c/83087ebc47ab0396ce092363001fa37c17153119ee282700c0713a195853/prettytable-3.17.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/f1/d9/7fb5aa316bc299258e68c73ba3bddbc499654a07f151cba08f6153988714/pathspec-1.1.1-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/f2/f2/728f041460f1b9739b85ee23b45fa5a505962ea11fd85bdbe2a02b021373/darkdetect-0.8.0-py3-none-any.whl @@ -1540,6 +1548,7 @@ environments: - pypi: https://files.pythonhosted.org/packages/2c/19/04f9b178c2d8a15b076c8b5140708fa6ffc5601fb6f1e975537072df5b2a/mergedeep-1.3.4-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/2c/7d/49777a3e20b55863d4794384a38acd460c04157b0a00f8602b0d508b8431/propcache-0.5.2-cp312-cp312-macosx_11_0_arm64.whl - pypi: https://files.pythonhosted.org/packages/32/28/79f0f8de97cce916d5ae88a7bee1ad724855e83e6019c0b4d5b3fabc80f3/mkdocstrings_python-2.0.3-py3-none-any.whl + - pypi: https://files.pythonhosted.org/packages/32/46/9cb0e58b2deb7f82b84065f37f3bffeb12413f947f9388e4cac22c4621ce/sortedcontainers-2.4.0-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/32/a0/37fb236da6040e337381dd656cafb97d09eacb998c5db3057547f5ffddd9/pycifrw-5.0.1-cp312-cp312-macosx_11_0_arm64.whl - pypi: https://files.pythonhosted.org/packages/33/6b/e0547afaf41bf2c42e52430072fa5658766e3d65bd4b03a563d1b6336f57/distlib-0.4.0-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/33/b2/986d1220f6ee931e338d272bc1f3ec02cfe5f9b5fad84e95afdad57f1ebc/format_docstring-0.2.7-py3-none-any.whl @@ -1641,6 +1650,7 @@ environments: - pypi: https://files.pythonhosted.org/packages/e0/f9/0595336914c5619e5f28a1fb793285925a8cd4b432c9da0a987836c7f822/shellingham-1.5.4-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/e7/05/c19819d5e3d95294a6f5947fb9b9629efb316b96de511b418c53d245aae6/cycler-0.12.1-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/eb/be/b257e12f9710819fde40adc972578bee6b72c5992da1bc8369bef2597756/nbmake-1.5.5-py3-none-any.whl + - pypi: https://files.pythonhosted.org/packages/ec/6e/e735f27ac1a530a4cd0a31cd970ec495a3a11830fdc5d281cc292593b330/hypothesis-6.155.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/ee/8c/83087ebc47ab0396ce092363001fa37c17153119ee282700c0713a195853/prettytable-3.17.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/f1/d9/7fb5aa316bc299258e68c73ba3bddbc499654a07f151cba08f6153988714/pathspec-1.1.1-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/f2/f2/728f041460f1b9739b85ee23b45fa5a505962ea11fd85bdbe2a02b021373/darkdetect-0.8.0-py3-none-any.whl @@ -1855,6 +1865,7 @@ environments: - pypi: https://files.pythonhosted.org/packages/2c/19/04f9b178c2d8a15b076c8b5140708fa6ffc5601fb6f1e975537072df5b2a/mergedeep-1.3.4-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/30/f5/310d104ddf41eb5a70f4c268d22508dfb0c3c8e86fec152be34d0d2ed819/greenlet-3.5.1-cp312-cp312-win_amd64.whl - pypi: https://files.pythonhosted.org/packages/32/28/79f0f8de97cce916d5ae88a7bee1ad724855e83e6019c0b4d5b3fabc80f3/mkdocstrings_python-2.0.3-py3-none-any.whl + - pypi: https://files.pythonhosted.org/packages/32/46/9cb0e58b2deb7f82b84065f37f3bffeb12413f947f9388e4cac22c4621ce/sortedcontainers-2.4.0-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/32/60/fca69c534602a7ced04280c952a246ad1edde2a6ca3a164f65d32ac41fe7/chardet-7.4.3-cp312-cp312-win_amd64.whl - pypi: https://files.pythonhosted.org/packages/33/6b/e0547afaf41bf2c42e52430072fa5658766e3d65bd4b03a563d1b6336f57/distlib-0.4.0-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/33/b2/986d1220f6ee931e338d272bc1f3ec02cfe5f9b5fad84e95afdad57f1ebc/format_docstring-0.2.7-py3-none-any.whl @@ -1958,6 +1969,7 @@ environments: - pypi: https://files.pythonhosted.org/packages/e7/05/c19819d5e3d95294a6f5947fb9b9629efb316b96de511b418c53d245aae6/cycler-0.12.1-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/eb/be/b257e12f9710819fde40adc972578bee6b72c5992da1bc8369bef2597756/nbmake-1.5.5-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/eb/f2/53be7a4ba5816e13c39be0f728facac4bcb39cf4903ceeec54b006511c8f/gemmi-0.7.5-cp312-cp312-win_amd64.whl + - pypi: https://files.pythonhosted.org/packages/ec/6e/e735f27ac1a530a4cd0a31cd970ec495a3a11830fdc5d281cc292593b330/hypothesis-6.155.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/ee/8c/83087ebc47ab0396ce092363001fa37c17153119ee282700c0713a195853/prettytable-3.17.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/f1/d9/7fb5aa316bc299258e68c73ba3bddbc499654a07f151cba08f6153988714/pathspec-1.1.1-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/f2/f2/728f041460f1b9739b85ee23b45fa5a505962ea11fd85bdbe2a02b021373/darkdetect-0.8.0-py3-none-any.whl @@ -2196,6 +2208,7 @@ environments: - pypi: https://files.pythonhosted.org/packages/2f/40/dc34d1a8d5f1e51fc64640b62b191684da52ca469da9cd74e84936ffa4a6/msgpack-1.1.2-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl - pypi: https://files.pythonhosted.org/packages/31/26/ef168b184a25701f9995e8fb7e503fafd7a99c1c77cda1bc1a26ea2ed486/sqlalchemy-2.0.50-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl - pypi: https://files.pythonhosted.org/packages/32/28/79f0f8de97cce916d5ae88a7bee1ad724855e83e6019c0b4d5b3fabc80f3/mkdocstrings_python-2.0.3-py3-none-any.whl + - pypi: https://files.pythonhosted.org/packages/32/46/9cb0e58b2deb7f82b84065f37f3bffeb12413f947f9388e4cac22c4621ce/sortedcontainers-2.4.0-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/32/5d/f7e914f7d9325abff4057cee62c0fa70263683189f774473cbfb534cd13b/matplotlib-3.10.9-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl - pypi: https://files.pythonhosted.org/packages/33/6b/e0547afaf41bf2c42e52430072fa5658766e3d65bd4b03a563d1b6336f57/distlib-0.4.0-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/33/b2/986d1220f6ee931e338d272bc1f3ec02cfe5f9b5fad84e95afdad57f1ebc/format_docstring-0.2.7-py3-none-any.whl @@ -2296,6 +2309,7 @@ environments: - pypi: https://files.pythonhosted.org/packages/e7/05/c19819d5e3d95294a6f5947fb9b9629efb316b96de511b418c53d245aae6/cycler-0.12.1-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/e7/f9/b06c934a6aa8bc91f566bd2a214fd04c30506c2d9e2b6b171953216a65b6/kiwisolver-1.5.0-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl - pypi: https://files.pythonhosted.org/packages/eb/be/b257e12f9710819fde40adc972578bee6b72c5992da1bc8369bef2597756/nbmake-1.5.5-py3-none-any.whl + - pypi: https://files.pythonhosted.org/packages/ec/6e/e735f27ac1a530a4cd0a31cd970ec495a3a11830fdc5d281cc292593b330/hypothesis-6.155.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/ee/8c/83087ebc47ab0396ce092363001fa37c17153119ee282700c0713a195853/prettytable-3.17.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/f1/16/d905e7f53e661ce2c24686c38048d8e2b750ffc4350009d41c4e6c6c9826/h5py-3.16.0-cp314-cp314-manylinux_2_28_x86_64.whl - pypi: https://files.pythonhosted.org/packages/f1/d9/7fb5aa316bc299258e68c73ba3bddbc499654a07f151cba08f6153988714/pathspec-1.1.1-py3-none-any.whl @@ -2519,6 +2533,7 @@ environments: - pypi: https://files.pythonhosted.org/packages/2c/01/bc663630c510822c95c47a66af9fa7a443c295b47d5f041e5e6ae62ef659/mkdocs_material-9.7.6-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/2c/19/04f9b178c2d8a15b076c8b5140708fa6ffc5601fb6f1e975537072df5b2a/mergedeep-1.3.4-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/32/28/79f0f8de97cce916d5ae88a7bee1ad724855e83e6019c0b4d5b3fabc80f3/mkdocstrings_python-2.0.3-py3-none-any.whl + - pypi: https://files.pythonhosted.org/packages/32/46/9cb0e58b2deb7f82b84065f37f3bffeb12413f947f9388e4cac22c4621ce/sortedcontainers-2.4.0-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/33/6b/e0547afaf41bf2c42e52430072fa5658766e3d65bd4b03a563d1b6336f57/distlib-0.4.0-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/33/b2/986d1220f6ee931e338d272bc1f3ec02cfe5f9b5fad84e95afdad57f1ebc/format_docstring-0.2.7-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/33/f0/3fe8c6e69135a845f4106f2ff8b6805638d4e85c264e70114e8126689587/tokenize_rt-6.2.0-py2.py3-none-any.whl @@ -2620,6 +2635,7 @@ environments: - pypi: https://files.pythonhosted.org/packages/e0/f9/0595336914c5619e5f28a1fb793285925a8cd4b432c9da0a987836c7f822/shellingham-1.5.4-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/e7/05/c19819d5e3d95294a6f5947fb9b9629efb316b96de511b418c53d245aae6/cycler-0.12.1-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/eb/be/b257e12f9710819fde40adc972578bee6b72c5992da1bc8369bef2597756/nbmake-1.5.5-py3-none-any.whl + - pypi: https://files.pythonhosted.org/packages/ec/6e/e735f27ac1a530a4cd0a31cd970ec495a3a11830fdc5d281cc292593b330/hypothesis-6.155.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/ee/8c/83087ebc47ab0396ce092363001fa37c17153119ee282700c0713a195853/prettytable-3.17.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/f1/4f/733c48f270565d78b4544f2baddc2fb2a245e5a8640254b12c36ac7ac68e/multidict-6.7.1-cp314-cp314-macosx_11_0_arm64.whl - pypi: https://files.pythonhosted.org/packages/f1/d9/7fb5aa316bc299258e68c73ba3bddbc499654a07f151cba08f6153988714/pathspec-1.1.1-py3-none-any.whl @@ -2834,6 +2850,7 @@ environments: - pypi: https://files.pythonhosted.org/packages/2c/01/bc663630c510822c95c47a66af9fa7a443c295b47d5f041e5e6ae62ef659/mkdocs_material-9.7.6-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/2c/19/04f9b178c2d8a15b076c8b5140708fa6ffc5601fb6f1e975537072df5b2a/mergedeep-1.3.4-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/32/28/79f0f8de97cce916d5ae88a7bee1ad724855e83e6019c0b4d5b3fabc80f3/mkdocstrings_python-2.0.3-py3-none-any.whl + - pypi: https://files.pythonhosted.org/packages/32/46/9cb0e58b2deb7f82b84065f37f3bffeb12413f947f9388e4cac22c4621ce/sortedcontainers-2.4.0-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/33/6b/e0547afaf41bf2c42e52430072fa5658766e3d65bd4b03a563d1b6336f57/distlib-0.4.0-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/33/b2/986d1220f6ee931e338d272bc1f3ec02cfe5f9b5fad84e95afdad57f1ebc/format_docstring-0.2.7-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/33/f0/3fe8c6e69135a845f4106f2ff8b6805638d4e85c264e70114e8126689587/tokenize_rt-6.2.0-py2.py3-none-any.whl @@ -2937,6 +2954,7 @@ environments: - pypi: https://files.pythonhosted.org/packages/e0/f9/0595336914c5619e5f28a1fb793285925a8cd4b432c9da0a987836c7f822/shellingham-1.5.4-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/e7/05/c19819d5e3d95294a6f5947fb9b9629efb316b96de511b418c53d245aae6/cycler-0.12.1-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/eb/be/b257e12f9710819fde40adc972578bee6b72c5992da1bc8369bef2597756/nbmake-1.5.5-py3-none-any.whl + - pypi: https://files.pythonhosted.org/packages/ec/6e/e735f27ac1a530a4cd0a31cd970ec495a3a11830fdc5d281cc292593b330/hypothesis-6.155.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/ee/8c/83087ebc47ab0396ce092363001fa37c17153119ee282700c0713a195853/prettytable-3.17.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/f1/d9/7fb5aa316bc299258e68c73ba3bddbc499654a07f151cba08f6153988714/pathspec-1.1.1-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/f2/f2/728f041460f1b9739b85ee23b45fa5a505962ea11fd85bdbe2a02b021373/darkdetect-0.8.0-py3-none-any.whl @@ -8779,6 +8797,7 @@ packages: - docstripy ; extra == 'dev' - format-docstring ; extra == 'dev' - gitpython ; extra == 'dev' + - hypothesis ; extra == 'dev' - interrogate ; extra == 'dev' - jinja2 ; extra == 'dev' - jupyterquiz ; extra == 'dev' @@ -9648,6 +9667,10 @@ packages: - griffelib>=2.0 - typing-extensions>=4.0 ; python_full_version < '3.11' requires_python: '>=3.10' +- pypi: https://files.pythonhosted.org/packages/32/46/9cb0e58b2deb7f82b84065f37f3bffeb12413f947f9388e4cac22c4621ce/sortedcontainers-2.4.0-py2.py3-none-any.whl + name: sortedcontainers + version: 2.4.0 + sha256: a163dcaede0f1c021485e957a39245190e74249897e2ae4b2aa38595db237ee0 - pypi: https://files.pythonhosted.org/packages/32/5d/f7e914f7d9325abff4057cee62c0fa70263683189f774473cbfb534cd13b/matplotlib-3.10.9-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl name: matplotlib version: 3.10.9 @@ -12714,6 +12737,49 @@ packages: version: 0.7.5 sha256: a1fdb6f72006495b5119e3a8bb5c3185efa708b785bd4a5ce4397ef7abb3fec7 requires_python: '>=3.9' +- pypi: https://files.pythonhosted.org/packages/ec/6e/e735f27ac1a530a4cd0a31cd970ec495a3a11830fdc5d281cc292593b330/hypothesis-6.155.2-py3-none-any.whl + name: hypothesis + version: 6.155.2 + sha256: c85ce6dcd630a90ce501f1d1dd1bc84b97f5649ca8a27e134c8cbf5aa480b1a5 + requires_dist: + - exceptiongroup>=1.0.0 ; python_full_version < '3.11' + - sortedcontainers>=2.1.0,<3.0.0 + - click>=7.0 ; extra == 'cli' + - black>=20.8b0 ; extra == 'cli' + - rich>=9.0.0 ; extra == 'cli' + - libcst>=0.3.16 ; extra == 'codemods' + - black>=20.8b0 ; extra == 'ghostwriter' + - pytz>=2014.1 ; extra == 'pytz' + - python-dateutil>=1.4 ; extra == 'dateutil' + - lark>=0.10.1 ; extra == 'lark' + - numpy>=1.21.6 ; extra == 'numpy' + - pandas>=1.1 ; extra == 'pandas' + - pytest>=4.6 ; extra == 'pytest' + - dpcontracts>=0.4 ; extra == 'dpcontracts' + - redis>=3.0.0 ; extra == 'redis' + - hypothesis-crosshair>=0.0.28 ; extra == 'crosshair' + - crosshair-tool>=0.0.106 ; extra == 'crosshair' + - tzdata>=2026.2 ; (sys_platform == 'emscripten' and extra == 'zoneinfo') or (sys_platform == 'win32' and extra == 'zoneinfo') + - django>=5.2 ; extra == 'django' + - watchdog>=4.0.0 ; extra == 'watchdog' + - black>=20.8b0 ; extra == 'all' + - click>=7.0 ; extra == 'all' + - crosshair-tool>=0.0.106 ; extra == 'all' + - django>=5.2 ; extra == 'all' + - dpcontracts>=0.4 ; extra == 'all' + - hypothesis-crosshair>=0.0.28 ; extra == 'all' + - lark>=0.10.1 ; extra == 'all' + - libcst>=0.3.16 ; extra == 'all' + - numpy>=1.21.6 ; extra == 'all' + - pandas>=1.1 ; extra == 'all' + - pytest>=4.6 ; extra == 'all' + - python-dateutil>=1.4 ; extra == 'all' + - pytz>=2014.1 ; extra == 'all' + - redis>=3.0.0 ; extra == 'all' + - rich>=9.0.0 ; extra == 'all' + - tzdata>=2026.2 ; (sys_platform == 'emscripten' and extra == 'all') or (sys_platform == 'win32' and extra == 'all') + - watchdog>=4.0.0 ; extra == 'all' + requires_python: '>=3.10' - pypi: https://files.pythonhosted.org/packages/ee/8c/83087ebc47ab0396ce092363001fa37c17153119ee282700c0713a195853/prettytable-3.17.0-py3-none-any.whl name: prettytable version: 3.17.0 diff --git a/pyproject.toml b/pyproject.toml index c2e2563a4..fde922ba3 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -63,6 +63,7 @@ dev = [ 'pytest', # Testing 'pytest-cov', # Test coverage 'pytest-xdist', # Enable parallel testing + 'hypothesis', # Property-based / input-domain testing 'ruff', # Linting and formatting code 'radon', # Code complexity and maintainability 'validate-pyproject[all]', # Validate pyproject.toml diff --git a/tests/conftest.py b/tests/conftest.py new file mode 100644 index 000000000..2feb4f438 --- /dev/null +++ b/tests/conftest.py @@ -0,0 +1,56 @@ +# SPDX-FileCopyrightText: 2026 EasyScience contributors +# SPDX-License-Identifier: BSD-3-Clause +"""Shared fixtures and configuration for the whole test suite. + +Defines the documented numeric-tolerance convention (one intra-engine +``rtol``/``atol`` pair and one looser cross-engine pair) and a +deterministic ``hypothesis`` profile, so property-based tests never flake +or depend on test ordering (see AGENTS.md testing rules and the +[Testing Guide](../docs/dev/testing-guide.md)). +""" + +from __future__ import annotations + +import pytest +from hypothesis import HealthCheck +from hypothesis import settings + +# Deterministic hypothesis profile: reproducible behaviour, no committed +# example database, no per-example deadline (engine-free property tests can +# still be a touch slow under coverage). +settings.register_profile( + 'easydiffraction', + derandomize=True, + database=None, + deadline=None, + suppress_health_check=[HealthCheck.too_slow], +) +settings.load_profile('easydiffraction') + +# Numeric-tolerance convention. Prefer these over ad-hoc per-test values. +# Intra-engine: a value against an expected scalar from the same engine. +# Cross-engine: patterns/parameters compared between calculation engines. +INTRA_ENGINE_RTOL = 1e-6 +INTRA_ENGINE_ATOL = 1e-8 +CROSS_ENGINE_RTOL = 1e-3 +CROSS_ENGINE_ATOL = 1e-5 + + +@pytest.fixture +def intra_engine_tol() -> dict[str, float]: + """Return the documented intra-engine ``rtol``/``atol`` pair.""" + return {'rtol': INTRA_ENGINE_RTOL, 'atol': INTRA_ENGINE_ATOL} + + +@pytest.fixture +def cross_engine_tol() -> dict[str, float]: + """Return the documented (looser) cross-engine ``rtol``/``atol`` pair.""" + return {'rtol': CROSS_ENGINE_RTOL, 'atol': CROSS_ENGINE_ATOL} + + +@pytest.fixture +def seeded_rng(): + """Return a NumPy generator with a fixed seed for reproducible tests.""" + import numpy as np + + return np.random.default_rng(12345) From 146ad57ed15008fbc009f8e4a2669c7da778fed7 Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Fri, 5 Jun 2026 22:29:22 -0700 Subject: [PATCH 12/57] Add property-based input-domain tests for validators --- docs/dev/plans/test-suite-and-validation.md | 2 +- .../core/test_validation_properties.py | 199 ++++++++++++++++++ 2 files changed, 200 insertions(+), 1 deletion(-) create mode 100644 tests/unit/easydiffraction/core/test_validation_properties.py diff --git a/docs/dev/plans/test-suite-and-validation.md b/docs/dev/plans/test-suite-and-validation.md index cec4cacf0..bd94d3399 100644 --- a/docs/dev/plans/test-suite-and-validation.md +++ b/docs/dev/plans/test-suite-and-validation.md @@ -244,7 +244,7 @@ files or generated artifacts. Do not run Phase 2 commands during Phase 1. guide. Commit: `Add hypothesis deterministic profile and shared test fixtures` -- [ ] **P1.8 — Input-domain property tests on validators (§4)** +- [x] **P1.8 — Input-domain property tests on validators (§4)** Property-based + explicit boundary-table tests against `core/validation.py` (`TypeValidator`, content `ValidatorBase` subclasses) through parameter (`core/variable.py`) and category diff --git a/tests/unit/easydiffraction/core/test_validation_properties.py b/tests/unit/easydiffraction/core/test_validation_properties.py new file mode 100644 index 000000000..8c272ba39 --- /dev/null +++ b/tests/unit/easydiffraction/core/test_validation_properties.py @@ -0,0 +1,199 @@ +# SPDX-FileCopyrightText: 2026 EasyScience contributors +# SPDX-License-Identifier: BSD-3-Clause +"""Input-domain (property-based) tests for the validation framework. + +These target the runtime validators directly — the single boundary +through which user input reaches the model — covering each input domain +the way one would test ``sqrt`` (negative / zero / positive, int / float, +and wrong types). Invalid input is *rejected by fallback*: the validator +logs and returns the ``default`` (or ``current``) value rather than +raising, so the assertions compare the returned value. The logger is kept +in ``WARN`` mode so that fallback path is exercised (a sibling test may +have left it in ``RAISE`` mode). +""" + +from __future__ import annotations + +import numpy as np +import pytest +from hypothesis import given +from hypothesis import strategies as st + +from easydiffraction.core.validation import AttributeSpec +from easydiffraction.core.validation import DataTypes +from easydiffraction.core.validation import MembershipValidator +from easydiffraction.core.validation import RangeValidator +from easydiffraction.core.validation import RegexValidator +from easydiffraction.utils.logging import log + + +def _warn() -> None: + """Keep the logger non-raising so validators take the fallback path.""" + log.configure(reaction=log.Reaction.WARN) + + +# --------------------------------------------------------------------------- +# TypeValidator (NUMERIC) — the sqrt-style domain sweep +# --------------------------------------------------------------------------- + +NUMERIC_DEFAULT = -1.0 + + +def _numeric_spec() -> AttributeSpec: + return AttributeSpec(data_type=DataTypes.NUMERIC, default=NUMERIC_DEFAULT) + + +@given(value=st.integers()) +def test_numeric_accepts_any_integer(value): + _warn() + assert _numeric_spec().validated(value, name='p') == value + + +@given(value=st.floats(allow_nan=False, allow_infinity=False)) +def test_numeric_accepts_any_finite_float(value): + _warn() + assert _numeric_spec().validated(value, name='p') == value + + +@given(value=st.text()) +def test_numeric_rejects_text_with_fallback(value): + _warn() + result = _numeric_spec().validated(value, name='p') + assert result == NUMERIC_DEFAULT + assert not isinstance(result, str) + + +@pytest.mark.parametrize( + 'value', + [0, -1, 1, 0.0, -2.5, 3.14, np.int64(5), np.float64(2.0)], +) +def test_numeric_accepts_boundary_table(value): + _warn() + assert _numeric_spec().validated(value, name='p') == value + + +@pytest.mark.parametrize('value', ['x', '', [1], {}, (1,)]) +def test_numeric_rejects_non_numeric_table(value): + _warn() + assert _numeric_spec().validated(value, name='p') == NUMERIC_DEFAULT + + +# --------------------------------------------------------------------------- +# RangeValidator — occupancy in [0, 1]; cell length > 0 +# --------------------------------------------------------------------------- + +OCC_DEFAULT = 0.5 + + +def _occupancy_spec() -> AttributeSpec: + return AttributeSpec( + data_type=DataTypes.NUMERIC, + default=OCC_DEFAULT, + validator=RangeValidator(ge=0.0, le=1.0), + ) + + +@given(value=st.floats(min_value=0.0, max_value=1.0)) +def test_occupancy_accepts_unit_interval(value): + _warn() + assert _occupancy_spec().validated(value, name='occ') == value + + +@given( + value=st.floats(allow_nan=False, allow_infinity=False).filter( + lambda v: v < 0.0 or v > 1.0 + ) +) +def test_occupancy_rejects_outside_unit_interval(value): + _warn() + assert _occupancy_spec().validated(value, name='occ') == OCC_DEFAULT + + +@pytest.mark.parametrize('value', [0.0, 1.0, 0.5, 0.999999]) +def test_occupancy_boundary_accepts(value): + _warn() + assert _occupancy_spec().validated(value, name='occ') == value + + +@pytest.mark.parametrize('value', [-0.0001, 1.0001, -5.0, 100.0]) +def test_occupancy_boundary_rejects(value): + _warn() + assert _occupancy_spec().validated(value, name='occ') == OCC_DEFAULT + + +CELL_DEFAULT = 1.0 + + +def _cell_length_spec() -> AttributeSpec: + return AttributeSpec( + data_type=DataTypes.NUMERIC, + default=CELL_DEFAULT, + validator=RangeValidator(gt=0.0), + ) + + +@given(value=st.floats(min_value=1e-6, max_value=1e6)) +def test_cell_length_accepts_positive(value): + _warn() + assert _cell_length_spec().validated(value, name='a') == value + + +@pytest.mark.parametrize('value', [0.0, -1.0, -1e-9]) +def test_cell_length_rejects_nonpositive(value): + _warn() + assert _cell_length_spec().validated(value, name='a') == CELL_DEFAULT + + +# --------------------------------------------------------------------------- +# MembershipValidator — space-group number in {1, ..., 230} +# --------------------------------------------------------------------------- + +SG_DEFAULT = 1 +SG_NUMBERS = tuple(range(1, 231)) + + +def _sg_spec() -> AttributeSpec: + return AttributeSpec( + data_type=DataTypes.INTEGER, + default=SG_DEFAULT, + validator=MembershipValidator(SG_NUMBERS), + ) + + +@given(value=st.integers(min_value=1, max_value=230)) +def test_space_group_accepts_valid_number(value): + _warn() + assert _sg_spec().validated(value, name='sg') == value + + +@given(value=st.integers().filter(lambda v: v < 1 or v > 230)) +def test_space_group_rejects_out_of_range(value): + _warn() + assert _sg_spec().validated(value, name='sg') == SG_DEFAULT + + +# --------------------------------------------------------------------------- +# RegexValidator — atom-site label like "La", "O1" +# --------------------------------------------------------------------------- + +LABEL_DEFAULT = 'X' + + +def _label_spec() -> AttributeSpec: + return AttributeSpec( + data_type=DataTypes.STRING, + default=LABEL_DEFAULT, + validator=RegexValidator(r'^[A-Za-z]{1,2}\d*$'), + ) + + +@given(value=st.from_regex(r'\A[A-Za-z]{1,2}[0-9]*\Z', fullmatch=True)) +def test_label_accepts_matching(value): + _warn() + assert _label_spec().validated(value, name='label') == value + + +@pytest.mark.parametrize('value', ['', '1A', 'Abc', 'La-1', ' O']) +def test_label_rejects_non_matching(value): + _warn() + assert _label_spec().validated(value, name='label') == LABEL_DEFAULT From dc90a36f2af6d5a5d66a27e04c8cfbcaab01b682 Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Fri, 5 Jun 2026 22:29:42 -0700 Subject: [PATCH 13/57] Raise coverage fail_under to 80 percent --- docs/dev/plans/test-suite-and-validation.md | 2 +- pyproject.toml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/dev/plans/test-suite-and-validation.md b/docs/dev/plans/test-suite-and-validation.md index bd94d3399..e701440c6 100644 --- a/docs/dev/plans/test-suite-and-validation.md +++ b/docs/dev/plans/test-suite-and-validation.md @@ -254,7 +254,7 @@ files or generated artifacts. Do not run Phase 2 commands during Phase 1. category tests). Commit: `Add property-based input-domain tests for validators` -- [ ] **P1.9 — Raise coverage gate to 80% (§4)** +- [x] **P1.9 — Raise coverage gate to 80% (§4)** Set `[tool.coverage.report] fail_under = 80`. (Resolve Open question 2 in Phase 2 if unit coverage is below 80 after P1.8.) Files: `pyproject.toml`. diff --git a/pyproject.toml b/pyproject.toml index fde922ba3..be88546c8 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -184,7 +184,7 @@ omit = [ [tool.coverage.report] show_missing = true # Show missing lines skip_covered = false # Skip files with 100% coverage in the report -fail_under = 65 # Minimum coverage percentage to pass +fail_under = 80 # Minimum coverage percentage to pass (ramp toward 90-95) ########################## # Configuration for pytest From aacc3e672002f3106818f1c0943f524017344a77 Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Fri, 5 Jun 2026 22:32:51 -0700 Subject: [PATCH 14/57] Add strict docs build and spell check on every push --- .github/workflows/test.yml | 19 ++++++++++++ docs/dev/plans/test-suite-and-validation.md | 2 +- pixi.lock | 33 +++++++++++++++++++++ pixi.toml | 8 +++++ pyproject.toml | 10 +++++++ 5 files changed, 71 insertions(+), 1 deletion(-) diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index ee7320f35..e3f1f88dd 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -69,6 +69,25 @@ jobs: echo 'pytest_marks=-m "not pr and not nightly"' >> "$GITHUB_OUTPUT" fi + # Job: Fast documentation checks on every push (tutorials NOT executed, + # unlike the slow docs.yml). Strict build catches broken nav/internal + # links; codespell catches typos. See ADR test-suite-and-validation §9. + docs-checks: + runs-on: ubuntu-latest + + steps: + - name: Checkout repository + uses: actions/checkout@v6 + + - name: Set up pixi + uses: ./.github/actions/setup-pixi + + - name: Strict documentation build (tutorials not executed) + run: pixi run docs-build-strict + + - name: Spell check + run: pixi run spell-check + # Job 2: Test code source-test: needs: env-prepare # depend on previous job diff --git a/docs/dev/plans/test-suite-and-validation.md b/docs/dev/plans/test-suite-and-validation.md index e701440c6..46031cfc4 100644 --- a/docs/dev/plans/test-suite-and-validation.md +++ b/docs/dev/plans/test-suite-and-validation.md @@ -260,7 +260,7 @@ files or generated artifacts. Do not run Phase 2 commands during Phase 1. Files: `pyproject.toml`. Commit: `Raise coverage fail_under to 80 percent` -- [ ] **P1.10 — Fast docs build gate (§9)** +- [x] **P1.10 — Fast docs build gate (§9)** Add `docs-build-strict` (`mkdocs build --strict`, tutorials not executed), `link-check` (`lychee`), and `spell-check` (`codespell`) pixi tasks with config and ignore lists; add a fast every-push job to diff --git a/pixi.lock b/pixi.lock index 07c98e205..7d69de5bd 100644 --- a/pixi.lock +++ b/pixi.lock @@ -247,6 +247,7 @@ environments: - pypi: https://files.pythonhosted.org/packages/3e/5c/fb93d3092640a24dfb7bd7727a24016d7c01774ca013e60efd3f683c8002/backrefs-7.0-py314-none-any.whl - pypi: https://files.pythonhosted.org/packages/3f/d0/7b958df957e4827837b590944008f0b28078f552b451f7407b4b3d54f574/asciichartpy-1.5.25-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/3f/f9/2b3ff4e56e5fa7debfaf9eb135d0da96f3e9a1d5b27222223c7296336e5f/typer-0.25.1-py3-none-any.whl + - pypi: https://files.pythonhosted.org/packages/42/a1/52fa05533e95fe45bcc09bcf8a503874b1c08f221a4e35608017e0938f55/codespell-2.4.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/43/e3/7d92a15f894aa0c9c4b49b8ee9ac9850d6e63b03c9c32c0367a13ae62209/mpmath-1.3.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/46/b4/0887c88ddfaba1d7140ea335144eb904af97550786ee58bdb295ff10d255/crysfml-0.6.2-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl - pypi: https://files.pythonhosted.org/packages/52/59/0782e51887ac6b07ffd1570e0364cf901ebc36345fea669969d2084baebb/simple_websocket-1.1.0-py3-none-any.whl @@ -570,6 +571,7 @@ environments: - pypi: https://files.pythonhosted.org/packages/3f/d0/7b958df957e4827837b590944008f0b28078f552b451f7407b4b3d54f574/asciichartpy-1.5.25-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/3f/f9/2b3ff4e56e5fa7debfaf9eb135d0da96f3e9a1d5b27222223c7296336e5f/typer-0.25.1-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/41/86/86231232fff41c9f8e4a1a7d7a597d349a02527109c3af7d618366122139/matplotlib-3.10.9-cp314-cp314-macosx_11_0_arm64.whl + - pypi: https://files.pythonhosted.org/packages/42/a1/52fa05533e95fe45bcc09bcf8a503874b1c08f221a4e35608017e0938f55/codespell-2.4.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/43/e3/7d92a15f894aa0c9c4b49b8ee9ac9850d6e63b03c9c32c0367a13ae62209/mpmath-1.3.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/49/b2/97980f3ad4fae37dd7fe31626e2bf75fbf8bdf5d303950ec1fab39a12da8/kiwisolver-1.5.0-cp314-cp314-macosx_11_0_arm64.whl - pypi: https://files.pythonhosted.org/packages/51/ac/b9d68ebddfe1b02c77af5bf81120e12b036b4432dc6af7a303d90e2bc38b/chardet-7.4.3-cp314-cp314-macosx_11_0_arm64.whl @@ -888,6 +890,7 @@ environments: - pypi: https://files.pythonhosted.org/packages/3e/5c/fb93d3092640a24dfb7bd7727a24016d7c01774ca013e60efd3f683c8002/backrefs-7.0-py314-none-any.whl - pypi: https://files.pythonhosted.org/packages/3f/d0/7b958df957e4827837b590944008f0b28078f552b451f7407b4b3d54f574/asciichartpy-1.5.25-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/3f/f9/2b3ff4e56e5fa7debfaf9eb135d0da96f3e9a1d5b27222223c7296336e5f/typer-0.25.1-py3-none-any.whl + - pypi: https://files.pythonhosted.org/packages/42/a1/52fa05533e95fe45bcc09bcf8a503874b1c08f221a4e35608017e0938f55/codespell-2.4.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/43/e3/7d92a15f894aa0c9c4b49b8ee9ac9850d6e63b03c9c32c0367a13ae62209/mpmath-1.3.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/4b/39/f0e8ea762a764a9dc52aa7dabcfad51a354819de1f0d4652b6a1122424d6/scipy-1.17.1-cp314-cp314-win_amd64.whl - pypi: https://files.pythonhosted.org/packages/52/59/0782e51887ac6b07ffd1570e0364cf901ebc36345fea669969d2084baebb/simple_websocket-1.1.0-py3-none-any.whl @@ -1233,6 +1236,7 @@ environments: - pypi: https://files.pythonhosted.org/packages/3f/d0/7b958df957e4827837b590944008f0b28078f552b451f7407b4b3d54f574/asciichartpy-1.5.25-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/3f/f9/2b3ff4e56e5fa7debfaf9eb135d0da96f3e9a1d5b27222223c7296336e5f/typer-0.25.1-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/40/69/b91cda0647df839483201545913514c2827ebea5e5ccdf931842763bc127/greenlet-3.5.1-cp312-cp312-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl + - pypi: https://files.pythonhosted.org/packages/42/a1/52fa05533e95fe45bcc09bcf8a503874b1c08f221a4e35608017e0938f55/codespell-2.4.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/43/e3/7d92a15f894aa0c9c4b49b8ee9ac9850d6e63b03c9c32c0367a13ae62209/mpmath-1.3.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/43/e3/fdc657359e919462369869f1c9f0e973f353f9a9ee295a39b1fea8ee1a77/pillow-12.2.0-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl - pypi: https://files.pythonhosted.org/packages/52/59/0782e51887ac6b07ffd1570e0364cf901ebc36345fea669969d2084baebb/simple_websocket-1.1.0-py3-none-any.whl @@ -1564,6 +1568,7 @@ environments: - pypi: https://files.pythonhosted.org/packages/3e/17/1f31d8562e6f970d64911f1abc330d233bc0c0601411cf7e19c1292be6da/spdx_headers-1.5.1-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/3f/d0/7b958df957e4827837b590944008f0b28078f552b451f7407b4b3d54f574/asciichartpy-1.5.25-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/3f/f9/2b3ff4e56e5fa7debfaf9eb135d0da96f3e9a1d5b27222223c7296336e5f/typer-0.25.1-py3-none-any.whl + - pypi: https://files.pythonhosted.org/packages/42/a1/52fa05533e95fe45bcc09bcf8a503874b1c08f221a4e35608017e0938f55/codespell-2.4.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/43/e3/7d92a15f894aa0c9c4b49b8ee9ac9850d6e63b03c9c32c0367a13ae62209/mpmath-1.3.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/52/59/0782e51887ac6b07ffd1570e0364cf901ebc36345fea669969d2084baebb/simple_websocket-1.1.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/53/3e/405b59cfa13021a56bba395a6b3aca8cec012b45bf177b0eaf7a202cde2c/contourpy-1.3.3-cp312-cp312-macosx_11_0_arm64.whl @@ -1880,6 +1885,7 @@ environments: - pypi: https://files.pythonhosted.org/packages/3f/d0/7b958df957e4827837b590944008f0b28078f552b451f7407b4b3d54f574/asciichartpy-1.5.25-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/3f/f9/2b3ff4e56e5fa7debfaf9eb135d0da96f3e9a1d5b27222223c7296336e5f/typer-0.25.1-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/40/8c/985c1d41ea1107c2534abd9870e4ed5c8e7669b5c308297835c001e7a1c4/pydantic_core-2.46.4-cp312-cp312-win_amd64.whl + - pypi: https://files.pythonhosted.org/packages/42/a1/52fa05533e95fe45bcc09bcf8a503874b1c08f221a4e35608017e0938f55/codespell-2.4.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/43/e3/7d92a15f894aa0c9c4b49b8ee9ac9850d6e63b03c9c32c0367a13ae62209/mpmath-1.3.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/52/59/0782e51887ac6b07ffd1570e0364cf901ebc36345fea669969d2084baebb/simple_websocket-1.1.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/54/20/4d324d65cc6d9205fabedc306948156824eb9f0ee1633355a8f7ec5c66bf/pluggy-1.6.0-py3-none-any.whl @@ -2224,6 +2230,7 @@ environments: - pypi: https://files.pythonhosted.org/packages/3e/5c/fb93d3092640a24dfb7bd7727a24016d7c01774ca013e60efd3f683c8002/backrefs-7.0-py314-none-any.whl - pypi: https://files.pythonhosted.org/packages/3f/d0/7b958df957e4827837b590944008f0b28078f552b451f7407b4b3d54f574/asciichartpy-1.5.25-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/3f/f9/2b3ff4e56e5fa7debfaf9eb135d0da96f3e9a1d5b27222223c7296336e5f/typer-0.25.1-py3-none-any.whl + - pypi: https://files.pythonhosted.org/packages/42/a1/52fa05533e95fe45bcc09bcf8a503874b1c08f221a4e35608017e0938f55/codespell-2.4.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/43/e3/7d92a15f894aa0c9c4b49b8ee9ac9850d6e63b03c9c32c0367a13ae62209/mpmath-1.3.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/46/b4/0887c88ddfaba1d7140ea335144eb904af97550786ee58bdb295ff10d255/crysfml-0.6.2-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl - pypi: https://files.pythonhosted.org/packages/52/59/0782e51887ac6b07ffd1570e0364cf901ebc36345fea669969d2084baebb/simple_websocket-1.1.0-py3-none-any.whl @@ -2547,6 +2554,7 @@ environments: - pypi: https://files.pythonhosted.org/packages/3f/d0/7b958df957e4827837b590944008f0b28078f552b451f7407b4b3d54f574/asciichartpy-1.5.25-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/3f/f9/2b3ff4e56e5fa7debfaf9eb135d0da96f3e9a1d5b27222223c7296336e5f/typer-0.25.1-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/41/86/86231232fff41c9f8e4a1a7d7a597d349a02527109c3af7d618366122139/matplotlib-3.10.9-cp314-cp314-macosx_11_0_arm64.whl + - pypi: https://files.pythonhosted.org/packages/42/a1/52fa05533e95fe45bcc09bcf8a503874b1c08f221a4e35608017e0938f55/codespell-2.4.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/43/e3/7d92a15f894aa0c9c4b49b8ee9ac9850d6e63b03c9c32c0367a13ae62209/mpmath-1.3.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/49/b2/97980f3ad4fae37dd7fe31626e2bf75fbf8bdf5d303950ec1fab39a12da8/kiwisolver-1.5.0-cp314-cp314-macosx_11_0_arm64.whl - pypi: https://files.pythonhosted.org/packages/51/ac/b9d68ebddfe1b02c77af5bf81120e12b036b4432dc6af7a303d90e2bc38b/chardet-7.4.3-cp314-cp314-macosx_11_0_arm64.whl @@ -2865,6 +2873,7 @@ environments: - pypi: https://files.pythonhosted.org/packages/3e/5c/fb93d3092640a24dfb7bd7727a24016d7c01774ca013e60efd3f683c8002/backrefs-7.0-py314-none-any.whl - pypi: https://files.pythonhosted.org/packages/3f/d0/7b958df957e4827837b590944008f0b28078f552b451f7407b4b3d54f574/asciichartpy-1.5.25-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/3f/f9/2b3ff4e56e5fa7debfaf9eb135d0da96f3e9a1d5b27222223c7296336e5f/typer-0.25.1-py3-none-any.whl + - pypi: https://files.pythonhosted.org/packages/42/a1/52fa05533e95fe45bcc09bcf8a503874b1c08f221a4e35608017e0938f55/codespell-2.4.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/43/e3/7d92a15f894aa0c9c4b49b8ee9ac9850d6e63b03c9c32c0367a13ae62209/mpmath-1.3.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/4b/39/f0e8ea762a764a9dc52aa7dabcfad51a354819de1f0d4652b6a1122424d6/scipy-1.17.1-cp314-cp314-win_amd64.whl - pypi: https://files.pythonhosted.org/packages/52/59/0782e51887ac6b07ffd1570e0364cf901ebc36345fea669969d2084baebb/simple_websocket-1.1.0-py3-none-any.whl @@ -8793,6 +8802,7 @@ packages: - uncertainties - varname - build ; extra == 'dev' + - codespell ; extra == 'dev' - copier ; extra == 'dev' - docstripy ; extra == 'dev' - format-docstring ; extra == 'dev' @@ -10078,6 +10088,29 @@ packages: - setuptools-scm>=7,<10 ; extra == 'dev' - setuptools>=64 ; extra == 'dev' requires_python: '>=3.10' +- pypi: https://files.pythonhosted.org/packages/42/a1/52fa05533e95fe45bcc09bcf8a503874b1c08f221a4e35608017e0938f55/codespell-2.4.2-py3-none-any.whl + name: codespell + version: 2.4.2 + sha256: 97e0c1060cf46bd1d5db89a936c98db8c2b804e1fdd4b5c645e82a1ec6b1f886 + requires_dist: + - build ; extra == 'dev' + - chardet ; extra == 'dev' + - pre-commit ; extra == 'dev' + - pytest ; extra == 'dev' + - pytest-cov ; extra == 'dev' + - pytest-dependency ; extra == 'dev' + - pygments ; extra == 'dev' + - ruff ; extra == 'dev' + - tomli ; extra == 'dev' + - twine ; extra == 'dev' + - chardet ; extra == 'hard-encoding-detection' + - tomli ; python_full_version < '3.11' and extra == 'toml' + - chardet>=5.1.0 ; extra == 'types' + - mypy ; extra == 'types' + - pytest ; extra == 'types' + - pytest-cov ; extra == 'types' + - pytest-dependency ; extra == 'types' + requires_python: '>=3.9' - pypi: https://files.pythonhosted.org/packages/43/e3/7d92a15f894aa0c9c4b49b8ee9ac9850d6e63b03c9c32c0367a13ae62209/mpmath-1.3.0-py3-none-any.whl name: mpmath version: 1.3.0 diff --git a/pixi.toml b/pixi.toml index 6bba5c773..0630d954c 100644 --- a/pixi.toml +++ b/pixi.toml @@ -246,6 +246,14 @@ docs-build = { cmd = 'pixi run docs-pre build -f docs/mkdocs.yml', depends-on = 'docs-sync-vendored-js', ] } docs-build-local = 'pixi run docs-build --no-directory-urls' +# Strict build: fails on broken nav / internal references. Tutorials are +# not executed (mkdocs-jupyter execute: false), so this is the fast +# every-push documentation gate. +docs-build-strict = { cmd = 'pixi run docs-pre build --strict -f docs/mkdocs.yml', depends-on = [ + 'docs-sync-vendored-js', +] } +# Spelling check over docs and source (config in [tool.codespell]). +spell-check = 'codespell docs src tools README.md CONTRIBUTING.md' docs-deploy-pre = 'mike deploy -F docs/mkdocs.yml --push --branch gh-pages --update-aliases --alias-type redirect' docs-set-default-pre = 'mike set-default -F docs/mkdocs.yml --push --branch gh-pages' diff --git a/pyproject.toml b/pyproject.toml index be88546c8..85e38e651 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -66,6 +66,7 @@ dev = [ 'hypothesis', # Property-based / input-domain testing 'ruff', # Linting and formatting code 'radon', # Code complexity and maintainability + 'codespell', # Spell checking docs and source 'validate-pyproject[all]', # Validate pyproject.toml 'versioningit', # Automatic versioning from git tags 'jupytext', # Jupyter notebook text format support @@ -211,6 +212,15 @@ filterwarnings = [ "ignore:'diffpy\\.structure\\.Structure\\.writeStr':DeprecationWarning", ] +############################## +# Configuration for codespell +############################## + +# 'codespell' -- Spell checker for docs and source comments/docstrings. +# https://github.com/codespell-project/codespell +[tool.codespell] +skip = '*.ipynb,*.lock,*.svg,*.min.js,*.json.gz,*/vendor/*,*/_vendored/*,docs/dev/package-structure/*,node_modules,.pixi' + ######################## # Configuration for ruff ######################## From 60b7d8aa6fbebb04116db445f95245adfa8c7e78 Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Fri, 5 Jun 2026 22:35:54 -0700 Subject: [PATCH 15/57] Add calculator support matrix query for verification --- docs/dev/plans/test-suite-and-validation.md | 2 +- .../analysis/calculators/support.py | 70 +++++++++++++++++++ .../analysis/calculators/test_support.py | 46 ++++++++++++ 3 files changed, 117 insertions(+), 1 deletion(-) create mode 100644 src/easydiffraction/analysis/calculators/support.py create mode 100644 tests/unit/easydiffraction/analysis/calculators/test_support.py diff --git a/docs/dev/plans/test-suite-and-validation.md b/docs/dev/plans/test-suite-and-validation.md index 46031cfc4..e931d08c3 100644 --- a/docs/dev/plans/test-suite-and-validation.md +++ b/docs/dev/plans/test-suite-and-validation.md @@ -269,7 +269,7 @@ files or generated artifacts. Do not run Phase 2 commands during Phase 1. `.codespellrc`, `lychee` config. Commit: `Add strict docs build, link, and spell checks on every push` -- [ ] **P1.11 — Apply/extend calculator support metadata (§6 prerequisite)** +- [x] **P1.11 — Apply/extend calculator support metadata (§6 prerequisite)** Build on the existing `Compatibility`/`CalculatorSupport` model in `core/metadata.py` (already declared per instrument category): add `radiation_probe` to `Compatibility` if missing, and add a small query diff --git a/src/easydiffraction/analysis/calculators/support.py b/src/easydiffraction/analysis/calculators/support.py new file mode 100644 index 000000000..92b9b3251 --- /dev/null +++ b/src/easydiffraction/analysis/calculators/support.py @@ -0,0 +1,70 @@ +# SPDX-FileCopyrightText: 2026 EasyScience contributors +# SPDX-License-Identifier: BSD-3-Clause +"""Calculator support matrix. + +Aggregates the per-instrument ``Compatibility`` and ``CalculatorSupport`` +metadata declared on the registered instrument categories into a single +queryable matrix: which calculation engines can compute which experiment +conditions. Used by the Verification documentation to enumerate +comparable engine x condition combinations. +""" + +from __future__ import annotations + +from dataclasses import dataclass + +from easydiffraction.core.metadata import Compatibility + + +@dataclass(frozen=True) +class SupportEntry: + """One instrument condition and the engines that can compute it. + + Attributes + ---------- + instrument_tag : str + The instrument category tag (for example ``'cwl-pd'``). + description : str + One-line human-readable description of the instrument. + compatibility : Compatibility + The experimental conditions (sample_form x scattering_type x + beam_mode x radiation_probe) the instrument supports. + calculators : frozenset + The ``CalculatorEnum`` engines declared able to handle it. + """ + + instrument_tag: str + description: str + compatibility: Compatibility + calculators: frozenset + + +def calculator_support_matrix() -> list[SupportEntry]: + """Return the engine x experiment-condition support matrix. + + One entry per registered instrument category, pairing its + ``Compatibility`` with the calculators declared able to handle it. + + Returns + ------- + list of SupportEntry + One entry per registered instrument category. + """ + # Lazy imports: ensure the instrument categories are registered and + # avoid a circular import at module load. + import easydiffraction.datablocks.experiment.categories.instrument # noqa: F401 + from easydiffraction.datablocks.experiment.categories.instrument.factory import ( + InstrumentFactory, + ) + + entries: list[SupportEntry] = [] + for klass in InstrumentFactory._supported_map().values(): + entries.append( + SupportEntry( + instrument_tag=klass.type_info.tag, + description=klass.type_info.description, + compatibility=klass.compatibility, + calculators=frozenset(klass.calculator_support.calculators), + ) + ) + return entries diff --git a/tests/unit/easydiffraction/analysis/calculators/test_support.py b/tests/unit/easydiffraction/analysis/calculators/test_support.py new file mode 100644 index 000000000..0f335ed7c --- /dev/null +++ b/tests/unit/easydiffraction/analysis/calculators/test_support.py @@ -0,0 +1,46 @@ +# SPDX-FileCopyrightText: 2026 EasyScience contributors +# SPDX-License-Identifier: BSD-3-Clause +"""Unit tests for the calculator support matrix.""" + +from __future__ import annotations + + +def test_matrix_covers_every_registered_instrument(): + from easydiffraction.analysis.calculators.support import calculator_support_matrix + from easydiffraction.datablocks.experiment.categories.instrument.factory import ( + InstrumentFactory, + ) + + entries = calculator_support_matrix() + + assert entries + assert {e.instrument_tag for e in entries} == set(InstrumentFactory.supported_tags()) + + +def test_matrix_entries_are_well_typed(): + from easydiffraction.analysis.calculators.support import calculator_support_matrix + from easydiffraction.core.metadata import Compatibility + from easydiffraction.datablocks.experiment.item.enums import CalculatorEnum + + for entry in calculator_support_matrix(): + assert isinstance(entry.compatibility, Compatibility) + assert all(isinstance(c, CalculatorEnum) for c in entry.calculators) + + +def test_cwl_pd_supports_all_three_engines(): + from easydiffraction.analysis.calculators.support import calculator_support_matrix + from easydiffraction.datablocks.experiment.item.enums import CalculatorEnum + + by_tag = {e.instrument_tag: e for e in calculator_support_matrix()} + + expected = {CalculatorEnum.CRYSPY, CalculatorEnum.CRYSFML, CalculatorEnum.PDFFIT} + assert expected <= by_tag['cwl-pd'].calculators + + +def test_cwl_sc_supports_cryspy_only(): + from easydiffraction.analysis.calculators.support import calculator_support_matrix + from easydiffraction.datablocks.experiment.item.enums import CalculatorEnum + + by_tag = {e.instrument_tag: e for e in calculator_support_matrix()} + + assert by_tag['cwl-sc'].calculators == frozenset({CalculatorEnum.CRYSPY}) From 8cc7e2aa081a65bd4a05721c4d83a25b3f6027ac Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Fri, 5 Jun 2026 22:45:00 -0700 Subject: [PATCH 16/57] Add cross-engine verification comparison pages and script wiring --- docs/dev/plans/test-suite-and-validation.md | 2 +- .../verification/cross-engine-bragg-cwl.ipynb | 223 ++++++++++++++++++ .../verification/cross-engine-bragg-cwl.py | 106 +++++++++ docs/docs/verification/index.md | 10 + docs/mkdocs.yml | 4 + pixi.toml | 6 +- tools/test_scripts.py | 8 +- 7 files changed, 353 insertions(+), 6 deletions(-) create mode 100644 docs/docs/verification/cross-engine-bragg-cwl.ipynb create mode 100644 docs/docs/verification/cross-engine-bragg-cwl.py create mode 100644 docs/docs/verification/index.md diff --git a/docs/dev/plans/test-suite-and-validation.md b/docs/dev/plans/test-suite-and-validation.md index e931d08c3..66b2014c0 100644 --- a/docs/dev/plans/test-suite-and-validation.md +++ b/docs/dev/plans/test-suite-and-validation.md @@ -283,7 +283,7 @@ files or generated artifacts. Do not run Phase 2 commands during Phase 1. `src/easydiffraction/analysis/calculators/**`. Commit: `Extend calculator support metadata with radiation probe` -- [ ] **P1.12 — Cross-engine verification pages + script wiring (§6)** +- [x] **P1.12 — Cross-engine verification pages + script wiring (§6)** Add the `Verification` nav node (between Tutorials and Command-Line) and calculation-only `.py` comparison pages (cryspy ↔ crysfml) across the supported experiment combinations, with closeness metrics, overlay diff --git a/docs/docs/verification/cross-engine-bragg-cwl.ipynb b/docs/docs/verification/cross-engine-bragg-cwl.ipynb new file mode 100644 index 000000000..eee4495bb --- /dev/null +++ b/docs/docs/verification/cross-engine-bragg-cwl.ipynb @@ -0,0 +1,223 @@ +{ + "cells": [ + { + "cell_type": "code", + "execution_count": null, + "id": "0", + "metadata": { + "tags": [ + "hide-in-docs" + ] + }, + "outputs": [], + "source": [ + "# Check whether easydiffraction is installed; install it if needed.\n", + "# Required for remote environments such as Google Colab.\n", + "import importlib.util\n", + "\n", + "if importlib.util.find_spec('easydiffraction') is None:\n", + " %pip install easydiffraction" + ] + }, + { + "cell_type": "markdown", + "id": "1", + "metadata": {}, + "source": [ + "# Cross-engine verification — neutron powder, constant wavelength (Bragg)\n", + "\n", + "This page calculates the **same** diffraction pattern for one structure\n", + "and one experiment with each supported engine, **without any fitting**,\n", + "and reports how closely the engines agree. It doubles as a regression\n", + "check run by `pixi run script-tests`." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "2", + "metadata": {}, + "outputs": [], + "source": [ + "import numpy as np\n", + "\n", + "import easydiffraction as ed\n", + "from easydiffraction.analysis.calculators.support import calculator_support_matrix\n", + "from easydiffraction.analysis.fit_helpers.metrics import get_reliability_inputs" + ] + }, + { + "cell_type": "markdown", + "id": "3", + "metadata": {}, + "source": [ + "## Build the project (La0.5Ba0.5CoO3, HRPT)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "4", + "metadata": {}, + "outputs": [], + "source": [ + "project = ed.Project()\n", + "project.structures.add_from_cif_path(ed.download_data(id=1, destination='data'))\n", + "project.experiments.add_from_cif_path(ed.download_data(id=2, destination='data'))\n", + "\n", + "experiment = project.experiments['hrpt']" + ] + }, + { + "cell_type": "markdown", + "id": "5", + "metadata": {}, + "source": [ + "## Engines to compare\n", + "\n", + "The calculator support matrix declares which engines can compute this\n", + "instrument condition (`cwl-pd`)." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "6", + "metadata": {}, + "outputs": [], + "source": [ + "by_tag = {entry.instrument_tag: entry for entry in calculator_support_matrix()}\n", + "declared = sorted(c.value for c in by_tag['cwl-pd'].calculators)\n", + "print('Engines declared for cwl-pd:', declared)\n", + "\n", + "ENGINES = ['cryspy', 'crysfml']" + ] + }, + { + "cell_type": "markdown", + "id": "7", + "metadata": {}, + "source": [ + "## Calculate the pattern with each engine (no fitting)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "8", + "metadata": {}, + "outputs": [], + "source": [ + "y_calc_by_engine = {}\n", + "for engine in ENGINES:\n", + " experiment.calculator.type = engine\n", + " assert experiment.calculator.type == engine\n", + " _, y_calc, _ = get_reliability_inputs(project.structures, [experiment])\n", + " y_calc_by_engine[engine] = np.asarray(y_calc, dtype=float)\n", + " # Per-engine measured-vs-calculated view (rendered in the docs build).\n", + " project.display.pattern(expt_name='hrpt')" + ] + }, + { + "cell_type": "markdown", + "id": "9", + "metadata": {}, + "source": [ + "## Closeness metrics between engines" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "10", + "metadata": {}, + "outputs": [], + "source": [ + "a = y_calc_by_engine['cryspy']\n", + "b = y_calc_by_engine['crysfml']\n", + "\n", + "assert a.shape == b.shape, 'engines returned patterns of different length'\n", + "assert np.all(np.isfinite(a)) and np.all(np.isfinite(b))\n", + "\n", + "rms = float(np.sqrt(np.mean((a - b) ** 2)))\n", + "norm = float(np.sqrt(np.mean(a**2)))\n", + "profile_diff_pct = 100.0 * rms / norm if norm else float('nan')\n", + "max_deviation = float(np.max(np.abs(a - b)))\n", + "intensity_ratio = float(a.sum() / b.sum()) if b.sum() else float('nan')\n", + "correlation = float(np.corrcoef(a, b)[0, 1])\n", + "\n", + "print(f'profile difference: {profile_diff_pct:.2f} %')\n", + "print(f'max point-wise deviation: {max_deviation:.4g}')\n", + "print(f'integrated-intensity ratio (cp/cf): {intensity_ratio:.4f}')\n", + "print(f'Pearson correlation: {correlation:.4f}')" + ] + }, + { + "cell_type": "markdown", + "id": "11", + "metadata": {}, + "source": [ + "## Overlay\n", + "\n", + "Both engines on one chart, in distinct colours and line styles." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "12", + "metadata": {}, + "outputs": [], + "source": [ + "import plotly.graph_objects as go\n", + "\n", + "x = np.arange(a.size)\n", + "fig = go.Figure()\n", + "fig.add_scatter(x=x, y=a, mode='lines', name='cryspy', line={'color': 'royalblue'})\n", + "fig.add_scatter(\n", + " x=x, y=b, mode='lines', name='crysfml', line={'color': 'crimson', 'dash': 'dot'}\n", + ")\n", + "fig.update_layout(\n", + " title='Calculated patterns: cryspy vs crysfml',\n", + " xaxis_title='point index',\n", + " yaxis_title='Icalc',\n", + ")\n", + "# Bare expression renders inline in the executed notebook; a no-op as a\n", + "# plain script, so `pixi run script-tests` stays headless-safe.\n", + "fig" + ] + }, + { + "cell_type": "markdown", + "id": "13", + "metadata": {}, + "source": [ + "## Regression assertions\n", + "\n", + "Both engines compute the same physics for the same structure, so their\n", + "calculated patterns share peak positions and are highly correlated. The\n", + "tolerance is intentionally loose for now and tightened as the\n", + "verification suite matures." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "14", + "metadata": {}, + "outputs": [], + "source": [ + "assert correlation > 0.8, f'cross-engine correlation unexpectedly low: {correlation:.4f}'" + ] + } + ], + "metadata": { + "jupytext": { + "cell_metadata_filter": "-all", + "main_language": "python", + "notebook_metadata_filter": "-all" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/docs/docs/verification/cross-engine-bragg-cwl.py b/docs/docs/verification/cross-engine-bragg-cwl.py new file mode 100644 index 000000000..8805a6df0 --- /dev/null +++ b/docs/docs/verification/cross-engine-bragg-cwl.py @@ -0,0 +1,106 @@ +# %% [markdown] +# # Cross-engine verification — neutron powder, constant wavelength (Bragg) +# +# This page calculates the **same** diffraction pattern for one structure +# and one experiment with each supported engine, **without any fitting**, +# and reports how closely the engines agree. It doubles as a regression +# check run by `pixi run script-tests`. + +# %% +import numpy as np + +import easydiffraction as ed +from easydiffraction.analysis.calculators.support import calculator_support_matrix +from easydiffraction.analysis.fit_helpers.metrics import get_reliability_inputs + +# %% [markdown] +# ## Build the project (La0.5Ba0.5CoO3, HRPT) + +# %% +project = ed.Project() +project.structures.add_from_cif_path(ed.download_data(id=1, destination='data')) +project.experiments.add_from_cif_path(ed.download_data(id=2, destination='data')) + +experiment = project.experiments['hrpt'] + +# %% [markdown] +# ## Engines to compare +# +# The calculator support matrix declares which engines can compute this +# instrument condition (`cwl-pd`). + +# %% +by_tag = {entry.instrument_tag: entry for entry in calculator_support_matrix()} +declared = sorted(c.value for c in by_tag['cwl-pd'].calculators) +print('Engines declared for cwl-pd:', declared) + +ENGINES = ['cryspy', 'crysfml'] + +# %% [markdown] +# ## Calculate the pattern with each engine (no fitting) + +# %% +y_calc_by_engine = {} +for engine in ENGINES: + experiment.calculator.type = engine + assert experiment.calculator.type == engine + _, y_calc, _ = get_reliability_inputs(project.structures, [experiment]) + y_calc_by_engine[engine] = np.asarray(y_calc, dtype=float) + # Per-engine measured-vs-calculated view (rendered in the docs build). + project.display.pattern(expt_name='hrpt') + +# %% [markdown] +# ## Closeness metrics between engines + +# %% +a = y_calc_by_engine['cryspy'] +b = y_calc_by_engine['crysfml'] + +assert a.shape == b.shape, 'engines returned patterns of different length' +assert np.all(np.isfinite(a)) and np.all(np.isfinite(b)) + +rms = float(np.sqrt(np.mean((a - b) ** 2))) +norm = float(np.sqrt(np.mean(a**2))) +profile_diff_pct = 100.0 * rms / norm if norm else float('nan') +max_deviation = float(np.max(np.abs(a - b))) +intensity_ratio = float(a.sum() / b.sum()) if b.sum() else float('nan') +correlation = float(np.corrcoef(a, b)[0, 1]) + +print(f'profile difference: {profile_diff_pct:.2f} %') +print(f'max point-wise deviation: {max_deviation:.4g}') +print(f'integrated-intensity ratio (cp/cf): {intensity_ratio:.4f}') +print(f'Pearson correlation: {correlation:.4f}') + +# %% [markdown] +# ## Overlay +# +# Both engines on one chart, in distinct colours and line styles. + +# %% +import plotly.graph_objects as go + +x = np.arange(a.size) +fig = go.Figure() +fig.add_scatter(x=x, y=a, mode='lines', name='cryspy', line={'color': 'royalblue'}) +fig.add_scatter( + x=x, y=b, mode='lines', name='crysfml', line={'color': 'crimson', 'dash': 'dot'} +) +fig.update_layout( + title='Calculated patterns: cryspy vs crysfml', + xaxis_title='point index', + yaxis_title='Icalc', +) +# Bare expression renders inline in the executed notebook; a no-op as a +# plain script, so `pixi run script-tests` stays headless-safe. +fig + +# %% [markdown] +# ## Regression assertions +# +# Both engines compute the same physics for the same structure, so their +# calculated patterns share peak positions and are highly correlated. The +# tolerance is intentionally loose for now and tightened as the +# verification suite matures. + +# %% +assert correlation > 0.8, f'cross-engine correlation unexpectedly low: {correlation:.4f}' diff --git a/docs/docs/verification/index.md b/docs/docs/verification/index.md new file mode 100644 index 000000000..bf41a17ea --- /dev/null +++ b/docs/docs/verification/index.md @@ -0,0 +1,10 @@ +# Verification + +This section compares EasyDiffraction's calculation engines against each +other (and, in future, against external software such as FullProf) on the +**same** input parameters, **without any fitting** — just calculated +diffraction patterns and clear closeness metrics. + +Each page also runs as a fast regression check (`pixi run script-tests`), +so cross-engine agreement is monitored over time. Coverage grows to span +every supported experiment and instrument combination. diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml index b7b7786fd..85a75d08c 100644 --- a/docs/mkdocs.yml +++ b/docs/mkdocs.yml @@ -243,6 +243,10 @@ nav: - Tb2TiO7 sg bumps-dream: tutorials/ed-22.ipynb - Workshops & Schools: - DMSC Summer School: tutorials/ed-13.ipynb + - Verification: + - Verification: verification/index.md + - Cross-engine: + - LBCO Bragg pd-neut-cwl: verification/cross-engine-bragg-cwl.ipynb - Command-Line: - Command-Line: cli/index.md - API Reference: diff --git a/pixi.toml b/pixi.toml index 0630d954c..28b5ab7bf 100644 --- a/pixi.toml +++ b/pixi.toml @@ -217,9 +217,9 @@ tutorial = { cmd = 'python', env = { EASYDIFFRACTION_ARTIFACT_ROOT = 'tmp/tutori tutorial-benchmarks = { cmd = 'python tools/benchmark_tutorials.py', env = { EASYDIFFRACTION_ARTIFACT_ROOT = 'tmp/tutorials' } } jupyter = { cmd = 'jupyter', env = { EASYDIFFRACTION_ARTIFACT_ROOT = 'tmp/tutorials' } } -notebook-convert = 'jupytext docs/docs/tutorials/*.py --from py:percent --to ipynb' -notebook-strip = 'nbstripout docs/docs/tutorials/*.ipynb' -notebook-tweak = 'python tools/tweak_notebooks.py docs/docs/tutorials/' +notebook-convert = 'jupytext docs/docs/tutorials/*.py docs/docs/verification/*.py --from py:percent --to ipynb' +notebook-strip = 'nbstripout docs/docs/tutorials/*.ipynb docs/docs/verification/*.ipynb' +notebook-tweak = 'python tools/tweak_notebooks.py docs/docs/tutorials/ docs/docs/verification/' notebook-exec = { cmd = 'python -m pytest --nbmake docs/docs/tutorials/ --overwrite --color=yes -n auto -v', env = { EASYDIFFRACTION_ARTIFACT_ROOT = 'tmp/tutorials', EASYDIFFRACTION_FIGURE_EMBED_MODE = 'shared' } } notebook-exec-ci = { cmd = 'python -m pytest --nbmake docs/docs/tutorials/ --nbmake-timeout=1200 --overwrite --color=yes -n auto -v', env = { EASYDIFFRACTION_ARTIFACT_ROOT = '.', EASYDIFFRACTION_FIGURE_EMBED_MODE = 'shared' } } diff --git a/tools/test_scripts.py b/tools/test_scripts.py index 2dcca083a..12df2e6ba 100644 --- a/tools/test_scripts.py +++ b/tools/test_scripts.py @@ -21,9 +21,13 @@ _repo_root = Path(__file__).resolve().parents[1] _src_root = _repo_root / 'src' -# Discover tutorial scripts, excluding temporary checkpoint files +# Discover tutorial and verification scripts, excluding checkpoint files. +_SCRIPT_DIRS = ('docs/docs/tutorials', 'docs/docs/verification') TUTORIALS = [ - p for p in Path('docs/docs/tutorials').rglob('*.py') if '.ipynb_checkpoints' not in p.parts + p + for directory in _SCRIPT_DIRS + for p in Path(directory).rglob('*.py') + if '.ipynb_checkpoints' not in p.parts ] From 0c9ffb7bb5c57197212021d39d9b3adda7db5e19 Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Fri, 5 Jun 2026 22:46:56 -0700 Subject: [PATCH 17/57] Add per-experiment performance benchmarks (nightly) --- .github/workflows/nightly.yml | 10 ++++ docs/dev/plans/test-suite-and-validation.md | 2 +- pixi.lock | 36 +++++++++++++++ pixi.toml | 8 +++- pyproject.toml | 1 + .../test_calculate_pattern_benchmark.py | 46 +++++++++++++++++++ 6 files changed, 100 insertions(+), 3 deletions(-) create mode 100644 tests/benchmarks/test_calculate_pattern_benchmark.py diff --git a/.github/workflows/nightly.yml b/.github/workflows/nightly.yml index 9645adb2a..52ae1c1fb 100644 --- a/.github/workflows/nightly.yml +++ b/.github/workflows/nightly.yml @@ -32,3 +32,13 @@ jobs: - name: Run nightly-tier tests run: pixi run nightly-tests + + - name: Run performance benchmarks + run: pixi run benchmarks + + - name: Upload benchmark results + if: ${{ !cancelled() }} + uses: ./.github/actions/upload-artifact + with: + name: benchmark-results + path: benchmark.json diff --git a/docs/dev/plans/test-suite-and-validation.md b/docs/dev/plans/test-suite-and-validation.md index 66b2014c0..ba2de3209 100644 --- a/docs/dev/plans/test-suite-and-validation.md +++ b/docs/dev/plans/test-suite-and-validation.md @@ -298,7 +298,7 @@ files or generated artifacts. Do not run Phase 2 commands during Phase 1. `*.ipynb`), `tools/test_scripts.py`, `pixi.toml`. Commit: `Add cross-engine verification comparison pages and script wiring` -- [ ] **P1.13 — Per-experiment performance benchmarks (§7)** +- [x] **P1.13 — Per-experiment performance benchmarks (§7)** Add `pytest-benchmark` (dev dep), `nightly`-marked benchmarks keyed by `beam_mode × radiation_probe × engine`, and a `benchmarks` pixi task emitting JSON as a CI artifact (data-repo history deferred). diff --git a/pixi.lock b/pixi.lock index 7d69de5bd..6b63ee45e 100644 --- a/pixi.lock +++ b/pixi.lock @@ -233,6 +233,7 @@ environments: - pypi: https://files.pythonhosted.org/packages/32/28/79f0f8de97cce916d5ae88a7bee1ad724855e83e6019c0b4d5b3fabc80f3/mkdocstrings_python-2.0.3-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/32/46/9cb0e58b2deb7f82b84065f37f3bffeb12413f947f9388e4cac22c4621ce/sortedcontainers-2.4.0-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/32/5d/f7e914f7d9325abff4057cee62c0fa70263683189f774473cbfb534cd13b/matplotlib-3.10.9-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl + - pypi: https://files.pythonhosted.org/packages/33/29/e756e715a48959f1c0045342088d7ca9762a2f509b945f362a316e9412b7/pytest_benchmark-5.2.3-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/33/6b/e0547afaf41bf2c42e52430072fa5658766e3d65bd4b03a563d1b6336f57/distlib-0.4.0-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/33/b2/986d1220f6ee931e338d272bc1f3ec02cfe5f9b5fad84e95afdad57f1ebc/format_docstring-0.2.7-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/33/eb/f9f1ded8e4db9638f9530c3782eb01f5ab04945f4cb9e597a51c203fa4c5/diffpy_pdffit2-1.6.0.tar.gz @@ -328,6 +329,7 @@ environments: - pypi: https://files.pythonhosted.org/packages/db/3c/33bac158f8ab7f89b2e59426d5fe2e4f63f7ed25df84c036890172b412b5/cfgv-3.5.0-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/dc/9b/47798a6c91d8bdb567fe2698fe81e0c6b7cb7ef4d13da4114b41d239f65d/typing_inspection-0.4.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/de/1f/77fa3081e4f66ca3576c896ae5d31c3002ac6607f9747d2e3aa49227e464/markdown-3.10.2-py3-none-any.whl + - pypi: https://files.pythonhosted.org/packages/e0/a9/023730ba63db1e494a271cb018dcd361bd2c917ba7004c3e49d5daf795a2/py_cpuinfo-9.0.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/e0/f9/0595336914c5619e5f28a1fb793285925a8cd4b432c9da0a987836c7f822/shellingham-1.5.4-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/e5/04/c5bb20d64417d20cba0105277235c51969444fa873000fbc26ac0a3fc5a8/gemmi-0.7.5-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl - pypi: https://files.pythonhosted.org/packages/e7/05/c19819d5e3d95294a6f5947fb9b9629efb316b96de511b418c53d245aae6/cycler-0.12.1-py3-none-any.whl @@ -558,6 +560,7 @@ environments: - pypi: https://files.pythonhosted.org/packages/2c/19/04f9b178c2d8a15b076c8b5140708fa6ffc5601fb6f1e975537072df5b2a/mergedeep-1.3.4-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/32/28/79f0f8de97cce916d5ae88a7bee1ad724855e83e6019c0b4d5b3fabc80f3/mkdocstrings_python-2.0.3-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/32/46/9cb0e58b2deb7f82b84065f37f3bffeb12413f947f9388e4cac22c4621ce/sortedcontainers-2.4.0-py2.py3-none-any.whl + - pypi: https://files.pythonhosted.org/packages/33/29/e756e715a48959f1c0045342088d7ca9762a2f509b945f362a316e9412b7/pytest_benchmark-5.2.3-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/33/6b/e0547afaf41bf2c42e52430072fa5658766e3d65bd4b03a563d1b6336f57/distlib-0.4.0-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/33/b2/986d1220f6ee931e338d272bc1f3ec02cfe5f9b5fad84e95afdad57f1ebc/format_docstring-0.2.7-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/33/f0/3fe8c6e69135a845f4106f2ff8b6805638d4e85c264e70114e8126689587/tokenize_rt-6.2.0-py2.py3-none-any.whl @@ -657,6 +660,7 @@ environments: - pypi: https://files.pythonhosted.org/packages/dc/9b/47798a6c91d8bdb567fe2698fe81e0c6b7cb7ef4d13da4114b41d239f65d/typing_inspection-0.4.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/de/1f/77fa3081e4f66ca3576c896ae5d31c3002ac6607f9747d2e3aa49227e464/markdown-3.10.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/df/32/10ac51b4be7cdecd7e93d069251c86dfbf70b7adbd7c67b48ccea6c49e1c/sqlalchemy-2.0.50-cp314-cp314-macosx_11_0_arm64.whl + - pypi: https://files.pythonhosted.org/packages/e0/a9/023730ba63db1e494a271cb018dcd361bd2c917ba7004c3e49d5daf795a2/py_cpuinfo-9.0.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/e0/f9/0595336914c5619e5f28a1fb793285925a8cd4b432c9da0a987836c7f822/shellingham-1.5.4-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/e7/05/c19819d5e3d95294a6f5947fb9b9629efb316b96de511b418c53d245aae6/cycler-0.12.1-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/eb/be/b257e12f9710819fde40adc972578bee6b72c5992da1bc8369bef2597756/nbmake-1.5.5-py3-none-any.whl @@ -876,6 +880,7 @@ environments: - pypi: https://files.pythonhosted.org/packages/2c/19/04f9b178c2d8a15b076c8b5140708fa6ffc5601fb6f1e975537072df5b2a/mergedeep-1.3.4-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/32/28/79f0f8de97cce916d5ae88a7bee1ad724855e83e6019c0b4d5b3fabc80f3/mkdocstrings_python-2.0.3-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/32/46/9cb0e58b2deb7f82b84065f37f3bffeb12413f947f9388e4cac22c4621ce/sortedcontainers-2.4.0-py2.py3-none-any.whl + - pypi: https://files.pythonhosted.org/packages/33/29/e756e715a48959f1c0045342088d7ca9762a2f509b945f362a316e9412b7/pytest_benchmark-5.2.3-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/33/6b/e0547afaf41bf2c42e52430072fa5658766e3d65bd4b03a563d1b6336f57/distlib-0.4.0-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/33/b2/986d1220f6ee931e338d272bc1f3ec02cfe5f9b5fad84e95afdad57f1ebc/format_docstring-0.2.7-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/33/f0/3fe8c6e69135a845f4106f2ff8b6805638d4e85c264e70114e8126689587/tokenize_rt-6.2.0-py2.py3-none-any.whl @@ -976,6 +981,7 @@ environments: - pypi: https://files.pythonhosted.org/packages/dc/9b/47798a6c91d8bdb567fe2698fe81e0c6b7cb7ef4d13da4114b41d239f65d/typing_inspection-0.4.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/de/1f/77fa3081e4f66ca3576c896ae5d31c3002ac6607f9747d2e3aa49227e464/markdown-3.10.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/df/ac/46de6dda46478f7942f839e094970be2d4a861e005c4b3bf07c92e291a09/numpy-2.4.6-cp314-cp314-win_amd64.whl + - pypi: https://files.pythonhosted.org/packages/e0/a9/023730ba63db1e494a271cb018dcd361bd2c917ba7004c3e49d5daf795a2/py_cpuinfo-9.0.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/e0/bf/52f25716bbe93745595800f36fb17b73711f14da59ed0bb2eba141bc9f0f/multidict-6.7.1-cp314-cp314-win_amd64.whl - pypi: https://files.pythonhosted.org/packages/e0/f9/0595336914c5619e5f28a1fb793285925a8cd4b432c9da0a987836c7f822/shellingham-1.5.4-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/e7/05/c19819d5e3d95294a6f5947fb9b9629efb316b96de511b418c53d245aae6/cycler-0.12.1-py3-none-any.whl @@ -1222,6 +1228,7 @@ environments: - pypi: https://files.pythonhosted.org/packages/32/28/79f0f8de97cce916d5ae88a7bee1ad724855e83e6019c0b4d5b3fabc80f3/mkdocstrings_python-2.0.3-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/32/46/9cb0e58b2deb7f82b84065f37f3bffeb12413f947f9388e4cac22c4621ce/sortedcontainers-2.4.0-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/32/91/d024616abdba99e83120e07a20658976f6a343646710760c4a51df126029/matplotlib-3.10.9-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl + - pypi: https://files.pythonhosted.org/packages/33/29/e756e715a48959f1c0045342088d7ca9762a2f509b945f362a316e9412b7/pytest_benchmark-5.2.3-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/33/6b/e0547afaf41bf2c42e52430072fa5658766e3d65bd4b03a563d1b6336f57/distlib-0.4.0-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/33/b2/986d1220f6ee931e338d272bc1f3ec02cfe5f9b5fad84e95afdad57f1ebc/format_docstring-0.2.7-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/33/eb/f9f1ded8e4db9638f9530c3782eb01f5ab04945f4cb9e597a51c203fa4c5/diffpy_pdffit2-1.6.0.tar.gz @@ -1323,6 +1330,7 @@ environments: - pypi: https://files.pythonhosted.org/packages/db/3c/33bac158f8ab7f89b2e59426d5fe2e4f63f7ed25df84c036890172b412b5/cfgv-3.5.0-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/dc/9b/47798a6c91d8bdb567fe2698fe81e0c6b7cb7ef4d13da4114b41d239f65d/typing_inspection-0.4.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/de/1f/77fa3081e4f66ca3576c896ae5d31c3002ac6607f9747d2e3aa49227e464/markdown-3.10.2-py3-none-any.whl + - pypi: https://files.pythonhosted.org/packages/e0/a9/023730ba63db1e494a271cb018dcd361bd2c917ba7004c3e49d5daf795a2/py_cpuinfo-9.0.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/e0/f9/0595336914c5619e5f28a1fb793285925a8cd4b432c9da0a987836c7f822/shellingham-1.5.4-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/e5/59/a32a241d861cf180853a11c8e5a67641cb1b2af13c3a5ccce83ec07e2c9f/chardet-7.4.3-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl - pypi: https://files.pythonhosted.org/packages/e7/05/c19819d5e3d95294a6f5947fb9b9629efb316b96de511b418c53d245aae6/cycler-0.12.1-py3-none-any.whl @@ -1554,6 +1562,7 @@ environments: - pypi: https://files.pythonhosted.org/packages/32/28/79f0f8de97cce916d5ae88a7bee1ad724855e83e6019c0b4d5b3fabc80f3/mkdocstrings_python-2.0.3-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/32/46/9cb0e58b2deb7f82b84065f37f3bffeb12413f947f9388e4cac22c4621ce/sortedcontainers-2.4.0-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/32/a0/37fb236da6040e337381dd656cafb97d09eacb998c5db3057547f5ffddd9/pycifrw-5.0.1-cp312-cp312-macosx_11_0_arm64.whl + - pypi: https://files.pythonhosted.org/packages/33/29/e756e715a48959f1c0045342088d7ca9762a2f509b945f362a316e9412b7/pytest_benchmark-5.2.3-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/33/6b/e0547afaf41bf2c42e52430072fa5658766e3d65bd4b03a563d1b6336f57/distlib-0.4.0-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/33/b2/986d1220f6ee931e338d272bc1f3ec02cfe5f9b5fad84e95afdad57f1ebc/format_docstring-0.2.7-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/33/f0/3fe8c6e69135a845f4106f2ff8b6805638d4e85c264e70114e8126689587/tokenize_rt-6.2.0-py2.py3-none-any.whl @@ -1652,6 +1661,7 @@ environments: - pypi: https://files.pythonhosted.org/packages/db/3c/33bac158f8ab7f89b2e59426d5fe2e4f63f7ed25df84c036890172b412b5/cfgv-3.5.0-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/dc/9b/47798a6c91d8bdb567fe2698fe81e0c6b7cb7ef4d13da4114b41d239f65d/typing_inspection-0.4.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/de/1f/77fa3081e4f66ca3576c896ae5d31c3002ac6607f9747d2e3aa49227e464/markdown-3.10.2-py3-none-any.whl + - pypi: https://files.pythonhosted.org/packages/e0/a9/023730ba63db1e494a271cb018dcd361bd2c917ba7004c3e49d5daf795a2/py_cpuinfo-9.0.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/e0/f9/0595336914c5619e5f28a1fb793285925a8cd4b432c9da0a987836c7f822/shellingham-1.5.4-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/e7/05/c19819d5e3d95294a6f5947fb9b9629efb316b96de511b418c53d245aae6/cycler-0.12.1-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/eb/be/b257e12f9710819fde40adc972578bee6b72c5992da1bc8369bef2597756/nbmake-1.5.5-py3-none-any.whl @@ -1872,6 +1882,7 @@ environments: - pypi: https://files.pythonhosted.org/packages/32/28/79f0f8de97cce916d5ae88a7bee1ad724855e83e6019c0b4d5b3fabc80f3/mkdocstrings_python-2.0.3-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/32/46/9cb0e58b2deb7f82b84065f37f3bffeb12413f947f9388e4cac22c4621ce/sortedcontainers-2.4.0-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/32/60/fca69c534602a7ced04280c952a246ad1edde2a6ca3a164f65d32ac41fe7/chardet-7.4.3-cp312-cp312-win_amd64.whl + - pypi: https://files.pythonhosted.org/packages/33/29/e756e715a48959f1c0045342088d7ca9762a2f509b945f362a316e9412b7/pytest_benchmark-5.2.3-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/33/6b/e0547afaf41bf2c42e52430072fa5658766e3d65bd4b03a563d1b6336f57/distlib-0.4.0-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/33/b2/986d1220f6ee931e338d272bc1f3ec02cfe5f9b5fad84e95afdad57f1ebc/format_docstring-0.2.7-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/33/f0/3fe8c6e69135a845f4106f2ff8b6805638d4e85c264e70114e8126689587/tokenize_rt-6.2.0-py2.py3-none-any.whl @@ -1971,6 +1982,7 @@ environments: - pypi: https://files.pythonhosted.org/packages/db/d9/c495884c6e548fce18a8f40568ff120bc3a4b7b99813081c8ac0c936fa64/watchdog-6.0.0-py3-none-win_amd64.whl - pypi: https://files.pythonhosted.org/packages/dc/9b/47798a6c91d8bdb567fe2698fe81e0c6b7cb7ef4d13da4114b41d239f65d/typing_inspection-0.4.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/de/1f/77fa3081e4f66ca3576c896ae5d31c3002ac6607f9747d2e3aa49227e464/markdown-3.10.2-py3-none-any.whl + - pypi: https://files.pythonhosted.org/packages/e0/a9/023730ba63db1e494a271cb018dcd361bd2c917ba7004c3e49d5daf795a2/py_cpuinfo-9.0.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/e0/f9/0595336914c5619e5f28a1fb793285925a8cd4b432c9da0a987836c7f822/shellingham-1.5.4-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/e7/05/c19819d5e3d95294a6f5947fb9b9629efb316b96de511b418c53d245aae6/cycler-0.12.1-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/eb/be/b257e12f9710819fde40adc972578bee6b72c5992da1bc8369bef2597756/nbmake-1.5.5-py3-none-any.whl @@ -2216,6 +2228,7 @@ environments: - pypi: https://files.pythonhosted.org/packages/32/28/79f0f8de97cce916d5ae88a7bee1ad724855e83e6019c0b4d5b3fabc80f3/mkdocstrings_python-2.0.3-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/32/46/9cb0e58b2deb7f82b84065f37f3bffeb12413f947f9388e4cac22c4621ce/sortedcontainers-2.4.0-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/32/5d/f7e914f7d9325abff4057cee62c0fa70263683189f774473cbfb534cd13b/matplotlib-3.10.9-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl + - pypi: https://files.pythonhosted.org/packages/33/29/e756e715a48959f1c0045342088d7ca9762a2f509b945f362a316e9412b7/pytest_benchmark-5.2.3-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/33/6b/e0547afaf41bf2c42e52430072fa5658766e3d65bd4b03a563d1b6336f57/distlib-0.4.0-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/33/b2/986d1220f6ee931e338d272bc1f3ec02cfe5f9b5fad84e95afdad57f1ebc/format_docstring-0.2.7-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/33/eb/f9f1ded8e4db9638f9530c3782eb01f5ab04945f4cb9e597a51c203fa4c5/diffpy_pdffit2-1.6.0.tar.gz @@ -2311,6 +2324,7 @@ environments: - pypi: https://files.pythonhosted.org/packages/db/3c/33bac158f8ab7f89b2e59426d5fe2e4f63f7ed25df84c036890172b412b5/cfgv-3.5.0-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/dc/9b/47798a6c91d8bdb567fe2698fe81e0c6b7cb7ef4d13da4114b41d239f65d/typing_inspection-0.4.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/de/1f/77fa3081e4f66ca3576c896ae5d31c3002ac6607f9747d2e3aa49227e464/markdown-3.10.2-py3-none-any.whl + - pypi: https://files.pythonhosted.org/packages/e0/a9/023730ba63db1e494a271cb018dcd361bd2c917ba7004c3e49d5daf795a2/py_cpuinfo-9.0.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/e0/f9/0595336914c5619e5f28a1fb793285925a8cd4b432c9da0a987836c7f822/shellingham-1.5.4-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/e5/04/c5bb20d64417d20cba0105277235c51969444fa873000fbc26ac0a3fc5a8/gemmi-0.7.5-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl - pypi: https://files.pythonhosted.org/packages/e7/05/c19819d5e3d95294a6f5947fb9b9629efb316b96de511b418c53d245aae6/cycler-0.12.1-py3-none-any.whl @@ -2541,6 +2555,7 @@ environments: - pypi: https://files.pythonhosted.org/packages/2c/19/04f9b178c2d8a15b076c8b5140708fa6ffc5601fb6f1e975537072df5b2a/mergedeep-1.3.4-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/32/28/79f0f8de97cce916d5ae88a7bee1ad724855e83e6019c0b4d5b3fabc80f3/mkdocstrings_python-2.0.3-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/32/46/9cb0e58b2deb7f82b84065f37f3bffeb12413f947f9388e4cac22c4621ce/sortedcontainers-2.4.0-py2.py3-none-any.whl + - pypi: https://files.pythonhosted.org/packages/33/29/e756e715a48959f1c0045342088d7ca9762a2f509b945f362a316e9412b7/pytest_benchmark-5.2.3-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/33/6b/e0547afaf41bf2c42e52430072fa5658766e3d65bd4b03a563d1b6336f57/distlib-0.4.0-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/33/b2/986d1220f6ee931e338d272bc1f3ec02cfe5f9b5fad84e95afdad57f1ebc/format_docstring-0.2.7-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/33/f0/3fe8c6e69135a845f4106f2ff8b6805638d4e85c264e70114e8126689587/tokenize_rt-6.2.0-py2.py3-none-any.whl @@ -2640,6 +2655,7 @@ environments: - pypi: https://files.pythonhosted.org/packages/dc/9b/47798a6c91d8bdb567fe2698fe81e0c6b7cb7ef4d13da4114b41d239f65d/typing_inspection-0.4.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/de/1f/77fa3081e4f66ca3576c896ae5d31c3002ac6607f9747d2e3aa49227e464/markdown-3.10.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/df/32/10ac51b4be7cdecd7e93d069251c86dfbf70b7adbd7c67b48ccea6c49e1c/sqlalchemy-2.0.50-cp314-cp314-macosx_11_0_arm64.whl + - pypi: https://files.pythonhosted.org/packages/e0/a9/023730ba63db1e494a271cb018dcd361bd2c917ba7004c3e49d5daf795a2/py_cpuinfo-9.0.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/e0/f9/0595336914c5619e5f28a1fb793285925a8cd4b432c9da0a987836c7f822/shellingham-1.5.4-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/e7/05/c19819d5e3d95294a6f5947fb9b9629efb316b96de511b418c53d245aae6/cycler-0.12.1-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/eb/be/b257e12f9710819fde40adc972578bee6b72c5992da1bc8369bef2597756/nbmake-1.5.5-py3-none-any.whl @@ -2859,6 +2875,7 @@ environments: - pypi: https://files.pythonhosted.org/packages/2c/19/04f9b178c2d8a15b076c8b5140708fa6ffc5601fb6f1e975537072df5b2a/mergedeep-1.3.4-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/32/28/79f0f8de97cce916d5ae88a7bee1ad724855e83e6019c0b4d5b3fabc80f3/mkdocstrings_python-2.0.3-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/32/46/9cb0e58b2deb7f82b84065f37f3bffeb12413f947f9388e4cac22c4621ce/sortedcontainers-2.4.0-py2.py3-none-any.whl + - pypi: https://files.pythonhosted.org/packages/33/29/e756e715a48959f1c0045342088d7ca9762a2f509b945f362a316e9412b7/pytest_benchmark-5.2.3-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/33/6b/e0547afaf41bf2c42e52430072fa5658766e3d65bd4b03a563d1b6336f57/distlib-0.4.0-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/33/b2/986d1220f6ee931e338d272bc1f3ec02cfe5f9b5fad84e95afdad57f1ebc/format_docstring-0.2.7-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/33/f0/3fe8c6e69135a845f4106f2ff8b6805638d4e85c264e70114e8126689587/tokenize_rt-6.2.0-py2.py3-none-any.whl @@ -2959,6 +2976,7 @@ environments: - pypi: https://files.pythonhosted.org/packages/dc/9b/47798a6c91d8bdb567fe2698fe81e0c6b7cb7ef4d13da4114b41d239f65d/typing_inspection-0.4.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/de/1f/77fa3081e4f66ca3576c896ae5d31c3002ac6607f9747d2e3aa49227e464/markdown-3.10.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/df/ac/46de6dda46478f7942f839e094970be2d4a861e005c4b3bf07c92e291a09/numpy-2.4.6-cp314-cp314-win_amd64.whl + - pypi: https://files.pythonhosted.org/packages/e0/a9/023730ba63db1e494a271cb018dcd361bd2c917ba7004c3e49d5daf795a2/py_cpuinfo-9.0.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/e0/bf/52f25716bbe93745595800f36fb17b73711f14da59ed0bb2eba141bc9f0f/multidict-6.7.1-cp314-cp314-win_amd64.whl - pypi: https://files.pythonhosted.org/packages/e0/f9/0595336914c5619e5f28a1fb793285925a8cd4b432c9da0a987836c7f822/shellingham-1.5.4-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/e7/05/c19819d5e3d95294a6f5947fb9b9629efb316b96de511b418c53d245aae6/cycler-0.12.1-py3-none-any.whl @@ -8826,6 +8844,7 @@ packages: - pre-commit ; extra == 'dev' - pydoclint ; extra == 'dev' - pytest ; extra == 'dev' + - pytest-benchmark ; extra == 'dev' - pytest-cov ; extra == 'dev' - pytest-xdist ; extra == 'dev' - pyyaml ; extra == 'dev' @@ -9732,6 +9751,19 @@ packages: - prettytable - ply - numpy +- pypi: https://files.pythonhosted.org/packages/33/29/e756e715a48959f1c0045342088d7ca9762a2f509b945f362a316e9412b7/pytest_benchmark-5.2.3-py3-none-any.whl + name: pytest-benchmark + version: 5.2.3 + sha256: bc839726ad20e99aaa0d11a127445457b4219bdb9e80a1afc4b51da7f96b0803 + requires_dist: + - pytest>=8.1 + - py-cpuinfo + - aspectlib ; extra == 'aspect' + - pygal ; extra == 'histogram' + - pygaljs ; extra == 'histogram' + - setuptools ; extra == 'histogram' + - elasticsearch ; extra == 'elasticsearch' + requires_python: '>=3.9' - pypi: https://files.pythonhosted.org/packages/33/6b/e0547afaf41bf2c42e52430072fa5658766e3d65bd4b03a563d1b6336f57/distlib-0.4.0-py2.py3-none-any.whl name: distlib version: 0.4.0 @@ -12709,6 +12741,10 @@ packages: version: 2.4.6 sha256: b507f5c4c1d508876d1819b6bf9a49d365b96320b5d4993426b33a23ca4b8261 requires_python: '>=3.11' +- pypi: https://files.pythonhosted.org/packages/e0/a9/023730ba63db1e494a271cb018dcd361bd2c917ba7004c3e49d5daf795a2/py_cpuinfo-9.0.0-py3-none-any.whl + name: py-cpuinfo + version: 9.0.0 + sha256: 859625bc251f64e21f077d099d4162689c762b5d6a4c3c97553d56241c9674d5 - pypi: https://files.pythonhosted.org/packages/e0/bf/52f25716bbe93745595800f36fb17b73711f14da59ed0bb2eba141bc9f0f/multidict-6.7.1-cp314-cp314-win_amd64.whl name: multidict version: 6.7.1 diff --git a/pixi.toml b/pixi.toml index 28b5ab7bf..8c45888be 100644 --- a/pixi.toml +++ b/pixi.toml @@ -106,8 +106,12 @@ user = { features = ['py-max', 'user'] } unit-tests = 'python -m pytest tests/unit/ --color=yes -v' functional-tests = 'python -m pytest tests/functional/ --color=yes -v' integration-tests = 'python -m pytest tests/integration/ --color=yes -n auto -v' -# Run only nightly-tier tests (e.g. performance benchmarks) across the suite. -nightly-tests = 'python -m pytest tests/ -m nightly --color=yes -n auto -v' +# Run nightly-tier tests across the suite (benchmarks have their own task +# because pytest-benchmark does not run under xdist). +nightly-tests = 'python -m pytest tests/ -m nightly --ignore=tests/benchmarks --color=yes -n auto -v' +# Performance benchmarks, per experiment type; writes benchmark.json for +# the nightly CI artifact. Informational (no regression gate yet). +benchmarks = 'python -m pytest tests/benchmarks/ --benchmark-only --benchmark-json=benchmark.json --color=yes -v' # Remove previously saved tutorial output projects (projects/ed_*) so the # tutorial-output checks cannot pass against a stale artifact from an earlier diff --git a/pyproject.toml b/pyproject.toml index 85e38e651..c00f22ad9 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -63,6 +63,7 @@ dev = [ 'pytest', # Testing 'pytest-cov', # Test coverage 'pytest-xdist', # Enable parallel testing + 'pytest-benchmark', # Performance benchmarking (nightly) 'hypothesis', # Property-based / input-domain testing 'ruff', # Linting and formatting code 'radon', # Code complexity and maintainability diff --git a/tests/benchmarks/test_calculate_pattern_benchmark.py b/tests/benchmarks/test_calculate_pattern_benchmark.py new file mode 100644 index 000000000..30c1eb52d --- /dev/null +++ b/tests/benchmarks/test_calculate_pattern_benchmark.py @@ -0,0 +1,46 @@ +# SPDX-FileCopyrightText: 2026 EasyScience contributors +# SPDX-License-Identifier: BSD-3-Clause +"""Performance benchmarks for diffraction-pattern calculation. + +Nightly-tier: these run real calculation engines and are excluded from +the per-push and per-PR runs. Each scenario is keyed by +``beam_mode x radiation_probe x engine`` so regressions can be localised. +Informational only for now — no regression gate yet (see ADR +test-suite-and-validation §7). Run with ``pixi run benchmarks``. +""" + +from __future__ import annotations + +import pytest + +# (id, structure download id, experiment download id, engine) +SCENARIOS = [ + ('neut-cwl-pd-cryspy', 1, 2, 'cryspy'), + ('neut-cwl-pd-crysfml', 1, 2, 'crysfml'), +] + + +@pytest.mark.nightly +@pytest.mark.parametrize( + ('label', 'structure_id', 'experiment_id', 'engine'), + SCENARIOS, + ids=[scenario[0] for scenario in SCENARIOS], +) +def test_calculate_pattern_benchmark(benchmark, label, structure_id, experiment_id, engine): + """Benchmark a single pattern calculation for one experiment x engine.""" + import easydiffraction as ed + from easydiffraction.analysis.fit_helpers.metrics import get_reliability_inputs + + project = ed.Project() + project.structures.add_from_cif_path(ed.download_data(id=structure_id, destination='data')) + project.experiments.add_from_cif_path(ed.download_data(id=experiment_id, destination='data')) + + experiment = project.experiments['hrpt'] + experiment.calculator.type = engine + + def _calculate(): + return get_reliability_inputs(project.structures, [experiment]) + + result = benchmark(_calculate) + + assert result is not None From 2dcb98d4ea47a6961b7d8d6acbfbeb553abf0176 Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Fri, 5 Jun 2026 22:48:59 -0700 Subject: [PATCH 18/57] Record cross-repo nightly harness and benchmarks as future work --- docs/dev/issues/open.md | 55 +++++++++++++++++++++ docs/dev/plans/test-suite-and-validation.md | 2 +- 2 files changed, 56 insertions(+), 1 deletion(-) diff --git a/docs/dev/issues/open.md b/docs/dev/issues/open.md index 89434ef8f..3c6b70d7b 100644 --- a/docs/dev/issues/open.md +++ b/docs/dev/issues/open.md @@ -1886,6 +1886,59 @@ only render the index column when no explicit id column is present. --- +## 113. 🟡 Cross-Repository Validation Harness (nightly) + +**Type:** Test infrastructure + +Deferred cross-repository work for the +[Test Suite and Validation Strategy](../adrs/suggestions/test-suite-and-validation.md) +ADR (§7, §8). The harness code lives in `diffraction-lib`; the corpus, +results database, and benchmark/reference history live in the +`diffraction` data repository, fetched at runtime and written back by a +nightly job that installs easydiffraction from PyPI (acceptance-style). + +**Items:** + +- **COD corpus check.** Download ~100–200 CIF files from the + Crystallography Open Database, load each, and record per-file status + (`ok` / `partial` + missing fields / `fail`) in a git-diffable CSV + keyed and ordered by COD id. Add a `--recheck-failed` flag to re-run + only the failed/partial entries after fixes (instead of new random + files). Extend with per-engine calculation results on the corpus. +- **Generative fuzzing.** Randomly generate ~100–200 structures (random + space group, cell, 1–10 atoms with random coordinates/ADP/occupancy), + compute patterns across engines, and record disagreements in the same + database. +- **Benchmark history + gate.** Commit `pytest-benchmark` baseline JSON + to the data repository and add a regression threshold once timing + variance is characterised on a controlled runner. (The serial + benchmark task itself now exists — see issue 16 — this is the + history/gating remainder.) +- **External-software comparison.** Add FullProf (then GSAS-II/TOPAS) + pre-calculated profiles as zipped projects in the data repository so + the Verification pages can overlay them against easydiffraction. + +**Depends on:** the `diffraction` data repository; cross-repo coordination. + +--- + +## 114. 🟢 External Link Checking in the Docs Gate + +**Type:** CI / Documentation + +The fast docs gate (`docs-build-strict` + `spell-check`) catches broken +nav/internal links and typos on every push, but does not yet check +external URLs. Add a `lychee` link checker (with an allowlist for +rate-limited/unstable domains), coordinated with the +[Documentation CI and Build Verification](../adrs/suggestions/documentation-ci-build.md) +ADR. Run it nightly or on pull requests to avoid flakiness from external +sites. Also covers link-checking of URLs that appear only inside executed +notebook output cells (a feature that does not exist yet). + +**Depends on:** nothing. + +--- + ## Summary | # | Issue | Severity | Type | @@ -1980,3 +2033,5 @@ only render the index column when no explicit id column is present. | 110 | Styled multi-line table cells in HTML backend | 🟢 Low | Display / Notebook parity | | 111 | Test coverage for `list_tutorials` rendering | 🟢 Low | Test coverage | | 112 | Suppress redundant row-index column in tables | 🟢 Low | Display / UX | +| 113 | Cross-repository validation harness (nightly) | 🟡 Med | Test infrastructure | +| 114 | External link checking in the docs gate | 🟢 Low | CI / Documentation | diff --git a/docs/dev/plans/test-suite-and-validation.md b/docs/dev/plans/test-suite-and-validation.md index ba2de3209..ec470d6f3 100644 --- a/docs/dev/plans/test-suite-and-validation.md +++ b/docs/dev/plans/test-suite-and-validation.md @@ -305,7 +305,7 @@ files or generated artifacts. Do not run Phase 2 commands during Phase 1. Files: `pyproject.toml`, `pixi.toml`, `tests/benchmarks/**`. Commit: `Add per-experiment performance benchmarks (nightly)` -- [ ] **P1.14 — Record cross-repository future work (§8 + deferred)** +- [x] **P1.14 — Record cross-repository future work (§8 + deferred)** Add prioritised entries to `docs/dev/issues/open.md` for the nightly COD harness + results DB + pip-install acceptance job, generative fuzzing, data-repo benchmark history, and external-software From d025df781842f84ad0ed444cd8f03578a3f4ee67 Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Fri, 5 Jun 2026 22:50:48 -0700 Subject: [PATCH 19/57] Promote test-suite-and-validation ADR to accepted --- docs/dev/adrs/accepted/test-strategy.md | 2 +- .../{suggestions => accepted}/test-suite-and-validation.md | 2 +- docs/dev/adrs/index.md | 2 +- docs/dev/issues/open.md | 2 +- docs/dev/plans/test-suite-and-validation.md | 4 ++-- docs/dev/testing-guide.md | 2 +- 6 files changed, 7 insertions(+), 7 deletions(-) rename docs/dev/adrs/{suggestions => accepted}/test-suite-and-validation.md (99%) diff --git a/docs/dev/adrs/accepted/test-strategy.md b/docs/dev/adrs/accepted/test-strategy.md index a4c23f5df..d5e03939b 100644 --- a/docs/dev/adrs/accepted/test-strategy.md +++ b/docs/dev/adrs/accepted/test-strategy.md @@ -42,7 +42,7 @@ structure makes missing coverage easier to spot. ## Amendments -[Test Suite and Validation Strategy](../suggestions/test-suite-and-validation.md) +[Test Suite and Validation Strategy](test-suite-and-validation.md) sharpens these layer definitions into strict, testable placement criteria and adds test cost tiers, coverage policy, codecov configuration, cross-engine verification documentation, and a nightly diff --git a/docs/dev/adrs/suggestions/test-suite-and-validation.md b/docs/dev/adrs/accepted/test-suite-and-validation.md similarity index 99% rename from docs/dev/adrs/suggestions/test-suite-and-validation.md rename to docs/dev/adrs/accepted/test-suite-and-validation.md index 5abeca8aa..61e1ec19f 100644 --- a/docs/dev/adrs/suggestions/test-suite-and-validation.md +++ b/docs/dev/adrs/accepted/test-suite-and-validation.md @@ -2,7 +2,7 @@ ## Status -Proposed. +Accepted. ## Date diff --git a/docs/dev/adrs/index.md b/docs/dev/adrs/index.md index 1e2b39bdc..5c815c062 100644 --- a/docs/dev/adrs/index.md +++ b/docs/dev/adrs/index.md @@ -45,7 +45,7 @@ folders. | Persistence | Accepted | Python and CIF Category Correspondence | Compares current Python paths and CIF tags, then records scoped one-to-one mapping for project-level categories. | [`python-cif-category-correspondence.md`](accepted/python-cif-category-correspondence.md) | | Quality | Accepted | Lint Complexity Thresholds | Treats ruff PLR complexity limits as design guardrails that should not be bypassed. | [`lint-complexity-thresholds.md`](accepted/lint-complexity-thresholds.md) | | Quality | Accepted | Test Strategy | Defines layered unit, functional, integration, script, and notebook testing. | [`test-strategy.md`](accepted/test-strategy.md) | -| Quality | Suggestion | Test Suite and Validation Strategy | Strict test layers, cost tiers, coverage/codecov policy, cross-engine verification docs, and a nightly validation harness. | [`test-suite-and-validation.md`](suggestions/test-suite-and-validation.md) | +| Quality | Accepted | Test Suite and Validation Strategy | Strict test layers, cost tiers, coverage/codecov policy, cross-engine verification docs, and a nightly validation harness. | [`test-suite-and-validation.md`](accepted/test-suite-and-validation.md) | | Structure model | Accepted | Type-Neutral ADP Parameters | Keeps ADP parameter object identities stable across B/U and iso/ani switches. | [`type-neutral-adp-parameters.md`](accepted/type-neutral-adp-parameters.md) | | Structure model | Accepted | Automatic Wyckoff Position Detection | Detects Wyckoff letter, multiplicity, and site symmetry from space group and coordinates; calculators consume them. | [`wyckoff-letter-detection.md`](accepted/wyckoff-letter-detection.md) | | Structure model | Accepted | Complete Space-Group Reference Database | One-time build of a complete space_groups.json.gz (all 230 groups) from cctbx, verified against multiple sources. | [`space-group-database.md`](accepted/space-group-database.md) | diff --git a/docs/dev/issues/open.md b/docs/dev/issues/open.md index 3c6b70d7b..a00bb03ac 100644 --- a/docs/dev/issues/open.md +++ b/docs/dev/issues/open.md @@ -1891,7 +1891,7 @@ only render the index column when no explicit id column is present. **Type:** Test infrastructure Deferred cross-repository work for the -[Test Suite and Validation Strategy](../adrs/suggestions/test-suite-and-validation.md) +[Test Suite and Validation Strategy](../adrs/accepted/test-suite-and-validation.md) ADR (§7, §8). The harness code lives in `diffraction-lib`; the corpus, results database, and benchmark/reference history live in the `diffraction` data repository, fetched at runtime and written back by a diff --git a/docs/dev/plans/test-suite-and-validation.md b/docs/dev/plans/test-suite-and-validation.md index ec470d6f3..b4d38bda4 100644 --- a/docs/dev/plans/test-suite-and-validation.md +++ b/docs/dev/plans/test-suite-and-validation.md @@ -21,7 +21,7 @@ branch, per the author's instruction. ## ADR Implements the suggestion ADR -[`test-suite-and-validation.md`](../adrs/suggestions/test-suite-and-validation.md) +[`test-suite-and-validation.md`](../adrs/accepted/test-suite-and-validation.md) (drafted via `/draft-adr`; review cycle closed). The plan owns this ADR. Per [`AGENTS.md`](../../../AGENTS.md) §Change Discipline, the ADR is promoted from `suggestions/` to `accepted/` as part of this change, @@ -313,7 +313,7 @@ files or generated artifacts. Do not run Phase 2 commands during Phase 1. Files: `docs/dev/issues/open.md`. Commit: `Record cross-repo nightly harness and benchmarks as future work` -- [ ] **P1.15 — Promote ADR to accepted (§Change Discipline)** +- [x] **P1.15 — Promote ADR to accepted (§Change Discipline)** `git mv docs/dev/adrs/suggestions/test-suite-and-validation.md docs/dev/adrs/accepted/`; set its `## Status` to `Accepted.`; flip the `docs/dev/adrs/index.md` row to `Accepted` with the `accepted/...` diff --git a/docs/dev/testing-guide.md b/docs/dev/testing-guide.md index 443c8ca7a..ee7ab9503 100644 --- a/docs/dev/testing-guide.md +++ b/docs/dev/testing-guide.md @@ -3,7 +3,7 @@ Practical placement rules for the easydiffraction test suite. The rationale is recorded in the ADRs [Test Strategy](adrs/accepted/test-strategy.md) and -[Test Suite and Validation Strategy](adrs/suggestions/test-suite-and-validation.md). +[Test Suite and Validation Strategy](adrs/accepted/test-suite-and-validation.md). ## Layers — what goes where From 78450c44a9bb660be9edeab8b93296898e25d7ba Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Fri, 5 Jun 2026 22:51:06 -0700 Subject: [PATCH 20/57] Reach Phase 1 review gate --- docs/dev/plans/test-suite-and-validation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/dev/plans/test-suite-and-validation.md b/docs/dev/plans/test-suite-and-validation.md index b4d38bda4..f3b55c646 100644 --- a/docs/dev/plans/test-suite-and-validation.md +++ b/docs/dev/plans/test-suite-and-validation.md @@ -322,7 +322,7 @@ files or generated artifacts. Do not run Phase 2 commands during Phase 1. Files: ADR file (moved), `docs/dev/adrs/index.md`. Commit: `Promote test-suite-and-validation ADR to accepted` -- [ ] **P1.16 — Phase 1 review gate (no code)** +- [x] **P1.16 — Phase 1 review gate (no code)** Confirm every box above is `[x]`. Mark this step and commit the checklist update alone. Commit: `Reach Phase 1 review gate` From a2a23a3a29a0657a1e442db15025a012448e7941 Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Fri, 5 Jun 2026 22:52:49 -0700 Subject: [PATCH 21/57] Run pr tier on pull_request events, not only main branches --- .github/workflows/test.yml | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index e3f1f88dd..debf84c0e 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -63,7 +63,11 @@ jobs: - name: Set test tier marker expression id: set-mark run: | - if [[ "${{ env.CI_BRANCH }}" == "master" || "${{ env.CI_BRANCH }}" == "develop" ]]; then + # Pull requests and develop/master pushes run the pr tier; + # feature-branch pushes run only the default (fast) tier. On a + # pull_request event CI_BRANCH is the contributor branch, so the + # event type must be checked explicitly. + if [[ "${{ github.event_name }}" == "pull_request" || "${{ env.CI_BRANCH }}" == "master" || "${{ env.CI_BRANCH }}" == "develop" ]]; then echo 'pytest_marks=-m "not nightly"' >> "$GITHUB_OUTPUT" else echo 'pytest_marks=-m "not pr and not nightly"' >> "$GITHUB_OUTPUT" From 7ccd28f5e1eb1e4ec2796a5457a015f7661cbd18 Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Fri, 5 Jun 2026 22:53:55 -0700 Subject: [PATCH 22/57] Share excluded-dirs set between structure check and docs --- docs/dev/package-structure/full.md | 15 ++++----------- docs/dev/package-structure/short.md | 12 ++---------- tools/_src_tree.py | 10 +++++++--- tools/generate_package_docs.py | 12 ++---------- 4 files changed, 15 insertions(+), 34 deletions(-) diff --git a/docs/dev/package-structure/full.md b/docs/dev/package-structure/full.md index 58991c95f..a242281cc 100644 --- a/docs/dev/package-structure/full.md +++ b/docs/dev/package-structure/full.md @@ -14,8 +14,10 @@ │ │ │ └── 🏷️ class CryspyCalculator │ │ ├── 📄 factory.py │ │ │ └── 🏷️ class CalculatorFactory -│ │ └── 📄 pdffit.py -│ │ └── 🏷️ class PdffitCalculator +│ │ ├── 📄 pdffit.py +│ │ │ └── 🏷️ class PdffitCalculator +│ │ └── 📄 support.py +│ │ └── 🏷️ class SupportEntry │ ├── 📁 categories │ │ ├── 📁 aliases │ │ │ ├── 📄 __init__.py @@ -507,8 +509,6 @@ │ │ │ ├── 📄 elements.py │ │ │ └── 📄 radii.py │ │ ├── 📁 renderers -│ │ │ ├── 📁 vendor -│ │ │ │ └── 📁 threejs │ │ │ ├── 📄 __init__.py │ │ │ ├── 📄 ascii.py │ │ │ │ ├── 🏷️ class _Orientation @@ -672,7 +672,6 @@ ├── 📁 report │ ├── 📁 templates │ │ ├── 📁 html -│ │ │ └── 📁 vendor │ │ └── 📁 tex │ ├── 📄 __init__.py │ ├── 📄 data_context.py @@ -685,12 +684,6 @@ │ ├── 📄 style.py │ └── 📄 tex_renderer.py ├── 📁 utils -│ ├── 📁 _vendored -│ │ ├── 📁 jupyter_dark_detect -│ │ │ ├── 📄 __init__.py -│ │ │ └── 📄 detector.py -│ │ ├── 📄 __init__.py -│ │ └── 📄 theme_detect.py │ ├── 📄 __init__.py │ ├── 📄 enums.py │ │ └── 🏷️ class VerbosityEnum diff --git a/docs/dev/package-structure/short.md b/docs/dev/package-structure/short.md index ec8c02af6..35b7863fb 100644 --- a/docs/dev/package-structure/short.md +++ b/docs/dev/package-structure/short.md @@ -9,7 +9,8 @@ │ │ ├── 📄 crysfml.py │ │ ├── 📄 cryspy.py │ │ ├── 📄 factory.py -│ │ └── 📄 pdffit.py +│ │ ├── 📄 pdffit.py +│ │ └── 📄 support.py │ ├── 📁 categories │ │ ├── 📁 aliases │ │ │ ├── 📄 __init__.py @@ -248,8 +249,6 @@ │ │ │ ├── 📄 elements.py │ │ │ └── 📄 radii.py │ │ ├── 📁 renderers -│ │ │ ├── 📁 vendor -│ │ │ │ └── 📁 threejs │ │ │ ├── 📄 __init__.py │ │ │ ├── 📄 ascii.py │ │ │ ├── 📄 base.py @@ -327,7 +326,6 @@ ├── 📁 report │ ├── 📁 templates │ │ ├── 📁 html -│ │ │ └── 📁 vendor │ │ └── 📁 tex │ ├── 📄 __init__.py │ ├── 📄 data_context.py @@ -338,12 +336,6 @@ │ ├── 📄 style.py │ └── 📄 tex_renderer.py ├── 📁 utils -│ ├── 📁 _vendored -│ │ ├── 📁 jupyter_dark_detect -│ │ │ ├── 📄 __init__.py -│ │ │ └── 📄 detector.py -│ │ ├── 📄 __init__.py -│ │ └── 📄 theme_detect.py │ ├── 📄 __init__.py │ ├── 📄 enums.py │ ├── 📄 environment.py diff --git a/tools/_src_tree.py b/tools/_src_tree.py index f57618226..eec5d371f 100644 --- a/tools/_src_tree.py +++ b/tools/_src_tree.py @@ -4,8 +4,8 @@ Shared by ``tools/test_structure_check.py`` (the unit-test mirror check) and ``tools/generate_package_docs.py`` (the package-structure docs), so -the two tools cannot drift on where the source tree lives or which -modules count as source. +the two tools cannot drift on where the source tree lives, which +directories are excluded, or which modules count as source. """ from __future__ import annotations @@ -20,8 +20,12 @@ # and tooling caches). EXCLUDED_DIRS: set[str] = { '_vendored', - '__pycache__', 'vendor', + '__pycache__', + '.pytest_cache', + '.mypy_cache', + '.ruff_cache', + '.ipynb_checkpoints', } # Source module stems that do not need a dedicated unit-test file. diff --git a/tools/generate_package_docs.py b/tools/generate_package_docs.py index c30cc5c3e..4411a200f 100644 --- a/tools/generate_package_docs.py +++ b/tools/generate_package_docs.py @@ -19,21 +19,13 @@ from pathlib import Path from typing import List +from _src_tree import EXCLUDED_DIRS from _src_tree import REPO_ROOT from _src_tree import SRC_ROOT DOCS_OUT_DIR = REPO_ROOT / 'docs' / 'dev' / 'package-structure' -IGNORE_DIRS = { - '__pycache__', - '.pytest_cache', - '.mypy_cache', - '.ruff_cache', - '.ipynb_checkpoints', -} - - @dataclass class Node: name: str @@ -68,7 +60,7 @@ def _walk(p: Path) -> Node: except PermissionError: entries = [] for child in entries: - if child.name in IGNORE_DIRS: + if child.name in EXCLUDED_DIRS: continue if child.is_dir(): node.children.append(_walk(child)) From 5edcc7b8dece30c1475d67629f70ec51252fb465 Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Fri, 5 Jun 2026 22:59:35 -0700 Subject: [PATCH 23/57] Add lychee link check to the docs gate --- .github/workflows/test.yml | 3 +++ lychee.toml | 22 ++++++++++++++++++ pixi.lock | 46 ++++++++++++++++++++++++++++++++++++++ pixi.toml | 4 ++++ 4 files changed, 75 insertions(+) create mode 100644 lychee.toml diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index debf84c0e..d5d5f7c3b 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -92,6 +92,9 @@ jobs: - name: Spell check run: pixi run spell-check + - name: Link check + run: pixi run link-check + # Job 2: Test code source-test: needs: env-prepare # depend on previous job diff --git a/lychee.toml b/lychee.toml new file mode 100644 index 000000000..db6de881d --- /dev/null +++ b/lychee.toml @@ -0,0 +1,22 @@ +# lychee link-checker configuration. +# +# Fast every-push docs gate: verify local/relative links only. External +# URL checking is deferred to avoid flakiness from rate-limited sites +# (see docs/dev/issues/open.md issue 114). + +# Skip online links entirely — deterministic and fast, no network. +offline = true + +# Quieter CI logs. +no_progress = true + +# Directories never traversed for links: third-party/tooling caches and +# generated artifacts (tutorial notebooks and package-structure docs are +# generated; their sources are checked elsewhere). +exclude_path = [ + "node_modules", + ".pixi", + "tmp", + "docs/docs/tutorials", + "docs/dev/package-structure", +] diff --git a/pixi.lock b/pixi.lock index 6b63ee45e..b0a0d316b 100644 --- a/pixi.lock +++ b/pixi.lock @@ -60,6 +60,7 @@ environments: - conda: https://conda.anaconda.org/conda-forge/linux-64/libuv-1.52.1-h280c20c_0.conda - conda: https://conda.anaconda.org/conda-forge/linux-64/libxcb-1.17.0-h8a09558_0.conda - conda: https://conda.anaconda.org/conda-forge/linux-64/libzlib-1.3.2-h25fd6f3_2.conda + - conda: https://conda.anaconda.org/conda-forge/linux-64/lychee-0.24.2-he64ecbb_0.conda - conda: https://conda.anaconda.org/conda-forge/linux-64/markupsafe-3.0.3-py314h67df5f8_1.conda - conda: https://conda.anaconda.org/conda-forge/linux-64/msgspec-0.21.1-py314h5bd0f2a_0.conda - conda: https://conda.anaconda.org/conda-forge/linux-64/ncurses-6.6-hdb14827_0.conda @@ -509,6 +510,7 @@ environments: - conda: https://conda.anaconda.org/conda-forge/osx-arm64/libuv-1.52.1-h1a92334_0.conda - conda: https://conda.anaconda.org/conda-forge/osx-arm64/libzlib-1.3.2-h8088a28_2.conda - conda: https://conda.anaconda.org/conda-forge/osx-arm64/llvm-openmp-22.1.6-hc7d1edf_0.conda + - conda: https://conda.anaconda.org/conda-forge/osx-arm64/lychee-0.24.2-h17e24d4_0.conda - conda: https://conda.anaconda.org/conda-forge/osx-arm64/markupsafe-3.0.3-py314h6e9b3f0_1.conda - conda: https://conda.anaconda.org/conda-forge/osx-arm64/msgspec-0.21.1-py314h6c2aa35_0.conda - conda: https://conda.anaconda.org/conda-forge/osx-arm64/ncurses-6.6-h1d4f5a5_0.conda @@ -826,6 +828,7 @@ environments: - conda: https://conda.anaconda.org/conda-forge/win-64/libxml2-2.15.3-h8ef44ab_0.conda - conda: https://conda.anaconda.org/conda-forge/win-64/libzlib-1.3.2-hfd05255_2.conda - conda: https://conda.anaconda.org/conda-forge/win-64/llvm-openmp-22.1.6-h4fa8253_0.conda + - conda: https://conda.anaconda.org/conda-forge/win-64/lychee-0.24.2-hb3eb754_0.conda - conda: https://conda.anaconda.org/conda-forge/win-64/markupsafe-3.0.3-py314h2359020_1.conda - conda: https://conda.anaconda.org/conda-forge/win-64/mkl-2026.0.0-hac47afa_908.conda - conda: https://conda.anaconda.org/conda-forge/win-64/msgspec-0.21.1-py314h5a2d7ad_0.conda @@ -1060,6 +1063,7 @@ environments: - conda: https://conda.anaconda.org/conda-forge/linux-64/libxcb-1.17.0-h8a09558_0.conda - conda: https://conda.anaconda.org/conda-forge/linux-64/libxcrypt-4.4.36-hd590300_1.conda - conda: https://conda.anaconda.org/conda-forge/linux-64/libzlib-1.3.2-h25fd6f3_2.conda + - conda: https://conda.anaconda.org/conda-forge/linux-64/lychee-0.24.2-he64ecbb_0.conda - conda: https://conda.anaconda.org/conda-forge/linux-64/markupsafe-3.0.3-py312h8a5da7c_1.conda - conda: https://conda.anaconda.org/conda-forge/linux-64/msgspec-0.21.1-py312h4c3975b_0.conda - conda: https://conda.anaconda.org/conda-forge/linux-64/ncurses-6.6-hdb14827_0.conda @@ -1507,6 +1511,7 @@ environments: - conda: https://conda.anaconda.org/conda-forge/osx-arm64/libuv-1.52.1-h1a92334_0.conda - conda: https://conda.anaconda.org/conda-forge/osx-arm64/libzlib-1.3.2-h8088a28_2.conda - conda: https://conda.anaconda.org/conda-forge/osx-arm64/llvm-openmp-22.1.6-hc7d1edf_0.conda + - conda: https://conda.anaconda.org/conda-forge/osx-arm64/lychee-0.24.2-h17e24d4_0.conda - conda: https://conda.anaconda.org/conda-forge/osx-arm64/markupsafe-3.0.3-py312h04c11ed_1.conda - conda: https://conda.anaconda.org/conda-forge/osx-arm64/msgspec-0.21.1-py312h2bbb03f_0.conda - conda: https://conda.anaconda.org/conda-forge/osx-arm64/ncurses-6.6-h1d4f5a5_0.conda @@ -1823,6 +1828,7 @@ environments: - conda: https://conda.anaconda.org/conda-forge/win-64/libxml2-2.15.3-h8ef44ab_0.conda - conda: https://conda.anaconda.org/conda-forge/win-64/libzlib-1.3.2-hfd05255_2.conda - conda: https://conda.anaconda.org/conda-forge/win-64/llvm-openmp-22.1.6-h4fa8253_0.conda + - conda: https://conda.anaconda.org/conda-forge/win-64/lychee-0.24.2-hb3eb754_0.conda - conda: https://conda.anaconda.org/conda-forge/win-64/markupsafe-3.0.3-py312h05f76fc_1.conda - conda: https://conda.anaconda.org/conda-forge/win-64/mkl-2026.0.0-hac47afa_908.conda - conda: https://conda.anaconda.org/conda-forge/win-64/msgspec-0.21.1-py312he06e257_0.conda @@ -2055,6 +2061,7 @@ environments: - conda: https://conda.anaconda.org/conda-forge/linux-64/libuv-1.52.1-h280c20c_0.conda - conda: https://conda.anaconda.org/conda-forge/linux-64/libxcb-1.17.0-h8a09558_0.conda - conda: https://conda.anaconda.org/conda-forge/linux-64/libzlib-1.3.2-h25fd6f3_2.conda + - conda: https://conda.anaconda.org/conda-forge/linux-64/lychee-0.24.2-he64ecbb_0.conda - conda: https://conda.anaconda.org/conda-forge/linux-64/markupsafe-3.0.3-py314h67df5f8_1.conda - conda: https://conda.anaconda.org/conda-forge/linux-64/msgspec-0.21.1-py314h5bd0f2a_0.conda - conda: https://conda.anaconda.org/conda-forge/linux-64/ncurses-6.6-hdb14827_0.conda @@ -2504,6 +2511,7 @@ environments: - conda: https://conda.anaconda.org/conda-forge/osx-arm64/libuv-1.52.1-h1a92334_0.conda - conda: https://conda.anaconda.org/conda-forge/osx-arm64/libzlib-1.3.2-h8088a28_2.conda - conda: https://conda.anaconda.org/conda-forge/osx-arm64/llvm-openmp-22.1.6-hc7d1edf_0.conda + - conda: https://conda.anaconda.org/conda-forge/osx-arm64/lychee-0.24.2-h17e24d4_0.conda - conda: https://conda.anaconda.org/conda-forge/osx-arm64/markupsafe-3.0.3-py314h6e9b3f0_1.conda - conda: https://conda.anaconda.org/conda-forge/osx-arm64/msgspec-0.21.1-py314h6c2aa35_0.conda - conda: https://conda.anaconda.org/conda-forge/osx-arm64/ncurses-6.6-h1d4f5a5_0.conda @@ -2821,6 +2829,7 @@ environments: - conda: https://conda.anaconda.org/conda-forge/win-64/libxml2-2.15.3-h8ef44ab_0.conda - conda: https://conda.anaconda.org/conda-forge/win-64/libzlib-1.3.2-hfd05255_2.conda - conda: https://conda.anaconda.org/conda-forge/win-64/llvm-openmp-22.1.6-h4fa8253_0.conda + - conda: https://conda.anaconda.org/conda-forge/win-64/lychee-0.24.2-hb3eb754_0.conda - conda: https://conda.anaconda.org/conda-forge/win-64/markupsafe-3.0.3-py314h2359020_1.conda - conda: https://conda.anaconda.org/conda-forge/win-64/mkl-2026.0.0-hac47afa_908.conda - conda: https://conda.anaconda.org/conda-forge/win-64/msgspec-0.21.1-py314h5a2d7ad_0.conda @@ -4380,6 +4389,19 @@ packages: purls: [] size: 63629 timestamp: 1774072609062 +- conda: https://conda.anaconda.org/conda-forge/linux-64/lychee-0.24.2-he64ecbb_0.conda + sha256: 3fd9108a9af3ad8f5124b9dc4ec4c0169e27e6379100374844b6a058ca8e8988 + md5: 5ef12daae06f2228dcb37c95a8db1881 + depends: + - libgcc >=14 + - __glibc >=2.17,<3.0.a0 + - openssl >=3.5.6,<4.0a0 + constrains: + - __glibc >=2.17 + license: Apache-2.0 OR MIT + purls: [] + size: 5789257 + timestamp: 1777662729855 - conda: https://conda.anaconda.org/conda-forge/linux-64/markupsafe-3.0.3-py312h8a5da7c_1.conda sha256: 5f3aad1f3a685ed0b591faad335957dbdb1b73abfd6fc731a0d42718e0653b33 md5: 93a4752d42b12943a355b682ee43285b @@ -7225,6 +7247,18 @@ packages: purls: [] size: 284850 timestamp: 1779340584016 +- conda: https://conda.anaconda.org/conda-forge/osx-arm64/lychee-0.24.2-h17e24d4_0.conda + sha256: f8d8397e8aa3077ebaeffdd18f7aaf0fce4188a3dbd08514ed85dc34acee4220 + md5: 0b13f105d1195e1b388706f8eb7f0a03 + depends: + - __osx >=11.0 + - openssl >=3.5.6,<4.0a0 + constrains: + - __osx >=11.0 + license: Apache-2.0 OR MIT + purls: [] + size: 5382476 + timestamp: 1777662876427 - conda: https://conda.anaconda.org/conda-forge/osx-arm64/markupsafe-3.0.3-py312h04c11ed_1.conda sha256: 330394fb9140995b29ae215a19fad46fcc6691bdd1b7654513d55a19aaa091c1 md5: 11d95ab83ef0a82cc2de12c1e0b47fe4 @@ -8238,6 +8272,18 @@ packages: purls: [] size: 347116 timestamp: 1779341186510 +- conda: https://conda.anaconda.org/conda-forge/win-64/lychee-0.24.2-hb3eb754_0.conda + sha256: 826077733e3c251831d7b7c9cdb27da4b661044363343c5614527a618cd94d40 + md5: f06076ddc8c0f46c64900f1c8b479325 + depends: + - vc >=14.3,<15 + - vc14_runtime >=14.44.35208 + - ucrt >=10.0.20348.0 + - openssl >=3.5.6,<4.0a0 + license: Apache-2.0 OR MIT + purls: [] + size: 6020917 + timestamp: 1777662847120 - conda: https://conda.anaconda.org/conda-forge/win-64/markupsafe-3.0.3-py312h05f76fc_1.conda sha256: b744287a780211ac4595126ef96a44309c791f155d4724021ef99092bae4aace md5: a73298d225c7852f97403ca105d10a13 diff --git a/pixi.toml b/pixi.toml index 8c45888be..cd1f91c5b 100644 --- a/pixi.toml +++ b/pixi.toml @@ -54,6 +54,7 @@ ipython = '*' # Interactive Python shell pixi-kernel = '*' # Pixi Jupyter kernel gsl = '*' # GNU Scientific Library; required for diffpy.pdffit2 tectonic = '*' # LaTeX engine for PDF report generation +lychee = '*' # Link checker for documentation [feature.dev.pypi-dependencies] pip = '*' @@ -258,6 +259,9 @@ docs-build-strict = { cmd = 'pixi run docs-pre build --strict -f docs/mkdocs.yml ] } # Spelling check over docs and source (config in [tool.codespell]). spell-check = 'codespell docs src tools README.md CONTRIBUTING.md' +# Local/relative link integrity across the docs (offline; external URL +# checking is deferred, see issue 114). Config in lychee.toml. +link-check = 'lychee --config lychee.toml docs README.md CONTRIBUTING.md' docs-deploy-pre = 'mike deploy -F docs/mkdocs.yml --push --branch gh-pages --update-aliases --alias-type redirect' docs-set-default-pre = 'mike set-default -F docs/mkdocs.yml --push --branch gh-pages' From f9a17300cb26f702bf2b21eb47429d4edace547b Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Fri, 5 Jun 2026 23:00:27 -0700 Subject: [PATCH 24/57] Run verification notebooks in notebook test and exec tasks --- pixi.toml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/pixi.toml b/pixi.toml index cd1f91c5b..de33de28f 100644 --- a/pixi.toml +++ b/pixi.toml @@ -121,7 +121,7 @@ clean-tutorial-projects = { cmd = 'python tools/clean_tutorial_projects.py', env script-tests = { cmd = 'python -m pytest tools/test_scripts.py --color=yes -n auto -v', depends-on = [ 'clean-tutorial-projects', ], env = { EASYDIFFRACTION_ARTIFACT_ROOT = 'tmp/tutorials' } } -notebook-tests = { cmd = 'python -m pytest --nbmake docs/docs/tutorials/ --nbmake-timeout=1200 --color=yes -n auto -v', depends-on = [ +notebook-tests = { cmd = 'python -m pytest --nbmake docs/docs/tutorials/ docs/docs/verification/ --nbmake-timeout=1200 --color=yes -n auto -v', depends-on = [ 'clean-tutorial-projects', ], env = { EASYDIFFRACTION_ARTIFACT_ROOT = 'tmp/tutorials' } } @@ -225,8 +225,8 @@ jupyter = { cmd = 'jupyter', env = { EASYDIFFRACTION_ARTIFACT_ROOT = 'tmp/tutori notebook-convert = 'jupytext docs/docs/tutorials/*.py docs/docs/verification/*.py --from py:percent --to ipynb' notebook-strip = 'nbstripout docs/docs/tutorials/*.ipynb docs/docs/verification/*.ipynb' notebook-tweak = 'python tools/tweak_notebooks.py docs/docs/tutorials/ docs/docs/verification/' -notebook-exec = { cmd = 'python -m pytest --nbmake docs/docs/tutorials/ --overwrite --color=yes -n auto -v', env = { EASYDIFFRACTION_ARTIFACT_ROOT = 'tmp/tutorials', EASYDIFFRACTION_FIGURE_EMBED_MODE = 'shared' } } -notebook-exec-ci = { cmd = 'python -m pytest --nbmake docs/docs/tutorials/ --nbmake-timeout=1200 --overwrite --color=yes -n auto -v', env = { EASYDIFFRACTION_ARTIFACT_ROOT = '.', EASYDIFFRACTION_FIGURE_EMBED_MODE = 'shared' } } +notebook-exec = { cmd = 'python -m pytest --nbmake docs/docs/tutorials/ docs/docs/verification/ --overwrite --color=yes -n auto -v', env = { EASYDIFFRACTION_ARTIFACT_ROOT = 'tmp/tutorials', EASYDIFFRACTION_FIGURE_EMBED_MODE = 'shared' } } +notebook-exec-ci = { cmd = 'python -m pytest --nbmake docs/docs/tutorials/ docs/docs/verification/ --nbmake-timeout=1200 --overwrite --color=yes -n auto -v', env = { EASYDIFFRACTION_ARTIFACT_ROOT = '.', EASYDIFFRACTION_FIGURE_EMBED_MODE = 'shared' } } notebook-prepare = { depends-on = [ 'notebook-convert', From 0262ed332bbc945c8c4f72edb0a2bb72d932664b Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Fri, 5 Jun 2026 23:01:16 -0700 Subject: [PATCH 25/57] Assert explicit tolerances for all verification metrics --- .../verification/cross-engine-bragg-cwl.ipynb | 32 ++++++++++++++++--- .../verification/cross-engine-bragg-cwl.py | 32 ++++++++++++++++--- 2 files changed, 54 insertions(+), 10 deletions(-) diff --git a/docs/docs/verification/cross-engine-bragg-cwl.ipynb b/docs/docs/verification/cross-engine-bragg-cwl.ipynb index eee4495bb..a60c7a864 100644 --- a/docs/docs/verification/cross-engine-bragg-cwl.ipynb +++ b/docs/docs/verification/cross-engine-bragg-cwl.ipynb @@ -194,10 +194,10 @@ "source": [ "## Regression assertions\n", "\n", - "Both engines compute the same physics for the same structure, so their\n", - "calculated patterns share peak positions and are highly correlated. The\n", - "tolerance is intentionally loose for now and tightened as the\n", - "verification suite matures." + "Explicit, named tolerances for each metric. These are intentionally\n", + "loose initial bounds — they catch a gross cross-engine divergence now\n", + "and are tightened once nightly runs establish the real spread for each\n", + "engine pair. Correlation is kept as an additional shape signal." ] }, { @@ -207,7 +207,29 @@ "metadata": {}, "outputs": [], "source": [ - "assert correlation > 0.8, f'cross-engine correlation unexpectedly low: {correlation:.4f}'" + "# Loose initial tolerances (tightened once real spreads are measured).\n", + "MAX_PROFILE_DIFFERENCE_PCT = 100.0\n", + "MAX_RELATIVE_DEVIATION = 2.0\n", + "MIN_INTENSITY_RATIO = 0.1\n", + "MAX_INTENSITY_RATIO = 10.0\n", + "MIN_CORRELATION = 0.8\n", + "\n", + "peak = float(np.max(np.abs(a)))\n", + "relative_deviation = max_deviation / peak if peak else float('nan')\n", + "\n", + "assert profile_diff_pct < MAX_PROFILE_DIFFERENCE_PCT, (\n", + " f'profile difference {profile_diff_pct:.2f}% exceeds {MAX_PROFILE_DIFFERENCE_PCT}%'\n", + ")\n", + "assert relative_deviation < MAX_RELATIVE_DEVIATION, (\n", + " f'relative max deviation {relative_deviation:.3f} exceeds {MAX_RELATIVE_DEVIATION}'\n", + ")\n", + "assert MIN_INTENSITY_RATIO < intensity_ratio < MAX_INTENSITY_RATIO, (\n", + " f'integrated-intensity ratio {intensity_ratio:.3f} outside '\n", + " f'[{MIN_INTENSITY_RATIO}, {MAX_INTENSITY_RATIO}]'\n", + ")\n", + "assert correlation > MIN_CORRELATION, (\n", + " f'cross-engine correlation {correlation:.4f} below {MIN_CORRELATION}'\n", + ")" ] } ], diff --git a/docs/docs/verification/cross-engine-bragg-cwl.py b/docs/docs/verification/cross-engine-bragg-cwl.py index 8805a6df0..a94f7b40a 100644 --- a/docs/docs/verification/cross-engine-bragg-cwl.py +++ b/docs/docs/verification/cross-engine-bragg-cwl.py @@ -97,10 +97,32 @@ # %% [markdown] # ## Regression assertions # -# Both engines compute the same physics for the same structure, so their -# calculated patterns share peak positions and are highly correlated. The -# tolerance is intentionally loose for now and tightened as the -# verification suite matures. +# Explicit, named tolerances for each metric. These are intentionally +# loose initial bounds — they catch a gross cross-engine divergence now +# and are tightened once nightly runs establish the real spread for each +# engine pair. Correlation is kept as an additional shape signal. # %% -assert correlation > 0.8, f'cross-engine correlation unexpectedly low: {correlation:.4f}' +# Loose initial tolerances (tightened once real spreads are measured). +MAX_PROFILE_DIFFERENCE_PCT = 100.0 +MAX_RELATIVE_DEVIATION = 2.0 +MIN_INTENSITY_RATIO = 0.1 +MAX_INTENSITY_RATIO = 10.0 +MIN_CORRELATION = 0.8 + +peak = float(np.max(np.abs(a))) +relative_deviation = max_deviation / peak if peak else float('nan') + +assert profile_diff_pct < MAX_PROFILE_DIFFERENCE_PCT, ( + f'profile difference {profile_diff_pct:.2f}% exceeds {MAX_PROFILE_DIFFERENCE_PCT}%' +) +assert relative_deviation < MAX_RELATIVE_DEVIATION, ( + f'relative max deviation {relative_deviation:.3f} exceeds {MAX_RELATIVE_DEVIATION}' +) +assert MIN_INTENSITY_RATIO < intensity_ratio < MAX_INTENSITY_RATIO, ( + f'integrated-intensity ratio {intensity_ratio:.3f} outside ' + f'[{MIN_INTENSITY_RATIO}, {MAX_INTENSITY_RATIO}]' +) +assert correlation > MIN_CORRELATION, ( + f'cross-engine correlation {correlation:.4f} below {MIN_CORRELATION}' +) From eb3aee00711587551e09eeb3f306525c1e997904 Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Fri, 5 Jun 2026 23:08:40 -0700 Subject: [PATCH 26/57] Run full suite across all tiers in the nightly workflow --- .github/workflows/nightly.yml | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/.github/workflows/nightly.yml b/.github/workflows/nightly.yml index 52ae1c1fb..c8c5e85e0 100644 --- a/.github/workflows/nightly.yml +++ b/.github/workflows/nightly.yml @@ -1,6 +1,7 @@ -# Nightly run of the most expensive `nightly`-tier tests (for example the -# performance benchmarks). These are excluded from the per-push and -# per-PR runs in test.yml; see ADR test-suite-and-validation. +# Nightly run of the FULL test suite across all cost tiers (default, pr, +# and nightly), plus the performance benchmarks. The per-push and per-PR +# runs in test.yml deselect the pr/nightly tiers; the nightly run does +# not. See ADR test-suite-and-validation. name: Nightly tests @@ -30,8 +31,8 @@ jobs: - name: Set up pixi uses: ./.github/actions/setup-pixi - - name: Run nightly-tier tests - run: pixi run nightly-tests + - name: Run the full test suite (all tiers) + run: pixi run test-all - name: Run performance benchmarks run: pixi run benchmarks From 7d02a248e8a6bcc9feea6d1d57f0cc40eb1fd040 Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Fri, 5 Jun 2026 23:11:57 -0700 Subject: [PATCH 27/57] Scope verification to CWL powder now, defer rest to issue 115 --- .../adrs/accepted/test-suite-and-validation.md | 8 ++++++-- docs/dev/issues/open.md | 18 ++++++++++++++++++ docs/dev/plans/test-suite-and-validation.md | 8 +++++--- 3 files changed, 29 insertions(+), 5 deletions(-) diff --git a/docs/dev/adrs/accepted/test-suite-and-validation.md b/docs/dev/adrs/accepted/test-suite-and-validation.md index 61e1ec19f..65eb20aef 100644 --- a/docs/dev/adrs/accepted/test-suite-and-validation.md +++ b/docs/dev/adrs/accepted/test-suite-and-validation.md @@ -244,10 +244,14 @@ Add a new top-level **Verification** section to the documentation nav and an integrated-intensity ratio — with explicit tolerances. - **Overlay plots.** Plot all engines on one chart with distinct colours and line styles (solid/dotted/…) for visual comparison. -- **Coverage of conditions.** Include **every valid experiment × +- **Coverage of conditions.** Grow to cover **every valid experiment × instrument-parameter combination at least once** (powder/single crystal × constant-wavelength/time-of-flight × neutron/x-ray × - bragg/total, per the support matrix below). + bragg/total, per the support matrix below). The section ships with the + framework and the first cross-engine comparison (constant-wavelength + powder, cryspy ↔ crysfml); the remaining supported combinations + (time-of-flight powder, single crystal) are added **incrementally** and + tracked in the open-issues list. - **External software, incrementally.** External tools (FullProf first, then GSAS-II/TOPAS) are compared by loading a **pre-calculated profile from a zipped project** stored in the `diffraction` data repository diff --git a/docs/dev/issues/open.md b/docs/dev/issues/open.md index a00bb03ac..25bc25aaf 100644 --- a/docs/dev/issues/open.md +++ b/docs/dev/issues/open.md @@ -1939,6 +1939,23 @@ notebook output cells (a feature that does not exist yet). --- +## 115. 🟢 Expand Cross-Engine Verification Coverage + +**Type:** Test coverage / Documentation + +The Verification docs section ships with the framework and the first +cross-engine comparison page (constant-wavelength powder, cryspy ↔ +crysfml). Extend it to the remaining supported combinations declared by +the calculator support matrix — time-of-flight powder (cryspy ↔ crysfml) +and single crystal — so every valid experiment/instrument combination is +documented and regression-checked at least once. Each new page is a +calculation-only `.py` under `docs/docs/verification/` wired into +`script-tests` and `notebook-tests`, with explicit metric tolerances. + +**Depends on:** nothing. + +--- + ## Summary | # | Issue | Severity | Type | @@ -2035,3 +2052,4 @@ notebook output cells (a feature that does not exist yet). | 112 | Suppress redundant row-index column in tables | 🟢 Low | Display / UX | | 113 | Cross-repository validation harness (nightly) | 🟡 Med | Test infrastructure | | 114 | External link checking in the docs gate | 🟢 Low | CI / Documentation | +| 115 | Expand cross-engine verification coverage | 🟢 Low | Test coverage | diff --git a/docs/dev/plans/test-suite-and-validation.md b/docs/dev/plans/test-suite-and-validation.md index f3b55c646..e71908f77 100644 --- a/docs/dev/plans/test-suite-and-validation.md +++ b/docs/dev/plans/test-suite-and-validation.md @@ -285,9 +285,11 @@ files or generated artifacts. Do not run Phase 2 commands during Phase 1. - [x] **P1.12 — Cross-engine verification pages + script wiring (§6)** Add the `Verification` nav node (between Tutorials and Command-Line) - and calculation-only `.py` comparison pages (cryspy ↔ crysfml) across - the supported experiment combinations, with closeness metrics, overlay - plots, and metric-tolerance assertions. **Wire the new + and a calculation-only `.py` comparison page (cryspy ↔ crysfml) for the + **first** supported combination (constant-wavelength powder), with + closeness metrics, overlay plots, and metric-tolerance assertions. The + remaining supported combinations (time-of-flight powder, single + crystal) are added incrementally (issue 115). **Wire the new `docs/docs/verification/` directory into the script-test runner** — `tools/test_scripts.py` discovers only `docs/docs/tutorials/*.py` today (lines 24-27) — and into the notebook pipeline (`notebook-prepare` From 64168ae0afa5cd8e918caf27d4d0cb24b3a746a3 Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Fri, 5 Jun 2026 23:15:01 -0700 Subject: [PATCH 28/57] Lint and format verification pages like tutorials --- pixi.toml | 14 +++++++------- pyproject.toml | 7 +++++++ 2 files changed, 14 insertions(+), 7 deletions(-) diff --git a/pixi.toml b/pixi.toml index de33de28f..877089a45 100644 --- a/pixi.toml +++ b/pixi.toml @@ -152,9 +152,9 @@ test-all = { depends-on = [ pyproject-check = 'python -m validate_pyproject pyproject.toml' docstring-lint-check = 'pydoclint --quiet src/' -notebook-lint-check = 'nbqa ruff docs/docs/tutorials/' -py-lint-check = 'ruff check src/ tests/ docs/docs/tutorials/' -py-format-check = 'ruff format --check src/ tests/ docs/docs/tutorials/' +notebook-lint-check = 'nbqa ruff docs/docs/tutorials/ docs/docs/verification/' +py-lint-check = 'ruff check src/ tests/ docs/docs/tutorials/ docs/docs/verification/' +py-format-check = 'ruff format --check src/ tests/ docs/docs/tutorials/ docs/docs/verification/' nonpy-format-check = 'npx prettier --list-different --config=prettierrc.toml --ignore-unknown .' nonpy-format-check-modified = 'python tools/nonpy_prettier_modified.py' test-structure-check = 'python tools/test_structure_check.py' @@ -167,10 +167,10 @@ check = 'pre-commit run --hook-stage manual --all-files' docstring-transform = 'pixi run docstripy src/ -s=numpy -w' docstring-format-fix = 'format-docstring src/' -notebook-lint-fix = 'nbqa ruff --fix docs/docs/tutorials/' -py-lint-fix = 'ruff check --fix src/ tests/ docs/docs/tutorials/' -py-lint-fix-unsafe = 'ruff check --fix --unsafe-fixes src/ tests/ docs/docs/tutorials/' -py-format-fix = 'ruff format src/ tests/ docs/docs/tutorials/' +notebook-lint-fix = 'nbqa ruff --fix docs/docs/tutorials/ docs/docs/verification/' +py-lint-fix = 'ruff check --fix src/ tests/ docs/docs/tutorials/ docs/docs/verification/' +py-lint-fix-unsafe = 'ruff check --fix --unsafe-fixes src/ tests/ docs/docs/tutorials/ docs/docs/verification/' +py-format-fix = 'ruff format src/ tests/ docs/docs/tutorials/ docs/docs/verification/' nonpy-format-fix = 'npx prettier --write --list-different --config=prettierrc.toml --ignore-unknown .' nonpy-format-fix-modified = 'python tools/nonpy_prettier_modified.py --write' update-package-diagrams = 'python tools/generate_package_docs.py' diff --git a/pyproject.toml b/pyproject.toml index c00f22ad9..35e9c4a12 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -376,6 +376,13 @@ ignore = [ 'docs/docs/tutorials/**' = [ 'E402', # https://docs.astral.sh/ruff/rules/module-import-not-at-top-of-file/ ] +# Verification pages are notebook sources too: allow mid-cell imports, +# printed metric values, and a trailing bare expression (figure render). +'docs/docs/verification/**' = [ + 'E402', # module-import-not-at-top-of-file + 'T201', # print (metric values shown in the rendered notebook) + 'B018', # useless-expression (trailing `fig` renders in the notebook) +] # Intentional terminal rendering: these write raw/ASCII output that # `Console.print` would garble, so `print` is deliberate here. 'src/easydiffraction/display/plotters/ascii.py' = [ From 81e521d54b9296ef686d24d4c4909e563d525d43 Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Fri, 5 Jun 2026 23:27:49 -0700 Subject: [PATCH 29/57] Apply pixi run fix auto-fixes and resolve lint findings --- .../accepted/test-suite-and-validation.md | 96 +++--- docs/dev/adrs/index.md | 90 ++--- docs/dev/issues/open.md | 7 +- docs/dev/plans/test-suite-and-validation.md | 311 +++++++++--------- docs/dev/testing-guide.md | 38 ++- .../verification/cross-engine-bragg-cwl.ipynb | 7 +- .../verification/cross-engine-bragg-cwl.py | 7 +- docs/docs/verification/index.md | 11 +- pyproject.toml | 1 + .../analysis/calculators/support.py | 56 ++-- .../core/test_validation_properties.py | 10 +- 11 files changed, 320 insertions(+), 314 deletions(-) diff --git a/docs/dev/adrs/accepted/test-suite-and-validation.md b/docs/dev/adrs/accepted/test-suite-and-validation.md index 65eb20aef..6d0744057 100644 --- a/docs/dev/adrs/accepted/test-suite-and-validation.md +++ b/docs/dev/adrs/accepted/test-suite-and-validation.md @@ -18,8 +18,8 @@ EasyDiffraction now has five test layers — unit, functional, integration, script, and notebook — plus a tutorial-output regression check. The layers were established by [Test Strategy](../accepted/test-strategy.md), which defines each in a -single line and states that the unit tree mirrors the source tree -"where practical." +single line and states that the unit tree mirrors the source tree "where +practical." That high-level statement is no longer enough. Concrete problems have accumulated: @@ -31,8 +31,8 @@ accumulated: tests are slow (parametrised sampler/plotting/display cases) and some call `download_data()` (mocked, but undocumented). There is no written rule an author can apply to a borderline test. -- **Codecov patch status is always red.** `.codecov.yml` runs a - blocking `patch: target: auto` against a **unit-only** coverage upload +- **Codecov patch status is always red.** `.codecov.yml` runs a blocking + `patch: target: auto` against a **unit-only** coverage upload (`coverage.yml` uploads only `coverage-unit.xml`). Any pull request that touches code exercised mainly by functional, integration, or script tests scores near-zero patch coverage and fails the blocking @@ -62,8 +62,8 @@ accumulated: exercises EasyDiffraction against a large, varied corpus of CIF files to catch parsing/recognition failures before users hit them. - **Documentation drift is not caught on every push.** `docs.yml` - executes all tutorials and then builds and deploys the site; it is slow - and therefore runs on pull requests only. There is no fast, + executes all tutorials and then builds and deploys the site; it is + slow and therefore runs on pull requests only. There is no fast, every-push check that the site builds strictly, links resolve, and prose is clean. This overlaps the unimplemented [Documentation CI and Build Verification](documentation-ci-build.md) @@ -87,13 +87,13 @@ implemented in follow-up pull requests. Replace the one-line definitions with observable, testable rules. A test belongs to the **lowest** layer whose constraints it can satisfy. -| Layer | May use | Must NOT use | Speed | -| --- | --- | --- | --- | -| **unit** | one module under test; in-process logic; `tmp_path` | real calculation engine; network / `download_data()`; filesystem outside `tmp_path`; `sleep`; subprocess | sub-second | -| **functional** | several modules / a workflow; small bundled fixtures | real calculation engine; **network / `download_data()`** | seconds | -| **integration** | real engines, real fits, real downloaded data; the only layer allowed network and real backends | — | slow; xdist | -| **script** | full tutorial `.py` executed subprocess-isolated | (already correct) | slow; xdist | -| **notebook** | generated `.ipynb` executed via `nbmake` | — | slow | +| Layer | May use | Must NOT use | Speed | +| --------------- | ----------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------- | ----------- | +| **unit** | one module under test; in-process logic; `tmp_path` | real calculation engine; network / `download_data()`; filesystem outside `tmp_path`; `sleep`; subprocess | sub-second | +| **functional** | several modules / a workflow; small bundled fixtures | real calculation engine; **network / `download_data()`** | seconds | +| **integration** | real engines, real fits, real downloaded data; the only layer allowed network and real backends | — | slow; xdist | +| **script** | full tutorial `.py` executed subprocess-isolated | (already correct) | slow; xdist | +| **notebook** | generated `.ipynb` executed via `nbmake` | — | slow | Mocking a forbidden dependency (for example a mocked `download_data()`) keeps a test in a lower layer **only when the mock is explicit**; an @@ -113,7 +113,7 @@ Consequences for the current suite: ### 2. Test cost tiers via opt-in escalation markers Cost tiers are **orthogonal** to layers. The default is fast; expensive -tests opt *into* a heavier tier, so only the minority are tagged. +tests opt _into_ a heavier tier, so only the minority are tagged. - **default (unmarked):** fast. Runs on every push, every pull request, and nightly. @@ -125,7 +125,7 @@ tests opt *into* a heavier tier, so only the minority are tagged. Orthogonality holds at the **unit and functional** layers: those default to fast, and an individual test opts into `pr`/`nightly`. The -**integration** layer is the one principled exception — *every* +**integration** layer is the one principled exception — _every_ integration test uses a real engine and/or downloaded data, so the layer **defaults to the `pr` tier**, applied once in `tests/integration/conftest.py` rather than by tagging each of ~150 @@ -141,7 +141,7 @@ pull request + main: -m "not nightly" nightly schedule: (all markers, including -m nightly) ``` -The current `fast` marker (which today selects a *cheap subset* and is +The current `fast` marker (which today selects a _cheap subset_ and is applied to six integration files) is **retired**; its intent is inverted into the scheme above. Markers are registered in `[tool.pytest.ini_options].markers`. @@ -149,12 +149,13 @@ into the scheme above. Markers are registered in ### 3. Mirrored unit structure as a CI gate `tools/test_structure_check.py` remains the canonical enforcer of the -`src/easydiffraction//.py` → `tests/unit/easydiffraction//test_.py` -mirror, including its three match strategies (direct mirror, known -aliases such as `singleton → singletons` and `variable → parameters`, -and parent-level roll-up for `default.py`/`factory.py` category -packages). It is added to CI (the lint/format or test workflow) as a -fast, static gate so structural drift fails before merge. +`src/easydiffraction//.py` → +`tests/unit/easydiffraction//test_.py` mirror, including its +three match strategies (direct mirror, known aliases such as +`singleton → singletons` and `variable → parameters`, and parent-level +roll-up for `default.py`/`factory.py` category packages). It is added to +CI (the lint/format or test workflow) as a fast, static gate so +structural drift fails before merge. The check should be driven by a **single source-of-truth enumeration of the `src/` tree**. `tools/generate_package_docs.py` already walks that @@ -173,8 +174,9 @@ Two distinct bars, because line coverage and case coverage are different guarantees. - **Line/branch coverage.** Raise `fail_under` from 65 to **80 now**, - with a documented ramp toward **90–95** as the suite fills, and gate it - in CI through the codecov project status (§5) rather than only locally. + with a documented ramp toward **90–95** as the suite fills, and gate + it in CI through the codecov project status (§5) rather than only + locally. - **Validators are the input boundary.** All user input is verified at runtime through the project's custom validator framework in `src/easydiffraction/core/validation.py`: an `AttributeSpec` pairs a @@ -185,8 +187,8 @@ guarantees. that they accept the full valid domain and that they reject (or fall back, per their contract) on invalid values — rather than re-checking the same boundaries at every call site. This matches the project - principle of explicit handling at the boundary and no defensive padding - past it. + principle of explicit handling at the boundary and no defensive + padding past it. - **Input-domain coverage.** Adopt **property-based testing with `hypothesis`** for the validator-guarded numeric and crystallographic inputs: cell lengths (> 0), cell angles (valid ranges and lattice @@ -230,8 +232,8 @@ Adopt the recommendation from ### 6. Verification documentation (cross-engine pattern comparison) Add a new top-level **Verification** section to the documentation nav -(between Tutorials and Command-Line), generated like tutorials -(`.py` source → notebook via `pixi run notebook-prepare`, built with +(between Tutorials and Command-Line), generated like tutorials (`.py` +source → notebook via `pixi run notebook-prepare`, built with `execute: false`). - **Calculation-only comparisons (no minimisation).** Feed identical @@ -250,8 +252,8 @@ Add a new top-level **Verification** section to the documentation nav bragg/total, per the support matrix below). The section ships with the framework and the first cross-engine comparison (constant-wavelength powder, cryspy ↔ crysfml); the remaining supported combinations - (time-of-flight powder, single crystal) are added **incrementally** and - tracked in the open-issues list. + (time-of-flight powder, single crystal) are added **incrementally** + and tracked in the open-issues list. - **External software, incrementally.** External tools (FullProf first, then GSAS-II/TOPAS) are compared by loading a **pre-calculated profile from a zipped project** stored in the `diffraction` data repository @@ -267,8 +269,7 @@ Add a new top-level **Verification** section to the documentation nav ### 7. Performance-regression benchmarks Replace the ad-hoc `tools/benchmark_tutorials.py` CSV tool with -**`pytest-benchmark`**, matching the prior art in -`deps-pycrysfml`: +**`pytest-benchmark`**, matching the prior art in `deps-pycrysfml`: - Benchmark **per experiment type** (one benchmark per `beam_mode × radiation_probe × engine`) rather than whole-tutorial @@ -285,9 +286,9 @@ Replace the ad-hoc `tools/benchmark_tutorials.py` CSV tool with A robustness harness exercising EasyDiffraction against many real and synthetic structures. -- **Code vs data split.** Harness *code* lives in `diffraction-lib` +- **Code vs data split.** Harness _code_ lives in `diffraction-lib` (`tests/nightly/`, `@pytest.mark.nightly`) so it versions with the - code it checks. The *corpus*, *results database*, FullProf profiles, + code it checks. The _corpus_, _results database_, FullProf profiles, and benchmark baselines live in the **`diffraction` data repository** (consistent with `download_data()`), fetched at runtime. - **Acceptance-style run.** A scheduled nightly CI job installs @@ -300,10 +301,10 @@ synthetic structures. - `ok` — parsed, all recognised; - `partial` — parsed, some information missing (EasyDiffraction applied defaults), with a comment naming what was not recognised; - - `fail` — could not be parsed, with the error. - The status lets the harness (a) skip re-downloading already-`ok` - files on later nights and (b) flag genuine EasyDiffraction recognition - bugs vs malformed files; problematic files convert to issues. + - `fail` — could not be parsed, with the error. The status lets the + harness (a) skip re-downloading already-`ok` files on later nights + and (b) flag genuine EasyDiffraction recognition bugs vs malformed + files; problematic files convert to issues. - **Results database — CSV.** A git-diffable manifest, **one row per CIF keyed by COD id and ordered by id** (so new files insert in order): `id, parse_status, missing_fields, calc_status_per_engine, comment, last_checked`. @@ -338,7 +339,7 @@ which executes all tutorials and then builds and deploys — slow, and therefore pull-request-only. The detailed catalogue of documentation checks is owned by [Documentation CI and Build Verification](documentation-ci-build.md), -which this ADR coordinates with: that ADR defines *what* the checks are; +which this ADR coordinates with: that ADR defines _what_ the checks are; this ADR's decision is that the cheap, deterministic subset runs as part of the every-push test workflow. Promoting that ADR is part of this work. @@ -381,7 +382,8 @@ for the future rather than solved now. - Relocating functional/unit tests and retiring `fast` touches many existing test files in one pass. - New dependencies (`hypothesis`, `pytest-benchmark`, plus `codespell` - and a link checker for the docs job) add configuration and maintenance. + and a link checker for the docs job) add configuration and + maintenance. - The nightly harness and data-repository round-trip add CI and cross-repository coordination. - Raising `fail_under` to 80 and gating it can block merges until @@ -392,8 +394,8 @@ for the future rather than solved now. - **One combined ADR vs several focused ADRs.** A split (taxonomy / codecov / benchmarks / verification) was considered. Chosen: one combined ADR, because the goals share infrastructure (markers, the - data repository, CI triggers) and read as one quality story; - large sub-areas are phased instead. + data repository, CI triggers) and read as one quality story; large + sub-areas are phased instead. - **Upload combined coverage to codecov.** Rejected for now: slower, flakier (engine-dependent), and needs per-flag setup. Unit-only upload with a non-blocking patch status is simpler and fixes the reported @@ -440,9 +442,10 @@ drafting conversation, per the dependency-approval rule): - `hypothesis` — property-based / input-domain testing (§4). - `pytest-benchmark` — performance-regression benchmarks (§7). -Coordinated with [Documentation CI and Build Verification](documentation-ci-build.md), -which carries the documentation-check tools (`codespell`, a link -checker such as `lychee`, and later `Vale`) used by §9. +Coordinated with +[Documentation CI and Build Verification](documentation-ci-build.md), +which carries the documentation-check tools (`codespell`, a link checker +such as `lychee`, and later `Vale`) used by §9. ## Related ADRs @@ -456,4 +459,5 @@ checker such as `lychee`, and later `Vale`) used by §9. - [Factory Contracts and Metadata](../accepted/factory-contracts.md) — the `CalculatorSupport`/`Compatibility` metadata used by §6. - [Enum-Backed Closed Value Sets](../accepted/enum-backed-closed-values.md) - — any new closed set (engine tags, experiment axes) stays `(str, Enum)`. + — any new closed set (engine tags, experiment axes) stays + `(str, Enum)`. diff --git a/docs/dev/adrs/index.md b/docs/dev/adrs/index.md index 5c815c062..4733588fb 100644 --- a/docs/dev/adrs/index.md +++ b/docs/dev/adrs/index.md @@ -13,49 +13,49 @@ folders. ## ADR Index -| Group | Status | Title | Short description | Link | -| -------------------- | ---------- | --------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------- | -| Analysis and fitting | Accepted | Fit Mode Categories and Fit Execution API | Splits fitting configuration from execution and defines active sibling fit-mode categories. | [`fit-mode-categories.md`](accepted/fit-mode-categories.md) | -| Analysis and fitting | Accepted | Runtime Fit Results | Keeps full fit outputs runtime-only in the current design unless a narrower persistence ADR is accepted. | [`runtime-fit-results.md`](accepted/runtime-fit-results.md) | -| Analysis and fitting | Accepted | Analysis CIF Fit State | Defines the persisted fit-state projection in `analysis/analysis.cif` and `analysis/results.h5`. | [`analysis-cif-fit-state.md`](accepted/analysis-cif-fit-state.md) | -| Analysis and fitting | Accepted | Parameter Correlation Persistence | Persists deterministic and posterior correlation summaries in `_fit_parameter_correlation` | [`parameter-correlation-persistence.md`](accepted/parameter-correlation-persistence.md) | -| Analysis and fitting | Suggestion | Fit Output Files and Data Exports | Narrows remaining archive/export questions after adopting `results.csv` and `results.h5`. | [`fit-output-files-and-data-exports.md`](suggestions/fit-output-files-and-data-exports.md) | -| Analysis and fitting | Accepted | Minimizer Category Consolidation | Collapses the seven Bayesian categories into one owner-level switchable `minimizer` category with HDF5 sidecar. | [`minimizer-category-consolidation.md`](accepted/minimizer-category-consolidation.md) | -| Analysis and fitting | Accepted | Minimizer Input/Output Split | Keeps `analysis.minimizer` input-only and moves scalar fit outputs to paired `analysis.fit_result` classes. | [`minimizer-input-output-split.md`](accepted/minimizer-input-output-split.md) | -| Analysis and fitting | Superseded | Parameter-Level Posterior Projection | Superseded by minimizer-category consolidation; kept as historical context for `parameter.posterior`. | [`parameter-posterior-summary.md`](suggestions/parameter-posterior-summary.md) | -| Analysis and fitting | Accepted | Undo Fit | Builds rollback semantics and CLI behavior on already-persisted pre-fit scalar snapshots. | [`undo-fit.md`](accepted/undo-fit.md) | -| Core model | Accepted | Category Owners and Real Datablocks | Introduces `CategoryOwner` so singleton sections do not pretend to be real CIF datablocks. | [`category-owner-sections.md`](accepted/category-owner-sections.md) | -| Core model | Accepted | Enum-Backed Closed Value Sets | Requires finite option sets to use `(str, Enum)` classes for validation and dispatch. | [`enum-backed-closed-values.md`](accepted/enum-backed-closed-values.md) | -| Core model | Accepted | Guarded Public Properties | Uses property setters as the public writability contract for guarded objects. | [`guarded-public-properties.md`](accepted/guarded-public-properties.md) | -| Core model | Accepted | Two-Level Category Parameter Access | Keeps parameter access to `datablock.category.parameter` or `datablock.collection[id].parameter`. | [`category-parameter-access.md`](accepted/category-parameter-access.md) | -| Documentation | Accepted | Descriptor Property Docstring Template | Makes descriptor metadata the source of truth for public property docstrings and annotations. | [`property-docstring-template.md`](accepted/property-docstring-template.md) | -| Documentation | Accepted | Development Documentation Structure | Defines the `docs/dev` layout for ADRs, issues, plans, package structure, and roadmap. | [`development-docs-structure.md`](accepted/development-docs-structure.md) | -| Documentation | Accepted | Help Method Discoverability | Requires primary public objects and facades to expose consistent `help()` output. | [`help-discoverability.md`](accepted/help-discoverability.md) | -| Documentation | Accepted | Notebook Generation Source of Truth | Treats tutorial `.py` files as editable sources and notebooks as generated artifacts. | [`notebook-generation.md`](accepted/notebook-generation.md) | -| Documentation | Accepted | Plotting & Docs Performance for Interactive Figures | Self-hosts a lazy, shared figure runtime so docs pages load fast and progressively while staying interactive. | [`plotting-docs-performance.md`](accepted/plotting-docs-performance.md) | -| Documentation | Suggestion | Documentation CI and Build Verification | Proposes strict MkDocs builds, API-derived docs, snippet smoke tests, link checks, and prose/spelling checks. | [`documentation-ci-build.md`](suggestions/documentation-ci-build.md) | -| Experiment model | Accepted | Immutable Experiment Type | Makes experiment type axes creation-time state rather than mutable runtime state. | [`immutable-experiment-type.md`](accepted/immutable-experiment-type.md) | -| Experiment model | Accepted | Automatic Line-Segment Background Estimation | Detects line-segment background control points from the measured pattern, peak-insensitive and editable. | [`background-auto-estimate.md`](accepted/background-auto-estimate.md) | -| Factories | Accepted | Factory Contracts and Metadata | Standardizes factory construction, metadata, compatibility, and registration behavior. | [`factory-contracts.md`](accepted/factory-contracts.md) | -| Naming | Accepted | Factory Tag Naming | Defines canonical factory tag style and standard abbreviations. | [`factory-tag-naming.md`](accepted/factory-tag-naming.md) | -| Persistence | Accepted | Free-Flag CIF Encoding | Encodes fit free/fixed state through CIF uncertainty syntax instead of a separate free list. | [`free-flag-cif-encoding.md`](accepted/free-flag-cif-encoding.md) | -| Persistence | Accepted | Loop Category Keys and Identity Naming | Documents loop collection keys and naming rules aligned with CIF category keys. | [`loop-category-key-identity.md`](accepted/loop-category-key-identity.md) | -| Persistence | Accepted | Project Facade and Persistence Layout | Documents the current `Project` facade and saved directory layout. | [`project-facade-and-persistence.md`](accepted/project-facade-and-persistence.md) | -| Persistence | Accepted | IUCr CIF Tag Alignment | Aligns default CIF tags with IUCr dictionaries and adds a clean IUCr-aligned report export. | [`iucr-cif-tag-alignment.md`](accepted/iucr-cif-tag-alignment.md) | -| Persistence | Accepted | Python and CIF Category Correspondence | Compares current Python paths and CIF tags, then records scoped one-to-one mapping for project-level categories. | [`python-cif-category-correspondence.md`](accepted/python-cif-category-correspondence.md) | -| Quality | Accepted | Lint Complexity Thresholds | Treats ruff PLR complexity limits as design guardrails that should not be bypassed. | [`lint-complexity-thresholds.md`](accepted/lint-complexity-thresholds.md) | -| Quality | Accepted | Test Strategy | Defines layered unit, functional, integration, script, and notebook testing. | [`test-strategy.md`](accepted/test-strategy.md) | +| Group | Status | Title | Short description | Link | +| -------------------- | ---------- | --------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------- | +| Analysis and fitting | Accepted | Fit Mode Categories and Fit Execution API | Splits fitting configuration from execution and defines active sibling fit-mode categories. | [`fit-mode-categories.md`](accepted/fit-mode-categories.md) | +| Analysis and fitting | Accepted | Runtime Fit Results | Keeps full fit outputs runtime-only in the current design unless a narrower persistence ADR is accepted. | [`runtime-fit-results.md`](accepted/runtime-fit-results.md) | +| Analysis and fitting | Accepted | Analysis CIF Fit State | Defines the persisted fit-state projection in `analysis/analysis.cif` and `analysis/results.h5`. | [`analysis-cif-fit-state.md`](accepted/analysis-cif-fit-state.md) | +| Analysis and fitting | Accepted | Parameter Correlation Persistence | Persists deterministic and posterior correlation summaries in `_fit_parameter_correlation` | [`parameter-correlation-persistence.md`](accepted/parameter-correlation-persistence.md) | +| Analysis and fitting | Suggestion | Fit Output Files and Data Exports | Narrows remaining archive/export questions after adopting `results.csv` and `results.h5`. | [`fit-output-files-and-data-exports.md`](suggestions/fit-output-files-and-data-exports.md) | +| Analysis and fitting | Accepted | Minimizer Category Consolidation | Collapses the seven Bayesian categories into one owner-level switchable `minimizer` category with HDF5 sidecar. | [`minimizer-category-consolidation.md`](accepted/minimizer-category-consolidation.md) | +| Analysis and fitting | Accepted | Minimizer Input/Output Split | Keeps `analysis.minimizer` input-only and moves scalar fit outputs to paired `analysis.fit_result` classes. | [`minimizer-input-output-split.md`](accepted/minimizer-input-output-split.md) | +| Analysis and fitting | Superseded | Parameter-Level Posterior Projection | Superseded by minimizer-category consolidation; kept as historical context for `parameter.posterior`. | [`parameter-posterior-summary.md`](suggestions/parameter-posterior-summary.md) | +| Analysis and fitting | Accepted | Undo Fit | Builds rollback semantics and CLI behavior on already-persisted pre-fit scalar snapshots. | [`undo-fit.md`](accepted/undo-fit.md) | +| Core model | Accepted | Category Owners and Real Datablocks | Introduces `CategoryOwner` so singleton sections do not pretend to be real CIF datablocks. | [`category-owner-sections.md`](accepted/category-owner-sections.md) | +| Core model | Accepted | Enum-Backed Closed Value Sets | Requires finite option sets to use `(str, Enum)` classes for validation and dispatch. | [`enum-backed-closed-values.md`](accepted/enum-backed-closed-values.md) | +| Core model | Accepted | Guarded Public Properties | Uses property setters as the public writability contract for guarded objects. | [`guarded-public-properties.md`](accepted/guarded-public-properties.md) | +| Core model | Accepted | Two-Level Category Parameter Access | Keeps parameter access to `datablock.category.parameter` or `datablock.collection[id].parameter`. | [`category-parameter-access.md`](accepted/category-parameter-access.md) | +| Documentation | Accepted | Descriptor Property Docstring Template | Makes descriptor metadata the source of truth for public property docstrings and annotations. | [`property-docstring-template.md`](accepted/property-docstring-template.md) | +| Documentation | Accepted | Development Documentation Structure | Defines the `docs/dev` layout for ADRs, issues, plans, package structure, and roadmap. | [`development-docs-structure.md`](accepted/development-docs-structure.md) | +| Documentation | Accepted | Help Method Discoverability | Requires primary public objects and facades to expose consistent `help()` output. | [`help-discoverability.md`](accepted/help-discoverability.md) | +| Documentation | Accepted | Notebook Generation Source of Truth | Treats tutorial `.py` files as editable sources and notebooks as generated artifacts. | [`notebook-generation.md`](accepted/notebook-generation.md) | +| Documentation | Accepted | Plotting & Docs Performance for Interactive Figures | Self-hosts a lazy, shared figure runtime so docs pages load fast and progressively while staying interactive. | [`plotting-docs-performance.md`](accepted/plotting-docs-performance.md) | +| Documentation | Suggestion | Documentation CI and Build Verification | Proposes strict MkDocs builds, API-derived docs, snippet smoke tests, link checks, and prose/spelling checks. | [`documentation-ci-build.md`](suggestions/documentation-ci-build.md) | +| Experiment model | Accepted | Immutable Experiment Type | Makes experiment type axes creation-time state rather than mutable runtime state. | [`immutable-experiment-type.md`](accepted/immutable-experiment-type.md) | +| Experiment model | Accepted | Automatic Line-Segment Background Estimation | Detects line-segment background control points from the measured pattern, peak-insensitive and editable. | [`background-auto-estimate.md`](accepted/background-auto-estimate.md) | +| Factories | Accepted | Factory Contracts and Metadata | Standardizes factory construction, metadata, compatibility, and registration behavior. | [`factory-contracts.md`](accepted/factory-contracts.md) | +| Naming | Accepted | Factory Tag Naming | Defines canonical factory tag style and standard abbreviations. | [`factory-tag-naming.md`](accepted/factory-tag-naming.md) | +| Persistence | Accepted | Free-Flag CIF Encoding | Encodes fit free/fixed state through CIF uncertainty syntax instead of a separate free list. | [`free-flag-cif-encoding.md`](accepted/free-flag-cif-encoding.md) | +| Persistence | Accepted | Loop Category Keys and Identity Naming | Documents loop collection keys and naming rules aligned with CIF category keys. | [`loop-category-key-identity.md`](accepted/loop-category-key-identity.md) | +| Persistence | Accepted | Project Facade and Persistence Layout | Documents the current `Project` facade and saved directory layout. | [`project-facade-and-persistence.md`](accepted/project-facade-and-persistence.md) | +| Persistence | Accepted | IUCr CIF Tag Alignment | Aligns default CIF tags with IUCr dictionaries and adds a clean IUCr-aligned report export. | [`iucr-cif-tag-alignment.md`](accepted/iucr-cif-tag-alignment.md) | +| Persistence | Accepted | Python and CIF Category Correspondence | Compares current Python paths and CIF tags, then records scoped one-to-one mapping for project-level categories. | [`python-cif-category-correspondence.md`](accepted/python-cif-category-correspondence.md) | +| Quality | Accepted | Lint Complexity Thresholds | Treats ruff PLR complexity limits as design guardrails that should not be bypassed. | [`lint-complexity-thresholds.md`](accepted/lint-complexity-thresholds.md) | +| Quality | Accepted | Test Strategy | Defines layered unit, functional, integration, script, and notebook testing. | [`test-strategy.md`](accepted/test-strategy.md) | | Quality | Accepted | Test Suite and Validation Strategy | Strict test layers, cost tiers, coverage/codecov policy, cross-engine verification docs, and a nightly validation harness. | [`test-suite-and-validation.md`](accepted/test-suite-and-validation.md) | -| Structure model | Accepted | Type-Neutral ADP Parameters | Keeps ADP parameter object identities stable across B/U and iso/ani switches. | [`type-neutral-adp-parameters.md`](accepted/type-neutral-adp-parameters.md) | -| Structure model | Accepted | Automatic Wyckoff Position Detection | Detects Wyckoff letter, multiplicity, and site symmetry from space group and coordinates; calculators consume them. | [`wyckoff-letter-detection.md`](accepted/wyckoff-letter-detection.md) | -| Structure model | Accepted | Complete Space-Group Reference Database | One-time build of a complete space_groups.json.gz (all 230 groups) from cctbx, verified against multiple sources. | [`space-group-database.md`](accepted/space-group-database.md) | -| User-facing API | Accepted | Crystal Structure 3D Visualization | Adds a renderer-neutral scene model drawn by ASCII and interactive Three.js engines for viewing crystal structures. | [`crysview-structure-visualization.md`](accepted/crysview-structure-visualization.md) | -| User-facing API | Accepted | Display UX Facade | Defines `project.display` and `project.rendering` responsibilities and display method names. | [`display-ux.md`](accepted/display-ux.md) | -| User-facing API | Accepted | Fit Results Display Naming | Short, IUCr/GUM-aligned column headers (`s.u.`, `value`, `95% CI`) with a footnote glossary on every fit table. | [`fit-results-display-naming.md`](accepted/fit-results-display-naming.md) | -| User-facing API | Accepted | Project Summary Rendering | Defines project report configuration plus terminal, HTML, TeX, PDF, and clean report-CIF metadata policy. | [`project-summary-rendering.md`](accepted/project-summary-rendering.md) | -| User-facing API | Accepted | Selector Families | Distinguishes backend selectors, switchable-category selectors, and active-sibling selectors. | [`selector-families.md`](accepted/selector-families.md) | -| User-facing API | Accepted | String Paths and Live Descriptors | Separates persisted field selectors from references to live model parameters. | [`string-paths-and-live-descriptors.md`](accepted/string-paths-and-live-descriptors.md) | -| User-facing API | Accepted | Switchable Category API | Places multi-type category selectors on the owner and omits public selectors for fixed or single-type categories. | [`switchable-category-api.md`](accepted/switchable-category-api.md) | -| User-facing API | Accepted | Switchable Category Owned Selectors | Moves the writable `type` selector and `show_supported()` onto the category itself; collapses the CIF duplication. | [`switchable-category-owned-selectors.md`](accepted/switchable-category-owned-selectors.md) | -| User-facing API | Accepted | Unified Pattern View | `pattern()` always renders available data, drops `include`, and unifies single- and three-panel figure sizing. | [`pattern-display-unification.md`](accepted/pattern-display-unification.md) | -| User-facing API | Accepted | Value-Selector Discovery | Gives enumerated value fields a per-descriptor `show_supported()`, beside the three category-level selector families. | [`value-selector-discovery.md`](accepted/value-selector-discovery.md) | +| Structure model | Accepted | Type-Neutral ADP Parameters | Keeps ADP parameter object identities stable across B/U and iso/ani switches. | [`type-neutral-adp-parameters.md`](accepted/type-neutral-adp-parameters.md) | +| Structure model | Accepted | Automatic Wyckoff Position Detection | Detects Wyckoff letter, multiplicity, and site symmetry from space group and coordinates; calculators consume them. | [`wyckoff-letter-detection.md`](accepted/wyckoff-letter-detection.md) | +| Structure model | Accepted | Complete Space-Group Reference Database | One-time build of a complete space_groups.json.gz (all 230 groups) from cctbx, verified against multiple sources. | [`space-group-database.md`](accepted/space-group-database.md) | +| User-facing API | Accepted | Crystal Structure 3D Visualization | Adds a renderer-neutral scene model drawn by ASCII and interactive Three.js engines for viewing crystal structures. | [`crysview-structure-visualization.md`](accepted/crysview-structure-visualization.md) | +| User-facing API | Accepted | Display UX Facade | Defines `project.display` and `project.rendering` responsibilities and display method names. | [`display-ux.md`](accepted/display-ux.md) | +| User-facing API | Accepted | Fit Results Display Naming | Short, IUCr/GUM-aligned column headers (`s.u.`, `value`, `95% CI`) with a footnote glossary on every fit table. | [`fit-results-display-naming.md`](accepted/fit-results-display-naming.md) | +| User-facing API | Accepted | Project Summary Rendering | Defines project report configuration plus terminal, HTML, TeX, PDF, and clean report-CIF metadata policy. | [`project-summary-rendering.md`](accepted/project-summary-rendering.md) | +| User-facing API | Accepted | Selector Families | Distinguishes backend selectors, switchable-category selectors, and active-sibling selectors. | [`selector-families.md`](accepted/selector-families.md) | +| User-facing API | Accepted | String Paths and Live Descriptors | Separates persisted field selectors from references to live model parameters. | [`string-paths-and-live-descriptors.md`](accepted/string-paths-and-live-descriptors.md) | +| User-facing API | Accepted | Switchable Category API | Places multi-type category selectors on the owner and omits public selectors for fixed or single-type categories. | [`switchable-category-api.md`](accepted/switchable-category-api.md) | +| User-facing API | Accepted | Switchable Category Owned Selectors | Moves the writable `type` selector and `show_supported()` onto the category itself; collapses the CIF duplication. | [`switchable-category-owned-selectors.md`](accepted/switchable-category-owned-selectors.md) | +| User-facing API | Accepted | Unified Pattern View | `pattern()` always renders available data, drops `include`, and unifies single- and three-panel figure sizing. | [`pattern-display-unification.md`](accepted/pattern-display-unification.md) | +| User-facing API | Accepted | Value-Selector Discovery | Gives enumerated value fields a per-descriptor `show_supported()`, beside the three category-level selector families. | [`value-selector-discovery.md`](accepted/value-selector-discovery.md) | diff --git a/docs/dev/issues/open.md b/docs/dev/issues/open.md index 25bc25aaf..c63987cc9 100644 --- a/docs/dev/issues/open.md +++ b/docs/dev/issues/open.md @@ -1918,7 +1918,8 @@ nightly job that installs easydiffraction from PyPI (acceptance-style). pre-calculated profiles as zipped projects in the data repository so the Verification pages can overlay them against easydiffraction. -**Depends on:** the `diffraction` data repository; cross-repo coordination. +**Depends on:** the `diffraction` data repository; cross-repo +coordination. --- @@ -1932,8 +1933,8 @@ external URLs. Add a `lychee` link checker (with an allowlist for rate-limited/unstable domains), coordinated with the [Documentation CI and Build Verification](../adrs/suggestions/documentation-ci-build.md) ADR. Run it nightly or on pull requests to avoid flakiness from external -sites. Also covers link-checking of URLs that appear only inside executed -notebook output cells (a feature that does not exist yet). +sites. Also covers link-checking of URLs that appear only inside +executed notebook output cells (a feature that does not exist yet). **Depends on:** nothing. diff --git a/docs/dev/plans/test-suite-and-validation.md b/docs/dev/plans/test-suite-and-validation.md index e71908f77..67ab6a50c 100644 --- a/docs/dev/plans/test-suite-and-validation.md +++ b/docs/dev/plans/test-suite-and-validation.md @@ -4,15 +4,14 @@ This plan follows [`AGENTS.md`](../../../AGENTS.md) with one **declared exception** to the two-phase workflow. [`AGENTS.md`](../../../AGENTS.md) §Workflow keeps test creation in Phase 2 ("Phase 1 — Code and docs updates only ... Do not create or run tests unless the user explicitly -asks"). Because this ADR's subject *is* the test suite, Phase 1 here +asks"). Because this ADR's subject _is_ the test suite, Phase 1 here necessarily includes test relocation, shared fixtures, new unit/property tests, and benchmark tests as implementation work — deferring them to verification would leave Phase 1 empty of its actual deliverable. The implementer running `/draft-impl-1` will therefore edit and add files under `tests/**` during Phase 1. Phase 2 remains the standard verification gate (`pixi run fix`/`check`/`unit-tests`/ -`integration-tests`/`script-tests`) and does not author new test -suites. +`integration-tests`/`script-tests`) and does not author new test suites. A scope decision is also recorded below (§Scope): the cross-repository work is documented for the future rather than implemented in this @@ -53,6 +52,7 @@ matrix + cross-engine (cryspy ↔ crysfml) calculation-only comparison pages, §7 `pytest-benchmark` suite, §9 fast docs build gate. **Documented for the future (cross-repository, NOT implemented here):** + - §8 nightly COD corpus harness, its results database, and the pip-install acceptance CI job that writes results back to the `diffraction` data repository. @@ -71,18 +71,18 @@ These are captured in step P1.14 (a future-work record in **no real calculation engine and no network/`download_data()`**; those move to integration. A test goes to the lowest layer whose constraints it can satisfy. -- **Markers (§2):** opt-in escalation. Default (unmarked) = fast. - Add `@pytest.mark.pr` (PR + `develop`/`master`) and - `@pytest.mark.nightly` (scheduled only). **Retire the current `fast` - marker** (6 integration files). CI selection: - `not pr and not nightly` (feature push) / `not nightly` (PR + main) / - all (nightly schedule). Because markers are **orthogonal to layers**, - the selected expression is applied to **every** pytest invocation under - the policy — unit, functional, and integration, in both the source and - package CI jobs — not to integration alone as today. The **integration - layer defaults to the `pr` tier** (auto-marked once in - `tests/integration/conftest.py`) because every integration test uses a - real engine; unit/functional default to fast and escalate individually. +- **Markers (§2):** opt-in escalation. Default (unmarked) = fast. Add + `@pytest.mark.pr` (PR + `develop`/`master`) and `@pytest.mark.nightly` + (scheduled only). **Retire the current `fast` marker** (6 integration + files). CI selection: `not pr and not nightly` (feature push) / + `not nightly` (PR + main) / all (nightly schedule). Because markers + are **orthogonal to layers**, the selected expression is applied to + **every** pytest invocation under the policy — unit, functional, and + integration, in both the source and package CI jobs — not to + integration alone as today. The **integration layer defaults to the + `pr` tier** (auto-marked once in `tests/integration/conftest.py`) + because every integration test uses a real engine; unit/functional + default to fast and escalate individually. - **Structure gate (§3):** unify on a single `src/` tree walk shared by `tools/generate_package_docs.py` and `tools/test_structure_check.py`; run the check in CI as a gate. @@ -116,8 +116,8 @@ These are captured in step P1.14 (a future-work record in exist in `src/easydiffraction/core/metadata.py` (lines 19, 40, 88) and are populated per instrument category (e.g. `datablocks/experiment/categories/instrument/cwl.py` declares - `compatibility` and `calculator_support`). P1.11 therefore *applies - and extends* this model — notably adding `radiation_probe` to + `compatibility` and `calculator_support`). P1.11 therefore _applies + and extends_ this model — notably adding `radiation_probe` to `Compatibility` if missing, and exposing a query to enumerate comparable engine × condition combinations — rather than introducing new classes. **Stop and ask** about a dedicated ADR only if extending @@ -176,158 +176,157 @@ be staged with explicit paths and committed locally before moving to the next step or the Phase 1 review gate**, per [`AGENTS.md`](../../../AGENTS.md) §Commits. Keep commits atomic, single-purpose, and aligned to the step. Do not stage unrelated dirty -files or generated artifacts. Do not run Phase 2 commands during Phase 1. +files or generated artifacts. Do not run Phase 2 commands during +Phase 1. ## Implementation steps (Phase 1) -- [x] **P1.1 — Codecov status policy (§5)** - Edit `.codecov.yml`: add `informational: true` to `patch.default`; set - `project.default` to `target: 80%`, `informational: false`. Leave the - unit-only upload untouched. - Files: `.codecov.yml`. - Commit: `Make codecov patch informational and gate project at 80%` - -- [x] **P1.2 — Cost-tier markers and test retagging (§2)** - Register `pr` and `nightly` markers in - `[tool.pytest.ini_options].markers`; remove the `fast` marker - definition. Remove `@pytest.mark.fast` from the 6 - `tests/integration/fitting/*.py` files (retag the genuinely heavy ones - with `pr` where appropriate). - Files: `pyproject.toml`, `tests/integration/fitting/*.py`. - Commit: `Replace fast marker with pr and nightly test tiers` - -- [x] **P1.3 — CI marker selection across all layers and nightly job (§2)** - Update `.github/workflows/test.yml` mark logic to - `-m "not pr and not nightly"` (feature push) and `-m "not nightly"` - (PR + `develop`/`master`), and apply the selected expression to - **every** pytest invocation in both the source-test and package-test - jobs — unit, functional, and integration (today `-m` reaches only the - integration runs at lines 132 and 311; unit/functional run unfiltered). - Thread a marker passthrough into the `unit-tests` and `functional-tests` - pixi tasks (the `integration-tests` task already accepts an appended - expression). Add a `schedule:` trigger and a nightly job running - `-m nightly`. - Files: `.github/workflows/test.yml`, `pixi.toml`. - Commit: `Select test tiers per trigger across all test layers` +- [x] **P1.1 — Codecov status policy (§5)** Edit `.codecov.yml`: add + `informational: true` to `patch.default`; set `project.default` to + `target: 80%`, `informational: false`. Leave the unit-only upload + untouched. Files: `.codecov.yml`. Commit: + `Make codecov patch informational and gate project at 80%` + +- [x] **P1.2 — Cost-tier markers and test retagging (§2)** Register `pr` + and `nightly` markers in `[tool.pytest.ini_options].markers`; + remove the `fast` marker definition. Remove `@pytest.mark.fast` + from the 6 `tests/integration/fitting/*.py` files (retag the + genuinely heavy ones with `pr` where appropriate). Files: + `pyproject.toml`, `tests/integration/fitting/*.py`. Commit: + `Replace fast marker with pr and nightly test tiers` + +- [x] **P1.3 — CI marker selection across all layers and nightly job + (§2)** Update `.github/workflows/test.yml` mark logic to + `-m "not pr and not nightly"` (feature push) and + `-m "not nightly"` (PR + `develop`/`master`), and apply the + selected expression to **every** pytest invocation in both the + source-test and package-test jobs — unit, functional, and + integration (today `-m` reaches only the integration runs at lines + 132 and 311; unit/functional run unfiltered). Thread a marker + passthrough into the `unit-tests` and `functional-tests` pixi + tasks (the `integration-tests` task already accepts an appended + expression). Add a `schedule:` trigger and a nightly job running + `-m nightly`. Files: `.github/workflows/test.yml`, `pixi.toml`. + Commit: `Select test tiers per trigger across all test layers` - [x] **P1.4 — Unify src-tree walk and gate structure check (§3)** - Extract the `src/` enumeration so `tools/test_structure_check.py` and - `tools/generate_package_docs.py` share one walker; add the check to CI - (lint/format or test workflow) as a blocking gate. - Files: `tools/test_structure_check.py`, - `tools/generate_package_docs.py`, `.github/workflows/*.yml`, - `pixi.toml`. - Commit: `Gate unit-test structure check on shared src tree walk` - -- [x] **P1.5 — Strict layer-criteria testing guide (§1)** - Write the may/must-not criteria and the "where does this test go?" - decision list (location per Open question 5), and tighten the layer - wording referenced by the amended `test-strategy.md`. - Files: new `docs/dev/` testing guide (or `test-strategy.md` update). - Commit: `Document strict test layer placement criteria` - -- [x] **P1.6 — Test relocation pass (§1)** - Move functional tests that call real `download_data()` into - integration; relocate or correctly mark slow/engine/network-touching - unit tests (the 16 `download_data()` unit call sites must be explicit - mocks or move out). Keep `test-structure-check` green. - Files: `tests/functional/**`, `tests/integration/**`, - `tests/unit/**`. - Commit: `Relocate network and engine tests to correct layers` - -- [x] **P1.7 — Shared fixtures, hypothesis profile, tolerance convention (§4)** - Add `hypothesis` (dev dep) and a deterministic profile - (`derandomize`, fixed seed, no committed `.hypothesis` DB). Add a root - `tests/conftest.py` with seeded-RNG and one documented - `rtol`/`atol` pair (intra-engine) and one cross-engine pair. - Files: `pyproject.toml`, `pixi.toml`, `tests/conftest.py`, testing - guide. - Commit: `Add hypothesis deterministic profile and shared test fixtures` + Extract the `src/` enumeration so `tools/test_structure_check.py` + and `tools/generate_package_docs.py` share one walker; add the + check to CI (lint/format or test workflow) as a blocking gate. + Files: `tools/test_structure_check.py`, + `tools/generate_package_docs.py`, `.github/workflows/*.yml`, + `pixi.toml`. Commit: + `Gate unit-test structure check on shared src tree walk` + +- [x] **P1.5 — Strict layer-criteria testing guide (§1)** Write the + may/must-not criteria and the "where does this test go?" decision + list (location per Open question 5), and tighten the layer wording + referenced by the amended `test-strategy.md`. Files: new + `docs/dev/` testing guide (or `test-strategy.md` update). Commit: + `Document strict test layer placement criteria` + +- [x] **P1.6 — Test relocation pass (§1)** Move functional tests that + call real `download_data()` into integration; relocate or + correctly mark slow/engine/network-touching unit tests (the 16 + `download_data()` unit call sites must be explicit mocks or move + out). Keep `test-structure-check` green. Files: + `tests/functional/**`, `tests/integration/**`, `tests/unit/**`. + Commit: `Relocate network and engine tests to correct layers` + +- [x] **P1.7 — Shared fixtures, hypothesis profile, tolerance convention + (§4)** Add `hypothesis` (dev dep) and a deterministic profile + (`derandomize`, fixed seed, no committed `.hypothesis` DB). Add a + root `tests/conftest.py` with seeded-RNG and one documented + `rtol`/`atol` pair (intra-engine) and one cross-engine pair. + Files: `pyproject.toml`, `pixi.toml`, `tests/conftest.py`, testing + guide. Commit: + `Add hypothesis deterministic profile and shared test fixtures` - [x] **P1.8 — Input-domain property tests on validators (§4)** - Property-based + explicit boundary-table tests against - `core/validation.py` (`TypeValidator`, content `ValidatorBase` - subclasses) through parameter (`core/variable.py`) and category - (`core/category.py`): valid-domain acceptance and invalid/ wrong-type - rejection or fallback per contract. - Files: `tests/unit/easydiffraction/core/**` (validator/variable/ - category tests). - Commit: `Add property-based input-domain tests for validators` - -- [x] **P1.9 — Raise coverage gate to 80% (§4)** - Set `[tool.coverage.report] fail_under = 80`. (Resolve Open question 2 - in Phase 2 if unit coverage is below 80 after P1.8.) - Files: `pyproject.toml`. - Commit: `Raise coverage fail_under to 80 percent` - -- [x] **P1.10 — Fast docs build gate (§9)** - Add `docs-build-strict` (`mkdocs build --strict`, tutorials not - executed), `link-check` (`lychee`), and `spell-check` (`codespell`) - pixi tasks with config and ignore lists; add a fast every-push job to - `test.yml`. Add `codespell` dev dep; wire `lychee` (Open question 4). - Files: `pixi.toml`, `pyproject.toml`, `.github/workflows/test.yml`, - `.codespellrc`, `lychee` config. - Commit: `Add strict docs build, link, and spell checks on every push` - -- [x] **P1.11 — Apply/extend calculator support metadata (§6 prerequisite)** - Build on the existing `Compatibility`/`CalculatorSupport` model in - `core/metadata.py` (already declared per instrument category): add - `radiation_probe` to `Compatibility` if missing, and add a small query - helper to enumerate comparable engine × experiment-condition - combinations for the verification pages. Prefer this declared metadata - over the ad-hoc per-calculator `if beam_mode == …` checks. **Stop and - ask** only if extending the metadata shape proves structural (Open - question 1). - Files: `src/easydiffraction/core/metadata.py`, - `src/easydiffraction/datablocks/experiment/categories/instrument/**`, - `src/easydiffraction/analysis/calculators/**`. - Commit: `Extend calculator support metadata with radiation probe` + Property-based + explicit boundary-table tests against + `core/validation.py` (`TypeValidator`, content `ValidatorBase` + subclasses) through parameter (`core/variable.py`) and category + (`core/category.py`): valid-domain acceptance and invalid/ + wrong-type rejection or fallback per contract. Files: + `tests/unit/easydiffraction/core/**` (validator/variable/ category + tests). Commit: + `Add property-based input-domain tests for validators` + +- [x] **P1.9 — Raise coverage gate to 80% (§4)** Set + `[tool.coverage.report] fail_under = 80`. (Resolve Open question 2 + in Phase 2 if unit coverage is below 80 after P1.8.) Files: + `pyproject.toml`. Commit: + `Raise coverage fail_under to 80 percent` + +- [x] **P1.10 — Fast docs build gate (§9)** Add `docs-build-strict` + (`mkdocs build --strict`, tutorials not executed), `link-check` + (`lychee`), and `spell-check` (`codespell`) pixi tasks with config + and ignore lists; add a fast every-push job to `test.yml`. Add + `codespell` dev dep; wire `lychee` (Open question 4). Files: + `pixi.toml`, `pyproject.toml`, `.github/workflows/test.yml`, + `.codespellrc`, `lychee` config. Commit: + `Add strict docs build, link, and spell checks on every push` + +- [x] **P1.11 — Apply/extend calculator support metadata (§6 + prerequisite)** Build on the existing + `Compatibility`/`CalculatorSupport` model in `core/metadata.py` + (already declared per instrument category): add `radiation_probe` + to `Compatibility` if missing, and add a small query helper to + enumerate comparable engine × experiment-condition combinations + for the verification pages. Prefer this declared metadata over the + ad-hoc per-calculator `if beam_mode == …` checks. **Stop and ask** + only if extending the metadata shape proves structural (Open + question 1). Files: `src/easydiffraction/core/metadata.py`, + `src/easydiffraction/datablocks/experiment/categories/instrument/**`, + `src/easydiffraction/analysis/calculators/**`. Commit: + `Extend calculator support metadata with radiation probe` - [x] **P1.12 — Cross-engine verification pages + script wiring (§6)** - Add the `Verification` nav node (between Tutorials and Command-Line) - and a calculation-only `.py` comparison page (cryspy ↔ crysfml) for the - **first** supported combination (constant-wavelength powder), with - closeness metrics, overlay plots, and metric-tolerance assertions. The - remaining supported combinations (time-of-flight powder, single - crystal) are added incrementally (issue 115). **Wire the new - `docs/docs/verification/` directory into the script-test runner** — - `tools/test_scripts.py` discovers only `docs/docs/tutorials/*.py` - today (lines 24-27) — and into the notebook pipeline (`notebook-prepare` - / `notebook-convert` / `notebook-tests`, which target the tutorials - dir), so the pages are generated and exercised as regressions. Run - `pixi run notebook-prepare`. - Files: `docs/mkdocs.yml`, `docs/docs/verification/*.py` (+ generated - `*.ipynb`), `tools/test_scripts.py`, `pixi.toml`. - Commit: `Add cross-engine verification comparison pages and script wiring` - -- [x] **P1.13 — Per-experiment performance benchmarks (§7)** - Add `pytest-benchmark` (dev dep), `nightly`-marked benchmarks keyed by - `beam_mode × radiation_probe × engine`, and a `benchmarks` pixi task - emitting JSON as a CI artifact (data-repo history deferred). - Files: `pyproject.toml`, `pixi.toml`, `tests/benchmarks/**`. - Commit: `Add per-experiment performance benchmarks (nightly)` + Add the `Verification` nav node (between Tutorials and + Command-Line) and a calculation-only `.py` comparison page (cryspy + ↔ crysfml) for the **first** supported combination + (constant-wavelength powder), with closeness metrics, overlay + plots, and metric-tolerance assertions. The remaining supported + combinations (time-of-flight powder, single crystal) are added + incrementally (issue 115). **Wire the new + `docs/docs/verification/` directory into the script-test runner** + — `tools/test_scripts.py` discovers only + `docs/docs/tutorials/*.py` today (lines 24-27) — and into the + notebook pipeline (`notebook-prepare` / `notebook-convert` / + `notebook-tests`, which target the tutorials dir), so the pages + are generated and exercised as regressions. Run + `pixi run notebook-prepare`. Files: `docs/mkdocs.yml`, + `docs/docs/verification/*.py` (+ generated `*.ipynb`), + `tools/test_scripts.py`, `pixi.toml`. Commit: + `Add cross-engine verification comparison pages and script wiring` + +- [x] **P1.13 — Per-experiment performance benchmarks (§7)** Add + `pytest-benchmark` (dev dep), `nightly`-marked benchmarks keyed by + `beam_mode × radiation_probe × engine`, and a `benchmarks` pixi + task emitting JSON as a CI artifact (data-repo history deferred). + Files: `pyproject.toml`, `pixi.toml`, `tests/benchmarks/**`. + Commit: `Add per-experiment performance benchmarks (nightly)` - [x] **P1.14 — Record cross-repository future work (§8 + deferred)** - Add prioritised entries to `docs/dev/issues/open.md` for the nightly - COD harness + results DB + pip-install acceptance job, generative - fuzzing, data-repo benchmark history, and external-software - comparison data. Confirm the ADR Deferred Work covers them. - Files: `docs/dev/issues/open.md`. - Commit: `Record cross-repo nightly harness and benchmarks as future work` + Add prioritised entries to `docs/dev/issues/open.md` for the + nightly COD harness + results DB + pip-install acceptance job, + generative fuzzing, data-repo benchmark history, and + external-software comparison data. Confirm the ADR Deferred Work + covers them. Files: `docs/dev/issues/open.md`. Commit: + `Record cross-repo nightly harness and benchmarks as future work` - [x] **P1.15 — Promote ADR to accepted (§Change Discipline)** - `git mv docs/dev/adrs/suggestions/test-suite-and-validation.md - docs/dev/adrs/accepted/`; set its `## Status` to `Accepted.`; flip the - `docs/dev/adrs/index.md` row to `Accepted` with the `accepted/...` - link; fix any links that pointed at the `suggestions/` path - (`git grep -n`). - Files: ADR file (moved), `docs/dev/adrs/index.md`. - Commit: `Promote test-suite-and-validation ADR to accepted` - -- [x] **P1.16 — Phase 1 review gate (no code)** - Confirm every box above is `[x]`. Mark this step and commit the - checklist update alone. - Commit: `Reach Phase 1 review gate` + `git mv docs/dev/adrs/suggestions/test-suite-and-validation.md docs/dev/adrs/accepted/`; + set its `## Status` to `Accepted.`; flip the + `docs/dev/adrs/index.md` row to `Accepted` with the `accepted/...` + link; fix any links that pointed at the `suggestions/` path + (`git grep -n`). Files: ADR file (moved), + `docs/dev/adrs/index.md`. Commit: + `Promote test-suite-and-validation ADR to accepted` + +- [x] **P1.16 — Phase 1 review gate (no code)** Confirm every box above + is `[x]`. Mark this step and commit the checklist update alone. + Commit: `Reach Phase 1 review gate` ## Phase 2 verification diff --git a/docs/dev/testing-guide.md b/docs/dev/testing-guide.md index ee7ab9503..b3f3bf7b4 100644 --- a/docs/dev/testing-guide.md +++ b/docs/dev/testing-guide.md @@ -9,13 +9,13 @@ rationale is recorded in the ADRs A test belongs to the **lowest** layer whose constraints it can satisfy. -| Layer | May use | Must NOT use | Speed | -| --------------- | ------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------- | ----------- | -| **unit** | one module under test; in-process logic; `tmp_path` | real calculation engine; network / `download_data()`; filesystem outside `tmp_path`; `sleep`; subprocess | sub-second | -| **functional** | several modules / a workflow; small bundled fixtures | real calculation engine; network / `download_data()` | seconds | -| **integration** | real engines, real fits, real downloaded data (the only layer allowed network and real backends) | — | slow | -| **script** | a full tutorial `.py` executed subprocess-isolated | — | slow | -| **notebook** | a generated `.ipynb` executed via `nbmake` | — | slow | +| Layer | May use | Must NOT use | Speed | +| --------------- | ------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------- | ---------- | +| **unit** | one module under test; in-process logic; `tmp_path` | real calculation engine; network / `download_data()`; filesystem outside `tmp_path`; `sleep`; subprocess | sub-second | +| **functional** | several modules / a workflow; small bundled fixtures | real calculation engine; network / `download_data()` | seconds | +| **integration** | real engines, real fits, real downloaded data (the only layer allowed network and real backends) | — | slow | +| **script** | a full tutorial `.py` executed subprocess-isolated | — | slow | +| **notebook** | a generated `.ipynb` executed via `nbmake` | — | slow | Mocking a forbidden dependency (for example a mocked `download_data()`) keeps a test in a lower layer **only when the mock is explicit**; an @@ -38,13 +38,14 @@ test file for a new module with `tools/gen_tests_scaffold.py`. ## Cost tiers (orthogonal to layers) -Tiers select *when* a test runs in CI; they are independent of the layer. +Tiers select _when_ a test runs in CI; they are independent of the +layer. -| Tier | Marker | Runs on | -| ----------- | ----------------------- | ------------------------------------------------ | -| **fast** | (none — the default) | every push, every pull request, and nightly | -| **pr** | `@pytest.mark.pr` | pull requests and `develop`/`master` | -| **nightly** | `@pytest.mark.nightly` | the scheduled nightly job only (`nightly.yml`) | +| Tier | Marker | Runs on | +| ----------- | ---------------------- | ---------------------------------------------- | +| **fast** | (none — the default) | every push, every pull request, and nightly | +| **pr** | `@pytest.mark.pr` | pull requests and `develop`/`master` | +| **nightly** | `@pytest.mark.nightly` | the scheduled nightly job only (`nightly.yml`) | Integration tests are `pr`-tier by default — they are auto-marked in `tests/integration/conftest.py` because they use real engines. Escalate @@ -65,8 +66,9 @@ numerics, and one (looser) pair for cross-engine comparison. ## Input-domain coverage User input is validated at runtime through `core/validation.py` -(`AttributeSpec` pairs a `TypeValidator` with a content `ValidatorBase`). -Aim input-domain tests at the validators directly — both that they accept -the full valid domain and that they reject (or fall back on) invalid -values. Use `hypothesis` (deterministic profile) for generative coverage -and explicit parametrised tables for the known-critical boundaries. +(`AttributeSpec` pairs a `TypeValidator` with a content +`ValidatorBase`). Aim input-domain tests at the validators directly — +both that they accept the full valid domain and that they reject (or +fall back on) invalid values. Use `hypothesis` (deterministic profile) +for generative coverage and explicit parametrised tables for the +known-critical boundaries. diff --git a/docs/docs/verification/cross-engine-bragg-cwl.ipynb b/docs/docs/verification/cross-engine-bragg-cwl.ipynb index a60c7a864..621e18735 100644 --- a/docs/docs/verification/cross-engine-bragg-cwl.ipynb +++ b/docs/docs/verification/cross-engine-bragg-cwl.ipynb @@ -137,7 +137,8 @@ "b = y_calc_by_engine['crysfml']\n", "\n", "assert a.shape == b.shape, 'engines returned patterns of different length'\n", - "assert np.all(np.isfinite(a)) and np.all(np.isfinite(b))\n", + "assert np.all(np.isfinite(a)), 'cryspy pattern has non-finite values'\n", + "assert np.all(np.isfinite(b)), 'crysfml pattern has non-finite values'\n", "\n", "rms = float(np.sqrt(np.mean((a - b) ** 2)))\n", "norm = float(np.sqrt(np.mean(a**2)))\n", @@ -174,9 +175,7 @@ "x = np.arange(a.size)\n", "fig = go.Figure()\n", "fig.add_scatter(x=x, y=a, mode='lines', name='cryspy', line={'color': 'royalblue'})\n", - "fig.add_scatter(\n", - " x=x, y=b, mode='lines', name='crysfml', line={'color': 'crimson', 'dash': 'dot'}\n", - ")\n", + "fig.add_scatter(x=x, y=b, mode='lines', name='crysfml', line={'color': 'crimson', 'dash': 'dot'})\n", "fig.update_layout(\n", " title='Calculated patterns: cryspy vs crysfml',\n", " xaxis_title='point index',\n", diff --git a/docs/docs/verification/cross-engine-bragg-cwl.py b/docs/docs/verification/cross-engine-bragg-cwl.py index a94f7b40a..bcf606048 100644 --- a/docs/docs/verification/cross-engine-bragg-cwl.py +++ b/docs/docs/verification/cross-engine-bragg-cwl.py @@ -57,7 +57,8 @@ b = y_calc_by_engine['crysfml'] assert a.shape == b.shape, 'engines returned patterns of different length' -assert np.all(np.isfinite(a)) and np.all(np.isfinite(b)) +assert np.all(np.isfinite(a)), 'cryspy pattern has non-finite values' +assert np.all(np.isfinite(b)), 'crysfml pattern has non-finite values' rms = float(np.sqrt(np.mean((a - b) ** 2))) norm = float(np.sqrt(np.mean(a**2))) @@ -82,9 +83,7 @@ x = np.arange(a.size) fig = go.Figure() fig.add_scatter(x=x, y=a, mode='lines', name='cryspy', line={'color': 'royalblue'}) -fig.add_scatter( - x=x, y=b, mode='lines', name='crysfml', line={'color': 'crimson', 'dash': 'dot'} -) +fig.add_scatter(x=x, y=b, mode='lines', name='crysfml', line={'color': 'crimson', 'dash': 'dot'}) fig.update_layout( title='Calculated patterns: cryspy vs crysfml', xaxis_title='point index', diff --git a/docs/docs/verification/index.md b/docs/docs/verification/index.md index bf41a17ea..cf6926494 100644 --- a/docs/docs/verification/index.md +++ b/docs/docs/verification/index.md @@ -1,10 +1,11 @@ # Verification This section compares EasyDiffraction's calculation engines against each -other (and, in future, against external software such as FullProf) on the -**same** input parameters, **without any fitting** — just calculated +other (and, in future, against external software such as FullProf) on +the **same** input parameters, **without any fitting** — just calculated diffraction patterns and clear closeness metrics. -Each page also runs as a fast regression check (`pixi run script-tests`), -so cross-engine agreement is monitored over time. Coverage grows to span -every supported experiment and instrument combination. +Each page also runs as a fast regression check +(`pixi run script-tests`), so cross-engine agreement is monitored over +time. Coverage grows to span every supported experiment and instrument +combination. diff --git a/pyproject.toml b/pyproject.toml index 35e9c4a12..458a5713d 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -382,6 +382,7 @@ ignore = [ 'E402', # module-import-not-at-top-of-file 'T201', # print (metric values shown in the rendered notebook) 'B018', # useless-expression (trailing `fig` renders in the notebook) + 'S101', # assert (the regression checks in the verification script) ] # Intentional terminal rendering: these write raw/ASCII output that # `Console.print` would garble, so `print` is deliberate here. diff --git a/src/easydiffraction/analysis/calculators/support.py b/src/easydiffraction/analysis/calculators/support.py index 92b9b3251..a24e7742b 100644 --- a/src/easydiffraction/analysis/calculators/support.py +++ b/src/easydiffraction/analysis/calculators/support.py @@ -1,24 +1,34 @@ # SPDX-FileCopyrightText: 2026 EasyScience contributors # SPDX-License-Identifier: BSD-3-Clause -"""Calculator support matrix. +""" +Calculator support matrix. -Aggregates the per-instrument ``Compatibility`` and ``CalculatorSupport`` -metadata declared on the registered instrument categories into a single -queryable matrix: which calculation engines can compute which experiment -conditions. Used by the Verification documentation to enumerate -comparable engine x condition combinations. +Aggregates the per-instrument ``Compatibility`` and +``CalculatorSupport`` metadata declared on the registered instrument +categories into a single queryable matrix: which calculation engines can +compute which experiment conditions. Used by the Verification +documentation to enumerate comparable engine x condition combinations. """ from __future__ import annotations from dataclasses import dataclass +from typing import TYPE_CHECKING + +# Importing the instrument package registers every concrete instrument +# category, so the support matrix below is complete regardless of import +# order. +import easydiffraction.datablocks.experiment.categories.instrument # noqa: F401 +from easydiffraction.datablocks.experiment.categories.instrument.factory import InstrumentFactory -from easydiffraction.core.metadata import Compatibility +if TYPE_CHECKING: + from easydiffraction.core.metadata import Compatibility @dataclass(frozen=True) class SupportEntry: - """One instrument condition and the engines that can compute it. + """ + One instrument condition and the engines that can compute it. Attributes ---------- @@ -40,31 +50,23 @@ class SupportEntry: def calculator_support_matrix() -> list[SupportEntry]: - """Return the engine x experiment-condition support matrix. + """ + Return the engine x experiment-condition support matrix. One entry per registered instrument category, pairing its ``Compatibility`` with the calculators declared able to handle it. Returns ------- - list of SupportEntry + list[SupportEntry] One entry per registered instrument category. """ - # Lazy imports: ensure the instrument categories are registered and - # avoid a circular import at module load. - import easydiffraction.datablocks.experiment.categories.instrument # noqa: F401 - from easydiffraction.datablocks.experiment.categories.instrument.factory import ( - InstrumentFactory, - ) - - entries: list[SupportEntry] = [] - for klass in InstrumentFactory._supported_map().values(): - entries.append( - SupportEntry( - instrument_tag=klass.type_info.tag, - description=klass.type_info.description, - compatibility=klass.compatibility, - calculators=frozenset(klass.calculator_support.calculators), - ) + return [ + SupportEntry( + instrument_tag=klass.type_info.tag, + description=klass.type_info.description, + compatibility=klass.compatibility, + calculators=frozenset(klass.calculator_support.calculators), ) - return entries + for klass in InstrumentFactory._supported_map().values() + ] diff --git a/tests/unit/easydiffraction/core/test_validation_properties.py b/tests/unit/easydiffraction/core/test_validation_properties.py index 8c272ba39..631e7dd7e 100644 --- a/tests/unit/easydiffraction/core/test_validation_properties.py +++ b/tests/unit/easydiffraction/core/test_validation_properties.py @@ -14,6 +14,8 @@ from __future__ import annotations +import math + import numpy as np import pytest from hypothesis import given @@ -65,7 +67,7 @@ def test_numeric_rejects_text_with_fallback(value): @pytest.mark.parametrize( 'value', - [0, -1, 1, 0.0, -2.5, 3.14, np.int64(5), np.float64(2.0)], + [0, -1, 1, 0.0, -2.5, math.pi, np.int64(5), np.float64(2.0)], ) def test_numeric_accepts_boundary_table(value): _warn() @@ -99,11 +101,7 @@ def test_occupancy_accepts_unit_interval(value): assert _occupancy_spec().validated(value, name='occ') == value -@given( - value=st.floats(allow_nan=False, allow_infinity=False).filter( - lambda v: v < 0.0 or v > 1.0 - ) -) +@given(value=st.floats(allow_nan=False, allow_infinity=False).filter(lambda v: v < 0.0 or v > 1.0)) def test_occupancy_rejects_outside_unit_interval(value): _warn() assert _occupancy_spec().validated(value, name='occ') == OCC_DEFAULT From 7ce29db16959d629b357bf3ac6ab197f6523e319 Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Sat, 6 Jun 2026 00:00:49 -0700 Subject: [PATCH 30/57] Tighten cross-engine verification tolerances from measurement --- .../verification/cross-engine-bragg-cwl.ipynb | 21 ++++++++++--------- .../verification/cross-engine-bragg-cwl.py | 21 ++++++++++--------- 2 files changed, 22 insertions(+), 20 deletions(-) diff --git a/docs/docs/verification/cross-engine-bragg-cwl.ipynb b/docs/docs/verification/cross-engine-bragg-cwl.ipynb index 621e18735..dd4593e20 100644 --- a/docs/docs/verification/cross-engine-bragg-cwl.ipynb +++ b/docs/docs/verification/cross-engine-bragg-cwl.ipynb @@ -193,10 +193,12 @@ "source": [ "## Regression assertions\n", "\n", - "Explicit, named tolerances for each metric. These are intentionally\n", - "loose initial bounds — they catch a gross cross-engine divergence now\n", - "and are tightened once nightly runs establish the real spread for each\n", - "engine pair. Correlation is kept as an additional shape signal." + "Explicit, named tolerances for each metric. cryspy and crysfml agree\n", + "very closely here (a first measurement gives ≈0.5% profile difference,\n", + "intensity ratio ≈1.00, correlation ≈1.00), so these bounds keep a\n", + "generous cross-platform margin while still catching a real regression.\n", + "They are tightened further once multi-platform nightly runs establish\n", + "the spread for each engine pair." ] }, { @@ -206,12 +208,11 @@ "metadata": {}, "outputs": [], "source": [ - "# Loose initial tolerances (tightened once real spreads are measured).\n", - "MAX_PROFILE_DIFFERENCE_PCT = 100.0\n", - "MAX_RELATIVE_DEVIATION = 2.0\n", - "MIN_INTENSITY_RATIO = 0.1\n", - "MAX_INTENSITY_RATIO = 10.0\n", - "MIN_CORRELATION = 0.8\n", + "MAX_PROFILE_DIFFERENCE_PCT = 10.0 # measured ≈0.5%\n", + "MAX_RELATIVE_DEVIATION = 1.0 # scale-sensitive; kept generous\n", + "MIN_INTENSITY_RATIO = 0.8 # measured ≈1.00\n", + "MAX_INTENSITY_RATIO = 1.25\n", + "MIN_CORRELATION = 0.99 # measured ≈1.00\n", "\n", "peak = float(np.max(np.abs(a)))\n", "relative_deviation = max_deviation / peak if peak else float('nan')\n", diff --git a/docs/docs/verification/cross-engine-bragg-cwl.py b/docs/docs/verification/cross-engine-bragg-cwl.py index bcf606048..55276d567 100644 --- a/docs/docs/verification/cross-engine-bragg-cwl.py +++ b/docs/docs/verification/cross-engine-bragg-cwl.py @@ -96,18 +96,19 @@ # %% [markdown] # ## Regression assertions # -# Explicit, named tolerances for each metric. These are intentionally -# loose initial bounds — they catch a gross cross-engine divergence now -# and are tightened once nightly runs establish the real spread for each -# engine pair. Correlation is kept as an additional shape signal. +# Explicit, named tolerances for each metric. cryspy and crysfml agree +# very closely here (a first measurement gives ≈0.5% profile difference, +# intensity ratio ≈1.00, correlation ≈1.00), so these bounds keep a +# generous cross-platform margin while still catching a real regression. +# They are tightened further once multi-platform nightly runs establish +# the spread for each engine pair. # %% -# Loose initial tolerances (tightened once real spreads are measured). -MAX_PROFILE_DIFFERENCE_PCT = 100.0 -MAX_RELATIVE_DEVIATION = 2.0 -MIN_INTENSITY_RATIO = 0.1 -MAX_INTENSITY_RATIO = 10.0 -MIN_CORRELATION = 0.8 +MAX_PROFILE_DIFFERENCE_PCT = 10.0 # measured ≈0.5% +MAX_RELATIVE_DEVIATION = 1.0 # scale-sensitive; kept generous +MIN_INTENSITY_RATIO = 0.8 # measured ≈1.00 +MAX_INTENSITY_RATIO = 1.25 +MIN_CORRELATION = 0.99 # measured ≈1.00 peak = float(np.max(np.abs(a))) relative_deviation = max_deviation / peak if peak else float('nan') From 4f4de17ae788fb9a050b383c8e5261ee87e89fec Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Sat, 6 Jun 2026 00:36:08 -0700 Subject: [PATCH 31/57] Render small canvas in raster tests to speed them up --- .../structure/renderers/test_raster.py | 24 ++++++++++++++++--- 1 file changed, 21 insertions(+), 3 deletions(-) diff --git a/tests/unit/easydiffraction/display/structure/renderers/test_raster.py b/tests/unit/easydiffraction/display/structure/renderers/test_raster.py index bfa426634..4bc41e819 100644 --- a/tests/unit/easydiffraction/display/structure/renderers/test_raster.py +++ b/tests/unit/easydiffraction/display/structure/renderers/test_raster.py @@ -117,6 +117,24 @@ def _full_scene() -> StructureScene: ) +@pytest.fixture(autouse=True) +def _fast_canvas(request, monkeypatch): + """Render at a small canvas to keep these renderer tests fast. + + Almost every test checks render *behaviour* (PNG output, drawn vs + blank pixels, determinism), which is resolution-independent, so a + small canvas gives identical outcomes far faster — the production + 1800x1800 buffer is ~80x more pixels. Size assertions read + ``MUT._CANVAS`` so they stay correct. ``test_axis_labels_follow_ + rendered_arrow_tips`` opts out because it asserts exact label pixel + positions against the production ``_CANVAS`` (and it does no render, + so it is already fast). + """ + if request.node.name == 'test_axis_labels_follow_rendered_arrow_tips': + return + monkeypatch.setattr(MUT, '_CANVAS', 200) + + def test_module_import(): import easydiffraction.display.structure.renderers.raster as MUT @@ -160,9 +178,9 @@ def test_decodes_as_png(self): assert _open(png).format == 'PNG' def test_canvas_dimensions(self): - # The supersampled buffer is downsampled to a fixed 1800x1800 frame. + # The supersampled buffer is downsampled to a square _CANVAS frame. png = RasterStructureRenderer().render_png(_atom_scene(), features=frozenset({'atoms'})) - assert _open(png).size == (1800, 1800) + assert _open(png).size == (MUT._CANVAS, MUT._CANVAS) def test_rgb_mode(self): png = RasterStructureRenderer().render_png(_atom_scene(), features=frozenset({'atoms'})) @@ -251,7 +269,7 @@ def test_full_scene_renders(self): _full_scene(), features=RasterStructureRenderer.SUPPORTED ) assert png[:8] == PNG_MAGIC - assert _open(png).size == (1800, 1800) + assert _open(png).size == (MUT._CANVAS, MUT._CANVAS) assert _has_drawn_pixels(png) From 83bd6a58d80631d5d2fb59d475097a3daec1bbda Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Sat, 6 Jun 2026 00:53:00 -0700 Subject: [PATCH 32/57] Add icon to the Verification docs section --- docs/docs/verification/index.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/docs/docs/verification/index.md b/docs/docs/verification/index.md index cf6926494..c7497a0b7 100644 --- a/docs/docs/verification/index.md +++ b/docs/docs/verification/index.md @@ -1,4 +1,8 @@ -# Verification +--- +icon: material/check-decagram +--- + +# :material-check-decagram: Verification This section compares EasyDiffraction's calculation engines against each other (and, in future, against external software such as FullProf) on From ede35b7aff0ea5bce5d24f61da113e898b31f6f5 Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Sat, 6 Jun 2026 00:53:00 -0700 Subject: [PATCH 33/57] Fix tabel typo in docs CSS comments --- docs/docs/assets/stylesheets/extra.css | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/docs/assets/stylesheets/extra.css b/docs/docs/assets/stylesheets/extra.css index ae3c9bccd..2ab67a91f 100644 --- a/docs/docs/assets/stylesheets/extra.css +++ b/docs/docs/assets/stylesheets/extra.css @@ -182,13 +182,13 @@ label.md-nav__title[for="__drawer"] { padding-left: 0.5em; /* Default */ } -/* Change line height of the tabel cells */ +/* Change line height of the table cells */ .md-typeset td, .md-typeset th { line-height: 1.25 !important; } -/* Change vertical alignment of the icon inside the tabel cells */ +/* Change vertical alignment of the icon inside the table cells */ .md-typeset td .twemoji { vertical-align: sub !important; } From 9bc03a24bcf39988816d7f8c4489f2b9bf8e2af2 Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Sat, 6 Jun 2026 00:53:00 -0700 Subject: [PATCH 34/57] Fix promoted ADR links to documentation-ci-build --- docs/dev/adrs/accepted/test-suite-and-validation.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/dev/adrs/accepted/test-suite-and-validation.md b/docs/dev/adrs/accepted/test-suite-and-validation.md index 6d0744057..3cadc0b98 100644 --- a/docs/dev/adrs/accepted/test-suite-and-validation.md +++ b/docs/dev/adrs/accepted/test-suite-and-validation.md @@ -66,7 +66,7 @@ accumulated: slow and therefore runs on pull requests only. There is no fast, every-push check that the site builds strictly, links resolve, and prose is clean. This overlaps the unimplemented - [Documentation CI and Build Verification](documentation-ci-build.md) + [Documentation CI and Build Verification](../suggestions/documentation-ci-build.md) suggestion. This ADR amends [Test Strategy](../accepted/test-strategy.md): the @@ -338,7 +338,7 @@ prompt drift feedback. It is deliberately **separate from `docs.yml`**, which executes all tutorials and then builds and deploys — slow, and therefore pull-request-only. The detailed catalogue of documentation checks is owned by -[Documentation CI and Build Verification](documentation-ci-build.md), +[Documentation CI and Build Verification](../suggestions/documentation-ci-build.md), which this ADR coordinates with: that ADR defines _what_ the checks are; this ADR's decision is that the cheap, deterministic subset runs as part of the every-push test workflow. Promoting that ADR is part of this @@ -443,14 +443,14 @@ drafting conversation, per the dependency-approval rule): - `pytest-benchmark` — performance-regression benchmarks (§7). Coordinated with -[Documentation CI and Build Verification](documentation-ci-build.md), +[Documentation CI and Build Verification](../suggestions/documentation-ci-build.md), which carries the documentation-check tools (`codespell`, a link checker such as `lychee`, and later `Vale`) used by §9. ## Related ADRs - [Test Strategy](../accepted/test-strategy.md) — amended by this ADR. -- [Documentation CI and Build Verification](documentation-ci-build.md) — +- [Documentation CI and Build Verification](../suggestions/documentation-ci-build.md) — coordinated with §9. - [Lint Complexity Thresholds](../accepted/lint-complexity-thresholds.md) — sibling Quality guardrail. From 54b6d48fe7950178c91e8a6ab70b8d4995930a62 Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Sat, 6 Jun 2026 00:53:00 -0700 Subject: [PATCH 35/57] Scope codespell and lychee to pass on real sources --- pixi.toml | 2 +- pyproject.toml | 5 ++++- 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/pixi.toml b/pixi.toml index 877089a45..59a34bf9a 100644 --- a/pixi.toml +++ b/pixi.toml @@ -261,7 +261,7 @@ docs-build-strict = { cmd = 'pixi run docs-pre build --strict -f docs/mkdocs.yml spell-check = 'codespell docs src tools README.md CONTRIBUTING.md' # Local/relative link integrity across the docs (offline; external URL # checking is deferred, see issue 114). Config in lychee.toml. -link-check = 'lychee --config lychee.toml docs README.md CONTRIBUTING.md' +link-check = 'lychee --config lychee.toml docs/docs README.md CONTRIBUTING.md' docs-deploy-pre = 'mike deploy -F docs/mkdocs.yml --push --branch gh-pages --update-aliases --alias-type redirect' docs-set-default-pre = 'mike set-default -F docs/mkdocs.yml --push --branch gh-pages' diff --git a/pyproject.toml b/pyproject.toml index 458a5713d..53464e20a 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -220,7 +220,10 @@ filterwarnings = [ # 'codespell' -- Spell checker for docs and source comments/docstrings. # https://github.com/codespell-project/codespell [tool.codespell] -skip = '*.ipynb,*.lock,*.svg,*.min.js,*.json.gz,*/vendor/*,*/_vendored/*,docs/dev/package-structure/*,node_modules,.pixi' +skip = '*.ipynb,*.lock,*.svg,*.min.js,*.json.gz,*/vendor/*,*/_vendored/*,docs/dev/package-structure/*,docs/site/*,docs/overrides/*,*/structure/assets/*,node_modules,.pixi' +# British spelling and a crystallographic abbreviation (B/U) flagged as +# typos; element symbols live in skipped vendored files. +ignore-words-list = 'pre-emptively,bu' ######################## # Configuration for ruff From bb2c9671f904914a3363541e8cf29e856084f1a1 Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Sat, 6 Jun 2026 00:57:46 -0700 Subject: [PATCH 36/57] Run docs checks (spell, link, strict build) with the checks --- .github/workflows/lint-format.yml | 24 ++++++++++++++++++++++++ .github/workflows/test.yml | 22 ---------------------- .pre-commit-config.yaml | 14 ++++++++++++++ pixi.toml | 21 ++++++++++----------- 4 files changed, 48 insertions(+), 33 deletions(-) diff --git a/.github/workflows/lint-format.yml b/.github/workflows/lint-format.yml index 0f45cefb2..81bb01982 100644 --- a/.github/workflows/lint-format.yml +++ b/.github/workflows/lint-format.yml @@ -97,6 +97,24 @@ jobs: shell: bash run: pixi run test-structure-check + - name: Check spelling (codespell) + id: spell + continue-on-error: true + shell: bash + run: pixi run spell-check + + - name: Check documentation links (lychee) + id: link + continue-on-error: true + shell: bash + run: pixi run link-check + + - name: Build documentation strictly (no tutorial execution) + id: docs_build + continue-on-error: true + shell: bash + run: pixi run docs-build-strict + # Add summary - name: Add quality checks summary if: always() @@ -115,6 +133,9 @@ jobs: echo "| nonpy format | ${{ steps.nonpy_format.outcome == 'success' && '✅' || '❌' }} |" echo "| notebooks lint | ${{ steps.notebook_lint.outcome == 'success' && '✅' || '❌' }} |" echo "| test structure | ${{ steps.test_structure.outcome == 'success' && '✅' || '❌' }} |" + echo "| spelling | ${{ steps.spell.outcome == 'success' && '✅' || '❌' }} |" + echo "| doc links | ${{ steps.link.outcome == 'success' && '✅' || '❌' }} |" + echo "| docs strict build| ${{ steps.docs_build.outcome == 'success' && '✅' || '❌' }} |" } >> "$GITHUB_STEP_SUMMARY" # Fail job if any check failed @@ -128,5 +149,8 @@ jobs: || steps.nonpy_format.outcome == 'failure' || steps.notebook_lint.outcome == 'failure' || steps.test_structure.outcome == 'failure' + || steps.spell.outcome == 'failure' + || steps.link.outcome == 'failure' + || steps.docs_build.outcome == 'failure' shell: bash run: exit 1 diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index d5d5f7c3b..83eaed8c8 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -73,28 +73,6 @@ jobs: echo 'pytest_marks=-m "not pr and not nightly"' >> "$GITHUB_OUTPUT" fi - # Job: Fast documentation checks on every push (tutorials NOT executed, - # unlike the slow docs.yml). Strict build catches broken nav/internal - # links; codespell catches typos. See ADR test-suite-and-validation §9. - docs-checks: - runs-on: ubuntu-latest - - steps: - - name: Checkout repository - uses: actions/checkout@v6 - - - name: Set up pixi - uses: ./.github/actions/setup-pixi - - - name: Strict documentation build (tutorials not executed) - run: pixi run docs-build-strict - - - name: Spell check - run: pixi run spell-check - - - name: Link check - run: pixi run link-check - # Job 2: Test code source-test: needs: env-prepare # depend on previous job diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 770dbca2f..07a0e43b4 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -60,6 +60,20 @@ repos: pass_filenames: false stages: [manual] + - id: pixi-spell-check + name: pixi run spell-check + entry: pixi run spell-check + language: system + pass_filenames: false + stages: [manual] + + - id: pixi-link-check + name: pixi run link-check + entry: pixi run link-check + language: system + pass_filenames: false + stages: [manual] + - id: pixi-unit-tests name: pixi run unit-tests entry: pixi run unit-tests diff --git a/pixi.toml b/pixi.toml index 59a34bf9a..0d162a047 100644 --- a/pixi.toml +++ b/pixi.toml @@ -158,6 +158,16 @@ py-format-check = 'ruff format --check src/ tests/ docs/docs/tutorials/ docs/doc nonpy-format-check = 'npx prettier --list-different --config=prettierrc.toml --ignore-unknown .' nonpy-format-check-modified = 'python tools/nonpy_prettier_modified.py' test-structure-check = 'python tools/test_structure_check.py' +# Spelling over docs and source (config in [tool.codespell]). +spell-check = 'codespell docs src tools README.md CONTRIBUTING.md' +# Local/relative documentation link integrity (offline; external URL +# checking is deferred, see issue 114). Config in lychee.toml. +link-check = 'lychee --config lychee.toml docs/docs README.md CONTRIBUTING.md' +# Strict documentation build: fails on broken nav / internal references. +# Tutorials are not executed (mkdocs-jupyter execute: false). +docs-build-strict = { cmd = 'pixi run docs-pre build --strict -f docs/mkdocs.yml', depends-on = [ + 'docs-sync-vendored-js', +] } check = 'pre-commit run --hook-stage manual --all-files' @@ -251,17 +261,6 @@ docs-build = { cmd = 'pixi run docs-pre build -f docs/mkdocs.yml', depends-on = 'docs-sync-vendored-js', ] } docs-build-local = 'pixi run docs-build --no-directory-urls' -# Strict build: fails on broken nav / internal references. Tutorials are -# not executed (mkdocs-jupyter execute: false), so this is the fast -# every-push documentation gate. -docs-build-strict = { cmd = 'pixi run docs-pre build --strict -f docs/mkdocs.yml', depends-on = [ - 'docs-sync-vendored-js', -] } -# Spelling check over docs and source (config in [tool.codespell]). -spell-check = 'codespell docs src tools README.md CONTRIBUTING.md' -# Local/relative link integrity across the docs (offline; external URL -# checking is deferred, see issue 114). Config in lychee.toml. -link-check = 'lychee --config lychee.toml docs/docs README.md CONTRIBUTING.md' docs-deploy-pre = 'mike deploy -F docs/mkdocs.yml --push --branch gh-pages --update-aliases --alias-type redirect' docs-set-default-pre = 'mike set-default -F docs/mkdocs.yml --push --branch gh-pages' From 2dbb6f0295856bdd6b013b4a9949e8a3e43e4ee7 Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Sat, 6 Jun 2026 01:35:48 -0700 Subject: [PATCH 37/57] Reflow ADR prose after link fix --- docs/dev/adrs/accepted/test-suite-and-validation.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/dev/adrs/accepted/test-suite-and-validation.md b/docs/dev/adrs/accepted/test-suite-and-validation.md index 3cadc0b98..b35aea302 100644 --- a/docs/dev/adrs/accepted/test-suite-and-validation.md +++ b/docs/dev/adrs/accepted/test-suite-and-validation.md @@ -450,8 +450,8 @@ such as `lychee`, and later `Vale`) used by §9. ## Related ADRs - [Test Strategy](../accepted/test-strategy.md) — amended by this ADR. -- [Documentation CI and Build Verification](../suggestions/documentation-ci-build.md) — - coordinated with §9. +- [Documentation CI and Build Verification](../suggestions/documentation-ci-build.md) + — coordinated with §9. - [Lint Complexity Thresholds](../accepted/lint-complexity-thresholds.md) — sibling Quality guardrail. - [Notebook Generation Source of Truth](../accepted/notebook-generation.md) From 2f2a40da05a85ad40c76015f39b0d5f38266c365 Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Sat, 6 Jun 2026 01:35:48 -0700 Subject: [PATCH 38/57] Make check static-only and add full pre-PR all task --- .pre-commit-config.yaml | 14 -------------- pixi.toml | 8 ++++++++ 2 files changed, 8 insertions(+), 14 deletions(-) diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 07a0e43b4..1476cec18 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -73,17 +73,3 @@ repos: language: system pass_filenames: false stages: [manual] - - - id: pixi-unit-tests - name: pixi run unit-tests - entry: pixi run unit-tests - language: system - pass_filenames: false - stages: [manual] - - - id: pixi-functional-tests - name: pixi run functional-tests - entry: pixi run functional-tests - language: system - pass_filenames: false - stages: [manual] diff --git a/pixi.toml b/pixi.toml index 0d162a047..32761e2b0 100644 --- a/pixi.toml +++ b/pixi.toml @@ -146,6 +146,14 @@ test-all = { depends-on = [ 'script-tests', ] } +# Full local gate before opening a PR: every static check, unit coverage, +# all tests, tutorial execution + output verification (as scripts AND +# notebooks), and a strict documentation build (which validates the built +# site's nav/links). Steps run sequentially because the tutorial tasks +# share tmp/tutorials/projects/. Slow but comprehensive — run `pixi run +# fix` first to auto-format. +all = 'pixi run check && pixi run unit-tests-coverage && pixi run functional-tests && pixi run integration-tests && pixi run script-tests-checked && pixi run notebook-tests-checked && pixi run docs-build-strict' + ########### # ✔️ Checks ########### From c1afbd2cd86eec425e6fea5c26beb963c7b2326d Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Sat, 6 Jun 2026 01:49:17 -0700 Subject: [PATCH 39/57] Always build docs strictly; drop docs-build-strict --- .github/workflows/lint-format.yml | 2 +- docs/dev/issues/open.md | 6 +++--- pixi.toml | 21 ++++++++------------- 3 files changed, 12 insertions(+), 17 deletions(-) diff --git a/.github/workflows/lint-format.yml b/.github/workflows/lint-format.yml index 81bb01982..ed78c69c2 100644 --- a/.github/workflows/lint-format.yml +++ b/.github/workflows/lint-format.yml @@ -113,7 +113,7 @@ jobs: id: docs_build continue-on-error: true shell: bash - run: pixi run docs-build-strict + run: pixi run docs-build # Add summary - name: Add quality checks summary diff --git a/docs/dev/issues/open.md b/docs/dev/issues/open.md index c63987cc9..b58e822c5 100644 --- a/docs/dev/issues/open.md +++ b/docs/dev/issues/open.md @@ -1927,9 +1927,9 @@ coordination. **Type:** CI / Documentation -The fast docs gate (`docs-build-strict` + `spell-check`) catches broken -nav/internal links and typos on every push, but does not yet check -external URLs. Add a `lychee` link checker (with an allowlist for +The fast docs gate (`docs-build` + `link-check` + `spell-check`) catches +broken nav/internal links and typos on every push, but does not yet +check external URLs. Add a `lychee` link checker (with an allowlist for rate-limited/unstable domains), coordinated with the [Documentation CI and Build Verification](../adrs/suggestions/documentation-ci-build.md) ADR. Run it nightly or on pull requests to avoid flakiness from external diff --git a/pixi.toml b/pixi.toml index 32761e2b0..1baf8cf6e 100644 --- a/pixi.toml +++ b/pixi.toml @@ -152,7 +152,7 @@ test-all = { depends-on = [ # site's nav/links). Steps run sequentially because the tutorial tasks # share tmp/tutorials/projects/. Slow but comprehensive — run `pixi run # fix` first to auto-format. -all = 'pixi run check && pixi run unit-tests-coverage && pixi run functional-tests && pixi run integration-tests && pixi run script-tests-checked && pixi run notebook-tests-checked && pixi run docs-build-strict' +all = 'pixi run check && pixi run unit-tests-coverage && pixi run functional-tests && pixi run integration-tests && pixi run script-tests-checked && pixi run notebook-tests-checked && pixi run docs-build-local' ########### # ✔️ Checks @@ -171,12 +171,6 @@ spell-check = 'codespell docs src tools README.md CONTRIBUTING.md' # Local/relative documentation link integrity (offline; external URL # checking is deferred, see issue 114). Config in lychee.toml. link-check = 'lychee --config lychee.toml docs/docs README.md CONTRIBUTING.md' -# Strict documentation build: fails on broken nav / internal references. -# Tutorials are not executed (mkdocs-jupyter execute: false). -docs-build-strict = { cmd = 'pixi run docs-pre build --strict -f docs/mkdocs.yml', depends-on = [ - 'docs-sync-vendored-js', -] } - check = 'pre-commit run --hook-stage manual --all-files' ########## @@ -225,11 +219,7 @@ functional-tests-coverage = 'pixi run functional-tests --cov=src/easydiffraction integration-tests-coverage = 'pixi run integration-tests --cov=src/easydiffraction --cov-report=term-missing' docstring-coverage = 'interrogate -c pyproject.toml src/easydiffraction' -cov = { depends-on = [ - 'docstring-coverage', - 'unit-tests-coverage', - 'integration-tests-coverage', -] } +cov = { depends-on = ['docstring-coverage', 'unit-tests-coverage'] } ######################## # 📓 Notebook Management @@ -265,9 +255,14 @@ docs-serve = { cmd = 'pixi run docs-pre serve -f docs/mkdocs.yml', depends-on = 'docs-sync-vendored-js', ] } docs-serve-dirty = 'pixi run docs-serve --dirty' -docs-build = { cmd = 'pixi run docs-pre build -f docs/mkdocs.yml', depends-on = [ +# Strict build (fails on broken nav / internal references); used for the +# deploy and the CI docs check. Tutorials are not executed (mkdocs-jupyter +# execute: false). docs-serve is intentionally left non-strict. +docs-build = { cmd = 'pixi run docs-pre build --strict -f docs/mkdocs.yml', depends-on = [ 'docs-sync-vendored-js', ] } +# Inherits --strict; --no-directory-urls makes the built site browsable +# from disk (used by `all` so you can inspect it after verifying). docs-build-local = 'pixi run docs-build --no-directory-urls' docs-deploy-pre = 'mike deploy -F docs/mkdocs.yml --push --branch gh-pages --update-aliases --alias-type redirect' From aee2a7628cdd7c04c8053d12501bb018c8dc6df8 Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Sat, 6 Jun 2026 01:55:31 -0700 Subject: [PATCH 40/57] Gate docstring coverage as a check, not in coverage.yml --- .github/workflows/coverage.yml | 19 +++---------------- .github/workflows/lint-format.yml | 8 ++++++++ .pre-commit-config.yaml | 7 +++++++ 3 files changed, 18 insertions(+), 16 deletions(-) diff --git a/.github/workflows/coverage.yml b/.github/workflows/coverage.yml index 5208a3333..002bfc82f 100644 --- a/.github/workflows/coverage.yml +++ b/.github/workflows/coverage.yml @@ -26,21 +26,8 @@ env: CI_BRANCH: ${{ github.head_ref || github.ref_name }} jobs: - # Job 1: Run docstring coverage - docstring-coverage: - runs-on: ubuntu-latest - - steps: - - name: Check-out repository - uses: actions/checkout@v6 - - - name: Set up pixi - uses: ./.github/actions/setup-pixi - - - name: Run docstring coverage - run: pixi run docstring-coverage - - # Job 2: Run unit tests with coverage and upload to Codecov + # Job 1: Run unit tests with coverage and upload to Codecov. + # Docstring coverage is gated as a static check in lint-format.yml. unit-tests-coverage: runs-on: ubuntu-latest @@ -65,6 +52,6 @@ jobs: # Job 4: Build and publish dashboard (reusable workflow) run-reusable-workflows: - needs: [docstring-coverage, unit-tests-coverage] # depend on the previous jobs + needs: [unit-tests-coverage] # depend on the previous job uses: ./.github/workflows/dashboard.yml secrets: inherit diff --git a/.github/workflows/lint-format.yml b/.github/workflows/lint-format.yml index ed78c69c2..f1308cd7f 100644 --- a/.github/workflows/lint-format.yml +++ b/.github/workflows/lint-format.yml @@ -79,6 +79,12 @@ jobs: shell: bash run: pixi run docstring-lint-check + - name: Check docstring coverage (interrogate) + id: docstring_coverage + continue-on-error: true + shell: bash + run: pixi run docstring-coverage + - name: Check formatting of non-Python files (md, toml, etc.) id: nonpy_format continue-on-error: true @@ -130,6 +136,7 @@ jobs: echo "| py lint | ${{ steps.py_lint.outcome == 'success' && '✅' || '❌' }} |" echo "| py format | ${{ steps.py_format.outcome == 'success' && '✅' || '❌' }} |" echo "| docstring lint | ${{ steps.docstring_lint.outcome == 'success' && '✅' || '❌' }} |" + echo "| docstring cover | ${{ steps.docstring_coverage.outcome == 'success' && '✅' || '❌' }} |" echo "| nonpy format | ${{ steps.nonpy_format.outcome == 'success' && '✅' || '❌' }} |" echo "| notebooks lint | ${{ steps.notebook_lint.outcome == 'success' && '✅' || '❌' }} |" echo "| test structure | ${{ steps.test_structure.outcome == 'success' && '✅' || '❌' }} |" @@ -146,6 +153,7 @@ jobs: || steps.py_lint.outcome == 'failure' || steps.py_format.outcome == 'failure' || steps.docstring_lint.outcome == 'failure' + || steps.docstring_coverage.outcome == 'failure' || steps.nonpy_format.outcome == 'failure' || steps.notebook_lint.outcome == 'failure' || steps.test_structure.outcome == 'failure' diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 1476cec18..6fc64e208 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -39,6 +39,13 @@ repos: pass_filenames: false stages: [manual] + - id: pixi-docstring-coverage + name: pixi run docstring-coverage + entry: pixi run docstring-coverage + language: system + pass_filenames: false + stages: [manual] + - id: pixi-nonpy-format-check name: pixi run nonpy-format-check entry: pixi run nonpy-format-check From 5046b059c7d4279024cc89b3583e492e4040cc7d Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Sat, 6 Jun 2026 05:02:47 -0700 Subject: [PATCH 41/57] Restore logger reaction mode in validation-properties test Add an autouse fixture that snapshots and restores Logger._reaction so the WARN mode set by _warn() cannot leak into later raise-expecting tests (e.g. core/test_guard, core/test_parameters). --- .../easydiffraction/core/test_validation_properties.py | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/tests/unit/easydiffraction/core/test_validation_properties.py b/tests/unit/easydiffraction/core/test_validation_properties.py index 631e7dd7e..0803dde04 100644 --- a/tests/unit/easydiffraction/core/test_validation_properties.py +++ b/tests/unit/easydiffraction/core/test_validation_properties.py @@ -29,6 +29,14 @@ from easydiffraction.utils.logging import log +@pytest.fixture(autouse=True) +def _restore_logger_reaction(): + """Restore the global logger reaction so WARN does not leak out.""" + saved = log._reaction + yield + log._reaction = saved + + def _warn() -> None: """Keep the logger non-raising so validators take the fallback path.""" log.configure(reaction=log.Reaction.WARN) From 6950251af38b679e88e8a0e4f9b97db2268aae57 Mon Sep 17 00:00:00 2001 From: Andrew Sazonov Date: Sat, 6 Jun 2026 05:02:55 -0700 Subject: [PATCH 42/57] Expand unit-test coverage across analysis, display, report Add and extend supplementary coverage tests for analysis (analysis, fitting, minimizers/base), display (plotting, progress), report (data_context, html_renderer), background estimate, and the package __main__ entry point. Raises overall unit coverage to ~84.5%. --- .../analysis/minimizers/test_base_coverage.py | 578 +++++ .../analysis/test_analysis_coverage.py | 1441 +++++++++++++ .../analysis/test_fitting_coverage.py | 742 +++++++ .../background/test_estimate_coverage.py | 395 ++++ .../display/test_plotting_coverage.py | 1903 +++++++++++++++++ .../display/test_progress_coverage.py | 1006 +++++++++ .../report/test_data_context_coverage.py | 579 +++++ .../report/test_html_renderer_coverage.py | 687 ++++++ .../easydiffraction/test___main___coverage.py | 526 +++++ 9 files changed, 7857 insertions(+) create mode 100644 tests/unit/easydiffraction/analysis/minimizers/test_base_coverage.py create mode 100644 tests/unit/easydiffraction/analysis/test_fitting_coverage.py create mode 100644 tests/unit/easydiffraction/datablocks/experiment/categories/background/test_estimate_coverage.py create mode 100644 tests/unit/easydiffraction/display/test_progress_coverage.py create mode 100644 tests/unit/easydiffraction/report/test_data_context_coverage.py create mode 100644 tests/unit/easydiffraction/report/test_html_renderer_coverage.py create mode 100644 tests/unit/easydiffraction/test___main___coverage.py diff --git a/tests/unit/easydiffraction/analysis/minimizers/test_base_coverage.py b/tests/unit/easydiffraction/analysis/minimizers/test_base_coverage.py new file mode 100644 index 000000000..5732cd45d --- /dev/null +++ b/tests/unit/easydiffraction/analysis/minimizers/test_base_coverage.py @@ -0,0 +1,578 @@ +# SPDX-FileCopyrightText: 2025 EasyScience contributors +# SPDX-License-Identifier: BSD-3-Clause + +"""Supplementary unit tests raising coverage for MinimizerBase.""" + +from dataclasses import FrozenInstanceError + +import numpy as np +import pytest + +from easydiffraction.analysis.minimizers import base as base_mod +from easydiffraction.analysis.minimizers.base import BOUNDARY_PROXIMITY_FRACTION +from easydiffraction.analysis.minimizers.base import MinimizerBase +from easydiffraction.analysis.minimizers.base import MinimizerFitOptions +from easydiffraction.utils.enums import VerbosityEnum + + +class _DummyParam: + """Minimal stand-in for an EasyDiffraction parameter.""" + + def __init__( + self, + value, + *, + fit_min=-np.inf, + fit_max=np.inf, + phys_lo=-np.inf, + phys_hi=np.inf, + name=None, + ): + self.value = value + self.fit_min = fit_min + self.fit_max = fit_max + self.unique_name = name or f'param_{value}' + self._phys_lo = phys_lo + self._phys_hi = phys_hi + self._outside_physical_limits = None + + def _physical_lower_bound(self): + return self._phys_lo + + def _physical_upper_bound(self): + return self._phys_hi + + +class _Minimal(MinimizerBase): + """Concrete minimizer with trivial abstract-method bodies.""" + + def _prepare_solver_args(self, parameters): + del parameters + return {} + + def _run_solver(self, objective_function, **kwargs): + del objective_function, kwargs + + def _sync_result_to_parameters(self, parameters, raw_result): + del parameters, raw_result + + def _check_success(self, raw_result): + del raw_result + return True + + +@pytest.fixture +def captured_warnings(monkeypatch): + """Capture messages passed to ``base.log.warning``.""" + messages = [] + + def _capture(msg): + messages.append(msg) + + monkeypatch.setattr(base_mod.log, 'warning', _capture) + return messages + + +# --------------------------------------------------------------------------- +# MinimizerFitOptions defaults +# --------------------------------------------------------------------------- + + +def test_fit_options_defaults(): + options = MinimizerFitOptions() + + assert options.finalize_tracking is True + assert options.use_physical_limits is False + assert options.random_seed is None + assert options.resume is False + assert options.extra_steps is None + + +def test_fit_options_is_frozen(): + options = MinimizerFitOptions() + + with pytest.raises(FrozenInstanceError): + options.resume = True + + +# --------------------------------------------------------------------------- +# Static helper hooks (default values) +# --------------------------------------------------------------------------- + + +def test_tracking_mode_default_is_fit(): + assert _Minimal()._tracking_mode() == 'fit' + + +def test_tracks_progress_via_solver_monitor_default_false(): + assert _Minimal()._tracks_progress_via_solver_monitor() is False + + +# --------------------------------------------------------------------------- +# _finalize_timing — idempotency / inactive guard (line 104) +# --------------------------------------------------------------------------- + + +def test_finalize_timing_noop_when_not_tracking(): + minimizer = _Minimal() + + # Tracking never started: should be a no-op and not raise. + assert minimizer._tracking_active is False + minimizer._finalize_timing() + assert minimizer._timing_finalized is False + + +def test_finalize_timing_noop_when_already_finalized(): + minimizer = _Minimal() + minimizer._tracking_active = True + minimizer._timing_finalized = True + + # Already finalized — must not call stop_timer again. + minimizer.tracker.stop_timer = lambda: (_ for _ in ()).throw( + AssertionError('stop_timer should not be called') + ) + minimizer._finalize_timing() + + +# --------------------------------------------------------------------------- +# _stop_tracking — inactive branch flushes deferred warnings (lines 113-114) +# --------------------------------------------------------------------------- + + +def test_stop_tracking_when_inactive_emits_deferred_warnings(captured_warnings): + minimizer = _Minimal() + minimizer._tracking_active = False + minimizer._deferred_warning_messages = ['deferred one', 'deferred two'] + + minimizer._stop_tracking() + + assert captured_warnings == ['deferred one', 'deferred two'] + assert minimizer._deferred_warning_messages == [] + + +# --------------------------------------------------------------------------- +# _warn_after_tracking — defer vs immediate (lines 123-127) +# --------------------------------------------------------------------------- + + +def test_warn_after_tracking_defers_while_active(captured_warnings): + minimizer = _Minimal() + minimizer._tracking_active = True + + minimizer._warn_after_tracking('hold this') + + # Deferred, not logged yet. + assert captured_warnings == [] + assert minimizer._deferred_warning_messages == ['hold this'] + + +def test_warn_after_tracking_logs_immediately_when_inactive(captured_warnings): + minimizer = _Minimal() + minimizer._tracking_active = False + + minimizer._warn_after_tracking('emit now') + + assert captured_warnings == ['emit now'] + assert minimizer._deferred_warning_messages == [] + + +# --------------------------------------------------------------------------- +# _emit_deferred_warnings — flush order (line 132) +# --------------------------------------------------------------------------- + + +def test_emit_deferred_warnings_flushes_in_fifo_order(captured_warnings): + minimizer = _Minimal() + minimizer._deferred_warning_messages = ['a', 'b', 'c'] + + minimizer._emit_deferred_warnings() + + assert captured_warnings == ['a', 'b', 'c'] + assert minimizer._deferred_warning_messages == [] + + +# --------------------------------------------------------------------------- +# _warn_boundary_parameters — unbounded/one-sided + bounded branches +# (lines 254, 261-273) +# --------------------------------------------------------------------------- + + +def test_warn_boundary_one_sided_lower_bound(captured_warnings): + # Finite lower bound, infinite upper => span is inf, not finite. + param = _DummyParam(10.0, fit_min=10.0, fit_max=np.inf, name='lo') + + MinimizerBase._warn_boundary_parameters([param]) + + assert len(captured_warnings) == 1 + assert 'lower' in captured_warnings[0] + assert 'fit_min' in captured_warnings[0] + + +def test_warn_boundary_one_sided_upper_bound(captured_warnings): + # Finite upper bound, infinite lower => span not finite. + param = _DummyParam(5.0, fit_min=-np.inf, fit_max=5.0, name='hi') + + MinimizerBase._warn_boundary_parameters([param]) + + assert len(captured_warnings) == 1 + assert 'upper' in captured_warnings[0] + assert 'fit_max' in captured_warnings[0] + + +def test_warn_boundary_one_sided_not_near_bound_is_silent(captured_warnings): + # Value far from the finite one-sided bound: no warning. + param = _DummyParam(100.0, fit_min=10.0, fit_max=np.inf) + + MinimizerBase._warn_boundary_parameters([param]) + + assert captured_warnings == [] + + +def test_warn_boundary_bounded_near_lower(captured_warnings): + # span = 100, tol = 1.0; value within tol of lower bound. + param = _DummyParam(0.5, fit_min=0.0, fit_max=100.0, name='lo') + + MinimizerBase._warn_boundary_parameters([param]) + + assert len(captured_warnings) == 1 + assert 'lower' in captured_warnings[0] + + +def test_warn_boundary_bounded_near_upper(captured_warnings): + param = _DummyParam(99.5, fit_min=0.0, fit_max=100.0, name='hi') + + MinimizerBase._warn_boundary_parameters([param]) + + assert len(captured_warnings) == 1 + assert 'upper' in captured_warnings[0] + + +def test_warn_boundary_bounded_centred_is_silent(captured_warnings): + # Value in the middle of [0, 100]; comfortably outside both tolerances. + param = _DummyParam(50.0, fit_min=0.0, fit_max=100.0) + + MinimizerBase._warn_boundary_parameters([param]) + + assert captured_warnings == [] + + +def test_warn_boundary_centred_param_followed_by_another(captured_warnings): + # Two bounded params: the first is centred (no warning, loop continues + # to the next iteration) and the second sits at its upper bound. + centred = _DummyParam(50.0, fit_min=0.0, fit_max=100.0) + edge = _DummyParam(99.9, fit_min=0.0, fit_max=100.0, name='edge') + + MinimizerBase._warn_boundary_parameters([centred, edge]) + + assert len(captured_warnings) == 1 + assert 'upper' in captured_warnings[0] + assert 'edge' in captured_warnings[0] + + +def test_warn_boundary_zero_span_is_silent(captured_warnings): + # fit_min == fit_max => span is finite but not > 0: neither branch runs. + param = _DummyParam(5.0, fit_min=5.0, fit_max=5.0) + + MinimizerBase._warn_boundary_parameters([param]) + + assert captured_warnings == [] + + +def test_warn_boundary_uses_proximity_fraction_constant(): + # Sanity-check the documented constant the tolerance derives from. + assert BOUNDARY_PROXIMITY_FRACTION == 0.01 + + +# --------------------------------------------------------------------------- +# _apply_physical_limits — replace infinite fit bounds (lines 292-300) +# --------------------------------------------------------------------------- + + +def test_apply_physical_limits_fills_both_infinite_bounds(): + param = _DummyParam(0.5, fit_min=-np.inf, fit_max=np.inf, phys_lo=0.0, phys_hi=1.0) + + MinimizerBase._apply_physical_limits([param]) + + assert param.fit_min == 0.0 + assert param.fit_max == 1.0 + + +def test_apply_physical_limits_skips_when_physical_bound_infinite(): + # Physical bounds are infinite => fit bounds remain unchanged. + param = _DummyParam(0.5, fit_min=-np.inf, fit_max=np.inf, phys_lo=-np.inf, phys_hi=np.inf) + + MinimizerBase._apply_physical_limits([param]) + + assert param.fit_min == -np.inf + assert param.fit_max == np.inf + + +def test_apply_physical_limits_leaves_finite_fit_bounds_untouched(): + param = _DummyParam(0.5, fit_min=-2.0, fit_max=2.0, phys_lo=0.0, phys_hi=1.0) + + MinimizerBase._apply_physical_limits([param]) + + # fit bounds were already finite => physical bounds are not applied. + assert param.fit_min == -2.0 + assert param.fit_max == 2.0 + + +# --------------------------------------------------------------------------- +# _warn_physical_limit_violations — below/above + flag (lines 320-330) +# --------------------------------------------------------------------------- + + +def test_warn_physical_violation_below_lower(captured_warnings): + param = _DummyParam(-1.0, phys_lo=0.0, phys_hi=10.0, name='below') + + MinimizerBase._warn_physical_limit_violations([param]) + + assert len(captured_warnings) == 1 + assert 'below' in captured_warnings[0] + assert param._outside_physical_limits is True + + +def test_warn_physical_violation_above_upper(captured_warnings): + param = _DummyParam(11.0, phys_lo=0.0, phys_hi=10.0, name='above') + + MinimizerBase._warn_physical_limit_violations([param]) + + assert len(captured_warnings) == 1 + assert 'above' in captured_warnings[0] + assert param._outside_physical_limits is True + + +def test_warn_physical_violation_within_limits_clears_flag(captured_warnings): + param = _DummyParam(5.0, phys_lo=0.0, phys_hi=10.0) + param._outside_physical_limits = True # stale True from a prior run + + MinimizerBase._warn_physical_limit_violations([param]) + + assert captured_warnings == [] + assert param._outside_physical_limits is False + + +def test_warn_physical_violation_infinite_limits_no_warning(captured_warnings): + param = _DummyParam(1e9, phys_lo=-np.inf, phys_hi=np.inf) + + MinimizerBase._warn_physical_limit_violations([param]) + + assert captured_warnings == [] + assert param._outside_physical_limits is False + + +# --------------------------------------------------------------------------- +# _resolve_random_seed — None passthrough vs unsupported raise (lines 356-362) +# --------------------------------------------------------------------------- + + +def test_resolve_random_seed_none_returns_none(): + minimizer = _Minimal() + + assert minimizer._resolve_random_seed(None) is None + assert minimizer._resolved_random_seed is None + + +def test_resolve_random_seed_unsupported_raises_with_name(): + minimizer = _Minimal(name='my-minimizer') + + with pytest.raises(ValueError, match="'my-minimizer' does not support random_seed"): + minimizer._resolve_random_seed(7) + + +def test_resolve_random_seed_unsupported_falls_back_to_class_name(): + minimizer = _Minimal() # no name => uses class name + + with pytest.raises(ValueError, match='_Minimal'): + minimizer._resolve_random_seed(7) + + +# --------------------------------------------------------------------------- +# fit — resume + physical-limit + random-seed + method-suffix branches +# (lines 401-403, 406, 411->414, 419) +# --------------------------------------------------------------------------- + + +def test_fit_resume_unsupported_raises(): + minimizer = _Minimal(name='dummy') + + with pytest.raises(NotImplementedError, match="'dummy' does not support resume"): + minimizer.fit( + parameters=[], + objective_function=lambda _: np.array([0.0]), + options=MinimizerFitOptions(resume=True), + ) + + +def test_fit_applies_physical_limits_when_requested(): + applied = [] + + class M(_Minimal): + @staticmethod + def _apply_physical_limits(parameters): + applied.append(parameters) + + def _prepare_solver_args(self, parameters): + del parameters + return {} + + def _run_solver(self, objective_function, **kwargs): + del objective_function, kwargs + return object() + + def _sync_result_to_parameters(self, parameters, raw_result): + del parameters, raw_result + + def _check_success(self, raw_result): + del raw_result + return True + + minimizer = M(name='m') + params = [_DummyParam(1.0)] + minimizer.fit( + parameters=params, + objective_function=lambda _: np.array([0.0]), + verbosity=VerbosityEnum.SILENT, + options=MinimizerFitOptions(use_physical_limits=True), + ) + + assert applied == [params] + + +def test_fit_injects_resolved_random_seed_into_solver_args(): + seen = {} + + class M(_Minimal): + def _resolve_random_seed(self, random_seed): + # Pretend this minimizer supports a seed. + self._resolved_random_seed = random_seed + return random_seed + + def _prepare_solver_args(self, parameters): + del parameters + return {} + + def _run_solver(self, objective_function, **kwargs): + del objective_function + seen.update(kwargs) + return object() + + def _sync_result_to_parameters(self, parameters, raw_result): + del parameters, raw_result + + def _check_success(self, raw_result): + del raw_result + return True + + minimizer = M(name='m') + minimizer.fit( + parameters=[_DummyParam(1.0)], + objective_function=lambda _: np.array([0.0]), + verbosity=VerbosityEnum.SILENT, + options=MinimizerFitOptions(random_seed=123), + ) + + assert seen.get('random_seed') == 123 + + +def test_fit_appends_method_suffix_to_name(): + captured_names = [] + + class M(_Minimal): + def _start_tracking(self, minimizer_name, verbosity=VerbosityEnum.FULL): + captured_names.append(minimizer_name) + + def _stop_tracking(self): + pass + + def _prepare_solver_args(self, parameters): + del parameters + return {} + + def _run_solver(self, objective_function, **kwargs): + del objective_function, kwargs + return object() + + def _sync_result_to_parameters(self, parameters, raw_result): + del parameters, raw_result + + def _check_success(self, raw_result): + del raw_result + return True + + minimizer = M(name='Solver', method='leastsq') + minimizer.fit( + parameters=[_DummyParam(1.0)], + objective_function=lambda _: np.array([0.0]), + ) + + assert captured_names == ['Solver (leastsq)'] + + +def test_fit_does_not_double_append_method_suffix(): + captured_names = [] + + class M(_Minimal): + def _start_tracking(self, minimizer_name, verbosity=VerbosityEnum.FULL): + captured_names.append(minimizer_name) + + def _stop_tracking(self): + pass + + def _prepare_solver_args(self, parameters): + del parameters + return {} + + def _run_solver(self, objective_function, **kwargs): + del objective_function, kwargs + return object() + + def _sync_result_to_parameters(self, parameters, raw_result): + del parameters, raw_result + + def _check_success(self, raw_result): + del raw_result + return True + + # Name already contains '(leastsq)' => the suffix is not re-appended. + minimizer = M(name='Solver (leastsq)', method='leastsq') + minimizer.fit( + parameters=[_DummyParam(1.0)], + objective_function=lambda _: np.array([0.0]), + ) + + assert captured_names == ['Solver (leastsq)'] + + +def test_fit_uses_default_name_when_unnamed(): + captured_names = [] + + class M(_Minimal): + def _start_tracking(self, minimizer_name, verbosity=VerbosityEnum.FULL): + captured_names.append(minimizer_name) + + def _stop_tracking(self): + pass + + def _prepare_solver_args(self, parameters): + del parameters + return {} + + def _run_solver(self, objective_function, **kwargs): + del objective_function, kwargs + return object() + + def _sync_result_to_parameters(self, parameters, raw_result): + del parameters, raw_result + + def _check_success(self, raw_result): + del raw_result + return True + + minimizer = M() # no name, no method + minimizer.fit( + parameters=[_DummyParam(1.0)], + objective_function=lambda _: np.array([0.0]), + ) + + assert captured_names == ['Unnamed Minimizer'] diff --git a/tests/unit/easydiffraction/analysis/test_analysis_coverage.py b/tests/unit/easydiffraction/analysis/test_analysis_coverage.py index 70c2f8e28..da7e68173 100644 --- a/tests/unit/easydiffraction/analysis/test_analysis_coverage.py +++ b/tests/unit/easydiffraction/analysis/test_analysis_coverage.py @@ -7,6 +7,41 @@ import numpy as np +def _make_parameter(name, value): + from easydiffraction.core.validation import AttributeSpec + from easydiffraction.core.variable import Parameter + from easydiffraction.io.cif.handler import CifHandler + + return Parameter( + name=name, + value_spec=AttributeSpec(default=value), + cif_handler=CifHandler(names=[f'_{name}.value']), + ) + + +def _make_project_with_parameters(structure_params, experiment_params=None): + experiment_params = experiment_params or [] + + class StructureColl: + def __init__(self, params): + self.parameters = list(params) + + class ExperimentColl: + def __init__(self, params): + self.parameters = list(params) + self.names = [] + + def values(self): + return [] + + return SimpleNamespace( + structures=StructureColl(structure_params), + experiments=ExperimentColl(experiment_params), + info=SimpleNamespace(path=None), + _varname='proj', + ) + + def _make_project(): class ExpCol: def __init__(self): @@ -308,3 +343,1409 @@ def __getitem__(self, name): sidecar['predictive_datasets']['hrpt']['best_sample_prediction'], np.asarray([3.0, 4.0], dtype=float), ) + + +# ------------------------------------------------------------------ +# Module-level helpers: _parameter_display_units / _int_or_none +# ------------------------------------------------------------------ + + +class TestModuleHelpers: + def test_parameter_display_units_prefers_resolver(self): + from easydiffraction.analysis.analysis import _parameter_display_units + + class WithResolver: + def resolve_display_units(self, context): + assert context == 'gui' + return 'Å' + + assert _parameter_display_units(WithResolver()) == 'Å' + + def test_parameter_display_units_none_units_maps_to_empty(self): + from easydiffraction.analysis.analysis import _parameter_display_units + + param = SimpleNamespace(units='none') + assert _parameter_display_units(param) == '' + + def test_parameter_display_units_passes_through_plain_units(self): + from easydiffraction.analysis.analysis import _parameter_display_units + + param = SimpleNamespace(units='deg') + assert _parameter_display_units(param) == 'deg' + + def test_parameter_display_units_missing_attribute_falls_back(self): + from easydiffraction.analysis.analysis import _parameter_display_units + + assert _parameter_display_units(object()) == 'N/A' + + def test_int_or_none_passes_none_through(self): + from easydiffraction.analysis.analysis import _int_or_none + + assert _int_or_none(None) is None + + def test_int_or_none_coerces_value(self): + from easydiffraction.analysis.analysis import _int_or_none + + assert _int_or_none(3.9) == 3 + assert _int_or_none('5') == 5 + + +# ------------------------------------------------------------------ +# Static numeric helpers +# ------------------------------------------------------------------ + + +class TestNumericStatics: + def test_finite_float_handles_none_and_nonfinite(self): + from easydiffraction.analysis.analysis import Analysis + + assert Analysis._finite_float(None) is None + assert Analysis._finite_float('not-a-number') is None + assert Analysis._finite_float(float('inf')) is None + assert Analysis._finite_float(float('nan')) is None + assert Analysis._finite_float('2.5') == 2.5 + assert Analysis._finite_float(3) == 3.0 + + def test_finite_metric_filters_nonfinite(self): + from easydiffraction.analysis.analysis import Analysis + + assert Analysis._finite_metric(1.5) == 1.5 + assert Analysis._finite_metric(float('nan')) is None + assert Analysis._finite_metric(float('inf')) is None + + def test_int_sampler_setting_defaults_and_none(self): + from easydiffraction.analysis.analysis import Analysis + + assert Analysis._int_sampler_setting({}, 'missing') == 0 + assert Analysis._int_sampler_setting({'nsteps': None}, 'nsteps') == 0 + assert Analysis._int_sampler_setting({'nsteps': 100}, 'nsteps') == 100 + + def test_sampler_sample_count_uses_steps_and_population(self): + from easydiffraction.analysis.analysis import Analysis + + emcee_settings = {'nsteps': 10, 'nwalkers': 4} + assert Analysis._sampler_sample_count(emcee_settings, n_parameters=3) == 120 + + dream_settings = {'steps': 5, 'pop': 2} + assert Analysis._sampler_sample_count(dream_settings, n_parameters=3) == 30 + + def test_sampler_sample_count_clamps_negatives_to_zero(self): + from easydiffraction.analysis.analysis import Analysis + + assert Analysis._sampler_sample_count({'steps': -5, 'pop': 4}, n_parameters=3) == 0 + assert Analysis._sampler_sample_count({}, n_parameters=3) == 0 + + +# ------------------------------------------------------------------ +# Software-provenance helpers +# ------------------------------------------------------------------ + + +class TestSoftwareValues: + def test_type_info_tag_reads_enum_value(self): + from easydiffraction.analysis.analysis import Analysis + + engine = SimpleNamespace(type_info=SimpleNamespace(tag=SimpleNamespace(value='cryspy'))) + assert Analysis._type_info_tag(engine) == 'cryspy' + + def test_type_info_tag_reads_plain_string(self): + from easydiffraction.analysis.analysis import Analysis + + engine = SimpleNamespace(type_info=SimpleNamespace(tag='lmfit (leastsq)')) + assert Analysis._type_info_tag(engine) == 'lmfit (leastsq)' + + def test_type_info_tag_missing_returns_empty(self): + from easydiffraction.analysis.analysis import Analysis + + assert Analysis._type_info_tag(object()) == '' + + def test_software_version_unknown_engine_is_none(self): + from easydiffraction.analysis.analysis import Analysis + + assert Analysis._software_version('not-a-real-engine') is None + + def test_software_version_known_engine_delegates(self, monkeypatch): + import easydiffraction.analysis.analysis as mod + from easydiffraction.analysis.analysis import Analysis + + monkeypatch.setattr(mod, 'package_version', lambda name: f'{name}-9.9') + # 'pdffit' maps to the 'diffpy.pdffit2' package name. + assert Analysis._software_version('pdffit') == 'diffpy.pdffit2-9.9' + assert Analysis._software_version('lmfit') == 'lmfit-9.9' + + def test_software_package_name_strips_minimizer_settings(self): + from easydiffraction.analysis.analysis import Analysis + + a = Analysis(project=_make_project()) + engine = SimpleNamespace(type_info=SimpleNamespace(tag='lmfit (leastsq)')) + assert a._software_package_name(engine) == 'lmfit' + + def test_software_values_returns_name_version_url(self, monkeypatch): + import easydiffraction.analysis.analysis as mod + from easydiffraction.analysis.analysis import Analysis + + monkeypatch.setattr(mod, 'package_version', lambda name: '1.2.3') + a = Analysis(project=_make_project()) + engine = SimpleNamespace( + type_info=SimpleNamespace(tag='lmfit (leastsq)'), + url='https://lmfit.example', + ) + assert a._software_values(engine) == ('lmfit', '1.2.3', 'https://lmfit.example') + + def test_combine_software_values_joins_unique_sorted(self): + from easydiffraction.analysis.analysis import Analysis + + values = [ + ('cryspy', '1.0', 'u1'), + ('cryspy', '1.0', 'u1'), + ('pdffit', '2.0', 'u2'), + ] + name, version, url = Analysis._combine_software_values(values) + assert name == 'cryspy, pdffit' + assert version == '1.0, 2.0' + assert url == 'u1, u2' + + def test_combine_software_values_empty_returns_none_triple(self): + from easydiffraction.analysis.analysis import Analysis + + assert Analysis._combine_software_values([]) == (None, None, None) + + def test_combine_software_values_all_blank_fields_collapse_to_none(self): + from easydiffraction.analysis.analysis import Analysis + + name, version, url = Analysis._combine_software_values([('engine', None, None)]) + assert name == 'engine' + assert version is None + assert url is None + + def test_set_software_role_assigns_fields(self): + from easydiffraction.analysis.analysis import Analysis + + a = Analysis(project=_make_project()) + Analysis._set_software_role(a.software.framework, ('EasyDiffraction', '9.9', 'url')) + assert a.software.framework.name.value == 'EasyDiffraction' + assert a.software.framework.version.value == '9.9' + assert a.software.framework.url.value == 'url' + + def test_has_software_provenance_tracks_stamping(self): + from easydiffraction.analysis.analysis import Analysis + + a = Analysis(project=_make_project()) + assert a._has_software_provenance() is False + a.software.framework.name = 'EasyDiffraction' + assert a._has_software_provenance() is True + + +# ------------------------------------------------------------------ +# IUCr / R-factor metric helpers (engine-free, array-driven) +# ------------------------------------------------------------------ + + +def _intensity_experiment(*, meas, calc, su): + data = SimpleNamespace( + intensity_meas=np.asarray(meas, dtype=float), + intensity_calc=np.asarray(calc, dtype=float), + intensity_meas_su=np.asarray(su, dtype=float), + ) + return SimpleNamespace(data=data) + + +class TestMetricHelpers: + def test_fit_data_point_count_sums_measured_sizes(self): + from easydiffraction.analysis.analysis import Analysis + + experiments = [ + _intensity_experiment(meas=[1.0, 2.0, 3.0], calc=[1, 2, 3], su=[1, 1, 1]), + _intensity_experiment(meas=[4.0, 5.0], calc=[4, 5], su=[1, 1]), + ] + assert Analysis._fit_data_point_count(experiments) == 5 + + def test_fit_intensity_arrays_drops_nonfinite_and_nonpositive_su(self): + from easydiffraction.analysis.analysis import Analysis + + experiment = _intensity_experiment( + meas=[1.0, 2.0, np.nan, 4.0, 5.0], + calc=[1.1, 2.1, 3.0, np.inf, 5.5], + su=[0.1, 0.0, 0.1, 0.1, 0.2], + ) + observed, calculated, uncertainties = Analysis._fit_intensity_arrays([experiment]) + # Only rows 0 and 4 survive: row1 su<=0, row2 meas nan, row3 calc inf. + assert np.allclose(observed, [1.0, 5.0]) + assert np.allclose(calculated, [1.1, 5.5]) + assert np.allclose(uncertainties, [0.1, 0.2]) + + def test_r_factor_or_none_empty_returns_none(self): + from easydiffraction.analysis.analysis import Analysis + + empty = np.asarray([], dtype=float) + assert Analysis._r_factor_or_none(empty, empty) is None + + def test_r_factor_or_none_computes_value(self): + from easydiffraction.analysis.analysis import Analysis + + observed = np.asarray([10.0, 10.0], dtype=float) + calculated = np.asarray([9.0, 11.0], dtype=float) + # sum|obs-calc| / sum|obs| = 2 / 20 = 0.1 + assert Analysis._r_factor_or_none(observed, calculated) == 0.1 + + def test_weighted_r_factor_or_none_empty_and_zero_denominator(self): + from easydiffraction.analysis.analysis import Analysis + + empty = np.asarray([], dtype=float) + assert Analysis._weighted_r_factor_or_none(empty, empty, empty) is None + + observed = np.asarray([0.0, 0.0], dtype=float) + calculated = np.asarray([0.0, 0.0], dtype=float) + uncertainties = np.asarray([1.0, 1.0], dtype=float) + assert Analysis._weighted_r_factor_or_none(observed, calculated, uncertainties) is None + + def test_weighted_r_factor_or_none_computes_value(self): + from easydiffraction.analysis.analysis import Analysis + + observed = np.asarray([10.0, 10.0], dtype=float) + calculated = np.asarray([10.0, 10.0], dtype=float) + uncertainties = np.asarray([1.0, 1.0], dtype=float) + assert Analysis._weighted_r_factor_or_none(observed, calculated, uncertainties) == 0.0 + + def test_expected_weighted_r_factor_guards(self): + from easydiffraction.analysis.analysis import Analysis + + observed = np.asarray([10.0, 10.0], dtype=float) + uncertainties = np.asarray([1.0, 1.0], dtype=float) + # Empty observations -> None. + empty = np.asarray([], dtype=float) + assert Analysis._expected_weighted_r_factor(empty, empty, 1) is None + # Non-positive degrees of freedom -> None. + assert Analysis._expected_weighted_r_factor(observed, uncertainties, 0) is None + # Valid: sqrt(dof / sum(w * obs^2)) = sqrt(1 / 200). + value = Analysis._expected_weighted_r_factor(observed, uncertainties, 1) + assert value is not None + assert np.isclose(value, np.sqrt(1.0 / 200.0)) + + def test_gt_observation_mask_uses_three_sigma(self): + from easydiffraction.analysis.analysis import Analysis + + observed = np.asarray([2.0, 4.0], dtype=float) + uncertainties = np.asarray([1.0, 1.0], dtype=float) + mask = Analysis._gt_observation_mask(observed, uncertainties) + assert mask.tolist() == [False, True] + + +# ------------------------------------------------------------------ +# Powder / category-type / reflection helpers +# ------------------------------------------------------------------ + + +class TestExperimentClassificationHelpers: + def test_is_powder_fit_detects_powder(self): + from easydiffraction.analysis.analysis import Analysis + from easydiffraction.datablocks.experiment.item.enums import SampleFormEnum + + powder = SimpleNamespace( + type=SimpleNamespace(sample_form=SimpleNamespace(value=SampleFormEnum.POWDER.value)) + ) + single = SimpleNamespace( + type=SimpleNamespace( + sample_form=SimpleNamespace(value=SampleFormEnum.SINGLE_CRYSTAL.value) + ) + ) + assert Analysis._is_powder_fit([single, powder]) is True + assert Analysis._is_powder_fit([single]) is False + + def test_unique_category_type_names_dedupes_and_handles_missing(self): + from easydiffraction.analysis.analysis import Analysis + + e1 = SimpleNamespace(peak=SimpleNamespace(type=SimpleNamespace(value='gaussian'))) + e2 = SimpleNamespace(peak=SimpleNamespace(type=SimpleNamespace(value='gaussian'))) + e3 = SimpleNamespace(peak=SimpleNamespace(type=SimpleNamespace(value='lorentzian'))) + e4 = SimpleNamespace() # no 'peak' attribute -> skipped + names = Analysis._unique_category_type_names([e1, e2, e3, e4], 'peak') + assert names == 'gaussian, lorentzian' + + def test_unique_category_type_names_all_missing_returns_none(self): + from easydiffraction.analysis.analysis import Analysis + + assert Analysis._unique_category_type_names([SimpleNamespace()], 'peak') is None + + def test_reflection_counts_no_reflections_returns_none_pair(self): + from easydiffraction.analysis.analysis import Analysis + + # Empty refln and missing refln both yield no reflections. + empty_refln = [] + e1 = SimpleNamespace(refln=empty_refln) + e2 = SimpleNamespace() + assert Analysis._reflection_counts([e1, e2]) == (None, None) + + def test_reflection_counts_thresholds_observations(self): + from easydiffraction.analysis.analysis import Analysis + + class Refln: + def __init__(self, meas, su): + self.intensity_meas = meas + self.intensity_meas_su = su + + def __len__(self): + return len(self.intensity_meas) + + # meas > 3 * su for index 1 only (10 > 3); index 0: 2 < 3. + refln = Refln([2.0, 10.0], [1.0, 1.0]) + experiment = SimpleNamespace(refln=refln) + total, greater_than = Analysis._reflection_counts([experiment]) + assert total == 2 + assert greater_than == 1 + + def test_reflection_counts_shape_mismatch_skips_threshold(self): + from easydiffraction.analysis.analysis import Analysis + + class Refln: + def __init__(self, meas, su): + self.intensity_meas = meas + self.intensity_meas_su = su + + def __len__(self): + return len(self.intensity_meas) + + refln = Refln([2.0, 10.0, 20.0], [1.0, 1.0]) # mismatched shapes + experiment = SimpleNamespace(refln=refln) + total, greater_than = Analysis._reflection_counts([experiment]) + assert total == 3 + assert greater_than == 0 + + +# ------------------------------------------------------------------ +# Shift-over-su and covariance/correlation helpers +# ------------------------------------------------------------------ + + +def _shift_param(*, value, start, uncertainty): + return SimpleNamespace(value=value, _fit_start_value=start, uncertainty=uncertainty) + + +class TestShiftAndCovariance: + def test_shift_over_su_values_skips_incomplete_and_nonpositive(self): + from easydiffraction.analysis.analysis import Analysis + + params = [ + _shift_param(value=4.2, start=4.0, uncertainty=0.1), # |0.2|/0.1 = 2.0 + _shift_param(value=4.2, start=None, uncertainty=0.1), # no start + _shift_param(value=4.2, start=4.0, uncertainty=None), # no uncertainty + _shift_param(value=4.2, start=4.0, uncertainty=0.0), # su <= 0 + ] + values = Analysis._shift_over_su_values(params) + assert np.allclose(values, [2.0]) + + def test_shift_over_su_summary_empty_returns_none_pair(self): + from easydiffraction.analysis.analysis import Analysis + + assert Analysis._shift_over_su_summary([]) == (None, None) + + def test_shift_over_su_summary_max_and_mean(self): + from easydiffraction.analysis.analysis import Analysis + + params = [ + _shift_param(value=4.2, start=4.0, uncertainty=0.1), # 2.0 + _shift_param(value=4.0, start=4.4, uncertainty=0.1), # 4.0 + ] + shift_max, shift_mean = Analysis._shift_over_su_summary(params) + assert np.isclose(shift_max, 4.0) + assert np.isclose(shift_mean, 3.0) + + def test_resolve_covariance_matrix_reads_covar_attribute(self): + from easydiffraction.analysis.analysis import Analysis + from easydiffraction.analysis.fit_helpers.reporting import FitResults + + covar = np.asarray([[1.0, 0.5], [0.5, 2.0]], dtype=float) + results = FitResults(success=True, engine_result=SimpleNamespace(covar=covar)) + resolved = Analysis._resolve_covariance_matrix(results) + assert np.allclose(resolved, covar) + + def test_resolve_covariance_matrix_rejects_nonsquare(self): + from easydiffraction.analysis.analysis import Analysis + from easydiffraction.analysis.fit_helpers.reporting import FitResults + + nonsquare = np.asarray([[1.0, 0.5, 0.2]], dtype=float) + results = FitResults(success=True, engine_result=SimpleNamespace(covar=nonsquare)) + assert Analysis._resolve_covariance_matrix(results) is None + + def test_resolve_covariance_matrix_missing_returns_none(self): + from easydiffraction.analysis.analysis import Analysis + from easydiffraction.analysis.fit_helpers.reporting import FitResults + + results = FitResults(success=True, engine_result=SimpleNamespace()) + assert Analysis._resolve_covariance_matrix(results) is None + + def test_correlation_matrix_from_covariance_normalizes(self): + from easydiffraction.analysis.analysis import Analysis + + covariance = np.asarray([[4.0, 2.0], [2.0, 9.0]], dtype=float) + correlation = Analysis._correlation_matrix_from_covariance(covariance) + assert correlation is not None + # off-diagonal = 2 / (2 * 3) = 1/3 + assert np.isclose(correlation[0, 1], 2.0 / 6.0) + assert np.allclose(np.diag(correlation), [1.0, 1.0]) + + def test_correlation_matrix_from_covariance_nonpositive_diag_returns_none(self): + from easydiffraction.analysis.analysis import Analysis + + covariance = np.asarray([[0.0, 0.0], [0.0, 1.0]], dtype=float) + assert Analysis._correlation_matrix_from_covariance(covariance) is None + + def test_resolve_objective_value_passthrough_and_none(self): + from easydiffraction.analysis.analysis import Analysis + from easydiffraction.analysis.fit_helpers.reporting import FitResults + + results = FitResults(success=True) + results.chi_square = None + assert Analysis._resolve_objective_value(results) is None + results.chi_square = 12.5 + assert Analysis._resolve_objective_value(results) == 12.5 + + +# ------------------------------------------------------------------ +# Correlation projection storage +# ------------------------------------------------------------------ + + +class TestCorrelationProjection: + def test_store_correlation_projection_single_name_is_noop(self): + from easydiffraction.analysis.analysis import Analysis + from easydiffraction.analysis.enums import FitCorrelationSourceEnum + + a = Analysis(project=_make_project()) + a._store_correlation_projection( + unique_names=['only'], + correlation_matrix=np.asarray([[1.0]], dtype=float), + source_kind=FitCorrelationSourceEnum.DETERMINISTIC, + ) + assert len(a.fit_parameter_correlations) == 0 + + def test_store_correlation_projection_shape_mismatch_is_noop(self): + from easydiffraction.analysis.analysis import Analysis + from easydiffraction.analysis.enums import FitCorrelationSourceEnum + + a = Analysis(project=_make_project()) + a._store_correlation_projection( + unique_names=['a', 'b'], + correlation_matrix=np.asarray([[1.0]], dtype=float), + source_kind=FitCorrelationSourceEnum.DETERMINISTIC, + ) + assert len(a.fit_parameter_correlations) == 0 + + def test_store_correlation_projection_writes_upper_triangle_clipped(self): + from easydiffraction.analysis.analysis import Analysis + from easydiffraction.analysis.enums import FitCorrelationSourceEnum + + a = Analysis(project=_make_project()) + # Off-diagonal above 1.0 must be clipped to 1.0; non-finite skipped. + matrix = np.asarray( + [ + [1.0, 1.5, np.nan], + [1.5, 1.0, 0.4], + [np.nan, 0.4, 1.0], + ], + dtype=float, + ) + a._store_correlation_projection( + unique_names=['p1', 'p2', 'p3'], + correlation_matrix=matrix, + source_kind=FitCorrelationSourceEnum.POSTERIOR, + ) + # Pairs: (p1,p2) clipped to 1.0, (p1,p3) nan -> skipped, (p2,p3)=0.4. + assert len(a.fit_parameter_correlations) == 2 + correlations = sorted(row.correlation.value for row in a.fit_parameter_correlations) + assert np.allclose(correlations, [0.4, 1.0]) + + +# ------------------------------------------------------------------ +# Predictive dataset payload +# ------------------------------------------------------------------ + + +class TestPredictiveDatasetPayload: + def test_payload_includes_only_present_optional_arrays(self): + from easydiffraction.analysis.analysis import Analysis + from easydiffraction.analysis.fit_helpers.bayesian import PosteriorPredictiveSummary + + summary = PosteriorPredictiveSummary( + experiment_name='hrpt', + x_axis_name='two_theta', + x=np.asarray([1.0, 2.0], dtype=float), + best_sample_prediction=np.asarray([3.0, 4.0], dtype=float), + ) + payload = Analysis._predictive_dataset_payload(summary) + assert payload['x_axis_name'] == 'two_theta' + assert np.allclose(payload['x'], [1.0, 2.0]) + assert np.allclose(payload['best_sample_prediction'], [3.0, 4.0]) + # Optional arrays not provided are omitted. + for optional in ('lower_95', 'upper_95', 'lower_68', 'upper_68', 'draws'): + assert optional not in payload + + def test_payload_includes_all_optional_arrays_when_present(self): + from easydiffraction.analysis.analysis import Analysis + from easydiffraction.analysis.fit_helpers.bayesian import PosteriorPredictiveSummary + + summary = PosteriorPredictiveSummary( + experiment_name='hrpt', + x_axis_name='two_theta', + x=np.asarray([1.0], dtype=float), + best_sample_prediction=np.asarray([3.0], dtype=float), + lower_95=np.asarray([2.5], dtype=float), + upper_95=np.asarray([3.5], dtype=float), + lower_68=np.asarray([2.8], dtype=float), + upper_68=np.asarray([3.2], dtype=float), + draws=np.asarray([[3.0, 3.1]], dtype=float), + ) + payload = Analysis._predictive_dataset_payload(summary) + for optional in ('lower_95', 'upper_95', 'lower_68', 'upper_68', 'draws'): + assert optional in payload + + +# ------------------------------------------------------------------ +# Posterior contour / pair-metadata statics +# ------------------------------------------------------------------ + + +class TestPosteriorPairStatics: + def test_contour_levels_scale_with_density_max(self): + from easydiffraction.analysis.analysis import Analysis + + density = np.asarray([0.0, 2.0, 1.0], dtype=float) + levels = Analysis._posterior_pair_contour_levels(density) + assert np.allclose( + levels, + 2.0 * np.asarray([0.20, 0.35, 0.50, 0.65, 0.80, 0.95]), + ) + + def test_contour_levels_nonpositive_max_returns_empty(self): + from easydiffraction.analysis.analysis import Analysis + + density = np.asarray([0.0, 0.0], dtype=float) + assert Analysis._posterior_pair_contour_levels(density).size == 0 + + def test_ordered_pair_metadata_sorts_by_name(self): + from easydiffraction.analysis.analysis import Analysis + + names = ['z_param', 'a_param'] + x_index, y_index, x_name, y_name = Analysis._ordered_pair_metadata(names, 0, 1) + # z_param > a_param so the pair is swapped into name order. + assert (x_index, y_index) == (1, 0) + assert (x_name, y_name) == ('a_param', 'z_param') + + def test_ordered_pair_metadata_keeps_order_when_already_sorted(self): + from easydiffraction.analysis.analysis import Analysis + + names = ['a_param', 'z_param'] + x_index, y_index, x_name, y_name = Analysis._ordered_pair_metadata(names, 0, 1) + assert (x_index, y_index) == (0, 1) + assert (x_name, y_name) == ('a_param', 'z_param') + + +# ------------------------------------------------------------------ +# Bayesian restore statics +# ------------------------------------------------------------------ + + +class TestBayesianRestoreStatics: + def test_restored_bayesian_converged_true_when_thresholds_met(self): + from easydiffraction.analysis.analysis import Analysis + from easydiffraction.analysis.fit_helpers.bayesian import ESS_BULK_CONVERGENCE_THRESHOLD + from easydiffraction.analysis.fit_helpers.bayesian import R_HAT_CONVERGENCE_THRESHOLD + + assert ( + Analysis._restored_bayesian_converged( + max_r_hat=R_HAT_CONVERGENCE_THRESHOLD, + min_ess_bulk=ESS_BULK_CONVERGENCE_THRESHOLD, + ) + is True + ) + + def test_restored_bayesian_converged_false_when_r_hat_high(self): + from easydiffraction.analysis.analysis import Analysis + from easydiffraction.analysis.fit_helpers.bayesian import ESS_BULK_CONVERGENCE_THRESHOLD + + assert ( + Analysis._restored_bayesian_converged( + max_r_hat=10.0, + min_ess_bulk=ESS_BULK_CONVERGENCE_THRESHOLD, + ) + is False + ) + + def test_restored_bayesian_converged_false_when_value_missing(self): + from easydiffraction.analysis.analysis import Analysis + + assert Analysis._restored_bayesian_converged(max_r_hat=None, min_ess_bulk=1000.0) is False + assert Analysis._restored_bayesian_converged(max_r_hat=1.0, min_ess_bulk=None) is False + + def test_bayesian_result_random_seed_handles_none(self): + from easydiffraction.analysis.analysis import Analysis + from easydiffraction.analysis.fit_helpers.bayesian import BayesianFitResults + + results = BayesianFitResults(success=True, sampler_settings={'random_seed': None}) + assert Analysis._bayesian_result_random_seed(results) is None + + results = BayesianFitResults(success=True, sampler_settings={'random_seed': 7}) + assert Analysis._bayesian_result_random_seed(results) == 7 + + def test_restored_bayesian_random_seed_prefers_persisted_value(self): + from easydiffraction.analysis.analysis import Analysis + from easydiffraction.analysis.categories.fit_result.bayesian import BayesianFitResult + + a = Analysis(project=_make_project()) + a._fit_result._parent = None + a._fit_result = BayesianFitResult() + a._fit_result._parent = a + a.fit_result._set_resolved_random_seed(99) + assert a._restored_bayesian_random_seed({'random_seed': 1}) == 99 + + def test_restored_bayesian_random_seed_falls_back_to_settings(self): + from easydiffraction.analysis.analysis import Analysis + from easydiffraction.analysis.categories.fit_result.bayesian import BayesianFitResult + + a = Analysis(project=_make_project()) + a._fit_result._parent = None + a._fit_result = BayesianFitResult() + a._fit_result._parent = a + # resolved_random_seed defaults to None -> use sampler settings. + assert a._restored_bayesian_random_seed({'random_seed': 5}) == 5 + + +# ------------------------------------------------------------------ +# Fit-request validation +# ------------------------------------------------------------------ + + +class TestFitRequestValidation: + def test_validate_resume_extra_steps_rejects_bad_values(self): + import pytest + + from easydiffraction.analysis.analysis import Analysis + + for bad in (None, True, 0, -1, 2.5, 'x'): + with pytest.raises(ValueError, match='positive integer'): + Analysis._validate_resume_extra_steps(bad) + + def test_validate_resume_extra_steps_accepts_positive_int(self): + from easydiffraction.analysis.analysis import Analysis + + assert Analysis._validate_resume_extra_steps(3) == 3 + + def test_validate_fit_request_extra_steps_without_resume_raises(self): + import pytest + + from easydiffraction.analysis.analysis import Analysis + from easydiffraction.analysis.enums import FitModeEnum + + a = Analysis(project=_make_project()) + with pytest.raises(ValueError, match='extra_steps is only valid when resume=True'): + a._validate_fit_request(mode=FitModeEnum.SINGLE, resume=False, extra_steps=5) + + def test_validate_fit_request_resume_requires_single_mode(self): + import pytest + + from easydiffraction.analysis.analysis import Analysis + from easydiffraction.analysis.enums import FitModeEnum + + a = Analysis(project=_make_project()) + a.minimizer.type = 'emcee' + with pytest.raises(ValueError, match='single fit mode only'): + a._validate_fit_request(mode=FitModeEnum.JOINT, resume=True, extra_steps=None) + + def test_validate_fit_request_resume_requires_emcee(self): + import pytest + + from easydiffraction.analysis.analysis import Analysis + from easydiffraction.analysis.enums import FitModeEnum + + a = Analysis(project=_make_project()) # default lmfit minimizer + with pytest.raises(ValueError, match=r"analysis.minimizer.type = 'emcee'"): + a._validate_fit_request(mode=FitModeEnum.SINGLE, resume=True, extra_steps=None) + + +# ------------------------------------------------------------------ +# Sequential data-dir resolution +# ------------------------------------------------------------------ + + +class TestSequentialDataDir: + def test_absolute_data_dir_returned_directly(self, tmp_path): + from easydiffraction.analysis.analysis import Analysis + + project = SimpleNamespace( + experiments=SimpleNamespace(values=list), + structures=object(), + info=SimpleNamespace(path=None), + _varname='proj', + ) + a = Analysis(project=project) + a.sequential_fit.data_dir.value = str(tmp_path) + assert a._resolve_sequential_data_dir() == tmp_path + + def test_relative_data_dir_requires_saved_project(self): + import pytest + + from easydiffraction.analysis.analysis import Analysis + + project = SimpleNamespace( + experiments=SimpleNamespace(values=list), + structures=object(), + info=SimpleNamespace(path=None), + _varname='proj', + ) + a = Analysis(project=project) + a.sequential_fit.data_dir.value = 'scans' + with pytest.raises(ValueError, match='Project must be saved'): + a._resolve_sequential_data_dir() + + def test_relative_data_dir_joined_to_project_path(self, tmp_path): + from easydiffraction.analysis.analysis import Analysis + + project = SimpleNamespace( + experiments=SimpleNamespace(values=list), + structures=object(), + info=SimpleNamespace(path=tmp_path), + _varname='proj', + ) + a = Analysis(project=project) + a.sequential_fit.data_dir.value = 'scans' + assert a._resolve_sequential_data_dir() == tmp_path / 'scans' + + +# ------------------------------------------------------------------ +# Resumable emcee sidecar detection (filesystem in tmp_path only) +# ------------------------------------------------------------------ + + +class TestResumableEmceeSidecar: + def test_no_project_path_returns_false(self): + from easydiffraction.analysis.analysis import Analysis + + project = SimpleNamespace( + experiments=SimpleNamespace(values=list), + structures=object(), + info=SimpleNamespace(path=None), + _varname='proj', + ) + a = Analysis(project=project) + assert a._has_resumable_emcee_sidecar() is False + + def test_missing_sidecar_file_returns_false(self, tmp_path): + from easydiffraction.analysis.analysis import Analysis + + project = SimpleNamespace( + experiments=SimpleNamespace(values=list), + structures=object(), + info=SimpleNamespace(path=tmp_path), + _varname='proj', + ) + a = Analysis(project=project) + assert a._has_resumable_emcee_sidecar() is False + + def test_sidecar_with_positive_iteration_returns_true(self, tmp_path): + import h5py + + from easydiffraction.analysis.analysis import Analysis + from easydiffraction.analysis.minimizers.emcee import EMCEE_CHAIN_GROUP + + analysis_dir = tmp_path / 'analysis' + analysis_dir.mkdir() + sidecar = analysis_dir / 'results.h5' + with h5py.File(sidecar, 'w') as handle: + group = handle.create_group(EMCEE_CHAIN_GROUP) + group.attrs['iteration'] = 12 + + project = SimpleNamespace( + experiments=SimpleNamespace(values=list), + structures=object(), + info=SimpleNamespace(path=tmp_path), + _varname='proj', + ) + a = Analysis(project=project) + assert a._has_resumable_emcee_sidecar() is True + + def test_sidecar_with_zero_iteration_returns_false(self, tmp_path): + import h5py + + from easydiffraction.analysis.analysis import Analysis + from easydiffraction.analysis.minimizers.emcee import EMCEE_CHAIN_GROUP + + analysis_dir = tmp_path / 'analysis' + analysis_dir.mkdir() + sidecar = analysis_dir / 'results.h5' + with h5py.File(sidecar, 'w') as handle: + group = handle.create_group(EMCEE_CHAIN_GROUP) + group.attrs['iteration'] = 0 + + project = SimpleNamespace( + experiments=SimpleNamespace(values=list), + structures=object(), + info=SimpleNamespace(path=tmp_path), + _varname='proj', + ) + a = Analysis(project=project) + assert a._has_resumable_emcee_sidecar() is False + + def test_sidecar_without_chain_group_returns_false(self, tmp_path): + import h5py + + from easydiffraction.analysis.analysis import Analysis + + analysis_dir = tmp_path / 'analysis' + analysis_dir.mkdir() + sidecar = analysis_dir / 'results.h5' + with h5py.File(sidecar, 'w') as handle: + handle.create_group('some_other_group') + + project = SimpleNamespace( + experiments=SimpleNamespace(values=list), + structures=object(), + info=SimpleNamespace(path=tmp_path), + _varname='proj', + ) + a = Analysis(project=project) + assert a._has_resumable_emcee_sidecar() is False + + +# ------------------------------------------------------------------ +# Joint-fit preparation +# ------------------------------------------------------------------ + + +class TestPrepareJointFit: + def _project_with_named_experiments(self, names): + class Experiments: + def __init__(self, names): + self._names = names + + def __len__(self): + return len(self._names) + + @property + def names(self): + return self._names + + return SimpleNamespace( + experiments=Experiments(names), + structures=object(), + info=SimpleNamespace(path=None), + _varname='proj', + ) + + def test_requires_at_least_two_experiments(self): + import pytest + + from easydiffraction.analysis.analysis import Analysis + + a = Analysis(project=self._project_with_named_experiments(['e1'])) + with pytest.raises(ValueError, match='at least 2 experiments'): + a._prepare_joint_fit() + + def test_auto_populates_missing_rows(self): + from easydiffraction.analysis.analysis import Analysis + + a = Analysis(project=self._project_with_named_experiments(['e1', 'e2'])) + a._prepare_joint_fit() + ids = sorted(item.experiment_id.value for item in a.joint_fit) + assert ids == ['e1', 'e2'] + + def test_rejects_rows_not_in_project(self): + import pytest + + from easydiffraction.analysis.analysis import Analysis + + a = Analysis(project=self._project_with_named_experiments(['e1', 'e2'])) + a.joint_fit.create(experiment_id='ghost', weight=1.0) + with pytest.raises(ValueError, match='not present in the project'): + a._prepare_joint_fit() + + +# ------------------------------------------------------------------ +# Help filter for mode-specific categories +# ------------------------------------------------------------------ + + +class TestHelpFilter: + def test_single_mode_hides_joint_and_sequential(self): + from easydiffraction.analysis.analysis import Analysis + + a = Analysis(project=_make_project()) + a._set_fitting_mode_type('single') + properties = ['minimizer', 'joint_fit', 'sequential_fit', 'sequential_fit_extract'] + filtered, methods = a._help_filter(properties, ['fit']) + assert filtered == ['minimizer'] + assert methods == ['fit'] + + def test_joint_mode_keeps_joint_hides_sequential(self): + from easydiffraction.analysis.analysis import Analysis + + a = Analysis(project=_make_project()) + a._set_fitting_mode_type('joint') + properties = ['joint_fit', 'sequential_fit', 'sequential_fit_extract'] + filtered, _ = a._help_filter(properties, []) + assert filtered == ['joint_fit'] + + def test_sequential_mode_hides_only_joint(self): + from easydiffraction.analysis.analysis import Analysis + + a = Analysis(project=_make_project()) + a._set_fitting_mode_type('sequential') + properties = ['joint_fit', 'sequential_fit', 'sequential_fit_extract'] + filtered, _ = a._help_filter(properties, []) + assert filtered == ['sequential_fit', 'sequential_fit_extract'] + + +# ------------------------------------------------------------------ +# Random-seed resolution +# ------------------------------------------------------------------ + + +class TestResolvedFitRandomSeed: + def test_explicit_seed_takes_precedence(self): + from easydiffraction.analysis.analysis import Analysis + + a = Analysis(project=_make_project()) + assert a._resolved_fit_random_seed(42) == 42 + + def test_falls_back_to_minimizer_seed(self, monkeypatch): + from easydiffraction.analysis.analysis import Analysis + + a = Analysis(project=_make_project()) + monkeypatch.setattr(a.minimizer, '_native_kwargs', lambda: {'random_seed': 7}) + assert a._resolved_fit_random_seed(None) == 7 + + def test_returns_none_when_no_seed_available(self, monkeypatch): + from easydiffraction.analysis.analysis import Analysis + + a = Analysis(project=_make_project()) + monkeypatch.setattr(a.minimizer, '_native_kwargs', lambda: {'random_seed': None}) + assert a._resolved_fit_random_seed(None) is None + + +# ------------------------------------------------------------------ +# Engine sync from minimizer category +# ------------------------------------------------------------------ + + +class TestSyncEngineFromMinimizer: + def test_sync_warns_for_unsupported_setting(self, monkeypatch): + import easydiffraction.analysis.analysis as mod + from easydiffraction.analysis.analysis import Analysis + + a = Analysis(project=_make_project()) + warnings = [] + monkeypatch.setattr(mod.log, 'warning', warnings.append) + + # Engine that lacks the 'method' attribute the kwargs reference. + engine = SimpleNamespace() + a.fitter.minimizer = engine + monkeypatch.setattr(a.minimizer, '_native_kwargs', lambda: {'method': 'leastsq'}) + monkeypatch.setattr( + type(a.minimizer), '_engine_sync_skip_keys', frozenset(), raising=False + ) + + a._sync_engine_from_minimizer_category() + + assert any('is not supported by' in message for message in warnings) + + def test_sync_applies_supported_settings_and_skips_keys(self, monkeypatch): + from easydiffraction.analysis.analysis import Analysis + + a = Analysis(project=_make_project()) + engine = SimpleNamespace(tolerance=0.0, skipme=None) + a.fitter.minimizer = engine + monkeypatch.setattr( + a.minimizer, + '_native_kwargs', + lambda: {'tolerance': 1e-6, 'skipme': 'should-not-apply'}, + ) + monkeypatch.setattr( + type(a.minimizer), + '_engine_sync_skip_keys', + frozenset({'skipme'}), + raising=False, + ) + + a._sync_engine_from_minimizer_category() + + assert engine.tolerance == 1e-6 + assert engine.skipme is None + + +# ------------------------------------------------------------------ +# Restored posterior samples shape validation +# ------------------------------------------------------------------ + + +class TestRestoredPosteriorSamples: + def test_no_parameter_samples_returns_none(self): + from easydiffraction.analysis.analysis import Analysis + + a = Analysis(project=_make_project()) + a._persisted_fit_state_sidecar = {'posterior': {}} + assert a._restored_posterior_samples() is None + + def test_invalid_ndim_warns_and_returns_none(self, monkeypatch): + import easydiffraction.analysis.analysis as mod + from easydiffraction.analysis.analysis import Analysis + + a = Analysis(project=_make_project()) + warnings = [] + monkeypatch.setattr(mod.log, 'warning', warnings.append) + # 2-D array (not 3-D) is rejected. + a._persisted_fit_state_sidecar = { + 'posterior': {'parameter_samples': [[1.0, 2.0], [3.0, 4.0]]} + } + assert a._restored_posterior_samples() is None + assert any('invalid shape' in message for message in warnings) + + def test_parameter_count_mismatch_warns_and_returns_none(self, monkeypatch): + import easydiffraction.analysis.analysis as mod + from easydiffraction.analysis.analysis import Analysis + + a = Analysis(project=_make_project()) + warnings = [] + monkeypatch.setattr(mod.log, 'warning', warnings.append) + # No persisted posterior rows -> parameter_names is empty, but the + # 3-D array has 2 parameters on its last axis -> mismatch. + a._persisted_fit_state_sidecar = { + 'posterior': {'parameter_samples': np.zeros((4, 2, 2)).tolist()} + } + assert a._restored_posterior_samples() is None + assert any('do not match' in message for message in warnings) + + +# ------------------------------------------------------------------ +# Common fit-result projection storage +# ------------------------------------------------------------------ + + +class TestCommonFitResultProjection: + def test_stores_shared_fields_and_sets_persisted_flag(self): + from easydiffraction.analysis.analysis import Analysis + from easydiffraction.analysis.enums import FitResultKindEnum + from easydiffraction.analysis.fit_helpers.reporting import FitResults + + a = Analysis(project=_make_project()) + results = FitResults(success=True, reduced_chi_square=2.5, fitting_time=1.5) + results.message = 'converged' + results.iterations = 42 + + a._store_common_fit_result_projection( + results, + result_kind=FitResultKindEnum.DETERMINISTIC, + ) + + assert a.fit_result.result_kind.value == FitResultKindEnum.DETERMINISTIC.value + assert a.fit_result.success.value is True + assert a.fit_result.message.value == 'converged' + assert a.fit_result.iterations.value == 42 + assert a.fit_result.fitting_time.value == 1.5 + assert a.fit_result.reduced_chi_square.value == 2.5 + assert a._has_persisted_fit_state() is True + + +# ------------------------------------------------------------------ +# Pre-fit parameter capture and selection +# ------------------------------------------------------------------ + + +class TestParameterCaptureAndSelection: + def test_capture_fit_parameter_state_creates_rows(self): + from easydiffraction.analysis.analysis import Analysis + + length_a = _make_parameter('length_a', 3.9) + length_a.value = 4.1 + length_a.uncertainty = 0.05 + length_a.fit_min = 3.5 + length_a.fit_max = 4.5 + project = _make_project_with_parameters([length_a]) + a = Analysis(project=project) + + a._capture_fit_parameter_state([length_a]) + + assert a._has_persisted_fit_state() is True + assert len(a.fit_parameters) == 1 + row = a.fit_parameters[length_a.unique_name] + assert row.start_value.value == 4.1 + assert row.start_uncertainty.value == 0.05 + assert row.fit_min.value == 3.5 + assert row.fit_max.value == 4.5 + + def test_selected_parameters_dedupe_across_structures_and_experiments(self): + from easydiffraction.analysis.analysis import Analysis + + shared = _make_parameter('scale', 1.0) + structure_only = _make_parameter('length_a', 4.0) + experiment_only = _make_parameter('wavelength', 1.5) + + project = _make_project_with_parameters([shared, structure_only]) + a = Analysis(project=project) + + experiment = SimpleNamespace(parameters=[shared, experiment_only]) + selected = a._selected_parameters_for_fit([experiment]) + + unique_names = [param.unique_name for param in selected] + # 'scale' appears in both but must be counted once. + assert unique_names.count(shared.unique_name) == 1 + assert structure_only.unique_name in unique_names + assert experiment_only.unique_name in unique_names + assert len(selected) == 3 + + def test_selected_parameters_ignores_non_parameter_descriptors(self): + from easydiffraction.analysis.analysis import Analysis + + real = _make_parameter('scale', 1.0) + not_a_parameter = SimpleNamespace(unique_name='descriptor') + project = _make_project_with_parameters([real, not_a_parameter]) + a = Analysis(project=project) + + selected = a._selected_parameters_for_fit([]) + assert [param.unique_name for param in selected] == [real.unique_name] + + +# ------------------------------------------------------------------ +# Fit-result state category selection +# ------------------------------------------------------------------ + + +class TestFitResultStateCategories: + def test_valid_result_kind_returns_common_categories(self): + from easydiffraction.analysis.analysis import Analysis + from easydiffraction.analysis.enums import FitResultKindEnum + + a = Analysis(project=_make_project()) + a.fit_result._set_result_kind(FitResultKindEnum.DETERMINISTIC.value) + + categories = a._fit_result_state_categories() + assert a.fit_result in categories + assert a.fit_parameter_correlations in categories + + def test_invalid_result_kind_warns_and_returns_common_only(self, monkeypatch): + import easydiffraction.analysis.analysis as mod + from easydiffraction.analysis.analysis import Analysis + + a = Analysis(project=_make_project()) + # The validating setter would reject an unknown kind and fall back, + # so force the raw descriptor value to exercise the except branch. + a.fit_result._result_kind._value = 'bogus-kind' + warnings = [] + monkeypatch.setattr(mod.log, 'warning', warnings.append) + + categories = a._fit_result_state_categories() + assert categories == [a.fit_result, a.fit_parameter_correlations] + assert any('Unsupported fit_result.result_kind' in message for message in warnings) + + +# ------------------------------------------------------------------ +# Restored Bayesian reduced chi-square branches +# ------------------------------------------------------------------ + + +class TestRestoredBayesianReducedChiSquare: + def _bayesian_analysis(self): + from easydiffraction.analysis.analysis import Analysis + from easydiffraction.analysis.categories.fit_result.bayesian import BayesianFitResult + + a = Analysis(project=_make_project()) + a._fit_result._parent = None + a._fit_result = BayesianFitResult() + a._fit_result._parent = a + return a + + def test_finite_persisted_value_passes_through(self): + a = self._bayesian_analysis() + value = a._restored_bayesian_reduced_chi_square(2.5, restored_parameters=[object()]) + assert value == 2.5 + + def test_missing_log_posterior_returns_none(self): + a = self._bayesian_analysis() + # best_log_posterior defaults to None -> cannot reconstruct. + value = a._restored_bayesian_reduced_chi_square( + float('nan'), + restored_parameters=[object()], + ) + assert value is None + + def test_nonpositive_dof_returns_none(self, monkeypatch): + a = self._bayesian_analysis() + a.fit_result._set_best_log_posterior(-10.0) + # Two data points, two parameters -> dof = 0 -> None. + monkeypatch.setattr(a, '_fit_data_point_count', lambda experiments: 2) + value = a._restored_bayesian_reduced_chi_square( + float('nan'), + restored_parameters=[object(), object()], + ) + assert value is None + + +# ------------------------------------------------------------------ +# Restored posterior summaries and predictive datasets +# ------------------------------------------------------------------ + + +class TestRestoredPosteriorSummariesAndPredictive: + def _analysis_with_posterior_row(self): + from easydiffraction.analysis.analysis import Analysis + + a = Analysis(project=_make_project()) + a.fit_parameters.create( + param_unique_name='alpha', + fit_min=0.0, + fit_max=2.0, + start_value=1.0, + start_uncertainty=0.1, + ) + row = a.fit_parameters['alpha'] + row._set_posterior_best_sample_value(1.2) + row._set_posterior_median(1.1) + row._set_posterior_uncertainty(0.1) + row._set_posterior_interval_68_low(1.0) + row._set_posterior_interval_68_high(1.2) + row._set_posterior_interval_95_low(0.9) + row._set_posterior_interval_95_high(1.3) + return a + + def test_restored_posterior_summaries_uses_unique_name_when_param_missing(self): + a = self._analysis_with_posterior_row() + summaries = a._restored_posterior_summaries() + assert len(summaries) == 1 + # No live parameter named 'alpha' -> display name falls back to id. + assert summaries[0].display_name == 'alpha' + + def test_restored_posterior_samples_valid_path_returns_samples(self): + a = self._analysis_with_posterior_row() + samples = np.zeros((4, 2, 1), dtype=float) + a._persisted_fit_state_sidecar = { + 'posterior': { + 'parameter_samples': samples.tolist(), + 'log_posterior': np.zeros((4, 2)).tolist(), + 'draw_index': [0, 1, 2, 3], + } + } + restored = a._restored_posterior_samples() + assert restored is not None + assert restored.parameter_names == ['alpha'] + assert restored.parameter_samples.shape == (4, 2, 1) + assert restored.log_posterior is not None + assert restored.draw_index is not None + + def test_restored_predictive_summaries_builds_cache_keys(self): + from easydiffraction.analysis.analysis import Analysis + from easydiffraction.analysis.fit_helpers.bayesian import posterior_predictive_cache_key + + a = Analysis(project=_make_project()) + a._persisted_fit_state_sidecar = { + 'predictive_datasets': { + 'hrpt': { + 'x_axis_name': 'two_theta', + 'x': [1.0, 2.0], + 'best_sample_prediction': [3.0, 4.0], + 'lower_95': [2.5, 3.5], + 'upper_95': [3.5, 4.5], + 'draws': [[3.0, 3.1], [4.0, 4.1]], + } + } + } + restored = a._restored_predictive_summaries() + assert 'hrpt' in restored + no_draws_key = posterior_predictive_cache_key('hrpt', 'two_theta', include_draws=False) + draws_key = posterior_predictive_cache_key('hrpt', 'two_theta', include_draws=True) + assert no_draws_key in restored + assert draws_key in restored + assert np.allclose(restored['hrpt'].best_sample_prediction, [3.0, 4.0]) + + +# ------------------------------------------------------------------ +# Least-squares result projection (engine-free, array-driven) +# ------------------------------------------------------------------ + + +class TestLeastSquaresResultProjection: + def _powder_experiment(self, *, meas, calc, su): + data = SimpleNamespace( + intensity_meas=np.asarray(meas, dtype=float), + intensity_calc=np.asarray(calc, dtype=float), + intensity_meas_su=np.asarray(su, dtype=float), + ) + from easydiffraction.datablocks.experiment.item.enums import SampleFormEnum + + return SimpleNamespace( + data=data, + type=SimpleNamespace(sample_form=SimpleNamespace(value=SampleFormEnum.POWDER.value)), + peak=SimpleNamespace(type=SimpleNamespace(value='gaussian')), + background=SimpleNamespace(type=SimpleNamespace(value='chebyshev')), + parameters=[], + ) + + def test_store_least_squares_result_projection_populates_statistics(self): + from easydiffraction.analysis.analysis import Analysis + from easydiffraction.analysis.fit_helpers.reporting import FitResults + + a = Analysis(project=_make_project()) + experiment = self._powder_experiment( + meas=[10.0, 10.0, 10.0, 10.0], + calc=[9.0, 11.0, 10.0, 10.0], + su=[1.0, 1.0, 1.0, 1.0], + ) + + fitted = _make_parameter('scale', 1.0) + fitted.value = 1.1 + fitted.uncertainty = 0.05 + fitted._fit_start_value = 1.0 + + results = FitResults(success=True, reduced_chi_square=1.0) + results.chi_square = 4.0 + results.message = 'converged' + + a._store_least_squares_result_projection( + results, + experiments=[experiment], + fitted_parameters=[fitted], + ) + + assert a.fit_result.objective_name.value == 'chi_square' + assert a.fit_result.objective_value.value == 4.0 + assert a.fit_result.n_data_points.value == 4 + assert a.fit_result.n_free_parameters.value == 1 + assert a.fit_result.degrees_of_freedom.value == 3 + assert a.fit_result.covariance_available.value is False + assert a.fit_result.r_factor_all.value is not None + # Powder profile R factors are populated for powder fits. + assert a.fit_result.prof_r_factor.value is not None + assert a.fit_result.profile_function.value == 'gaussian' + assert a.fit_result.background_function.value == 'chebyshev' + + def test_store_least_squares_with_covariance_records_correlations(self): + from easydiffraction.analysis.analysis import Analysis + from easydiffraction.analysis.fit_helpers.reporting import FitResults + + a = Analysis(project=_make_project()) + experiment = self._powder_experiment( + meas=[10.0, 10.0], + calc=[10.0, 10.0], + su=[1.0, 1.0], + ) + + p1 = _make_parameter('p1', 1.0) + p1.value = 1.0 + p1.uncertainty = 0.1 + p1._fit_start_value = 1.0 + p2 = _make_parameter('p2', 2.0) + p2.value = 2.0 + p2.uncertainty = 0.2 + p2._fit_start_value = 2.0 + + covar = np.asarray([[0.01, 0.005], [0.005, 0.04]], dtype=float) + results = FitResults(success=True, engine_result=SimpleNamespace(covar=covar)) + results.chi_square = 0.0 + results.message = 'ok' + + a._store_least_squares_result_projection( + results, + experiments=[experiment], + fitted_parameters=[p1, p2], + ) + + assert a.fit_result.covariance_available.value is True + assert a.fit_result.correlation_available.value is True + assert len(a.fit_parameter_correlations) == 1 diff --git a/tests/unit/easydiffraction/analysis/test_fitting_coverage.py b/tests/unit/easydiffraction/analysis/test_fitting_coverage.py new file mode 100644 index 000000000..e83589766 --- /dev/null +++ b/tests/unit/easydiffraction/analysis/test_fitting_coverage.py @@ -0,0 +1,742 @@ +# SPDX-FileCopyrightText: 2026 EasyScience contributors +# SPDX-License-Identifier: BSD-3-Clause + +from types import SimpleNamespace + +import numpy as np +import pytest + +from easydiffraction.analysis.fitting import FitterFitOptions +from easydiffraction.analysis.fitting import _resolve_fit_result_chi_square +from easydiffraction.analysis.fitting import _resolve_fit_result_iterations +from easydiffraction.analysis.fitting import _resolve_fit_result_message + +# --------------------------------------------------------------------------- +# FitterFitOptions.as_minimizer_options +# --------------------------------------------------------------------------- + + +def test_as_minimizer_options_maps_fields_and_forces_no_finalize(): + options = FitterFitOptions( + use_physical_limits=True, + random_seed=42, + resume=True, + extra_steps=7, + ) + + minimizer_options = options.as_minimizer_options() + + # finalize_tracking is always forced off; the fitter owns finalization. + assert minimizer_options.finalize_tracking is False + assert minimizer_options.use_physical_limits is True + assert minimizer_options.random_seed == 42 + assert minimizer_options.resume is True + assert minimizer_options.extra_steps == 7 + + +def test_fitter_fit_options_defaults(): + options = FitterFitOptions() + + assert options.use_physical_limits is False + assert options.random_seed is None + assert options.resume is False + assert options.extra_steps is None + + +# --------------------------------------------------------------------------- +# _resolve_fit_result_message +# --------------------------------------------------------------------------- + + +def test_resolve_message_prefers_existing_message(): + results = SimpleNamespace(message='converged', engine_result=object()) + + assert _resolve_fit_result_message(results) == 'converged' + + +def test_resolve_message_falls_back_to_engine_result_attribute(): + results = SimpleNamespace( + message='', + engine_result=SimpleNamespace(message='engine says hi'), + ) + + assert _resolve_fit_result_message(results) == 'engine says hi' + + +def test_resolve_message_handles_missing_engine_attribute(): + results = SimpleNamespace(message='', engine_result=object()) + + assert _resolve_fit_result_message(results) == '' + + +def test_resolve_message_handles_none_engine_message(): + results = SimpleNamespace( + message='', + engine_result=SimpleNamespace(message=None), + ) + + assert _resolve_fit_result_message(results) == '' + + +# --------------------------------------------------------------------------- +# _resolve_fit_result_iterations +# --------------------------------------------------------------------------- + + +def test_resolve_iterations_prefers_existing_value(): + results = SimpleNamespace(iterations=5, engine_result=object()) + + assert _resolve_fit_result_iterations(results) == 5 + + +def test_resolve_iterations_reads_first_available_engine_attribute(): + # 'nfev' is checked before 'nit'; the first non-None wins. + results = SimpleNamespace( + iterations=0, + engine_result=SimpleNamespace(nfev=11, nit=99), + ) + + assert _resolve_fit_result_iterations(results) == 11 + + +def test_resolve_iterations_skips_none_attributes_in_order(): + results = SimpleNamespace( + iterations=0, + engine_result=SimpleNamespace(nfev=None, nit=None, iterations=None, niter=33), + ) + + assert _resolve_fit_result_iterations(results) == 33 + + +def test_resolve_iterations_returns_zero_when_nothing_found(): + results = SimpleNamespace(iterations=0, engine_result=object()) + + assert _resolve_fit_result_iterations(results) == 0 + + +# --------------------------------------------------------------------------- +# _resolve_fit_result_chi_square +# --------------------------------------------------------------------------- + + +def test_resolve_chi_square_prefers_existing_value(): + results = SimpleNamespace(chi_square=2.5, engine_result=object()) + + assert _resolve_fit_result_chi_square(results) == 2.5 + + +def test_resolve_chi_square_uses_engine_chisqr(): + results = SimpleNamespace( + chi_square=None, + engine_result=SimpleNamespace(chisqr=3.0), + ) + + assert _resolve_fit_result_chi_square(results) == 3.0 + + +def test_resolve_chi_square_uses_scalar_fun(): + results = SimpleNamespace( + chi_square=None, + engine_result=SimpleNamespace(chisqr=None, fun=4.0), + ) + + assert _resolve_fit_result_chi_square(results) == 4.0 + + +def test_resolve_chi_square_sums_squares_of_array_fun(): + results = SimpleNamespace( + chi_square=None, + engine_result=SimpleNamespace(chisqr=None, fun=np.array([3.0, 4.0])), + ) + + # 3**2 + 4**2 = 25 + assert _resolve_fit_result_chi_square(results) == 25.0 + + +def test_resolve_chi_square_returns_none_when_no_source(): + results = SimpleNamespace(chi_square=None, engine_result=object()) + + assert _resolve_fit_result_chi_square(results) is None + + +def test_resolve_chi_square_returns_none_when_fun_is_none(): + results = SimpleNamespace( + chi_square=None, + engine_result=SimpleNamespace(chisqr=None, fun=None), + ) + + assert _resolve_fit_result_chi_square(results) is None + + +# --------------------------------------------------------------------------- +# Fitter helpers and branches +# --------------------------------------------------------------------------- + + +def _make_fitter_with_dummy_minimizer(): + from easydiffraction.analysis.fitting import Fitter + + fitter = Fitter() + fitter.minimizer = SimpleNamespace() + return fitter + + +def _make_param(name): + from easydiffraction.core.validation import AttributeSpec + from easydiffraction.core.variable import Parameter + from easydiffraction.io.cif.handler import CifHandler + + return Parameter( + name=name, + value_spec=AttributeSpec(default=0.0), + cif_handler=CifHandler(names=[f'_param.{name}']), + ) + + +def test_collect_fit_parameters_filters_constrained_and_fixed(): + from easydiffraction.analysis.fitting import Fitter + + free = _make_param('free') + free.free = True + + fixed = _make_param('fixed') + fixed.free = False + + constrained = _make_param('constrained') + constrained.free = True + constrained._user_constrained = True + + experiment = SimpleNamespace(parameters=[free, fixed, constrained, 'not-a-parameter']) + structures = SimpleNamespace(free_parameters=[]) + + collected = Fitter._collect_fit_parameters(structures, [experiment]) + + assert collected == [free] + + +def test_collect_fit_parameters_includes_structure_free_parameters(): + from easydiffraction.analysis.fitting import Fitter + + struct_param = _make_param('struct') + expt_param = _make_param('expt') + expt_param.free = True + + experiment = SimpleNamespace(parameters=[expt_param]) + structures = SimpleNamespace(free_parameters=[struct_param]) + + collected = Fitter._collect_fit_parameters(structures, [experiment]) + + assert collected == [struct_param, expt_param] + + +def test_fit_no_params_resume_raises_value_error(): + fitter = _make_fitter_with_dummy_minimizer() + + with pytest.raises(ValueError, match='Resume requires the same free parameters'): + fitter.fit( + structures=_NoStructures(), + experiments=[], + options=FitterFitOptions(resume=True), + ) + + +def test_fit_no_params_clears_analysis_state(monkeypatch): + from easydiffraction.utils.logging import log + + monkeypatch.setattr(log, '_reaction', log.Reaction.WARN, raising=True) + + fitter = _make_fitter_with_dummy_minimizer() + events = [] + analysis = SimpleNamespace( + _clear_persisted_fit_state=lambda: events.append('clear'), + fit_results='stale', + ) + + fitter.fit(structures=_NoStructures(), experiments=[], analysis=analysis) + + assert events == ['clear'] + assert analysis.fit_results is None + assert fitter.results is None + + +def test_fit_resume_with_params_invokes_resume_validation(monkeypatch): + from easydiffraction.analysis.fitting import Fitter + + param = SimpleNamespace(value=1.0, unique_name='a', _fit_start_value=None) + + class DummyMin: + def __init__(self): + self.tracker = SimpleNamespace(track=lambda residuals, parameters: residuals) + + def fit(self, params, obj, verbosity=None, **kwargs): + del params, obj, verbosity, kwargs + return SimpleNamespace( + message='ok', + iterations=1, + chi_square=1.0, + minimizer_type=None, + engine_result=object(), + fitting_time=2.0, + ) + + def _finalize_timing(self): + return None + + def _stop_tracking(self): + return None + + fitter = Fitter() + fitter.minimizer = DummyMin() + monkeypatch.setattr( + fitter, + '_collect_fit_parameters', + lambda structures, experiments: [param], + ) + + validate_calls = [] + + def fake_validate(*, params, analysis): + validate_calls.append((params, analysis)) + + # resume path runs the resume validation instead of capturing state. + monkeypatch.setattr(fitter, '_validate_resume_parameter_set', fake_validate) + + persisted = [SimpleNamespace(param_unique_name=SimpleNamespace(value='a'))] + analysis = SimpleNamespace( + fit_parameters=persisted, + fit_result=SimpleNamespace(), + _store_fit_result_projection=lambda results, experiments, fitted_parameters: None, + ) + + fitter.fit( + structures=_NoStructures(), + experiments=[], + analysis=analysis, + options=FitterFitOptions(resume=True), + ) + + assert len(validate_calls) == 1 + assert validate_calls[0][0] == [param] + assert validate_calls[0][1] is analysis + + +def test_validate_resume_parameter_set_no_persisted_names_is_noop(): + from easydiffraction.analysis.fitting import Fitter + + analysis = SimpleNamespace(fit_parameters=[]) + param = SimpleNamespace(unique_name='a') + + # No persisted names -> returns without raising. + Fitter._validate_resume_parameter_set(params=[param], analysis=analysis) + + +def test_validate_resume_parameter_set_matching_names_ok(): + from easydiffraction.analysis.fitting import Fitter + + persisted = [SimpleNamespace(param_unique_name=SimpleNamespace(value='a'))] + analysis = SimpleNamespace(fit_parameters=persisted) + param = SimpleNamespace(unique_name='a') + + Fitter._validate_resume_parameter_set(params=[param], analysis=analysis) + + +def test_validate_resume_parameter_set_mismatch_raises(): + from easydiffraction.analysis.fitting import Fitter + + persisted = [SimpleNamespace(param_unique_name=SimpleNamespace(value='a'))] + analysis = SimpleNamespace(fit_parameters=persisted) + param = SimpleNamespace(unique_name='b') + + with pytest.raises(ValueError, match='differs from the saved emcee chain'): + Fitter._validate_resume_parameter_set(params=[param], analysis=analysis) + + +def test_set_minimizer_sidecar_path_noop_when_analysis_none(): + fitter = _make_fitter_with_dummy_minimizer() + fitter.minimizer._sidecar_path = 'unchanged' + + fitter._set_minimizer_sidecar_path(None) + + assert fitter.minimizer._sidecar_path == 'unchanged' + + +def test_set_minimizer_sidecar_path_noop_when_no_attribute(): + fitter = _make_fitter_with_dummy_minimizer() + # Minimizer without _sidecar_path attribute -> early return, no crash. + analysis = SimpleNamespace(project=SimpleNamespace(info=SimpleNamespace(path=None))) + + fitter._set_minimizer_sidecar_path(analysis) + + assert not hasattr(fitter.minimizer, '_sidecar_path') + + +def test_set_minimizer_sidecar_path_none_when_no_project_path(): + fitter = _make_fitter_with_dummy_minimizer() + fitter.minimizer._sidecar_path = 'unset' + analysis = SimpleNamespace(project=SimpleNamespace(info=SimpleNamespace(path=None))) + + fitter._set_minimizer_sidecar_path(analysis) + + assert fitter.minimizer._sidecar_path is None + + +def test_set_minimizer_sidecar_path_builds_results_path(tmp_path): + fitter = _make_fitter_with_dummy_minimizer() + fitter.minimizer._sidecar_path = None + analysis = SimpleNamespace( + project=SimpleNamespace(info=SimpleNamespace(path=tmp_path)), + ) + + fitter._set_minimizer_sidecar_path(analysis) + + assert fitter.minimizer._sidecar_path == tmp_path / 'analysis' / 'results.h5' + + +def test_backfill_persisted_fitting_time_noop_when_analysis_none(): + fitter = _make_fitter_with_dummy_minimizer() + fitter.results = SimpleNamespace(fitting_time=1.0) + + # Should not raise even though no analysis is provided. + fitter._backfill_persisted_fitting_time(None) + + +def test_backfill_persisted_fitting_time_noop_when_results_none(): + fitter = _make_fitter_with_dummy_minimizer() + fitter.results = None + analysis = SimpleNamespace(fit_result=SimpleNamespace()) + + fitter._backfill_persisted_fitting_time(analysis) + + +def test_backfill_persisted_fitting_time_sets_time_when_callable(): + fitter = _make_fitter_with_dummy_minimizer() + fitter.results = SimpleNamespace(fitting_time=9.5) + captured = {} + fit_result = SimpleNamespace(_set_fitting_time=lambda value: captured.setdefault('t', value)) + analysis = SimpleNamespace(fit_result=fit_result) + + fitter._backfill_persisted_fitting_time(analysis) + + assert captured == {'t': 9.5} + + +def test_backfill_persisted_fitting_time_skips_when_not_callable(): + fitter = _make_fitter_with_dummy_minimizer() + fitter.results = SimpleNamespace(fitting_time=9.5) + # fit_result has no _set_fitting_time -> getattr returns None -> skipped. + analysis = SimpleNamespace(fit_result=SimpleNamespace()) + + # Should not raise. + fitter._backfill_persisted_fitting_time(analysis) + + +def test_postprocess_fit_results_noop_when_results_none(): + fitter = _make_fitter_with_dummy_minimizer() + fitter.results = None + + # Should return immediately without touching analysis. + fitter._postprocess_fit_results( + analysis=object(), + experiments=[], + fitted_parameters=[], + ) + + +def test_postprocess_fit_results_normalizes_fields_without_analysis(): + fitter = _make_fitter_with_dummy_minimizer() + fitter.selection = 'lmfit' + fitter.results = SimpleNamespace( + message='', + iterations=0, + chi_square=None, + minimizer_type=None, + engine_result=SimpleNamespace(message='done', nfev=7, chisqr=2.0), + ) + + fitter._postprocess_fit_results( + analysis=None, + experiments=[], + fitted_parameters=[], + ) + + assert fitter.results.message == 'done' + assert fitter.results.iterations == 7 + assert fitter.results.chi_square == 2.0 + assert fitter.results.minimizer_type == 'lmfit' + + +def test_postprocess_fit_results_stores_projection_when_analysis_present(): + fitter = _make_fitter_with_dummy_minimizer() + fitter.selection = 'lmfit' + fitter.results = SimpleNamespace( + message='ok', + iterations=3, + chi_square=1.0, + minimizer_type=None, + engine_result=object(), + ) + stored = {} + analysis = SimpleNamespace( + _store_fit_result_projection=lambda results, experiments, fitted_parameters: stored.update( + results=results, + experiments=experiments, + fitted_parameters=fitted_parameters, + ), + ) + experiments = ['expt'] + fitted = ['param'] + + fitter._postprocess_fit_results( + analysis=analysis, + experiments=experiments, + fitted_parameters=fitted, + ) + + assert stored['results'] is fitter.results + assert stored['experiments'] == experiments + assert stored['fitted_parameters'] == fitted + + +# --------------------------------------------------------------------------- +# _process_fit_results +# --------------------------------------------------------------------------- + + +def test_process_fit_results_displays_when_results_present(monkeypatch): + fitter = _make_fitter_with_dummy_minimizer() + + monkeypatch.setattr( + 'easydiffraction.analysis.fitting.get_reliability_inputs', + lambda structures, experiments: ( + np.array([1.0]), + np.array([1.1]), + np.array([0.1]), + ), + ) + + displayed = {} + + def display_results(*, y_obs, y_calc, y_err, f_obs, f_calc): + displayed.update( + y_obs=y_obs, + y_calc=y_calc, + y_err=y_err, + f_obs=f_obs, + f_calc=f_calc, + ) + + fitter.results = SimpleNamespace(display_results=display_results) + + fitter._process_fit_results(structures=object(), experiments=[]) + + np.testing.assert_allclose(displayed['y_obs'], np.array([1.0])) + np.testing.assert_allclose(displayed['y_calc'], np.array([1.1])) + np.testing.assert_allclose(displayed['y_err'], np.array([0.1])) + assert displayed['f_obs'] is None + assert displayed['f_calc'] is None + + +def test_process_fit_results_skips_display_when_no_results(monkeypatch): + fitter = _make_fitter_with_dummy_minimizer() + fitter.results = None + + called = {'display': False} + + monkeypatch.setattr( + 'easydiffraction.analysis.fitting.get_reliability_inputs', + lambda structures, experiments: (None, None, None), + ) + + # Even though get_reliability_inputs runs, display must not be invoked. + fitter._process_fit_results(structures=object(), experiments=[]) + + assert called['display'] is False + + +# --------------------------------------------------------------------------- +# _residual_function +# --------------------------------------------------------------------------- + + +def test_residual_function_updates_structures_and_analysis(monkeypatch): + fitter = _make_fitter_with_dummy_minimizer() + + events = [] + + def sync(parameters, engine_params): + events.append('sync') + + fitter.minimizer._sync_result_to_parameters = sync + fitter.minimizer.tracker = SimpleNamespace( + track=lambda residuals, parameters: residuals * 10.0, + ) + + structure = SimpleNamespace( + _update_categories=lambda *, called_by_minimizer=False: events.append(( + 'struct', + called_by_minimizer, + )) + ) + analysis = SimpleNamespace( + _update_categories=lambda *, called_by_minimizer=False: events.append(( + 'analysis', + called_by_minimizer, + )) + ) + experiment = SimpleNamespace( + _update_categories=lambda *, called_by_minimizer=False: events.append(( + 'expt', + called_by_minimizer, + )) + ) + + monkeypatch.setattr( + 'easydiffraction.analysis.fitting.intensity_category_for', + lambda experiment: SimpleNamespace( + intensity_calc=np.array([1.0]), + intensity_meas=np.array([3.0]), + intensity_meas_su=np.array([1.0]), + ), + ) + + residuals = fitter._residual_function( + engine_params={}, + parameters=[], + structures=[structure], + experiments=[experiment], + weights=None, + analysis=analysis, + ) + + # diff = (3 - 1) / 1 = 2, weight = 1 (single experiment), tracker *10. + np.testing.assert_allclose(residuals, np.array([20.0])) + assert ('struct', True) in events + assert ('analysis', True) in events + assert ('expt', True) in events + assert 'sync' in events + + +def test_residual_function_applies_normalized_weights(monkeypatch): + fitter = _make_fitter_with_dummy_minimizer() + fitter.minimizer._sync_result_to_parameters = lambda parameters, engine_params: None + fitter.minimizer.tracker = SimpleNamespace(track=lambda residuals, parameters: residuals) + + def make_experiment(meas): + return SimpleNamespace( + _update_categories=lambda *, called_by_minimizer=False: None, + _meas=meas, + ) + + experiments = [make_experiment(3.0), make_experiment(5.0)] + + def category_for(experiment): + return SimpleNamespace( + intensity_calc=np.array([1.0]), + intensity_meas=np.array([experiment._meas]), + intensity_meas_su=np.array([1.0]), + ) + + monkeypatch.setattr( + 'easydiffraction.analysis.fitting.intensity_category_for', + category_for, + ) + + # Weights [1, 3] normalized to sum=2 (num experiments): [0.5, 1.5]. + residuals = fitter._residual_function( + engine_params={}, + parameters=[], + structures=[], + experiments=experiments, + weights=np.array([1.0, 3.0]), + analysis=None, + ) + + # diff0 = (3-1)/1 * sqrt(0.5); diff1 = (5-1)/1 * sqrt(1.5). + expected = np.array([2.0 * np.sqrt(0.5), 4.0 * np.sqrt(1.5)]) + np.testing.assert_allclose(residuals, expected) + + +def test_residual_function_handles_minimizer_without_solver_monitor_attr(monkeypatch): + fitter = _make_fitter_with_dummy_minimizer() + fitter.minimizer._sync_result_to_parameters = lambda parameters, engine_params: None + # No _tracks_progress_via_solver_monitor attribute -> default lambda False + # path -> tracker.track is used. + fitter.minimizer.tracker = SimpleNamespace( + track=lambda residuals, parameters: residuals + 100.0, + ) + + experiment = SimpleNamespace( + _update_categories=lambda *, called_by_minimizer=False: None, + ) + + monkeypatch.setattr( + 'easydiffraction.analysis.fitting.intensity_category_for', + lambda experiment: SimpleNamespace( + intensity_calc=np.array([1.0]), + intensity_meas=np.array([2.0]), + intensity_meas_su=np.array([1.0]), + ), + ) + + residuals = fitter._residual_function( + engine_params={}, + parameters=[], + structures=[], + experiments=[experiment], + weights=None, + analysis=None, + ) + + np.testing.assert_allclose(residuals, np.array([101.0])) + + +def test_build_objective_function_delegates_to_residual(monkeypatch): + fitter = _make_fitter_with_dummy_minimizer() + + captured = {} + + def fake_residual(*, engine_params, parameters, structures, experiments, weights, analysis): + captured.update( + engine_params=engine_params, + parameters=parameters, + structures=structures, + experiments=experiments, + weights=weights, + analysis=analysis, + ) + return np.array([7.0]) + + monkeypatch.setattr(fitter, '_residual_function', fake_residual) + + params = ['p'] + structures = 's' + experiments = ['e'] + weights = np.array([1.0]) + analysis = 'a' + + objective = fitter._build_objective_function( + params=params, + structures=structures, + experiments=experiments, + weights=weights, + analysis=analysis, + ) + + result = objective({'x': 1.0}) + + np.testing.assert_allclose(result, np.array([7.0])) + assert captured['engine_params'] == {'x': 1.0} + assert captured['parameters'] is params + assert captured['structures'] is structures + assert captured['experiments'] is experiments + assert captured['weights'] is weights + assert captured['analysis'] is analysis + + +class _NoStructures: + """Structures double with no structures and no free parameters.""" + + free_parameters: list = [] + + def __iter__(self): + return iter([]) diff --git a/tests/unit/easydiffraction/datablocks/experiment/categories/background/test_estimate_coverage.py b/tests/unit/easydiffraction/datablocks/experiment/categories/background/test_estimate_coverage.py new file mode 100644 index 000000000..655104256 --- /dev/null +++ b/tests/unit/easydiffraction/datablocks/experiment/categories/background/test_estimate_coverage.py @@ -0,0 +1,395 @@ +# SPDX-FileCopyrightText: 2026 EasyScience contributors +# SPDX-License-Identifier: BSD-3-Clause + +import numpy as np +import pytest + +from easydiffraction.datablocks.experiment.categories.background import estimate + +estimate_background_curve = estimate.estimate_background_curve +BackgroundEstimate = estimate.BackgroundEstimate + + +def _collect_warnings(monkeypatch): + """Replace the module logger with a recorder and return the records list.""" + records = [] + + class _Log: + def warning(self, message, *args, **kwargs): + records.append(str(message)) + + monkeypatch.setattr(estimate, 'log', _Log()) + return records + + +# --------------------------------------------------------------------------- +# _robust_noise +# --------------------------------------------------------------------------- + + +def test_robust_noise_zero_for_flat_input(): + y = np.full(50, 7.0) + assert estimate._robust_noise(y) == 0.0 + + +def test_robust_noise_zero_for_linear_input(): + # A pure ramp has a vanishing second difference, hence zero noise. + y = np.linspace(0.0, 100.0, 80) + assert estimate._robust_noise(y) == pytest.approx(0.0, abs=1e-12) + + +def test_robust_noise_positive_for_noisy_input(): + rng = np.random.default_rng(0) + y = rng.normal(0.0, 1.0, size=500) + sigma = estimate._robust_noise(y) + # Recovered sigma is in the right ballpark of the injected sigma (1.0). + assert 0.5 < sigma < 2.0 + + +# --------------------------------------------------------------------------- +# _measure_width +# --------------------------------------------------------------------------- + + +def test_measure_width_fallback_when_no_peaks(): + # Monotonic ramp has no local maxima -> fallback width, empty peaks. + y = np.linspace(0.0, 10.0, 100) + width, peaks = estimate._measure_width(y, sigma=0.1) + assert width == estimate._FALLBACK_WIDTH + assert peaks.size == 0 + + +def test_measure_width_detects_peak_and_returns_positive_width(): + x = np.linspace(0.0, 10.0, 400) + y = np.exp(-((x - 5.0) ** 2) / (2.0 * 0.2**2)) + width, peaks = estimate._measure_width(y, sigma=0.0) + assert peaks.size >= 1 + assert width >= 1.0 + + +def test_measure_width_floor_is_one_point(): + # A one-sample-wide spike yields a measured FWHM below 1 point, which + # must be floored to 1.0. + y = np.zeros(50) + y[25] = 100.0 + width, peaks = estimate._measure_width(y, sigma=0.0) + assert peaks.size == 1 + assert width == pytest.approx(1.0) + + +# --------------------------------------------------------------------------- +# _forbidden_from_peaks +# --------------------------------------------------------------------------- + + +def test_forbidden_from_peaks_masks_window_around_each_peak(): + mask = estimate._forbidden_from_peaks(20, np.array([10]), width=2.0) + assert mask.dtype == bool + assert mask.shape == (20,) + # +/- ceil(2) around index 10 -> indices 8..12 inclusive. + assert np.all(mask[8:13]) + assert not mask[7] + assert not mask[13] + + +def test_forbidden_from_peaks_clamps_at_edges(): + mask = estimate._forbidden_from_peaks(5, np.array([0, 4]), width=3.0) + # Windows clamp to [0, n); both edges and everything between are masked. + assert np.all(mask) + + +def test_forbidden_from_peaks_empty_peaks_all_false(): + mask = estimate._forbidden_from_peaks(10, np.array([], dtype=int), width=2.0) + assert not np.any(mask) + + +# --------------------------------------------------------------------------- +# _derive_lam +# --------------------------------------------------------------------------- + + +def test_derive_lam_returns_floor_for_small_grids(): + # n * width below the floor -> the floor wins. + assert estimate._derive_lam(2, 1.0) == estimate._LAM_FLOOR + + +def test_derive_lam_grows_with_size_and_width(): + big = estimate._derive_lam(1000, 5.0) + assert big == pytest.approx(1000 * 5.0) + assert big > estimate._derive_lam(1000, 2.0) + + +def test_derive_lam_clamps_width_to_one(): + # Width below 1.0 is treated as 1.0 in the penalty. + assert estimate._derive_lam(500, 0.1) == estimate._derive_lam(500, 1.0) + + +# --------------------------------------------------------------------------- +# _stage1_baseline error path +# --------------------------------------------------------------------------- + + +def test_stage1_baseline_rejects_unknown_method(): + x = np.linspace(0.0, 10.0, 50) + y = np.ones_like(x) + with pytest.raises(ValueError, match='Unsupported Stage-1 background method'): + estimate._stage1_baseline(x, y, 'nope', width=5.0, smoothness=None) + + +def test_stage1_baseline_arpls_honours_smoothness_override(): + x = np.linspace(0.0, 10.0, 80) + y = 3.0 + 0.1 * x + _, params = estimate._stage1_baseline(x, y, 'arpls', width=5.0, smoothness=1234.0) + assert params['lam'] == pytest.approx(1234.0) + + +# --------------------------------------------------------------------------- +# _rdp_indices +# --------------------------------------------------------------------------- + + +def test_rdp_keeps_only_endpoints_for_straight_line(): + x = np.linspace(0.0, 10.0, 50) + curve = 2.0 + 0.5 * x + idx = estimate._rdp_indices(x, curve, epsilon=1e-9) + assert idx.tolist() == [0, 49] + + +def test_rdp_span_zero_segment_skipped(): + # All-equal x: the root segment has zero span, so the only retained + # points are the two endpoints (the span<=0 branch is taken). + x = np.array([0.0, 0.0, 0.0]) + curve = np.array([0.0, 9.0, 0.0]) + idx = estimate._rdp_indices(x, curve, epsilon=0.1) + assert idx.tolist() == [0, 2] + + +def test_rdp_keeps_sharp_deviation(): + x = np.linspace(0.0, 10.0, 11) + curve = np.zeros(11) + curve[5] = 100.0 + idx = estimate._rdp_indices(x, curve, epsilon=1.0) + assert 5 in idx.tolist() + assert idx[0] == 0 + assert idx[-1] == 10 + + +# --------------------------------------------------------------------------- +# _drop_forbidden +# --------------------------------------------------------------------------- + + +def test_drop_forbidden_keeps_endpoints_even_when_masked(): + indices = np.array([0, 3, 5, 7, 9]) + forbidden = np.zeros(10, dtype=bool) + forbidden[0] = True # endpoint on a peak -> still kept + forbidden[5] = True # interior on a peak -> dropped + kept = estimate._drop_forbidden(indices, forbidden, n=10) + assert kept.tolist() == [0, 3, 7, 9] + + +def test_drop_forbidden_deduplicates_and_sorts(): + indices = np.array([9, 0, 3, 3]) + forbidden = np.zeros(10, dtype=bool) + kept = estimate._drop_forbidden(indices, forbidden, n=10) + assert kept.tolist() == [0, 3, 9] + + +# --------------------------------------------------------------------------- +# _cap_by_deviation +# --------------------------------------------------------------------------- + + +def test_cap_by_deviation_noop_when_within_budget(): + x = np.linspace(0.0, 10.0, 20) + curve = np.sin(x) + indices = np.array([0, 5, 10, 19]) + out = estimate._cap_by_deviation(x, curve, indices, n_points=5) + assert np.array_equal(out, indices) + + +def test_cap_by_deviation_keeps_endpoints_and_most_deviating(): + x = np.linspace(0.0, 10.0, 11) + # Chord between endpoints is flat at 0; index 5 deviates the most. + curve = np.zeros(11) + curve[5] = 50.0 + curve[2] = 5.0 + indices = np.arange(11) + out = estimate._cap_by_deviation(x, curve, indices, n_points=3) + assert out.tolist() == [0, 5, 10] + + +def test_cap_by_deviation_keep_count_zero_returns_endpoints(): + x = np.linspace(0.0, 10.0, 11) + curve = np.sin(x) + indices = np.arange(11) + out = estimate._cap_by_deviation(x, curve, indices, n_points=2) + assert out.tolist() == [0, 10] + + +def test_cap_by_deviation_zero_span_uses_first_interior_slice(): + # Degenerate x (all equal) -> span<=0 branch: take the first + # ``keep_count`` interior anchors in order. + x = np.zeros(11) + curve = np.arange(11, dtype=float) + indices = np.arange(11) + out = estimate._cap_by_deviation(x, curve, indices, n_points=4) + # Endpoints 0 and 10 plus the first two interior indices (1, 2). + assert out.tolist() == [0, 1, 2, 10] + + +# --------------------------------------------------------------------------- +# _thin_to_anchors +# --------------------------------------------------------------------------- + + +def test_thin_to_anchors_uncapped_returns_rdp_result(): + x = np.linspace(0.0, 10.0, 50) + curve = 1.0 + 0.2 * x + forbidden = np.zeros(50, dtype=bool) + out = estimate._thin_to_anchors(x, curve, epsilon=1e-9, forbidden=forbidden, n_points=None) + assert out.tolist() == [0, 49] + + +def test_thin_to_anchors_grows_tolerance_to_meet_cap(): + # A finely-wiggling curve yields many RDP anchors at a small tolerance; + # the growth loop must reduce them to <= n_points. + x = np.linspace(0.0, 10.0, 400) + curve = np.sin(8.0 * x) + forbidden = np.zeros(400, dtype=bool) + out = estimate._thin_to_anchors(x, curve, epsilon=1e-6, forbidden=forbidden, n_points=8) + assert out.size <= 8 + assert out[0] == 0 + assert out[-1] == 399 + + +def test_thin_to_anchors_growth_alone_meets_cap(): + # A smooth single-frequency curve: growing the RDP tolerance alone + # brings the anchor count to the target, so the deviation cap is not + # needed (the size-already-within-budget branch after the loop). + x = np.linspace(0.0, 10.0, 400) + curve = np.sin(1.2 * x) + forbidden = np.zeros(400, dtype=bool) + out = estimate._thin_to_anchors(x, curve, epsilon=1e-3, forbidden=forbidden, n_points=12) + assert out.size <= 12 + assert out[0] == 0 + assert out[-1] == 399 + + +def test_thin_to_anchors_zero_tolerance_falls_back_to_deviation_cap(): + # With epsilon == 0 the growth loop cannot shrink the count (tolerance + # stays 0), so the deviation-based cap must enforce the bound. + x = np.linspace(0.0, 10.0, 200) + curve = np.sin(5.0 * x) + forbidden = np.zeros(200, dtype=bool) + out = estimate._thin_to_anchors(x, curve, epsilon=0.0, forbidden=forbidden, n_points=4) + assert out.size <= 4 + assert out[0] == 0 + assert out[-1] == 199 + + +# --------------------------------------------------------------------------- +# _flat_estimate +# --------------------------------------------------------------------------- + + +def test_flat_estimate_levels_at_data_minimum(): + x = np.linspace(0.0, 5.0, 6) + y = np.array([4.0, 2.0, 3.0, 5.0, 2.5, 6.0]) + est = estimate._flat_estimate(x, y, 'arpls', width=None, noise=0.1) + assert np.all(est.curve == 2.0) + assert est.anchors.tolist() == [[0.0, 2.0], [5.0, 2.0]] + assert est.width == estimate._FALLBACK_WIDTH + assert est.tolerance == pytest.approx(estimate._NOISE_TOLERANCE_FACTOR * 0.1) + assert est.backend_params == {} + + +def test_flat_estimate_respects_supplied_width(): + x = np.linspace(0.0, 1.0, 5) + y = np.ones(5) + est = estimate._flat_estimate(x, y, 'snip', width=42.0, noise=0.0) + assert est.width == 42.0 + assert est.method == 'snip' + + +def test_flat_estimate_handles_empty_arrays(): + est = estimate._flat_estimate(np.array([]), np.array([]), 'arpls', width=None, noise=0.0) + assert est.curve.size == 0 + assert est.anchors.shape == (0, 2) + assert est.width == estimate._FALLBACK_WIDTH + + +# --------------------------------------------------------------------------- +# estimate_background_curve short-pattern path +# --------------------------------------------------------------------------- + + +def test_short_pattern_returns_flat_estimate_with_warning(monkeypatch): + records = _collect_warnings(monkeypatch) + x = np.linspace(0.0, 1.0, 4) # below _MIN_POINTS (5) + y = np.array([3.0, 1.0, 2.0, 4.0]) + result = estimate_background_curve(x, y) + assert isinstance(result, BackgroundEstimate) + assert result.anchors.shape == (2, 2) + assert result.curve.size == 4 + assert np.all(result.curve == 1.0) # flat at the data minimum + assert any('too short' in r.lower() for r in records) + + +def test_short_pattern_keeps_supplied_width(monkeypatch): + _collect_warnings(monkeypatch) + x = np.linspace(0.0, 1.0, 3) + y = np.array([2.0, 2.0, 2.0]) + result = estimate_background_curve(x, y, width=7.0) + assert result.width == 7.0 + + +# --------------------------------------------------------------------------- +# estimate_background_curve width/peaks resolution branches +# --------------------------------------------------------------------------- + + +def test_supplied_width_and_peaks_skip_detection(): + # Both width and peaks supplied: the detection branch (524->529) is + # skipped entirely and the supplied values are honoured. + x = np.linspace(0.0, 10.0, 200) + y = 4.0 + 0.2 * x + 5.0 * np.exp(-((x - 5.0) ** 2) / (2.0 * 0.2**2)) + mask = (x > 4.5) & (x < 5.5) + result = estimate_background_curve(x, y, width=8.0, peaks=mask) + assert result.width == 8.0 + # No interior anchor falls inside the supplied forbidden mask. + for ax in result.anchors[1:-1, 0]: + idx = int(np.argmin(np.abs(x - ax))) + assert not mask[idx] + + +def test_supplied_peaks_only_still_measures_width(): + # peaks supplied, width None -> width is derived but the supplied mask + # is used (the width-only branch of 524->529). + x = np.linspace(0.0, 10.0, 200) + y = 4.0 + 5.0 * np.exp(-((x - 5.0) ** 2) / (2.0 * 0.2**2)) + mask = np.zeros(200, dtype=bool) + result = estimate_background_curve(x, y, peaks=mask, width=None) + assert result.width > 0 + + +def test_supplied_width_only_derives_mask_from_data(monkeypatch): + # width supplied, peaks None: the detection branch still runs to build + # the forbidden mask, but the supplied width is honoured verbatim. + _collect_warnings(monkeypatch) + x = np.linspace(0.0, 10.0, 300) + rng = np.random.default_rng(3) + y = 4.0 + 6.0 * np.exp(-((x - 5.0) ** 2) / (2.0 * 0.2**2)) + rng.normal(0.0, 0.05, size=300) + result = estimate_background_curve(x, y, width=6.0, peaks=None) + assert result.width == 6.0 + interior = result.anchors[1:-1, 0] + # No interior anchor sits on the planted peak (centre 5.0). + assert not np.any(np.abs(interior - 5.0) < 0.4) + + +def test_no_peaks_detected_emits_unreliable_warning(monkeypatch): + records = _collect_warnings(monkeypatch) + x = np.linspace(0.0, 10.0, 60) + y = np.full(60, 3.0) # perfectly flat: no peaks + estimate_background_curve(x, y, peaks=None) + assert any('unreliable' in r.lower() for r in records) diff --git a/tests/unit/easydiffraction/display/test_plotting_coverage.py b/tests/unit/easydiffraction/display/test_plotting_coverage.py index ed2d9d342..8e004e9c1 100644 --- a/tests/unit/easydiffraction/display/test_plotting_coverage.py +++ b/tests/unit/easydiffraction/display/test_plotting_coverage.py @@ -3,6 +3,7 @@ """Additional unit tests for display/plotting.py to cover patch gaps.""" import numpy as np +import pytest # ------------------------------------------------------------------ # PlotterEngineEnum @@ -570,3 +571,1905 @@ def test_plot_meas_vs_calc_without_residual(self, monkeypatch): assert calls[0][0] == 'powder_meas_vs_calc' assert calls[0][1].y_resid is None assert calls[0][1].bragg_tick_sets == () + + +# ------------------------------------------------------------------ +# PlotterEngineEnum.description fallback (unknown member) +# ------------------------------------------------------------------ + + +class TestPlotterEngineDescriptionFallback: + def test_unknown_member_returns_empty_string(self, monkeypatch): + """A StrEnum member that is neither ASCII nor PLOTLY returns ''.""" + from easydiffraction.display.plotting import PlotterEngineEnum + + # Build a stand-in that is *not* one of the known singletons so + # both `is` comparisons fall through to the empty-string return. + class FakeMember: + description = PlotterEngineEnum.description + + assert FakeMember().description() == '' + + +# ------------------------------------------------------------------ +# Plotter._series_column_names +# ------------------------------------------------------------------ + + +class TestSeriesColumnNames: + def test_unique_name_then_name(self): + from types import SimpleNamespace + + from easydiffraction.display.plotting import Plotter + + param = SimpleNamespace(unique_name='u', name='n') + assert Plotter._series_column_names(param) == ['u', 'n'] + + def test_duplicate_unique_and_name_deduplicated(self): + from types import SimpleNamespace + + from easydiffraction.display.plotting import Plotter + + param = SimpleNamespace(unique_name='same', name='same') + assert Plotter._series_column_names(param) == ['same'] + + def test_only_name_when_unique_name_blank(self): + from types import SimpleNamespace + + from easydiffraction.display.plotting import Plotter + + param = SimpleNamespace(unique_name='', name='n') + assert Plotter._series_column_names(param) == ['n'] + + def test_empty_when_no_string_attributes(self): + from easydiffraction.display.plotting import Plotter + + assert Plotter._series_column_names(object()) == [] + + +# ------------------------------------------------------------------ +# Plotter._numeric_series_values +# ------------------------------------------------------------------ + + +class TestNumericSeriesValues: + def test_bool_dtype_converted_to_float(self): + from easydiffraction.display.plotting import Plotter + + assert Plotter._numeric_series_values([True, False, True]) == [1.0, 0.0, 1.0] + + def test_string_truth_values_mapped(self): + from easydiffraction.display.plotting import Plotter + + assert Plotter._numeric_series_values(['True', 'False', 'true', 'false']) == [ + 1.0, + 0.0, + 1.0, + 0.0, + ] + + def test_numeric_strings_parsed(self): + from easydiffraction.display.plotting import Plotter + + assert Plotter._numeric_series_values(['1.5', '2.5']) == [1.5, 2.5] + + def test_invalid_string_raises(self): + import pytest + + from easydiffraction.display.plotting import Plotter + + with pytest.raises(ValueError, match='Unable to parse string'): + Plotter._numeric_series_values(['not_a_number']) + + +# ------------------------------------------------------------------ +# Plotter.plot_param_series routing +# ------------------------------------------------------------------ + + +class TestPlotParamSeriesRouting: + def test_warns_when_no_column_name(self, monkeypatch, capsys): + from easydiffraction.display.plotting import Plotter + from easydiffraction.utils.logging import Logger + + monkeypatch.setattr(Logger, '_reaction', Logger.Reaction.WARN, raising=True) + + p = Plotter() + # object() exposes neither unique_name nor name -> no columns + p.plot_param_series(object()) + out = capsys.readouterr().out + assert 'does not expose a CSV column name' in out + + def test_falls_back_to_snapshots_without_csv(self, monkeypatch): + from types import SimpleNamespace + + from easydiffraction.display.plotting import Plotter + + captured = {} + + class FakeAnalysis: + _parameter_snapshots = {'expt1': {'param_a': {}}} + + class FakeProject: + info = SimpleNamespace(path=None) + experiments = {'expt1': object()} + analysis = FakeAnalysis() + + p = Plotter() + p._set_project(FakeProject()) + + def fake_snapshots(unique_name, versus, experiments, snapshots): + captured['unique_name'] = unique_name + captured['versus'] = versus + captured['snapshots'] = snapshots + + p.plot_param_series_from_snapshots = fake_snapshots + p.plot_param_series(SimpleNamespace(unique_name='param_a', name='param_a'), versus='v') + + assert captured['unique_name'] == 'param_a' + assert captured['versus'] == 'v' + assert captured['snapshots'] == {'expt1': {'param_a': {}}} + + def test_uses_csv_when_results_file_present(self, monkeypatch, tmp_path): + from types import SimpleNamespace + + from easydiffraction.display.plotting import Plotter + + analysis_dir = tmp_path / 'analysis' + analysis_dir.mkdir(parents=True) + (analysis_dir / 'results.csv').write_text('param_a\n1.0\n') + + captured = {} + + class FakeProject: + info = SimpleNamespace(path=str(tmp_path)) + experiments = {} + analysis = SimpleNamespace(_parameter_snapshots={}) + + p = Plotter() + p._set_project(FakeProject()) + + def fake_from_csv(*, csv_path, column_names, param_descriptor, versus_path): + captured['csv_path'] = csv_path + captured['column_names'] = column_names + + p._plot_param_series_from_csv = fake_from_csv + p.plot_param_series(SimpleNamespace(unique_name='param_a', name='param_a')) + + assert captured['csv_path'].endswith('results.csv') + assert captured['column_names'] == ['param_a'] + + +# ------------------------------------------------------------------ +# Plotter.plot_all_param_series +# ------------------------------------------------------------------ + + +class TestPlotAllParamSeries: + def test_warns_when_no_fitted_params(self, monkeypatch, capsys): + from types import SimpleNamespace + + from easydiffraction.display.plotting import Plotter + from easydiffraction.utils.logging import Logger + + monkeypatch.setattr(Logger, '_reaction', Logger.Reaction.WARN, raising=True) + + class FakeProject: + info = SimpleNamespace(path=None) + analysis = SimpleNamespace(_parameter_snapshots={}) + + p = Plotter() + p._set_project(FakeProject()) + p.plot_all_param_series() + out = capsys.readouterr().out + assert 'No fitted parameters found to plot' in out + + def test_plots_each_descriptor_and_skips_missing(self, monkeypatch, capsys): + from types import SimpleNamespace + + from easydiffraction.display.plotting import Plotter + from easydiffraction.utils.logging import Logger + + monkeypatch.setattr(Logger, '_reaction', Logger.Reaction.WARN, raising=True) + + p = Plotter() + # Two fitted names; only one has a descriptor in the project. + monkeypatch.setattr( + Plotter, + '_collect_fitted_param_unique_names', + lambda self: ['present', 'missing'], + ) + descriptor = SimpleNamespace(unique_name='present') + monkeypatch.setattr( + Plotter, + '_fitted_param_descriptors_by_unique_name', + lambda self: {'present': descriptor}, + ) + + plotted = [] + p.plot_param_series = lambda *, param, versus: plotted.append((param, versus)) + + p.plot_all_param_series(versus='temp') + + out = capsys.readouterr().out + assert plotted == [(descriptor, 'temp')] + assert "'missing' not found in project" in out + + +# ------------------------------------------------------------------ +# Plotter._collect_fitted_param_unique_names +# ------------------------------------------------------------------ + + +class TestCollectFittedParamUniqueNames: + def test_from_csv_filters_meta_and_diffrn_and_uncertainty(self, tmp_path): + from types import SimpleNamespace + + from easydiffraction.analysis.sequential import _META_COLUMNS + from easydiffraction.display.plotting import Plotter + + analysis_dir = tmp_path / 'analysis' + analysis_dir.mkdir(parents=True) + meta_col = next(iter(_META_COLUMNS)) + header = f'{meta_col},param_a,param_a.uncertainty,diffrn.temp\n' + (analysis_dir / 'results.csv').write_text(header + '1,2,0.1,300\n') + + class FakeProject: + info = SimpleNamespace(path=str(tmp_path)) + analysis = SimpleNamespace(_parameter_snapshots={}) + + p = Plotter() + p._set_project(FakeProject()) + assert p._collect_fitted_param_unique_names() == ['param_a'] + + def test_from_snapshots_when_no_csv(self): + from types import SimpleNamespace + + from easydiffraction.display.plotting import Plotter + + class FakeProject: + info = SimpleNamespace(path=None) + analysis = SimpleNamespace(_parameter_snapshots={'e1': {'param_a': {}, 'param_b': {}}}) + + p = Plotter() + p._set_project(FakeProject()) + assert p._collect_fitted_param_unique_names() == ['param_a', 'param_b'] + + def test_empty_when_no_csv_and_no_snapshots(self): + from types import SimpleNamespace + + from easydiffraction.display.plotting import Plotter + + class FakeProject: + info = SimpleNamespace(path=None) + analysis = SimpleNamespace(_parameter_snapshots={}) + + p = Plotter() + p._set_project(FakeProject()) + assert p._collect_fitted_param_unique_names() == [] + + +# ------------------------------------------------------------------ +# Plotter._versus_field_name / _versus_axis_label +# ------------------------------------------------------------------ + + +class TestVersusHelpers: + def test_field_name_none(self): + from easydiffraction.display.plotting import Plotter + + assert Plotter._versus_field_name(None) is None + + def test_field_name_strips_diffrn_prefix(self): + from easydiffraction.display.plotting import Plotter + + assert Plotter._versus_field_name('diffrn.ambient_temperature') == 'ambient_temperature' + + def test_field_name_passthrough_without_prefix(self): + from easydiffraction.display.plotting import Plotter + + assert Plotter._versus_field_name('ambient_temperature') == 'ambient_temperature' + + def test_axis_label_descriptor_with_units(self): + from types import SimpleNamespace + + from easydiffraction.display.plotting import Plotter + + descriptor = SimpleNamespace(description='Temperature', name='t', units='K') + assert Plotter._versus_axis_label('diffrn.t', descriptor) == 'Temperature (K)' + + def test_axis_label_descriptor_without_units(self): + from types import SimpleNamespace + + from easydiffraction.display.plotting import Plotter + + descriptor = SimpleNamespace(description='Temperature', name='t', units=None) + assert Plotter._versus_axis_label('diffrn.t', descriptor) == 'Temperature' + + def test_axis_label_falls_back_to_field_name(self): + from easydiffraction.display.plotting import Plotter + + assert Plotter._versus_axis_label('diffrn.ambient_temperature', None) == ( + 'ambient temperature' + ) + + def test_axis_label_default_experiment_number(self): + from easydiffraction.display.plotting import Plotter + + assert Plotter._versus_axis_label(None, None) == 'Experiment No.' + + +# ------------------------------------------------------------------ +# Plotter._resolve_versus_descriptor_from_path +# ------------------------------------------------------------------ + + +class TestResolveVersusDescriptorFromPath: + def test_none_path_returns_none(self): + from easydiffraction.display.plotting import Plotter + + assert Plotter()._resolve_versus_descriptor_from_path(None) is None + + def test_no_project_returns_none(self): + from easydiffraction.display.plotting import Plotter + + p = Plotter() + p._project = None + assert p._resolve_versus_descriptor_from_path('diffrn.ambient_temperature') is None + + def test_no_experiments_returns_none(self): + from types import SimpleNamespace + + from easydiffraction.display.plotting import Plotter + + p = Plotter() + p._set_project(SimpleNamespace(experiments={})) + assert p._resolve_versus_descriptor_from_path('diffrn.ambient_temperature') is None + + def test_resolves_descriptor_from_first_experiment(self): + from types import SimpleNamespace + + from easydiffraction.display.plotting import Plotter + + sentinel = object() + + class Diffrn: + ambient_temperature = sentinel + + experiment = SimpleNamespace(diffrn=Diffrn()) + p = Plotter() + p._set_project(SimpleNamespace(experiments={'e1': experiment})) + result = p._resolve_versus_descriptor_from_path('diffrn.ambient_temperature') + assert result is sentinel + + +# ------------------------------------------------------------------ +# Plotter._validated_max_parameter_count +# ------------------------------------------------------------------ + + +class TestValidatedMaxParameterCount: + def test_valid_value_passes_through(self): + from easydiffraction.display.plotting import Plotter + + assert Plotter._validated_max_parameter_count(5, minimum=1) == 5 + + def test_non_integer_raises_type_error(self): + import pytest + + from easydiffraction.display.plotting import Plotter + + with pytest.raises(TypeError, match='must be an integer'): + Plotter._validated_max_parameter_count(2.5, minimum=1) + + def test_bool_rejected_as_non_integer(self): + import pytest + + from easydiffraction.display.plotting import Plotter + + bool_value = True + with pytest.raises(TypeError, match='must be an integer'): + Plotter._validated_max_parameter_count(bool_value, minimum=1) + + def test_below_minimum_raises_value_error(self): + import pytest + + from easydiffraction.display.plotting import Plotter + + with pytest.raises(ValueError, match='at least 2'): + Plotter._validated_max_parameter_count(1, minimum=2) + + +# ------------------------------------------------------------------ +# Plotter._auto_filtered_correlation_dataframe +# ------------------------------------------------------------------ + + +class TestAutoFilteredCorrelationDataframe: + def test_returns_unchanged_when_within_limit(self): + import pandas as pd + + from easydiffraction.display.plotting import Plotter + + corr = pd.DataFrame( + [[1.0, 0.5], [0.5, 1.0]], + index=['a', 'b'], + columns=['a', 'b'], + ) + result, threshold = Plotter._auto_filtered_correlation_dataframe(corr, max_parameters=4) + assert threshold == 0.0 + assert list(result.index) == ['a', 'b'] + + def test_picks_threshold_to_fit_within_limit(self): + import pandas as pd + + from easydiffraction.display.plotting import Plotter + + corr = pd.DataFrame( + [ + [1.0, 0.9, 0.1], + [0.9, 1.0, 0.1], + [0.1, 0.1, 1.0], + ], + index=['a', 'b', 'c'], + columns=['a', 'b', 'c'], + ) + result, threshold = Plotter._auto_filtered_correlation_dataframe(corr, max_parameters=2) + assert list(result.index) == ['a', 'b'] + assert threshold > 0.0 + + def test_no_off_diagonal_correlation_truncates(self): + import numpy as np + import pandas as pd + + from easydiffraction.display.plotting import Plotter + + corr = pd.DataFrame( + np.eye(3), + index=['a', 'b', 'c'], + columns=['a', 'b', 'c'], + ) + result, threshold = Plotter._auto_filtered_correlation_dataframe(corr, max_parameters=2) + assert list(result.index) == ['a', 'b'] + assert threshold == 0.0 + + def test_falls_back_to_top_parameters_by_strength(self): + import pandas as pd + + from easydiffraction.display.plotting import Plotter + + # No single threshold lands the count exactly in [1, 2]; the + # final top-strength fallback selects the two strongest params. + corr = pd.DataFrame( + [ + [1.0, 0.8, 0.8, 0.8], + [0.8, 1.0, 0.8, 0.8], + [0.8, 0.8, 1.0, 0.8], + [0.8, 0.8, 0.8, 1.0], + ], + index=['a', 'b', 'c', 'd'], + columns=['a', 'b', 'c', 'd'], + ) + result, threshold = Plotter._auto_filtered_correlation_dataframe(corr, max_parameters=2) + assert result.shape == (2, 2) + assert threshold == 0.0 + + +# ------------------------------------------------------------------ +# Plotter._correlation_filtered_title / _posterior_pair_title +# ------------------------------------------------------------------ + + +class TestTitleHelpers: + def test_filtered_title_without_threshold(self): + from easydiffraction.display.plotting import Plotter + + assert Plotter._correlation_filtered_title('Base', 0.0) == 'Base' + + def test_filtered_title_with_threshold(self): + from easydiffraction.display.plotting import Plotter + + title = Plotter._correlation_filtered_title('Base', 0.5) + assert title.startswith('Base with |correlation|') + assert '0.50' in title + + def test_posterior_pair_title_none(self): + from easydiffraction.display.plotting import Plotter + + assert Plotter._posterior_pair_title(None) == 'Posterior pair plot' + + def test_posterior_pair_title_with_multiplier(self): + from easydiffraction.display.plotting import Plotter + + title = Plotter._posterior_pair_title(2.0) + assert '2' in title + assert 'uncertainty region' in title + + +# ------------------------------------------------------------------ +# Plotter._posterior_pair_uncertainty_multiplier +# ------------------------------------------------------------------ + + +class TestPosteriorPairUncertaintyMultiplier: + def _fit_results(self, multipliers): + from types import SimpleNamespace + + parameters = [ + SimpleNamespace(unique_name=f'p{i}', fit_bounds_uncertainty_multiplier=mult) + for i, mult in enumerate(multipliers) + ] + return SimpleNamespace(parameters=parameters) + + def test_shared_multiplier_returned(self): + from easydiffraction.display.plotting import Plotter + + fit_results = self._fit_results([3.0, 3.0]) + result = Plotter._posterior_pair_uncertainty_multiplier(fit_results, ['p0', 'p1']) + assert result == 3.0 + + def test_missing_parameter_returns_none(self): + from easydiffraction.display.plotting import Plotter + + fit_results = self._fit_results([3.0]) + assert Plotter._posterior_pair_uncertainty_multiplier(fit_results, ['missing']) is None + + def test_none_multiplier_returns_none(self): + from easydiffraction.display.plotting import Plotter + + fit_results = self._fit_results([None]) + assert Plotter._posterior_pair_uncertainty_multiplier(fit_results, ['p0']) is None + + def test_inconsistent_multipliers_return_none(self): + from easydiffraction.display.plotting import Plotter + + fit_results = self._fit_results([3.0, 5.0]) + assert Plotter._posterior_pair_uncertainty_multiplier(fit_results, ['p0', 'p1']) is None + + +# ------------------------------------------------------------------ +# Plotter._get_fit_result_for_correlation +# ------------------------------------------------------------------ + + +class TestGetFitResultForCorrelation: + def test_warns_when_no_project(self, monkeypatch, capsys): + from easydiffraction.display.plotting import Plotter + from easydiffraction.utils.logging import Logger + + monkeypatch.setattr(Logger, '_reaction', Logger.Reaction.WARN, raising=True) + + p = Plotter() + p._project = None + assert p._get_fit_result_for_correlation() is None + assert 'not attached to a project' in capsys.readouterr().out + + def test_warns_when_no_fit_results(self, monkeypatch, capsys): + from types import SimpleNamespace + + from easydiffraction.display.plotting import Plotter + from easydiffraction.utils.logging import Logger + + monkeypatch.setattr(Logger, '_reaction', Logger.Reaction.WARN, raising=True) + + p = Plotter() + p._set_project(SimpleNamespace(analysis=SimpleNamespace(fit_results=None))) + assert p._get_fit_result_for_correlation() is None + assert 'No fit results available' in capsys.readouterr().out + + def test_returns_fit_results_when_present(self): + from types import SimpleNamespace + + from easydiffraction.display.plotting import Plotter + + sentinel = object() + p = Plotter() + p._set_project(SimpleNamespace(analysis=SimpleNamespace(fit_results=sentinel))) + assert p._get_fit_result_for_correlation() is sentinel + + +# ------------------------------------------------------------------ +# Plotter._correlation_from_covariance / _get_correlation_labels +# ------------------------------------------------------------------ + + +class TestCorrelationFromCovariance: + def test_valid_covariance(self): + import numpy as np + + from easydiffraction.display.plotting import Plotter + + covar = np.array([[4.0, 1.0], [1.0, 9.0]]) + corr = Plotter._correlation_from_covariance(covar, ['p1', 'p2'], []) + np.testing.assert_allclose(np.diag(corr.to_numpy()), [1.0, 1.0]) + np.testing.assert_allclose(corr.iloc[0, 1], 1.0 / 6.0) + + def test_non_square_returns_none(self, monkeypatch, capsys): + import numpy as np + + from easydiffraction.display.plotting import Plotter + from easydiffraction.utils.logging import Logger + + monkeypatch.setattr(Logger, '_reaction', Logger.Reaction.WARN, raising=True) + + covar = np.array([[1.0, 0.0, 0.0], [0.0, 1.0, 0.0]]) + assert Plotter._correlation_from_covariance(covar, ['a', 'b'], []) is None + assert 'invalid covariance matrix' in capsys.readouterr().out + + def test_size_mismatch_with_var_names_returns_none(self, monkeypatch, capsys): + import numpy as np + + from easydiffraction.display.plotting import Plotter + from easydiffraction.utils.logging import Logger + + monkeypatch.setattr(Logger, '_reaction', Logger.Reaction.WARN, raising=True) + + covar = np.array([[4.0, 1.0], [1.0, 9.0]]) + assert Plotter._correlation_from_covariance(covar, ['only_one'], []) is None + assert 'does not match the fitted parameter list' in capsys.readouterr().out + + def test_labels_use_minimizer_uid_mapping(self): + from types import SimpleNamespace + + from easydiffraction.display.plotting import Plotter + + params = [ + SimpleNamespace(_minimizer_uid='u1', unique_name='phase.a', name='a'), + SimpleNamespace(_minimizer_uid='u2', unique_name='phase.b', name='b'), + ] + labels = Plotter._get_correlation_labels(params, ['u1', 'u2', 'u3']) + assert labels == ['phase.a', 'phase.b', 'u3'] + + +# ------------------------------------------------------------------ +# Plotter._get_param_correlation_dataframe_from_engine_params +# ------------------------------------------------------------------ + + +class TestEngineParamsCorrelation: + def test_no_params_returns_none(self): + from types import SimpleNamespace + + from easydiffraction.display.plotting import Plotter + + raw = SimpleNamespace(params=None, var_names=['p1']) + assert Plotter()._get_param_correlation_dataframe_from_engine_params(raw, []) is None + + def test_no_correlations_returns_none(self): + from types import SimpleNamespace + + from easydiffraction.display.plotting import Plotter + + raw = SimpleNamespace( + params={'p1': SimpleNamespace(correl=None), 'p2': SimpleNamespace(correl={})}, + var_names=['p1', 'p2'], + ) + assert Plotter()._get_param_correlation_dataframe_from_engine_params(raw, []) is None + + def test_builds_symmetric_matrix_from_correl(self): + from types import SimpleNamespace + + import numpy as np + + from easydiffraction.display.plotting import Plotter + + raw = SimpleNamespace( + params={ + 'p1': SimpleNamespace(correl={'p2': 0.5, 'unknown': 0.9}), + 'p2': SimpleNamespace(correl={'p1': 0.5}), + }, + var_names=['p1', 'p2'], + ) + params = [ + SimpleNamespace(_minimizer_uid='p1', unique_name='a', name='a'), + SimpleNamespace(_minimizer_uid='p2', unique_name='b', name='b'), + ] + corr = Plotter()._get_param_correlation_dataframe_from_engine_params(raw, params) + np.testing.assert_allclose(corr.to_numpy(), [[1.0, 0.5], [0.5, 1.0]]) + assert list(corr.index) == ['a', 'b'] + + +# ------------------------------------------------------------------ +# Plotter._excluded_ranges +# ------------------------------------------------------------------ + + +class TestExcludedRanges: + def test_none_excluded_regions_returns_empty(self): + from easydiffraction.display.plotting import Plotter + + assert Plotter._excluded_ranges(experiment=object(), x_min=0.0, x_max=10.0) == () + + def test_clips_to_view_window(self): + from types import SimpleNamespace + + from easydiffraction.display.plotting import Plotter + + region = SimpleNamespace( + start=SimpleNamespace(value=1.0), + end=SimpleNamespace(value=5.0), + ) + experiment = SimpleNamespace(excluded_regions=[region]) + result = Plotter._excluded_ranges(experiment=experiment, x_min=2.0, x_max=4.0) + assert result == ((2.0, 4.0),) + + def test_drops_regions_outside_window(self): + from types import SimpleNamespace + + from easydiffraction.display.plotting import Plotter + + inside = SimpleNamespace( + start=SimpleNamespace(value=2.0), + end=SimpleNamespace(value=3.0), + ) + outside = SimpleNamespace( + start=SimpleNamespace(value=20.0), + end=SimpleNamespace(value=30.0), + ) + experiment = SimpleNamespace(excluded_regions=[inside, outside]) + result = Plotter._excluded_ranges(experiment=experiment, x_min=0.0, x_max=10.0) + assert result == ((2.0, 3.0),) + + def test_uses_infinite_bounds_when_none(self): + from types import SimpleNamespace + + from easydiffraction.display.plotting import Plotter + + region = SimpleNamespace( + start=SimpleNamespace(value=1.0), + end=SimpleNamespace(value=5.0), + ) + experiment = SimpleNamespace(excluded_regions=[region]) + result = Plotter._excluded_ranges(experiment=experiment, x_min=None, x_max=None) + assert result == ((1.0, 5.0),) + + +# ------------------------------------------------------------------ +# Plotter._bragg_tick_d_spacing +# ------------------------------------------------------------------ + + +class TestBraggTickDSpacing: + def test_two_theta_path(self): + from types import SimpleNamespace + + import numpy as np + + from easydiffraction.display.plotting import Plotter + from easydiffraction.utils.utils import twotheta_to_d + + refln = SimpleNamespace(two_theta=np.array([20.0, 40.0])) + experiment = SimpleNamespace( + instrument=SimpleNamespace(setup_wavelength=SimpleNamespace(value=1.5)) + ) + result = Plotter._bragg_tick_d_spacing(refln=refln, experiment=experiment) + np.testing.assert_allclose(result, twotheta_to_d(np.array([20.0, 40.0]), 1.5)) + + def test_time_of_flight_path(self): + from types import SimpleNamespace + + import numpy as np + + from easydiffraction.display.plotting import Plotter + from easydiffraction.utils.utils import tof_to_d + + refln = SimpleNamespace(time_of_flight=np.array([1000.0, 2000.0])) + # Intentionally has no two_theta attribute so the tof branch runs. + instrument = SimpleNamespace( + calib_d_to_tof_offset=SimpleNamespace(value=0.0), + calib_d_to_tof_linear=SimpleNamespace(value=1.0), + calib_d_to_tof_quad=SimpleNamespace(value=0.0), + ) + experiment = SimpleNamespace(instrument=instrument) + result = Plotter._bragg_tick_d_spacing(refln=refln, experiment=experiment) + np.testing.assert_allclose(result, tof_to_d(np.array([1000.0, 2000.0]), 0.0, 1.0, 0.0)) + + def test_falls_back_to_explicit_d_spacing(self): + from types import SimpleNamespace + + import numpy as np + + from easydiffraction.display.plotting import Plotter + + refln = SimpleNamespace(d_spacing=np.array([1.1, 2.2])) + result = Plotter._bragg_tick_d_spacing(refln=refln, experiment=object()) + np.testing.assert_allclose(result, np.array([1.1, 2.2])) + + +# ------------------------------------------------------------------ +# Plotter._bragg_tick_x_values / _bragg_tick_attr / _bragg_tick_arrays +# ------------------------------------------------------------------ + + +class TestBraggTickResolution: + def test_unsupported_axis_warns_and_returns_none(self, monkeypatch, capsys): + from types import SimpleNamespace + + from easydiffraction.display.plotting import Plotter + from easydiffraction.utils.logging import Logger + + monkeypatch.setattr(Logger, '_reaction', Logger.Reaction.WARN, raising=True) + + result = Plotter._bragg_tick_x_values( + refln=SimpleNamespace(), + experiment=object(), + expt_name='E1', + x_axis='sin_theta_over_lambda', + ) + assert result is None + assert 'Unsupported Bragg tick x axis' in capsys.readouterr().out + + def test_attr_missing_warns_and_returns_none(self, monkeypatch, capsys): + from types import SimpleNamespace + + from easydiffraction.display.plotting import Plotter + from easydiffraction.utils.logging import Logger + + monkeypatch.setattr(Logger, '_reaction', Logger.Reaction.WARN, raising=True) + + result = Plotter._bragg_tick_attr(SimpleNamespace(two_theta=None), 'two_theta', 'E1') + assert result is None + assert "does not expose 'two_theta'" in capsys.readouterr().out + + def test_arrays_missing_field_warns_and_returns_none(self, monkeypatch, capsys): + from types import SimpleNamespace + + import numpy as np + + from easydiffraction.display.plotting import Plotter + from easydiffraction.utils.logging import Logger + + monkeypatch.setattr(Logger, '_reaction', Logger.Reaction.WARN, raising=True) + + # phase_id present but f_calc missing -> returns None. + refln = SimpleNamespace( + phase_id=np.array(['a']), + index_h=np.array([1]), + index_k=np.array([0]), + index_l=np.array([1]), + f_squared_calc=np.array([1.0]), + f_calc=None, + ) + assert Plotter._bragg_tick_arrays(refln=refln, expt_name='E1') is None + assert "missing 'f_calc'" in capsys.readouterr().out + + +# ------------------------------------------------------------------ +# Plotter._square_matrix_axis_title_label(s) +# ------------------------------------------------------------------ + + +class TestSquareMatrixAxisTitleLabel: + def test_plain_name_without_dot(self): + from easydiffraction.display.plotting import Plotter + + assert Plotter._square_matrix_axis_title_label('length_a') == 'length_a' + + def test_dotted_name_split_into_lines(self): + from easydiffraction.display.plotting import Plotter + + assert Plotter._square_matrix_axis_title_label('phase.cell.length_a') == ( + 'phase.
cell.
length_a' + ) + + def test_empty_name_returns_empty(self): + from easydiffraction.display.plotting import Plotter + + assert Plotter._square_matrix_axis_title_label(' ') == '' + + def test_labels_helper_maps_each_name(self): + from easydiffraction.display.plotting import Plotter + + result = Plotter._square_matrix_axis_title_labels(['a', 'x.y']) + assert result == ['a', 'x.
y'] + + def test_line_count(self): + from easydiffraction.display.plotting import Plotter + + assert Plotter._square_matrix_axis_title_line_count('') == 1 + assert Plotter._square_matrix_axis_title_line_count('a
b
c') == 3 + + +# ------------------------------------------------------------------ +# Plotter._posterior_density_axis_range +# ------------------------------------------------------------------ + + +class TestPosteriorDensityAxisRange: + def test_empty_returns_none(self): + import numpy as np + + from easydiffraction.display.plotting import Plotter + + assert Plotter._posterior_density_axis_range(np.array([])) is None + + def test_all_non_finite_returns_none(self): + import numpy as np + + from easydiffraction.display.plotting import Plotter + + assert Plotter._posterior_density_axis_range(np.array([np.nan, np.inf])) is None + + def test_positive_range_padded_above(self): + import numpy as np + + from easydiffraction.display.plotting import Plotter + + lower, upper = Plotter._posterior_density_axis_range(np.array([1.0, 2.0, 3.0])) + assert lower == 0.0 + assert upper > 3.0 + + def test_flat_values_use_fallback_padding(self): + import numpy as np + + from easydiffraction.display.plotting import Plotter + + lower, upper = Plotter._posterior_density_axis_range(np.array([5.0, 5.0])) + assert lower == 0.0 + assert upper > 5.0 + + +# ------------------------------------------------------------------ +# Plotter._posterior_axis_bounds +# ------------------------------------------------------------------ + + +class TestPosteriorAxisBounds: + def test_explicit_bounds_used_directly(self): + import numpy as np + + from easydiffraction.display.plotting import Plotter + + lower, upper = Plotter._posterior_axis_bounds( + np.array([1.0, 2.0, 3.0]), + lower_bound=-1.0, + upper_bound=10.0, + ) + assert (lower, upper) == (-1.0, 10.0) + + def test_data_driven_bounds_padded(self): + import numpy as np + + from easydiffraction.display.plotting import Plotter + + lower, upper = Plotter._posterior_axis_bounds( + np.array([1.0, 2.0, 3.0]), + lower_bound=None, + upper_bound=None, + ) + assert lower < 1.0 + assert upper > 3.0 + + def test_flat_data_uses_fallback_padding(self): + import numpy as np + + from easydiffraction.display.plotting import Plotter + + lower, upper = Plotter._posterior_axis_bounds( + np.array([4.0, 4.0]), + lower_bound=None, + upper_bound=None, + ) + assert lower < 4.0 + assert upper > 4.0 + + +# ------------------------------------------------------------------ +# Plotter._posterior_density_curve +# ------------------------------------------------------------------ + + +class TestPosteriorDensityCurve: + def test_too_few_samples_returns_none(self): + import numpy as np + + from easydiffraction.display.plotting import Plotter + + assert ( + Plotter._posterior_density_curve(np.array([1.0]), lower_bound=None, upper_bound=None) + is None + ) + + def test_constant_samples_make_narrow_peak(self): + import numpy as np + + from easydiffraction.display.plotting import Plotter + + grid, density = Plotter._posterior_density_curve( + np.array([2.0, 2.0, 2.0, 2.0]), + lower_bound=1.0, + upper_bound=3.0, + ) + assert grid.shape == density.shape + # Density integrates to ~1. + np.testing.assert_allclose(np.trapezoid(density, grid), 1.0, rtol=1e-6) + + def test_inverted_bounds_returns_none(self): + import numpy as np + + from easydiffraction.display.plotting import Plotter + + assert ( + Plotter._posterior_density_curve( + np.array([1.0, 2.0, 3.0]), + lower_bound=5.0, + upper_bound=1.0, + ) + is None + ) + + def test_varied_samples_normalize_to_unit_area(self): + import numpy as np + + from easydiffraction.display.plotting import Plotter + + rng = np.random.default_rng(0) + values = rng.normal(0.0, 1.0, size=500) + grid, density = Plotter._posterior_density_curve( + values, + lower_bound=None, + upper_bound=None, + ) + np.testing.assert_allclose(np.trapezoid(density, grid), 1.0, rtol=1e-3) + + +# ------------------------------------------------------------------ +# Plotter._selected_posterior_samples / _thin_posterior_samples +# ------------------------------------------------------------------ + + +class TestSelectedAndThinnedSamples: + def test_selected_reorders_columns(self): + from types import SimpleNamespace + + import numpy as np + + from easydiffraction.display.plotting import Plotter + + flattened = np.array([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]]) + posterior = SimpleNamespace( + parameter_names=['a', 'b', 'c'], + flattened=lambda: flattened, + ) + result = Plotter._selected_posterior_samples(posterior, ['c', 'a']) + np.testing.assert_allclose(result, np.array([[3.0, 1.0], [6.0, 4.0]])) + + def test_selected_unknown_name_returns_none(self): + from types import SimpleNamespace + + import numpy as np + + from easydiffraction.display.plotting import Plotter + + posterior = SimpleNamespace( + parameter_names=['a', 'b'], + flattened=lambda: np.zeros((2, 2)), + ) + assert Plotter._selected_posterior_samples(posterior, ['missing']) is None + + def test_selected_empty_names_returns_none(self): + from types import SimpleNamespace + + from easydiffraction.display.plotting import Plotter + + posterior = SimpleNamespace(parameter_names=[], flattened=lambda: None) + assert Plotter._selected_posterior_samples(posterior, ['a']) is None + + def test_thin_keeps_small_arrays(self): + import numpy as np + + from easydiffraction.display.plotting import Plotter + + samples = np.arange(10).reshape(5, 2) + np.testing.assert_array_equal( + Plotter._thin_posterior_samples(samples, max_points=10), + samples, + ) + + def test_thin_downsamples_large_arrays(self): + import numpy as np + + from easydiffraction.display.plotting import Plotter + + samples = np.arange(200).reshape(100, 2) + thinned = Plotter._thin_posterior_samples(samples, max_points=10) + assert thinned.shape == (10, 2) + + +# ------------------------------------------------------------------ +# Plotter._posterior_plot_labels +# ------------------------------------------------------------------ + + +class TestPosteriorPlotLabels: + def test_unknown_parameter_uses_name_directly(self): + from types import SimpleNamespace + + from easydiffraction.display.plotting import Plotter + + fit_results = SimpleNamespace(parameters=[]) + assert Plotter._posterior_plot_labels(fit_results, ['unknown']) == ['unknown'] + + def test_entry_name_prefixes_short_name(self): + from types import SimpleNamespace + + from easydiffraction.display.plotting import Plotter + + parameter = SimpleNamespace( + unique_name='phase.cell.length_a', + name='length_a', + _identity=SimpleNamespace(category_entry_name='cell'), + ) + fit_results = SimpleNamespace(parameters=[parameter]) + assert Plotter._posterior_plot_labels(fit_results, ['phase.cell.length_a']) == [ + 'cell length_a' + ] + + def test_no_entry_name_uses_short_name(self): + from types import SimpleNamespace + + from easydiffraction.display.plotting import Plotter + + parameter = SimpleNamespace( + unique_name='length_a', + name='length_a', + _identity=SimpleNamespace(category_entry_name=''), + ) + fit_results = SimpleNamespace(parameters=[parameter]) + assert Plotter._posterior_plot_labels(fit_results, ['length_a']) == ['length_a'] + + +# ------------------------------------------------------------------ +# Plotter._posterior_summary_by_name / _posterior_parameter_bounds +# ------------------------------------------------------------------ + + +class TestPosteriorSummaryAndBounds: + def test_summary_by_name_maps_unique_names(self): + from types import SimpleNamespace + + from easydiffraction.display.plotting import Plotter + + summary = SimpleNamespace(unique_name='a') + fit_results = SimpleNamespace(posterior_parameter_summaries=[summary]) + assert Plotter._posterior_summary_by_name(fit_results) == {'a': summary} + + def test_bounds_missing_parameter_returns_none_pair(self): + from types import SimpleNamespace + + from easydiffraction.display.plotting import Plotter + + fit_results = SimpleNamespace(parameters=[]) + assert Plotter._posterior_parameter_bounds( + fit_results=fit_results, parameter_name='missing' + ) == (None, None) + + def test_bounds_finite_values(self): + from types import SimpleNamespace + + from easydiffraction.display.plotting import Plotter + + parameter = SimpleNamespace(unique_name='a', fit_min=1.0, fit_max=5.0) + fit_results = SimpleNamespace(parameters=[parameter]) + assert Plotter._posterior_parameter_bounds( + fit_results=fit_results, parameter_name='a' + ) == (1.0, 5.0) + + def test_bounds_non_finite_values_dropped(self): + from types import SimpleNamespace + + import numpy as np + + from easydiffraction.display.plotting import Plotter + + parameter = SimpleNamespace(unique_name='a', fit_min=np.inf, fit_max=None) + fit_results = SimpleNamespace(parameters=[parameter]) + assert Plotter._posterior_parameter_bounds( + fit_results=fit_results, parameter_name='a' + ) == (None, None) + + +# ------------------------------------------------------------------ +# Plotter._posterior_pair_show_contours / contour panel counts +# ------------------------------------------------------------------ + + +class TestPosteriorPairContourDecisions: + def test_full_style_always_shows(self): + from easydiffraction.display.plotting import Plotter + from easydiffraction.display.plotting import PosteriorPairPlotStyleEnum + + assert ( + Plotter._posterior_pair_show_contours( + n_parameters=20, style=PosteriorPairPlotStyleEnum.FULL + ) + is True + ) + + def test_fast_style_never_shows(self): + from easydiffraction.display.plotting import Plotter + from easydiffraction.display.plotting import PosteriorPairPlotStyleEnum + + assert ( + Plotter._posterior_pair_show_contours( + n_parameters=2, style=PosteriorPairPlotStyleEnum.FAST + ) + is False + ) + + def test_auto_depends_on_parameter_count(self): + from easydiffraction.display.plotting import POSTERIOR_PAIR_AUTO_MAX_CONTOUR_PARAMETERS + from easydiffraction.display.plotting import Plotter + from easydiffraction.display.plotting import PosteriorPairPlotStyleEnum + + assert Plotter._posterior_pair_show_contours( + n_parameters=POSTERIOR_PAIR_AUTO_MAX_CONTOUR_PARAMETERS, + style=PosteriorPairPlotStyleEnum.AUTO, + ) + assert not Plotter._posterior_pair_show_contours( + n_parameters=POSTERIOR_PAIR_AUTO_MAX_CONTOUR_PARAMETERS + 1, + style=PosteriorPairPlotStyleEnum.AUTO, + ) + + def test_contour_panel_count_minimum(self): + from easydiffraction.display.plotting import Plotter + + assert Plotter._posterior_pair_contour_panel_count(1) == 1 + assert Plotter._posterior_pair_contour_panel_count(4) == 6 + + def test_validated_style_rejects_unknown(self): + import pytest + + from easydiffraction.display.plotting import Plotter + + with pytest.raises(ValueError, match='style must be one of'): + Plotter._validated_posterior_pair_plot_style('nope') + + def test_validated_style_accepts_enum_value(self): + from easydiffraction.display.plotting import Plotter + from easydiffraction.display.plotting import PosteriorPairPlotStyleEnum + + assert ( + Plotter._validated_posterior_pair_plot_style('full') is PosteriorPairPlotStyleEnum.FULL + ) + + +# ------------------------------------------------------------------ +# Plotter._posterior_pair_correlation_value / contour colorscales +# ------------------------------------------------------------------ + + +class TestPosteriorPairCorrelationValue: + def test_too_few_finite_points_returns_none(self): + import numpy as np + + from easydiffraction.display.plotting import Plotter + + x = np.array([1.0, np.nan]) + y = np.array([np.nan, 2.0]) + assert Plotter._posterior_pair_correlation_value(x, y) is None + + def test_positive_correlation(self): + import numpy as np + + from easydiffraction.display.plotting import Plotter + + x = np.array([1.0, 2.0, 3.0, 4.0]) + y = np.array([1.0, 2.0, 3.0, 4.0]) + assert Plotter._posterior_pair_correlation_value(x, y) > 0.99 + + def test_constant_input_returns_none(self): + import numpy as np + + from easydiffraction.display.plotting import Plotter + + x = np.array([1.0, 1.0, 1.0]) + y = np.array([1.0, 2.0, 3.0]) + # corrcoef yields nan for a constant series (and warns on the + # internal divide-by-zero); the helper detects the nan and + # returns None. + with np.errstate(invalid='ignore', divide='ignore'): + assert Plotter._posterior_pair_correlation_value(x, y) is None + + def test_colorscales_negative_correlation(self): + import numpy as np + + from easydiffraction.display.plotting import POSTERIOR_NEGATIVE_CONTOUR_FILL_COLORSCALE + from easydiffraction.display.plotting import Plotter + + x = np.array([1.0, 2.0, 3.0, 4.0]) + y = np.array([4.0, 3.0, 2.0, 1.0]) + fill, _ = Plotter._posterior_pair_contour_colorscales(x, y) + assert fill is POSTERIOR_NEGATIVE_CONTOUR_FILL_COLORSCALE + + def test_colorscales_positive_correlation(self): + import numpy as np + + from easydiffraction.display.plotting import POSTERIOR_CONTOUR_FILL_COLORSCALE + from easydiffraction.display.plotting import Plotter + + x = np.array([1.0, 2.0, 3.0, 4.0]) + y = np.array([1.0, 2.0, 3.0, 4.0]) + fill, _ = Plotter._posterior_pair_contour_colorscales(x, y) + assert fill is POSTERIOR_CONTOUR_FILL_COLORSCALE + + +# ------------------------------------------------------------------ +# Plotter._posterior_contour_levels +# ------------------------------------------------------------------ + + +class TestPosteriorContourLevels: + def test_uses_provided_finite_levels(self): + import numpy as np + + from easydiffraction.display.plotting import Plotter + + start, end, size = Plotter._posterior_contour_levels( + density=np.ones((3, 3)), + contour_levels=np.array([0.1, 0.2, 0.4]), + ) + assert start == 0.1 + assert end == 0.4 + assert size == pytest.approx(0.1) + + def test_single_level_uses_density_fallback_size(self): + import numpy as np + + from easydiffraction.display.plotting import Plotter + + # Single level cannot yield end>start; falls back to density max. + density = np.array([[0.0, 1.0], [1.0, 2.0]]) + start, end, size = Plotter._posterior_contour_levels( + density=density, + contour_levels=np.array([0.5]), + ) + assert end > start + assert size > 0 + + def test_none_levels_derive_from_density(self): + import numpy as np + + from easydiffraction.display.plotting import Plotter + + density = np.array([[0.0, 2.0], [4.0, 10.0]]) + start, end, size = Plotter._posterior_contour_levels( + density=density, + contour_levels=None, + ) + assert start == pytest.approx(2.0) + assert end == pytest.approx(9.5) + assert size == pytest.approx(1.5) + + +# ------------------------------------------------------------------ +# Plotter._correlation_heatmap_edges / centers / values / customdata +# ------------------------------------------------------------------ + + +class TestCorrelationHeatmapArrays: + def test_edges_zero_parameters(self): + import numpy as np + + from easydiffraction.display.plotting import Plotter + + np.testing.assert_array_equal(Plotter._correlation_heatmap_edges(0), np.array([0.0])) + + def test_edges_single_parameter_no_gap(self): + import numpy as np + + from easydiffraction.display.plotting import Plotter + + np.testing.assert_array_equal(Plotter._correlation_heatmap_edges(1), np.array([0.0, 1.0])) + + def test_edges_multiple_parameters_include_gaps(self): + from easydiffraction.display.plotting import Plotter + + edges = Plotter._correlation_heatmap_edges(3) + # 2*n - 1 widths => n + (n-1) gap segments => 2n edges. + assert len(edges) == 6 + assert edges[0] == 0.0 + + def test_centers_offset_by_half(self): + import numpy as np + + from easydiffraction.display.plotting import Plotter + + centers = Plotter._correlation_heatmap_centers(2) + np.testing.assert_allclose(centers[0], 0.5) + + def test_values_place_data_on_even_indices(self): + import numpy as np + import pandas as pd + + from easydiffraction.display.plotting import Plotter + + corr = pd.DataFrame( + [[1.0, 0.5], [0.5, 1.0]], + index=['a', 'b'], + columns=['a', 'b'], + ) + expanded = Plotter._correlation_heatmap_values(corr) + assert expanded.shape == (3, 3) + assert expanded[0, 0] == 1.0 + assert np.isnan(expanded[1, 1]) + + def test_customdata_labels_on_even_indices(self): + import pandas as pd + + from easydiffraction.display.plotting import Plotter + + corr = pd.DataFrame( + [[1.0, 0.5], [0.5, 1.0]], + index=['row0', 'row1'], + columns=['col0', 'col1'], + ) + custom = Plotter._correlation_heatmap_customdata(corr) + assert custom.shape == (3, 3, 2) + assert custom[0, 0, 0] == 'col0' + assert custom[0, 0, 1] == 'row0' + assert custom[1, 1, 0] == '' + + +# ------------------------------------------------------------------ +# Plotter._square_matrix_gap_data_width / plot extent +# ------------------------------------------------------------------ + + +class TestSquareMatrixGeometry: + def test_gap_width_zero_for_single_parameter(self): + from easydiffraction.display.plotting import Plotter + + assert Plotter._square_matrix_gap_data_width(1) == 0.0 + + def test_gap_width_positive_for_multiple(self): + from easydiffraction.display.plotting import Plotter + + assert Plotter._square_matrix_gap_data_width(4) > 0.0 + + def test_plot_extent_matches_parameter_count_without_gap(self): + from easydiffraction.display.plotting import Plotter + + assert Plotter._square_matrix_plot_extent(1) == 1.0 + + def test_extra_axis_title_margin_zero_for_single_line(self): + from easydiffraction.display.plotting import Plotter + + assert Plotter._square_matrix_extra_axis_title_margin(['a', 'b']) == 0 + + def test_extra_axis_title_margin_for_multiline(self): + from easydiffraction.display.plotting import SQUARE_MATRIX_AXIS_TITLE_LINE_HEIGHT_PIXELS + from easydiffraction.display.plotting import Plotter + + margin = Plotter._square_matrix_extra_axis_title_margin(['a
b
c']) + assert margin == 2 * SQUARE_MATRIX_AXIS_TITLE_LINE_HEIGHT_PIXELS + + def test_extra_axis_title_margin_empty_labels(self): + from easydiffraction.display.plotting import Plotter + + assert Plotter._square_matrix_extra_axis_title_margin([]) == 0 + + +# ------------------------------------------------------------------ +# Plotter._posterior_pair_cell_size_pixels +# ------------------------------------------------------------------ + + +class TestPosteriorPairCellSize: + def test_zero_parameters_returns_default(self): + from easydiffraction.display.plotting import PAIR_PLOT_CELL_SIZE_PIXELS + from easydiffraction.display.plotting import Plotter + + assert ( + Plotter._posterior_pair_cell_size_pixels(0, available_width_pixels=980) + == PAIR_PLOT_CELL_SIZE_PIXELS + ) + + def test_large_count_clamped_to_minimum(self): + from easydiffraction.display.plotting import PAIR_PLOT_MIN_CELL_SIZE_PIXELS + from easydiffraction.display.plotting import Plotter + + size = Plotter._posterior_pair_cell_size_pixels(100, available_width_pixels=980) + assert size >= PAIR_PLOT_MIN_CELL_SIZE_PIXELS + + +# ------------------------------------------------------------------ +# Plotter._show_background_enabled / _show_bragg_enabled +# ------------------------------------------------------------------ + + +class TestShowFlags: + def test_background_default_follows_availability(self): + from types import SimpleNamespace + + from easydiffraction.display.plotting import Plotter + + options = SimpleNamespace(show_background=None) + assert Plotter._show_background_enabled(options, background_available=True) is True + assert Plotter._show_background_enabled(options, background_available=False) is False + + def test_background_explicit_true_requires_availability(self): + from types import SimpleNamespace + + from easydiffraction.display.plotting import Plotter + + options = SimpleNamespace(show_background=True) + assert Plotter._show_background_enabled(options, background_available=False) is False + assert Plotter._show_background_enabled(options, background_available=True) is True + + def test_bragg_default_true(self): + from types import SimpleNamespace + + from easydiffraction.display.plotting import Plotter + + assert Plotter._show_bragg_enabled(SimpleNamespace(show_bragg=None)) is True + + def test_bragg_explicit_false(self): + from types import SimpleNamespace + + from easydiffraction.display.plotting import Plotter + + assert Plotter._show_bragg_enabled(SimpleNamespace(show_bragg=False)) is False + + +# ------------------------------------------------------------------ +# Plotter.plot_posterior_predictive style validation +# ------------------------------------------------------------------ + + +class TestPlotPosteriorPredictiveValidation: + def test_invalid_style_raises(self): + import pytest + + from easydiffraction.display.plotting import Plotter + + with pytest.raises(ValueError, match='style must be'): + Plotter().plot_posterior_predictive('hrpt', style='invalid') + + def test_warns_when_no_project(self, monkeypatch, capsys): + from easydiffraction.display.plotting import Plotter + from easydiffraction.utils.logging import Logger + + monkeypatch.setattr(Logger, '_reaction', Logger.Reaction.WARN, raising=True) + + p = Plotter() + p._project = None + p.plot_posterior_predictive('hrpt') + assert 'not attached to a project' in capsys.readouterr().out + + +# ------------------------------------------------------------------ +# Plotter._get_posterior_samples_and_fit_results +# ------------------------------------------------------------------ + + +class TestGetPosteriorSamplesAndFitResults: + def test_non_plotly_engine_warns(self, monkeypatch, capsys): + from easydiffraction.display.plotting import Plotter + from easydiffraction.utils.logging import Logger + + monkeypatch.setattr(Logger, '_reaction', Logger.Reaction.WARN, raising=True) + + p = Plotter() + p.engine = 'asciichartpy' + samples, results = p._get_posterior_samples_and_fit_results() + assert samples is None + assert results is None + assert 'require the Plotly plotting backend' in capsys.readouterr().out + + def test_no_posterior_samples_warns(self, monkeypatch, capsys): + from types import SimpleNamespace + + from easydiffraction.display.plotting import Plotter + from easydiffraction.utils.logging import Logger + + monkeypatch.setattr(Logger, '_reaction', Logger.Reaction.WARN, raising=True) + + p = Plotter() + p.engine = 'plotly' + monkeypatch.setattr( + Plotter, + '_get_fit_result_for_correlation', + lambda self: SimpleNamespace(posterior_samples=None), + ) + samples, results = p._get_posterior_samples_and_fit_results() + assert samples is None + assert results is None + assert 'Posterior samples are unavailable' in capsys.readouterr().out + + +# ------------------------------------------------------------------ +# Plotter._get_or_build_posterior_predictive_summary early returns +# ------------------------------------------------------------------ + + +class TestGetOrBuildPosteriorPredictiveEarlyReturns: + def test_no_fit_results_returns_none(self, monkeypatch): + from easydiffraction.display.plotting import Plotter + + p = Plotter() + monkeypatch.setattr(Plotter, '_get_fit_result_for_correlation', lambda self: None) + assert ( + p._get_or_build_posterior_predictive_summary( + experiment=object(), expt_name='e', x_axis='two_theta' + ) + is None + ) + + def test_no_posterior_predictive_returns_none(self, monkeypatch): + from types import SimpleNamespace + + from easydiffraction.display.plotting import Plotter + + p = Plotter() + monkeypatch.setattr( + Plotter, + '_get_fit_result_for_correlation', + lambda self: SimpleNamespace(posterior_predictive=None), + ) + assert ( + p._get_or_build_posterior_predictive_summary( + experiment=object(), expt_name='e', x_axis='two_theta' + ) + is None + ) + + +# ------------------------------------------------------------------ +# Plotter._posterior_predictive_draw_indices +# ------------------------------------------------------------------ + + +class TestPosteriorPredictiveDrawIndices: + def test_small_count_returns_all_indices(self): + import numpy as np + + from easydiffraction.display.plotting import Plotter + + np.testing.assert_array_equal( + Plotter._posterior_predictive_draw_indices(3), np.array([0, 1, 2]) + ) + + def test_large_count_evenly_spaced_and_unique(self): + import numpy as np + + from easydiffraction.display.plotting import DEFAULT_POSTERIOR_PREDICTIVE_DRAWS + from easydiffraction.display.plotting import Plotter + + indices = Plotter._posterior_predictive_draw_indices(10000) + assert indices[0] == 0 + assert indices[-1] == 9999 + assert len(indices) <= DEFAULT_POSTERIOR_PREDICTIVE_DRAWS + assert len(np.unique(indices)) == len(indices) + + +# ------------------------------------------------------------------ +# Plotter._correlation_from_posterior_samples edge cases +# ------------------------------------------------------------------ + + +class TestCorrelationFromPosteriorSamples: + def test_no_parameter_names_returns_none(self, monkeypatch, capsys): + from types import SimpleNamespace + + from easydiffraction.display.plotting import Plotter + from easydiffraction.utils.logging import Logger + + monkeypatch.setattr(Logger, '_reaction', Logger.Reaction.WARN, raising=True) + + posterior = SimpleNamespace(parameter_names=[], flattened=lambda: None) + assert Plotter._correlation_from_posterior_samples(posterior) is None + assert 'do not expose parameter names' in capsys.readouterr().out + + def test_wrong_shape_returns_none(self, monkeypatch, capsys): + from types import SimpleNamespace + + import numpy as np + + from easydiffraction.display.plotting import Plotter + from easydiffraction.utils.logging import Logger + + monkeypatch.setattr(Logger, '_reaction', Logger.Reaction.WARN, raising=True) + + # 3 columns but only 2 parameter names. + posterior = SimpleNamespace( + parameter_names=['a', 'b'], + flattened=lambda: np.zeros((4, 3)), + ) + assert Plotter._correlation_from_posterior_samples(posterior) is None + assert 'invalid shape' in capsys.readouterr().out + + def test_too_few_draws_returns_none(self, monkeypatch, capsys): + from types import SimpleNamespace + + import numpy as np + + from easydiffraction.display.plotting import Plotter + from easydiffraction.utils.logging import Logger + + monkeypatch.setattr(Logger, '_reaction', Logger.Reaction.WARN, raising=True) + + posterior = SimpleNamespace( + parameter_names=['a', 'b'], + flattened=lambda: np.zeros((1, 2)), + ) + assert Plotter._correlation_from_posterior_samples(posterior) is None + assert 'two posterior draws' in capsys.readouterr().out + + +# ------------------------------------------------------------------ +# Plotter._raw_fit_result_for_correlation +# ------------------------------------------------------------------ + + +class TestRawFitResultForCorrelation: + def test_no_raw_result_returns_none(self): + from types import SimpleNamespace + + from easydiffraction.display.plotting import Plotter + + fit_results = SimpleNamespace(result=None, engine_result=None) + assert Plotter()._raw_fit_result_for_correlation(fit_results) is None + + def test_no_var_names_returns_none(self): + from types import SimpleNamespace + + from easydiffraction.display.plotting import Plotter + + raw = SimpleNamespace(var_names=[]) + fit_results = SimpleNamespace(result=raw) + assert Plotter()._raw_fit_result_for_correlation(fit_results) is None + + def test_returns_raw_with_var_names(self): + from types import SimpleNamespace + + from easydiffraction.display.plotting import Plotter + + raw = SimpleNamespace(var_names=['p1']) + fit_results = SimpleNamespace(result=None, engine_result=raw) + assert Plotter()._raw_fit_result_for_correlation(fit_results) is raw + + +# ------------------------------------------------------------------ +# Plotter._trim_correlation_display_dataframe +# ------------------------------------------------------------------ + + +class TestTrimCorrelationDisplayDataframe: + def test_show_diagonal_keeps_all(self): + import numpy as np + import pandas as pd + + from easydiffraction.display.plotting import Plotter + + corr = pd.DataFrame(np.eye(3)) + result, rows, cols = Plotter._trim_correlation_display_dataframe( + corr, preserve_all_rows=True, show_diagonal=True + ) + assert result.shape == (3, 3) + assert rows == [1, 2, 3] + assert cols == [1, 2, 3] + + def test_preserve_all_rows_trims_last_column_only(self): + import numpy as np + import pandas as pd + + from easydiffraction.display.plotting import Plotter + + corr = pd.DataFrame(np.eye(3)) + result, rows, cols = Plotter._trim_correlation_display_dataframe( + corr, preserve_all_rows=True, show_diagonal=False + ) + assert result.shape == (3, 2) + assert rows == [1, 2, 3] + assert cols == [1, 2] + + def test_graphical_trims_first_row_and_last_column(self): + import numpy as np + import pandas as pd + + from easydiffraction.display.plotting import Plotter + + corr = pd.DataFrame(np.eye(3)) + result, rows, cols = Plotter._trim_correlation_display_dataframe( + corr, preserve_all_rows=False, show_diagonal=False + ) + assert result.shape == (2, 2) + assert rows == [2, 3] + assert cols == [1, 2] + + def test_single_dimension_not_trimmed(self): + import pandas as pd + + from easydiffraction.display.plotting import Plotter + + corr = pd.DataFrame([[1.0]]) + result, rows, cols = Plotter._trim_correlation_display_dataframe( + corr, preserve_all_rows=False, show_diagonal=False + ) + assert result.shape == (1, 1) + assert rows == [1] + assert cols == [1] + + +# ------------------------------------------------------------------ +# Plotter._mask_correlation_lower_triangle +# ------------------------------------------------------------------ + + +class TestMaskCorrelationLowerTriangle: + def test_upper_triangle_and_diagonal_masked(self): + import numpy as np + import pandas as pd + + from easydiffraction.display.plotting import Plotter + + corr = pd.DataFrame( + [[1.0, 0.5, 0.3], [0.5, 1.0, 0.2], [0.3, 0.2, 1.0]], + index=['a', 'b', 'c'], + columns=['a', 'b', 'c'], + ) + masked = Plotter._mask_correlation_lower_triangle(corr) + # Diagonal and upper triangle are NaN. + assert np.isnan(masked.iloc[0, 0]) + assert np.isnan(masked.iloc[0, 1]) + # Lower triangle retained. + assert masked.iloc[1, 0] == 0.5 + assert masked.iloc[2, 0] == 0.3 + + +# ------------------------------------------------------------------ +# Plotter._filter_correlation_dataframe +# ------------------------------------------------------------------ + + +class TestFilterCorrelationDataframe: + def test_zero_threshold_returns_unchanged(self): + import pandas as pd + + from easydiffraction.display.plotting import Plotter + + corr = pd.DataFrame([[1.0, 0.5], [0.5, 1.0]], index=['a', 'b'], columns=['a', 'b']) + result = Plotter._filter_correlation_dataframe(corr, threshold=0) + assert result is corr + + def test_above_one_raises(self): + import pytest + + from easydiffraction.display.plotting import Plotter + + with pytest.raises(ValueError, match='between 0 and 1'): + Plotter._filter_correlation_dataframe(object(), threshold=2.0) + + def test_no_pairs_above_threshold_returns_none(self, monkeypatch, capsys): + import pandas as pd + + from easydiffraction.display.plotting import Plotter + from easydiffraction.utils.logging import Logger + + monkeypatch.setattr(Logger, '_reaction', Logger.Reaction.WARN, raising=True) + + corr = pd.DataFrame( + [[1.0, 0.1], [0.1, 1.0]], + index=['a', 'b'], + columns=['a', 'b'], + ) + assert Plotter._filter_correlation_dataframe(corr, threshold=0.9) is None + assert 'No parameter pairs' in capsys.readouterr().out + + def test_keeps_correlated_pairs(self): + import pandas as pd + + from easydiffraction.display.plotting import Plotter + + corr = pd.DataFrame( + [ + [1.0, 0.95, 0.1], + [0.95, 1.0, 0.1], + [0.1, 0.1, 1.0], + ], + index=['a', 'b', 'c'], + columns=['a', 'b', 'c'], + ) + result = Plotter._filter_correlation_dataframe(corr, threshold=0.5) + assert list(result.index) == ['a', 'b'] diff --git a/tests/unit/easydiffraction/display/test_progress_coverage.py b/tests/unit/easydiffraction/display/test_progress_coverage.py new file mode 100644 index 000000000..b49e5d453 --- /dev/null +++ b/tests/unit/easydiffraction/display/test_progress_coverage.py @@ -0,0 +1,1006 @@ +# SPDX-FileCopyrightText: 2026 EasyScience contributors +# SPDX-License-Identifier: BSD-3-Clause +"""Supplementary coverage tests for display/progress.py.""" + +from __future__ import annotations + +import easydiffraction.display.progress as progress_mod +from easydiffraction.utils.enums import VerbosityEnum + +_BOOM = 'boom' + + +class _FakeHTML: + """Minimal stand-in for ``IPython.display.HTML``.""" + + def __init__(self, data): + self.data = data + + +class _FakeJavascript: + """Minimal stand-in for ``IPython.display.Javascript``.""" + + def __init__(self, data): + self.data = data + + +class _FakeDisplayHandle: + """Minimal stand-in for ``IPython.display.DisplayHandle``.""" + + def __init__(self): + self.displayed = [] + self.updated = [] + + def display(self, obj): + self.displayed.append(obj) + + def update(self, obj): + self.updated.append(obj) + + +# --------------------------------------------------------------------------- +# resolve_activity_terminal_style +# --------------------------------------------------------------------------- + + +def test_resolve_activity_terminal_style_default_when_no_console(): + style = progress_mod.resolve_activity_terminal_style(None) + + assert style == progress_mod.ACTIVITY_TERMINAL_STYLE + + +def test_resolve_activity_terminal_style_fallback_for_standard(): + class FakeConsole: + color_system = 'standard' + + style = progress_mod.resolve_activity_terminal_style(FakeConsole()) + + assert style == progress_mod.ACTIVITY_TERMINAL_FALLBACK_STYLE + + +def test_resolve_activity_terminal_style_fallback_for_windows(): + class FakeConsole: + color_system = 'windows' + + style = progress_mod.resolve_activity_terminal_style(FakeConsole()) + + assert style == progress_mod.ACTIVITY_TERMINAL_FALLBACK_STYLE + + +def test_resolve_activity_terminal_style_accent_for_truecolor(): + class FakeConsole: + color_system = 'truecolor' + + style = progress_mod.resolve_activity_terminal_style(FakeConsole()) + + assert style == progress_mod.ACTIVITY_TERMINAL_STYLE + + +# --------------------------------------------------------------------------- +# make_display_handle (Jupyter branch) +# --------------------------------------------------------------------------- + + +def test_make_display_handle_returns_display_handle_in_jupyter(monkeypatch): + monkeypatch.setattr(progress_mod, 'in_jupyter', lambda: True) + monkeypatch.setattr(progress_mod, 'DisplayHandle', _FakeDisplayHandle) + monkeypatch.setattr(progress_mod, 'HTML', _FakeHTML) + + handle = progress_mod.make_display_handle() + + assert isinstance(handle, _FakeDisplayHandle) + # An empty HTML payload was displayed to reserve the output area. + assert len(handle.displayed) == 1 + assert isinstance(handle.displayed[0], _FakeHTML) + assert handle.displayed[0].data == '' + + +# --------------------------------------------------------------------------- +# _TerminalLiveHandle internals +# --------------------------------------------------------------------------- + + +def test_terminal_live_handle_resolves_callable_renderable(monkeypatch): + monkeypatch.setattr(progress_mod, 'in_jupyter', lambda: False) + monkeypatch.setattr(progress_mod.ConsoleManager, 'get', lambda: 'console') + + captured = {} + + class FakeLive: + def __init__(self, **kwargs): + captured.update(kwargs) + self.refresh_calls = 0 + + def start(self): + pass + + def stop(self): + pass + + def refresh(self): + self.refresh_calls += 1 + + monkeypatch.setattr(progress_mod, 'Live', FakeLive) + + handle = progress_mod.make_display_handle() + # A callable renderable is invoked lazily by the live get_renderable hook. + handle.update(lambda: 'lazy-value') + + assert captured['get_renderable']() == 'lazy-value' + + +def test_terminal_live_handle_close_suppresses_stop_errors(monkeypatch): + monkeypatch.setattr(progress_mod, 'in_jupyter', lambda: False) + monkeypatch.setattr(progress_mod.ConsoleManager, 'get', lambda: 'console') + + class FakeLive: + def __init__(self, **kwargs): + pass + + def start(self): + pass + + def stop(self): + raise RuntimeError(_BOOM) + + def refresh(self): + pass + + monkeypatch.setattr(progress_mod, 'Live', FakeLive) + + handle = progress_mod.make_display_handle() + # Errors raised by Live.stop() are swallowed so teardown never fails. + handle.close() + + +# --------------------------------------------------------------------------- +# ActivityIndicator.start (Jupyter branch) and update/stop +# --------------------------------------------------------------------------- + + +def test_activity_indicator_start_in_jupyter_displays_html(monkeypatch): + monkeypatch.setattr(progress_mod, 'in_jupyter', lambda: True) + monkeypatch.setattr(progress_mod, 'DisplayHandle', _FakeDisplayHandle) + monkeypatch.setattr(progress_mod, 'HTML', _FakeHTML) + + indicator = progress_mod.ActivityIndicator(label='Fitting...', verbosity=VerbosityEnum.FULL) + indicator.start() + + assert isinstance(indicator._display_handle, _FakeDisplayHandle) + assert indicator._live is None + assert len(indicator._display_handle.displayed) == 1 + assert 'Fitting...' in indicator._display_handle.displayed[0].data + + +def test_activity_indicator_start_is_idempotent_when_running(monkeypatch): + monkeypatch.setattr(progress_mod, 'in_jupyter', lambda: True) + monkeypatch.setattr(progress_mod, 'DisplayHandle', _FakeDisplayHandle) + monkeypatch.setattr(progress_mod, 'HTML', _FakeHTML) + + indicator = progress_mod.ActivityIndicator(verbosity=VerbosityEnum.FULL) + indicator.start() + first_handle = indicator._display_handle + + indicator.start() + + # A second start() while running must not replace the handle. + assert indicator._display_handle is first_handle + + +def test_activity_indicator_update_keeps_label_and_content_when_none(monkeypatch): + monkeypatch.setattr(progress_mod, 'in_jupyter', lambda: True) + monkeypatch.setattr(progress_mod, 'DisplayHandle', _FakeDisplayHandle) + monkeypatch.setattr(progress_mod, 'HTML', _FakeHTML) + + indicator = progress_mod.ActivityIndicator(label='Fitting...', verbosity=VerbosityEnum.FULL) + indicator.start() + indicator.update(content='partial output') + + assert indicator._label == 'Fitting...' + assert indicator._content == 'partial output' + + indicator.update() # both None: label and content unchanged + + assert indicator._label == 'Fitting...' + assert indicator._content == 'partial output' + + +def test_activity_indicator_update_replaces_label(monkeypatch): + monkeypatch.setattr(progress_mod, 'in_jupyter', lambda: True) + monkeypatch.setattr(progress_mod, 'DisplayHandle', _FakeDisplayHandle) + monkeypatch.setattr(progress_mod, 'HTML', _FakeHTML) + + indicator = progress_mod.ActivityIndicator(label='Fitting...', verbosity=VerbosityEnum.FULL) + indicator.start() + indicator.update(label='Sampling...') + + assert indicator._label == 'Sampling...' + + +def test_activity_indicator_stop_with_final_label_keeps_label(monkeypatch): + monkeypatch.setattr(progress_mod, 'in_jupyter', lambda: True) + monkeypatch.setattr(progress_mod, 'DisplayHandle', _FakeDisplayHandle) + monkeypatch.setattr(progress_mod, 'HTML', _FakeHTML) + + indicator = progress_mod.ActivityIndicator(label='Fitting...', verbosity=VerbosityEnum.FULL) + indicator.start() + indicator.stop(final_label='Done') + + assert indicator._running is False + assert indicator._keep_stopped_label is True + assert indicator._label == 'Done' + assert indicator._display_handle is None + + +def test_activity_indicator_stop_without_final_label(monkeypatch): + monkeypatch.setattr(progress_mod, 'in_jupyter', lambda: False) + monkeypatch.setattr(progress_mod.ConsoleManager, 'get', lambda: 'console') + + class FakeLive: + def __init__(self, **kwargs): + self.stopped = False + + def start(self): + pass + + def stop(self): + self.stopped = True + + def refresh(self): + pass + + monkeypatch.setattr(progress_mod, 'Live', FakeLive) + + indicator = progress_mod.ActivityIndicator(verbosity=VerbosityEnum.FULL) + indicator.start() + live = indicator._live + indicator.stop() + + assert indicator._running is False + assert indicator._keep_stopped_label is False + assert live.stopped is True + assert indicator._live is None + + +def test_activity_indicator_stop_suppresses_live_stop_errors(monkeypatch): + monkeypatch.setattr(progress_mod, 'in_jupyter', lambda: False) + monkeypatch.setattr(progress_mod.ConsoleManager, 'get', lambda: 'console') + + class FakeLive: + def __init__(self, **kwargs): + pass + + def start(self): + pass + + def stop(self): + raise RuntimeError(_BOOM) + + def refresh(self): + pass + + monkeypatch.setattr(progress_mod, 'Live', FakeLive) + + indicator = progress_mod.ActivityIndicator(verbosity=VerbosityEnum.FULL) + indicator.start() + # stop() must not propagate Live.stop() errors. + indicator.stop() + + assert indicator._live is None + + +# --------------------------------------------------------------------------- +# _terminal_renderable +# --------------------------------------------------------------------------- + + +def test_terminal_renderable_empty_when_idle(): + indicator = progress_mod.ActivityIndicator(verbosity=VerbosityEnum.FULL) + # Not running, no content, no kept label -> empty Text. + result = indicator._terminal_renderable() + + assert isinstance(result, progress_mod.Text) + assert result.plain == '' + + +def test_terminal_renderable_single_indicator_line(): + indicator = progress_mod.ActivityIndicator( + label='Fitting...', verbosity=VerbosityEnum.FULL, animated=False + ) + indicator._running = True + result = indicator._terminal_renderable() + + # Only the indicator line present -> returned directly, not grouped. + assert isinstance(result, progress_mod.Text) + assert result.plain == 'Fitting...' + + +def test_terminal_renderable_groups_content_and_indicator(): + indicator = progress_mod.ActivityIndicator( + label='Fitting...', verbosity=VerbosityEnum.FULL, animated=False + ) + indicator._running = True + indicator._content = 'iteration 5' + result = indicator._terminal_renderable() + + # Content plus indicator line -> a Rich Group. + assert isinstance(result, progress_mod.Group) + + +def test_terminal_renderable_content_only_when_stopped(): + indicator = progress_mod.ActivityIndicator(verbosity=VerbosityEnum.FULL) + content = progress_mod.Text('final output') + indicator._content = content + result = indicator._terminal_renderable() + + # Stopped with no kept label: only the content renderable remains, and a + # single renderable is returned directly rather than wrapped in a Group. + assert result is content + assert not isinstance(result, progress_mod.Group) + + +# --------------------------------------------------------------------------- +# _terminal_content +# --------------------------------------------------------------------------- + + +def test_terminal_content_none_returns_none(): + indicator = progress_mod.ActivityIndicator(verbosity=VerbosityEnum.FULL) + + assert indicator._terminal_content() is None + + +def test_terminal_content_passes_through_renderable(): + indicator = progress_mod.ActivityIndicator(verbosity=VerbosityEnum.FULL) + renderable = progress_mod.Text('already renderable') + indicator._content = renderable + + assert indicator._terminal_content() is renderable + + +def test_terminal_content_wraps_non_renderable_in_text(): + indicator = progress_mod.ActivityIndicator(verbosity=VerbosityEnum.FULL) + indicator._content = 12345 + + result = indicator._terminal_content() + + assert isinstance(result, progress_mod.Text) + assert result.plain == '12345' + + +# --------------------------------------------------------------------------- +# _terminal_indicator_line edge cases +# --------------------------------------------------------------------------- + + +def test_terminal_indicator_line_keeps_stopped_label(): + indicator = progress_mod.ActivityIndicator(label='Done', verbosity=VerbosityEnum.FULL) + indicator._running = False + indicator._keep_stopped_label = True + + line = indicator._terminal_indicator_line() + + assert line is not None + assert line.plain == 'Done' + + +def test_terminal_indicator_line_none_when_idle_and_no_label(): + indicator = progress_mod.ActivityIndicator(verbosity=VerbosityEnum.FULL) + indicator._running = False + indicator._keep_stopped_label = False + + assert indicator._terminal_indicator_line() is None + + +def test_terminal_indicator_line_animated_uses_spinner_frame(): + indicator = progress_mod.ActivityIndicator(label='Fitting...', verbosity=VerbosityEnum.FULL) + indicator._running = True + indicator._current_frame = lambda: 'Z' + + line = indicator._terminal_indicator_line() + + assert line is not None + assert line.plain == 'Z Fitting...' + + +# --------------------------------------------------------------------------- +# _current_frame +# --------------------------------------------------------------------------- + + +def test_current_frame_returns_valid_spinner_frame(monkeypatch): + indicator = progress_mod.ActivityIndicator(verbosity=VerbosityEnum.FULL) + indicator._started_at = 0.0 + # Elapsed time selects frame index deterministically. + monkeypatch.setattr(progress_mod, 'monotonic', lambda: 0.25) + + frame = indicator._current_frame() + + expected_index = int(0.25 / progress_mod._SPINNER_FRAME_SECONDS) % len( + progress_mod.SPINNER_FRAMES + ) + assert frame == progress_mod.SPINNER_FRAMES[expected_index] + + +def test_current_frame_wraps_around(monkeypatch): + indicator = progress_mod.ActivityIndicator(verbosity=VerbosityEnum.FULL) + indicator._started_at = 0.0 + # Beyond a full cycle, index wraps back into range. + monkeypatch.setattr(progress_mod, 'monotonic', lambda: 100.0) + + frame = indicator._current_frame() + + assert frame in progress_mod.SPINNER_FRAMES + + +# --------------------------------------------------------------------------- +# _render_html and helpers +# --------------------------------------------------------------------------- + + +def test_render_html_empty_when_idle(): + indicator = progress_mod.ActivityIndicator(verbosity=VerbosityEnum.FULL) + # Idle: no content and no indicator -> empty string. + assert indicator._render_html() == '' + + +def test_render_html_includes_style_and_stack(): + indicator = progress_mod.ActivityIndicator(label='Fitting...', verbosity=VerbosityEnum.FULL) + indicator._running = True + + html = indicator._render_html() + + assert html.startswith('