You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
docs: update test-tier docs for PerformanceTests + perf.yml split
Reflects the project layout from this PR in the docs:
CLAUDE.md
- Adds tests/Terminal.Gui.Editor.PerformanceTests to the test
runner list, with the -c Release note + perf.yml pointer.
- "Testing tiers": now four projects with a workflow + OS column;
adds the "Performance gates" subsection that explains the two
layers (stopwatch smoke + BenchmarkDotNet baseline) and calls
out the `--job short` lowercase requirement.
specs/constitution.md
- §VI "Testing Tiers" table grows the fourth row + workflow + OS
matrix columns. Adds rationale for ubuntu-only and the manual
workflow_dispatch path for baseline.json refreshes.
specs/plan.md
- Repo-layout block adds tests/Terminal.Gui.Editor.PerformanceTests
and a benchmarks/ subtree. EditorBenchmarks placeholder removed
(it's not in the tree).
- DoD checkbox updated from "three test projects" to "four", with
the BenchmarkDotNet 3× baseline gate called out.
README.md
- Repo-layout adds benchmarks/ + examples/ rows.
- Build block adds the PerformanceTests run line, with the
ubuntu-latest-only / perf.yml pointer.
No behavior change; this is the doc tail of perf-suite-split.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three test projects, mirroring Terminal.Gui's convention. **All three run fully in parallel** — Terminal.Gui's `Application` lifetime is per-instance (`Application.Create()` returns an `IApplication` whose `Init`/`Begin`/`End`/`Dispose` track via `ThreadLocal<>`, not process globals). Tests must never call the static `Application.Init()` shortcut, and must never enable `ConfigurationManager` (`CM.Enable(...)`) — both reach for process-global state and would force serialization.
173
+
Four test projects, mirroring Terminal.Gui's convention. **The correctness projects all run fully in parallel** — Terminal.Gui's `Application` lifetime is per-instance (`Application.Create()` returns an `IApplication` whose `Init`/`Begin`/`End`/`Dispose` track via `ThreadLocal<>`, not process globals). Tests must never call the static `Application.Init()` shortcut, and must never enable `ConfigurationManager` (`CM.Enable(...)`) — both reach for process-global state and would force serialization.
169
174
170
-
-`Terminal.Gui.Editor.Tests` — pure, no UI, no static state. Target ≥90% coverage.
171
-
-`Terminal.Gui.Editor.Tests` — visual-line builder, wrap, caret/selection math, command handlers — anything that doesn't need an `IApplication`. Target ≥75%.
172
-
-`Terminal.Gui.Editor.IntegrationTests` — full key-input → render scenarios via `AppFixture<T>`, which boots a per-test `IApplication` from `Application.Create()`. Parallel by default.
175
+
-`Terminal.Gui.Editor.Tests` — pure, no UI, no static state. Target ≥90% coverage. Runs in `ci.yml`.
176
+
-`Terminal.Gui.Editor.IntegrationTests` — full key-input → render scenarios via `AppFixture<T>`, which boots a per-test `IApplication` from `Application.Create()`. Parallel by default. Runs in `ci.yml`.
177
+
-`Terminal.Gui.Editor.PerformanceTests` — stopwatch-based perf smoke tests. **Release only, ubuntu-latest only.** Lives in its own project and its own workflow (`.github/workflows/perf.yml`) because Windows/macOS GitHub-hosted runners are too noisy for wall-time assertions. The BenchmarkDotNet suite in `benchmarks/` runs from the same workflow.
173
178
174
-
New tests default to the parallel-by-name project. Promote to `IntegrationTests` only when an `IApplication` (driver, input injection, full layout/draw) is genuinely needed.
179
+
New tests default to the parallel-by-name project. Promote to `IntegrationTests` only when an `IApplication` (driver, input injection, full layout/draw) is genuinely needed. Promote to `PerformanceTests` only when you need a wall-time assertion — and remember it won't run on Windows/macOS CI, so don't put correctness checks there.
175
180
176
181
**The one allowed exception:** a test that legitimately mutates a process-global (e.g. `Logging.Logger`, `Trace.EnabledCategories`, anything `static`) must opt out of cross-collection parallelism via a `[CollectionDefinition(name, DisableParallelization = true)]` + `[Collection(name)]` pair. See `tests/Terminal.Gui.Editor.IntegrationTests/HostingTests.cs` for the canonical example. Do **not** add an assembly-wide `xunit.runner.json` to make the whole project serial — that's the wrong tool for one offending class.
177
182
183
+
### Performance gates
184
+
185
+
Two layers, both in `.github/workflows/perf.yml`:
186
+
187
+
1.**`Terminal.Gui.Editor.PerformanceTests`** — stopwatch smoke tests with deliberately loose thresholds (~5× typical wall time). They catch catastrophic regressions, not 10% drift.
188
+
2.**`benchmarks/compare-baseline.sh`** — runs the focused `*VisualLineBuild*` BenchmarkDotNet filter and compares against `benchmarks/baseline.json`. Fails on >3× regression, celebrates on <0.8× improvement. **Run `--job short`** (lowercase) — `ShortRun` makes BDN reject it and the comparison silently no-ops.
189
+
190
+
The full BenchmarkDotNet matrix (`Scrolling`, `EndToEndScroll`, `CaretMovement`, `DocumentAccess`) is opt-in via `workflow_dispatch` on the perf workflow with `full-suite: true`. That's the operator path for refreshing `baseline.json` — run, download artifact, commit the numbers.
191
+
178
192
### Diagnosing parallel-test hangs
179
193
180
194
When integration tests run individually but hang when run as a suite, **the cause is almost always shared mutable state, not the parallelism itself**. Do not reach for `xunit.runner.json` with `parallelizeTestCollections: false` to "fix" it — that hides the bug and slows the suite. Walk this checklist instead:
|`Terminal.Gui.Editor.Tests`|`ci.yml`| ubuntu, macos, windows| Pure, no UI, no static state | ≥ 90% |
145
+
|`Terminal.Gui.Editor.IntegrationTests`|`ci.yml`|ubuntu, macos, windows | Full key-input → render via `AppFixture<T>` (per-test `IApplication.Create()`) | Informational|
The lone exception: a class that mutates a process-global static (e.g. `HostingTests`) opts out of cross-collection parallelism via `[CollectionDefinition(..., DisableParallelization = true)]` + `[Collection(...)]`. Never disable parallelism at the assembly level.
148
+
The three correctness projects run **in parallel** within each assembly. The lone exception: a class that mutates a process-global static (e.g. `HostingTests`) opts out of cross-collection parallelism via `[CollectionDefinition(..., DisableParallelization = true)]` + `[Collection(...)]`. Never disable parallelism at the assembly level.
149
149
150
-
Tests run as **executables** (xUnit.v3): `dotnet run --project tests/<project>`.
150
+
`PerformanceTests` lives in its own workflow because Windows / macOS GitHub-hosted runners share hosts with neighbour VMs — wall-time assertions there are too noisy to gate on. Ubuntu runners are still noisy but consistent enough that a >3× BenchmarkDotNet baseline ratio is a real signal. The full BenchmarkDotNet matrix runs only on `workflow_dispatch` (`full-suite: true`); that's the operator path for refreshing `benchmarks/baseline.json`.
151
+
152
+
Tests run as **executables** (xUnit.v3): `dotnet run --project tests/<project>`. `PerformanceTests` always runs with `-c Release`.
@@ -104,7 +107,7 @@ Each criterion is testable. This is the merge-to-`main` gate.
104
107
105
108
-[ ] All features merged: drawing-overhaul, caret-anchors, read-only, find-and-replace, auto-indent, folding-ui, syntax-colorizer, word-wrap, multi-caret, clipboard.
106
109
-[ ]`dotnet build Terminal.Gui.Editor.slnx` clean on Linux/macOS/Windows on net10.0.
107
-
-[ ] All three test projects pass. Coverage: `Text.Tests` ≥ 90%, `Editor.Tests` ≥ 75%.
110
+
-[ ] All test projects pass. Coverage: `Editor.Tests` ≥ 90%. `PerformanceTests` smoke tests + the `*VisualLineBuild*` BenchmarkDotNet gate stay within 3× of `benchmarks/baseline.json`.
108
111
-[ ]`Editor.OnDrawingContent` does not iterate `text` by `char`. R1, R2, R4, R5 hold.
109
112
-[ ]`Editor.TabWidth`, `Editor.SyntaxHighlighter`, `Editor.SyntaxLanguage` all removed.
110
113
-[ ] No file under `src/Terminal.Gui.Editor/` references `Terminal.Gui`.
0 commit comments