Skip to content

Commit 3f142a7

Browse files
tigclaude
andcommitted
docs: update test-tier docs for PerformanceTests + perf.yml split
Reflects the project layout from this PR in the docs: CLAUDE.md - Adds tests/Terminal.Gui.Editor.PerformanceTests to the test runner list, with the -c Release note + perf.yml pointer. - "Testing tiers": now four projects with a workflow + OS column; adds the "Performance gates" subsection that explains the two layers (stopwatch smoke + BenchmarkDotNet baseline) and calls out the `--job short` lowercase requirement. specs/constitution.md - §VI "Testing Tiers" table grows the fourth row + workflow + OS matrix columns. Adds rationale for ubuntu-only and the manual workflow_dispatch path for baseline.json refreshes. specs/plan.md - Repo-layout block adds tests/Terminal.Gui.Editor.PerformanceTests and a benchmarks/ subtree. EditorBenchmarks placeholder removed (it's not in the tree). - DoD checkbox updated from "three test projects" to "four", with the BenchmarkDotNet 3× baseline gate called out. README.md - Repo-layout adds benchmarks/ + examples/ rows. - Build block adds the PerformanceTests run line, with the ubuntu-latest-only / perf.yml pointer. No behavior change; this is the doc tail of perf-suite-split. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent 443c67d commit 3f142a7

5 files changed

Lines changed: 52 additions & 26 deletions

File tree

.github/workflows/perf.yml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -42,12 +42,12 @@ jobs:
4242
dotnet-quality: 'preview'
4343

4444
- name: Restore
45-
run: dotnet restore Terminal.Gui.Text.slnx
45+
run: dotnet restore Terminal.Gui.Editor.slnx
4646

4747
# Smoke tests are stopwatch-based (PerformanceSmokeTests.cs) — Release config matters
4848
# because Debug-mode timings would force the thresholds to be useless.
4949
- name: Build (Release)
50-
run: dotnet build Terminal.Gui.Text.slnx -c Release --no-restore
50+
run: dotnet build Terminal.Gui.Editor.slnx -c Release --no-restore
5151

5252
- name: Terminal.Gui.Editor.PerformanceTests (smoke)
5353
run: dotnet run -c Release --project tests/Terminal.Gui.Editor.PerformanceTests --no-build

CLAUDE.md

Lines changed: 19 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -38,8 +38,13 @@ Tests are xUnit.v3 and run as **executables** (each test project sets `<OutputTy
3838
dotnet run --project tests/Terminal.Gui.Editor.Tests
3939
dotnet run --project tests/Terminal.Gui.Editor.Tests
4040
dotnet run --project tests/Terminal.Gui.Editor.IntegrationTests
41+
dotnet run --project tests/Terminal.Gui.Editor.PerformanceTests -c Release
4142
```
4243

44+
The `PerformanceTests` project is stopwatch-based and only meaningful in Release. It runs in
45+
the dedicated `.github/workflows/perf.yml` workflow (ubuntu-latest only), separately from the
46+
correctness-focused `ci.yml`. See the "Testing tiers" section below.
47+
4348
Run a single test by passing xUnit.v3 filter args after `--`:
4449

4550
```sh
@@ -165,16 +170,25 @@ private void ExtendCaretBy (int delta)
165170

166171
## Testing tiers
167172

168-
Three test projects, mirroring Terminal.Gui's convention. **All three run fully in parallel** — Terminal.Gui's `Application` lifetime is per-instance (`Application.Create()` returns an `IApplication` whose `Init`/`Begin`/`End`/`Dispose` track via `ThreadLocal<>`, not process globals). Tests must never call the static `Application.Init()` shortcut, and must never enable `ConfigurationManager` (`CM.Enable(...)`) — both reach for process-global state and would force serialization.
173+
Four test projects, mirroring Terminal.Gui's convention. **The correctness projects all run fully in parallel** — Terminal.Gui's `Application` lifetime is per-instance (`Application.Create()` returns an `IApplication` whose `Init`/`Begin`/`End`/`Dispose` track via `ThreadLocal<>`, not process globals). Tests must never call the static `Application.Init()` shortcut, and must never enable `ConfigurationManager` (`CM.Enable(...)`) — both reach for process-global state and would force serialization.
169174

170-
- `Terminal.Gui.Editor.Tests` — pure, no UI, no static state. Target ≥90% coverage.
171-
- `Terminal.Gui.Editor.Tests`visual-line builder, wrap, caret/selection math, command handlers — anything that doesn't need an `IApplication`. Target ≥75%.
172-
- `Terminal.Gui.Editor.IntegrationTests`full key-input → render scenarios via `AppFixture<T>`, which boots a per-test `IApplication` from `Application.Create()`. Parallel by default.
175+
- `Terminal.Gui.Editor.Tests` — pure, no UI, no static state. Target ≥90% coverage. Runs in `ci.yml`.
176+
- `Terminal.Gui.Editor.IntegrationTests`full key-input → render scenarios via `AppFixture<T>`, which boots a per-test `IApplication` from `Application.Create()`. Parallel by default. Runs in `ci.yml`.
177+
- `Terminal.Gui.Editor.PerformanceTests`stopwatch-based perf smoke tests. **Release only, ubuntu-latest only.** Lives in its own project and its own workflow (`.github/workflows/perf.yml`) because Windows/macOS GitHub-hosted runners are too noisy for wall-time assertions. The BenchmarkDotNet suite in `benchmarks/` runs from the same workflow.
173178

174-
New tests default to the parallel-by-name project. Promote to `IntegrationTests` only when an `IApplication` (driver, input injection, full layout/draw) is genuinely needed.
179+
New tests default to the parallel-by-name project. Promote to `IntegrationTests` only when an `IApplication` (driver, input injection, full layout/draw) is genuinely needed. Promote to `PerformanceTests` only when you need a wall-time assertion — and remember it won't run on Windows/macOS CI, so don't put correctness checks there.
175180

176181
**The one allowed exception:** a test that legitimately mutates a process-global (e.g. `Logging.Logger`, `Trace.EnabledCategories`, anything `static`) must opt out of cross-collection parallelism via a `[CollectionDefinition(name, DisableParallelization = true)]` + `[Collection(name)]` pair. See `tests/Terminal.Gui.Editor.IntegrationTests/HostingTests.cs` for the canonical example. Do **not** add an assembly-wide `xunit.runner.json` to make the whole project serial — that's the wrong tool for one offending class.
177182

183+
### Performance gates
184+
185+
Two layers, both in `.github/workflows/perf.yml`:
186+
187+
1. **`Terminal.Gui.Editor.PerformanceTests`** — stopwatch smoke tests with deliberately loose thresholds (~5× typical wall time). They catch catastrophic regressions, not 10% drift.
188+
2. **`benchmarks/compare-baseline.sh`** — runs the focused `*VisualLineBuild*` BenchmarkDotNet filter and compares against `benchmarks/baseline.json`. Fails on >3× regression, celebrates on <0.8× improvement. **Run `--job short`** (lowercase) — `ShortRun` makes BDN reject it and the comparison silently no-ops.
189+
190+
The full BenchmarkDotNet matrix (`Scrolling`, `EndToEndScroll`, `CaretMovement`, `DocumentAccess`) is opt-in via `workflow_dispatch` on the perf workflow with `full-suite: true`. That's the operator path for refreshing `baseline.json` — run, download artifact, commit the numbers.
191+
178192
### Diagnosing parallel-test hangs
179193

180194
When integration tests run individually but hang when run as a suite, **the cause is almost always shared mutable state, not the parallelism itself**. Do not reach for `xunit.runner.json` with `parallelizeTestCollections: false` to "fix" it — that hides the bug and slows the suite. Walk this checklist instead:

README.md

Lines changed: 11 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -18,10 +18,11 @@ Pre-alpha — see [`specs/00-plan.md`](specs/00-plan.md) for the full implementa
1818
## Repository layout
1919

2020
```
21-
specs/ Planning and design docs
22-
src/ Terminal.Gui.Editor library (document layer + Editor view)
23-
tests/ xUnit.v3 test projects
24-
examples/ ted — standalone demo app
21+
specs/ Planning and design docs
22+
src/ Terminal.Gui.Editor library (document layer + Editor view)
23+
tests/ xUnit.v3 test projects (correctness + perf smoke)
24+
benchmarks/ BenchmarkDotNet suite + CI baseline
25+
examples/ ted — standalone demo app
2526
```
2627

2728
## Build
@@ -31,8 +32,14 @@ Requires the .NET 10 SDK (preview).
3132
```sh
3233
dotnet restore Terminal.Gui.Editor.slnx
3334
dotnet build Terminal.Gui.Editor.slnx
35+
36+
# Correctness suites — run on every push/PR across ubuntu/macos/windows.
3437
dotnet run --project tests/Terminal.Gui.Editor.Tests
3538
dotnet run --project tests/Terminal.Gui.Editor.IntegrationTests
39+
40+
# Perf smoke + BenchmarkDotNet baseline gate — ubuntu-latest only in CI
41+
# (.github/workflows/perf.yml). Run locally in Release config.
42+
dotnet run --project tests/Terminal.Gui.Editor.PerformanceTests -c Release
3643
```
3744

3845
Run the demo:

specs/constitution.md

Lines changed: 10 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -137,17 +137,19 @@ The fork is **hard** — re-syncs are manual and deliberate, triggered only by u
137137

138138
## VI. Testing Tiers
139139

140-
Three test projects mirroring Terminal.Gui's convention:
140+
Four test projects mirroring Terminal.Gui's convention:
141141

142-
| Project | Parallel | Purpose | Coverage Target |
143-
|---------|----------|---------|-----------------|
144-
| `Terminal.Gui.Editor.Tests` | | Pure, no UI, no static state | ≥ 90% |
145-
| `Terminal.Gui.Editor.Tests` | | Visual-line builder, wrap, caret/selection, commands | ≥ 75% |
146-
| `Terminal.Gui.Editor.IntegrationTests` | | Full key-input → render via `AppFixture<T>` (per-test `IApplication.Create()`) | Informational |
142+
| Project | Workflow | OS matrix | Purpose | Coverage Target |
143+
|---------|----------|-----------|---------|-----------------|
144+
| `Terminal.Gui.Editor.Tests` | `ci.yml` | ubuntu, macos, windows | Pure, no UI, no static state | ≥ 90% |
145+
| `Terminal.Gui.Editor.IntegrationTests` | `ci.yml` | ubuntu, macos, windows | Full key-input → render via `AppFixture<T>` (per-test `IApplication.Create()`) | Informational |
146+
| `Terminal.Gui.Editor.PerformanceTests` | `perf.yml` | ubuntu only, Release only | Stopwatch perf smoke tests + BenchmarkDotNet baseline gate | Wall-time guards |
147147

148-
The lone exception: a class that mutates a process-global static (e.g. `HostingTests`) opts out of cross-collection parallelism via `[CollectionDefinition(..., DisableParallelization = true)]` + `[Collection(...)]`. Never disable parallelism at the assembly level.
148+
The three correctness projects run **in parallel** within each assembly. The lone exception: a class that mutates a process-global static (e.g. `HostingTests`) opts out of cross-collection parallelism via `[CollectionDefinition(..., DisableParallelization = true)]` + `[Collection(...)]`. Never disable parallelism at the assembly level.
149149

150-
Tests run as **executables** (xUnit.v3): `dotnet run --project tests/<project>`.
150+
`PerformanceTests` lives in its own workflow because Windows / macOS GitHub-hosted runners share hosts with neighbour VMs — wall-time assertions there are too noisy to gate on. Ubuntu runners are still noisy but consistent enough that a >3× BenchmarkDotNet baseline ratio is a real signal. The full BenchmarkDotNet matrix runs only on `workflow_dispatch` (`full-suite: true`); that's the operator path for refreshing `benchmarks/baseline.json`.
151+
152+
Tests run as **executables** (xUnit.v3): `dotnet run --project tests/<project>`. `PerformanceTests` always runs with `-c Release`.
151153

152154
## VII. Coding Standards
153155

specs/plan.md

Lines changed: 10 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -63,14 +63,17 @@ src/Terminal.Gui.Editor/ # the View
6363
Editor.cs / .Drawing / .Keyboard / .Mouse / .Selection / .Commands
6464
Rendering/ # rendering pipeline types
6565
tests/
66-
Terminal.Gui.Editor.Tests/ (parallel, pure)
67-
Terminal.Gui.Editor.Tests/ (parallel, logic)
68-
Terminal.Gui.Editor.IntegrationTests/ (parallel, per-test IApplication.Create())
66+
Terminal.Gui.Editor.Tests/ (parallel, pure) — ci.yml
67+
Terminal.Gui.Editor.IntegrationTests/ (parallel, IApplication) — ci.yml
68+
Terminal.Gui.Editor.PerformanceTests/(stopwatch perf smoke) — perf.yml (ubuntu, Release only)
69+
benchmarks/
70+
Terminal.Gui.Editor.Benchmarks/ (BenchmarkDotNet suite) — perf.yml
71+
baseline.json (gated metrics)
72+
compare-baseline.sh (CI compare script)
6973
examples/
70-
ted/ (TG demo app)
71-
EditorBenchmarks/ (placeholder)
74+
ted/ (TG demo app)
7275
third_party/AvaloniaEdit/
73-
LICENSE UPSTREAM.md (commit d7a6b63 pinned)
76+
LICENSE UPSTREAM.md (commit d7a6b63 pinned)
7477
```
7578

7679
## Dependencies
@@ -104,7 +107,7 @@ Each criterion is testable. This is the merge-to-`main` gate.
104107

105108
- [ ] All features merged: drawing-overhaul, caret-anchors, read-only, find-and-replace, auto-indent, folding-ui, syntax-colorizer, word-wrap, multi-caret, clipboard.
106109
- [ ] `dotnet build Terminal.Gui.Editor.slnx` clean on Linux/macOS/Windows on net10.0.
107-
- [ ] All three test projects pass. Coverage: `Text.Tests` ≥ 90%, `Editor.Tests` ≥ 75%.
110+
- [ ] All test projects pass. Coverage: `Editor.Tests` ≥ 90%. `PerformanceTests` smoke tests + the `*VisualLineBuild*` BenchmarkDotNet gate stay within 3× of `benchmarks/baseline.json`.
108111
- [ ] `Editor.OnDrawingContent` does not iterate `text` by `char`. R1, R2, R4, R5 hold.
109112
- [ ] `Editor.TabWidth`, `Editor.SyntaxHighlighter`, `Editor.SyntaxLanguage` all removed.
110113
- [ ] No file under `src/Terminal.Gui.Editor/` references `Terminal.Gui`.

0 commit comments

Comments
 (0)