Several pytest tests are flaky or weak because of unseeded randomness, long sweeps, and smoke-only plot checks

[vivian99cc-a11y]

Summary

Several pytest tests use random data without a fixed seed, perform broad/slow parameter sweeps, or only check that plotting functions did not crash. These tests are useful as exploratory smoke coverage, but they are weak as deterministic regression tests.

Examples

Unseeded randomness

Many tests call np.random.randn, np.random.rand, np.random.permutation, or similar global RNG APIs without a fixed seed. Examples include:

• python/tests/unit/aout/test_analyze_error_by_phase.py
• python/tests/unit/aout/test_analyze_error_by_value.py
• python/tests/unit/aout/test_decompose_harmonics.py
• python/tests/unit/calibration/test_verify_estimate_frequencies.py
• several python/tests/unit/spectrum/* tests

Calibration tests are print-heavy and sweep-heavy

python/tests/unit/calibration/test_verify_calibration_lite.py runs many broad sweeps and prints metrics such as weight error, SNDR, and ENOB, but several sweeps do not assert the expected bounds.

python/tests/unit/calibration/test_verify_estimate_frequencies.py prints whether frequency estimates are good or bad, while the actual threshold assertions are commented out.

Plot tests are often smoke-only

Some plotting tests only assert that a figure/file/result exists, for example:

• python/tests/unit/dout/test_plot_residual_scatter.py
• python/tests/unit/spectrum/test_sweep_performance_vs_osr.py
• many AOUT/Spectrum plot tests that only verify PNG creation

Smoke tests are useful, but they should be labeled as such and complemented with structural or numeric checks when the plotted data encodes important behavior.

Why this matters

• Unseeded tests can pass locally and fail in CI with a different random draw.
• Long sweeps slow down feedback and make failures harder to isolate.
• Print-only checks do not protect against regressions.
• Smoke-only plot assertions can pass even if the plotted data is wrong.

Suggested fixes

• Replace global RNG calls with np.random.default_rng(seed) or fixed RandomState where reproducibility matters.
• Convert printed pass/fail thresholds into explicit assertions.
• Split long calibration sweeps into a small deterministic regression test plus optional stress/performance tests.
• Mark long exploratory sweeps with a dedicated marker such as @pytest.mark.slow.
• For plot tests, assert returned data shape, axis count, labels, plotted line/bar counts, or representative numeric values in addition to checking that output files exist.

Expected result

The default pytest suite should be deterministic, reasonably fast, and should fail only when a meaningful behavior contract is violated.

[Arcadia-1]

#44 partially addressed this by moving broad calibration-lite sweeps behind the slow marker and adding deterministic assertions for the calibration/frequency tests. I am leaving this open because the broader issue is not fully resolved yet: there are still many global np.random calls without fixed per-test RNGs, and several plot-oriented tests remain mostly smoke coverage. Good next step: split this into a focused RNG determinism pass plus a plot-test assertion pass.

[Arcadia-1]

Follow-up after merging #44: keeping this open intentionally. #44 fixed the calibration-lite part of this issue by marking broad sweeps as slow, adding deterministic regression cases, and turning several print-only calibration/frequency checks into assertions.\n\nRemaining work I recommend before closing this issue:\n\n1. RNG determinism pass: replace remaining global np.random.randn/rand/randint/uniform calls in tests with per-test np.random.default_rng(seed), especially under tests/unit/aout, tests/unit/spectrum, and tests/unit/calibration.\n2. Plot assertion pass: for plot-heavy tests, keep file/figure smoke checks but add structural assertions such as axis count, line/bar count, labels, returned data shape, and representative numeric values.\n3. Slow-test audit: keep broad exploratory sweeps behind @pytest.mark.slow and make sure the default pytest run stays deterministic and reasonably quick.\n4. Verification target: run the default pytest suite repeatedly from a clean checkout and confirm no stochastic failures, then run pytest -m slow separately for the exploratory coverage.\n\nOnce those are done, this issue can be closed with confidence instead of just being mostly improved.