[perlinm]

Motivation

Coverage can't trace numba-compiled code. So far, each test of compiled code has had a hand-written plain-Python variant for the coverage check, in one of three styles: fixtures that swap hand-listed functions for their .py_func, the compilation_mode fixture, or compiled/interpreted parameters. The compiled variant carried pytest.mark.compiled, which the coverage check skipped and a CI job of its own ran. Developers had to keep the lists of functions, learn which style a module used, and mark the variant, and reviewers had to check all of it. This PR replaces it all with QLDPC_DISABLE_COMPILATION (#836), which runs qLDPC's compiled functions as plain Python, and the pytest options for compiled runs (#837). The tests that were too slow for this as plain Python were shrunk in #838.

Changes

• checks/coverage_.py sets QLDPC_DISABLE_COMPILATION, so that the coverage check runs qLDPC's compiled functions as plain Python, and it no longer skips any tests.
• The compiled CI job now makes the compiled runs, in parallel, with compilation on: the test files of the modules that define compiled functions (--compiled-module-tests), and the distance tests with the compiled-on-request set-up code compiled too (--compile-search-setup). "Pytest and coverage check" still depends on it.
• The per-test variants, name lists, compilation_mode, compiled/interpreted parameters and .py_func swaps are removed, and the marker is unregistered.
  • Tests that need a function's Python version on purpose use getattr(f, "py_func", f): a tracemalloc test, since tracemalloc traces only Python allocations, and a test that records calls.
  • scan_test::test_python_kernels, which existed only for coverage, becomes test_counts_with_tiny_chunks_and_forced_tables, since it alone covers some lines of scan.py.
  • completion_bounds_test::test_bounds_are_exact_relaxations checks the default 16-bit width only on its shorter tails, where that is fast as plain Python. Before, only its compiled variant checked that width, on every tail.
• Rules:
  • The testing standard replaces the marker rule. Every test must be fast as plain Python, and a test of behavior that only compiled code has must also pass as plain Python, and be in a test file that a compiled run selects.
  • The validation commands list the compiled runs, and suggest coverage_.py to check whether a module's tests pass and run fast as plain Python.
  • Compiled-on-request functions must be defined in qldpc.codes.distance, since only the distance tests run with them compiled. The _compilation.py docstring states this, and a new test in _compilation_test.py enforces it.
• The distance README points to the tests in _compilation_test.py for the rules of compiled code, rather than counting and listing them.

Testing

Every former compiled variant still runs compiled in one of the compiled runs, and the coverage check runs the same tests as before. Tests whose checks only mean something compiled, such as of a failed compilation or of agreement between plain and compiled code, pass trivially in the coverage check and check for real in a compiled run.

Additional notes

• Accepted trade-off: tests of code that calls compiled functions, such as code-family tests that compute distances, no longer run compiled anywhere in CI. So CI no longer catches argument types that only those callers pass to compiled functions.
• test_oracles_build_other_families_with_the_tuning_they_are_built_with replaces a compiled-on-request function, so it now requests plain_search_setup and runs with plain set-up code in every run. Before, its compiled variant ran with the set-up code compiled.
• information_sets_test::test_functions_agree_in_plain_python_and_compiled now requests plain_search_setup, as that fixture's docstring asks of tests that compare plain with compiled set-up code.
• .test_durations holds per-file durations measured with compiled code, so the coverage shards may be less balanced until it is regenerated.