Skip to content

Pytest suite contains non-collected and assertion-free tests #34

@vivian99cc-a11y

Description

@vivian99cc-a11y

Summary

The current pytest suite contains tests that either are not collected by pytest at all, or are collected but do not make meaningful assertions. These files can make the suite look broader than it really is.

Examples

  • python/tests/unit/test_cap2weight.py defines run_tests() and only executes it under if __name__ == "__main__", so pytest collects zero tests from the file.
  • Many comparison wrappers under python/tests/compare/ contain no direct assertions and rely entirely on the shared runner.
  • Many integration tests under python/tests/integration/ write CSV/PNG artifacts but contain no assert statements or np.testing checks.
  • python/tests/unit/calibration/test_verify_estimate_frequencies.py has pytest test functions but currently only prints pass/bad information; the intended accuracy threshold assertions are commented out.

Why this matters

  • Regressions can pass if a test prints a failure-like message but never asserts.
  • Files may appear to be part of the pytest suite while contributing no collected tests.
  • The suite is harder to trust as a CI signal because it mixes smoke scripts, generators, and actual assertions.

Suggested fixes

  • Convert test_cap2weight.py into normal pytest functions with explicit assert/np.testing checks.
  • Add real assertions to integration tests that currently only generate artifacts.
  • In frequency-estimation tests, turn the printed threshold checks into actual assertions.
  • Keep comparison wrapper assertions centralized if desired, but document that the wrapper is intentionally assertion-free and relies on run_comparison_suite() raising failures.

Verification target

A static audit should find no collected test_* functions with zero assertion-like behavior unless they are explicitly documented as runner wrappers.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions