Skip to content

test: fail closed on empty unit selections and expand hosted coverage - #708

Closed
Graffioh wants to merge 1 commit into
Luce-Org:mainfrom
Graffioh:codex/unit-test-quality-gates
Closed

Graffioh wants to merge 1 commit into
Luce-Org:mainfrom
Graffioh:codex/unit-test-quality-gates

Conversation

@Graffioh

@Graffioh Graffioh commented Sep 5, 2026 •

Copy link
Copy Markdown
Collaborator

Problem and behavior

Hosted Python CI currently runs only the DS4 benchmark tests. The shared C++ runner also lets an empty filter return skip (77), lets an empty binary succeed, and lets --exact match a tag belonging to another test. CTest can therefore report success without executing the intended case.

This change makes pyproject.toml the model-free pytest entry point and runs the existing harness/concurrency/helper suites in CI. It makes empty execution/discovery fail and restricts --exact to full test names, while preserving ordinary substring/tag selection and explicit hardware skips.

A shared pytest fixture compiles small C++ binaries without CUDA/HIP and exercises the real runner and CMake/CTest adapter. The tests cover assertion failures, exceptions, missing selections, tag/name collisions, explicit skips, and empty discovery. harness/tests/README.md documents scope and tool requirements.

Validation

  • 94 tests and 20 subtests passed with the CI-style pytest -q command on Linux.
  • The new contract suite against main's original runner/discovery: 8 failed, 8 passed; all 16 pass with this change.
  • ruff check . and git diff --check passed.
  • Full server/GPU builds and model-backed qualification were not run locally; inference code is unchanged.

Based on main and independent of #636 and #687. Their qualification policies and report formats are not changed here.

Review in cubic

@Graffioh Graffioh closed this Sep 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant