Repository navigation
Test fails but doesn't fail as expected (tests/interp/m98m99/12-M99-endless-main-program changed output) #4537
Description
The test in tests/interp/m98m99/12-M99-endless-main-program changed after @grandixximo #4441 merge we get the output:
--- expected.motion-logger 2026-08-26 13:18:16.903654087 +0200
+++ out.motion-logger 2026-09-14 15:41:11.731367523 +0200
@@ -194,6 +194,7 @@
JOG_ABORT joint=15
ABORT
SPINDLE_OFF
+SS_ENABLE
ENABLE
COORD
JOG_ABORT joint=0The real problem is that the test is marked xfail and that means that it is supposed to fail. However, it has not failed for a very long time and we've all gotten used to the message *** /home/runner/work/linuxcnc/linuxcnc/tests/interp/m98m99/12-M99-endless-main-program: XPASS: Passed, but was expected to fail.
Now the output changed and nobody caught it because CI lets it pass. We told it to let it pass...
Either the program marked xfail does fail and only passes if it fails. Or, we let it pass and it must always pass. This intermediate state we have now is the same as what happened in #4397 and it will happen again and again as long as we let non-failing tests that should fail pass.
The other solution is to disable or remove this test if it does not serve any proper purpose.
Activity
BsAtHome commentedon Sep 14, 2026
BTW, a test that is supposed to fail (using
xfail) still needs to test whether the failure mode is exactly the failure expected. That makes things very difficult. Tests should test the result and make the decision on that result. If the result changes, then the test should always be rejected as not passing.The test in question,
tests/interp/m98m99/12-M99-endless-main-program/is the only one using thexfailmarker. It would probably be prudent to remove that functionality completely. The test can do logic negation when necessary. No need for the runtests script to make things complicated by negating negations by negating the negative not negated by negation... (if you still can follow my line of thought)tests: drop the xfail marker and its runtests support