Flaky test_exec_bash in hooks_test.py on macOS still reproducing after previous fixes

[mangelajo]

Summary

TestHookExecutor::test_exec_bash in jumpstarter/exporter/hooks_test.py is still flaky on macOS despite previous fixes in #560, #733, and #826. The PTY output race condition on macOS has not been fully resolved.

Latest failing CI run: https://github.com/jumpstarter-dev/jumpstarter/actions/runs/28084374100/job/83146665529

Workflow: Python Tests — pytest-matrix (macos-15)

This is the 4th occurrence of this issue (#560, #733, #821, #826). Each fix has reduced the frequency but not eliminated the race condition.

Root Cause

The bug is in the PTY drain logic in hooks.py (the finally block of read_pty_output). On macOS, the PTY kernel buffer may not deliver data synchronously with process exit. Despite increasing DRAIN_MAX_EMPTY_POLLS to 10 in #826, the race still occurs under heavy CI load.

The CI runs make test -j4 (4 parallel test suites) with concurrent uv package installations, creating enough I/O and CPU contention for the macOS PTY buffer to miss the drain window.

Previous Fix Attempts

• test_exec_bash hook test fails on macOS due to PTY output not captured by mocked logger #560 — Added synchronous drain after async reader cancellation (closed 2026-04-16)
• Flaky test_exec_bash in hooks_test.py on macOS due to PTY output race condition #733 — Added select()-based drain with timeout in the finally block (closed 2026-06-05)
• fix: retry PTY drain on empty select() to prevent data loss on macOS #826 — Increased DRAIN_MAX_EMPTY_POLLS from 1 to 10 (~1s window) (merged 2026-06-23)

All fixes reduced the frequency but did not eliminate the race.

Tests Marked as xfail on macOS

The following tests are marked with @macos_pty_xfail (strict=False) to unblock CI while the root cause is investigated. All of these spawn real subprocesses via PTY and assert on captured logger output:

TestHookExecutor (output capture tests)

• test_hook_environment_variables
• test_real_time_output_logging
• test_post_lease_hook_execution_on_completion
• test_exec_bash
• test_exec_python3
• test_script_file_sh
• test_script_file_py_autodetects_python
• test_script_file_py_exec_override
• test_noninteractive_environment
• test_drain_captures_output_without_trailing_newline

TestHookExecutor (drain behavior tests with patched PTY)

• test_drain_reads_data_remaining_in_pty_buffer
• test_drain_select_oserror_exits_gracefully
• test_drain_select_valueerror_exits_gracefully
• test_drain_exits_when_deadline_exceeded_before_select
• test_drain_exception_is_suppressed
• test_drain_retries_empty_select_then_captures_data
• test_drain_terminates_after_max_empty_polls
• test_drain_empty_counter_resets_on_data

TestHookExecutorPRRegressions

• test_infrastructure_messages_at_debug_not_info

Environment

• Platform: macOS 15 (Apple Silicon, GitHub Actions runner)
• Python: 3.11, 3.13
• Passes on: Linux runners (same Python versions)

Goal for 0.10.0

Investigate the root cause properly and either:

1. Find a reliable fix for macOS PTY timing (possibly using waitpid + blocking drain, or pipe-based output capture instead of PTY)
2. Or redesign hook output capture to not depend on PTY timing

Once fixed, remove the macos_pty_xfail markers from all affected tests.

[mangelajo]

Another CI failure from the same root cause, this time hitting two different tests:

CI run: https://github.com/jumpstarter-dev/jumpstarter/actions/runs/28013357000/job/82911642124

FAILED jumpstarter/exporter/hooks_test.py::TestHookExecutor::test_script_file_sh - assert False
FAILED jumpstarter/exporter/hooks_test.py::TestHookExecutorPRRegressions::test_infrastructure_messages_at_debug_not_info - assert False

Same pattern — both tests run a subprocess via _execute_hook_process, assert that stdout output appears in the mocked logger, and fail because the PTY drain gave up too early. This confirms the bug is not specific to test_exec_bash but affects any hook test that depends on capturing PTY output under load.

The fix in #826 addresses all of these since it changes the shared drain logic.

[mangelajo]

This issue is still reproducing after the fix in #826 (DRAIN_MAX_EMPTY_POLLS=10). The PTY output race condition on macOS continues to cause random test failures in CI.

Latest failing CI run: https://github.com/jumpstarter-dev/jumpstarter/actions/runs/28084374100/job/83146665529

This is the 4th occurrence of this issue (#560, #733, #821, and now post-#826). Each fix has reduced the frequency but not eliminated the race condition.

Action taken:

• Re-opened this issue and moved milestone to 0.10.0 for a proper investigation and fix
• Marking all PTY-output-dependent hook tests as xfail on macOS to unblock CI while the root cause is investigated
• The following tests are marked as xfail(condition=sys.platform == "darwin"):
  • TestHookExecutor::test_hook_environment_variables
  • TestHookExecutor::test_real_time_output_logging
  • TestHookExecutor::test_post_lease_hook_execution_on_completion
  • TestHookExecutor::test_exec_bash
  • TestHookExecutor::test_exec_python3
  • TestHookExecutor::test_script_file_sh
  • TestHookExecutor::test_script_file_py_autodetects_python
  • TestHookExecutor::test_script_file_py_exec_override
  • TestHookExecutor::test_noninteractive_environment
  • TestHookExecutor::test_drain_captures_output_without_trailing_newline
  • TestHookExecutorPRRegressions::test_infrastructure_messages_at_debug_not_info

These are all tests that spawn real subprocesses via PTY and assert on captured logger output — the exact pattern affected by the macOS PTY kernel buffer timing issue.