Skip to content

test(daemon): vanished-transcript e2e tests hold up under load - #55

Open
HackPoint wants to merge 3 commits into
mainfrom
agents/daemon-vanished-test-flake
Open

HackPoint wants to merge 3 commits into
mainfrom
agents/daemon-vanished-test-flake

Conversation

@HackPoint

Copy link
Copy Markdown
Owner

What was flaky

The three e2e tests added in #49 failed 5 times in 125 runs under CPU load. Two causes, both in the tests:

  1. They took "the first turn was ingested" as proof that the notify watcher was registered, then made one directory. The first turn can come from the startup pass or the 3 s poll, so it says nothing about the watcher. Under load the directory landed before the watcher existed, its event was lost, and the wait timed out.
  2. They read faults[0] / faults.is_empty() over the whole spool. The harness picks WebSocket ports with free_port() (bind port 0, release), so daemons started at the same time can collide, and the loser spools a ws_restart ("Address already in use") ahead of anything the tests look at.

Change

Tests only, in crates/lumen-daemon/tests/e2e_pipeline.rs. No product code, no sleep added.

  • Readiness: each test makes a fresh unreadable-N.jsonl directory on every poll of wait_until, until a fault for one is spooled.
  • Deletion test: each round writes a transcript, deletes it at once, then makes the probe directory. The vanish comes first because the daemon records one ingest_failed/notify fault a minute: a probe made before the vanish would take that slot and hide a regression.
  • Spool reads go through ingest_failures(spool, named): kind ingest_failed, file name matching. The two "is not a fault" tests now assert that no such fault names the vanished path, not that the spool is empty. The "cannot read" test asserts a notify fault for an unreadable-N directory.
  • Waits name what they assert: turn rows by message id (has_turn) instead of a running count.

Evidence

Same loop before and after: the compiled test binary runs the three tests N times, 16 at a time, beside 16 busy-loop processes (load average 70 to 86 on a 10-core machine), killed afterwards.

tests load runs failures
current 10 workers, 12 busy 125 1
current 16 workers, 16 busy 250 4
fixed 16 workers, 16 busy 500 0

The 5 failures on the current tests: 4 ws_restart port collisions read as the first fault (three tests), 1 lost watcher event (timed out waiting for the directory fault). A lighter profile (125 runs) did not fail.

Both causes also reproduce on demand with a local, uncommitted daemon change:

daemon change current tests fixed tests
watcher registration delayed 5 s 2 of 3 time out 3 pass
an unrelated ws_restart spooled at startup 3 of 3 fail 3 pass

Each assertion still goes red when the behaviour it guards breaks (local, reverted, not committed):

  • ErrorKind::NotFound handling in ingest_from removed: the deletion test and the listed-then-gone test fail in 45 of 45 runs (5 direct, 40 under load); the "cannot read" test stays green.
  • Non-NotFound read errors made silent: the "cannot read" test fails (timed out waiting for the fault); the deletion test fails too, since its probe fault never arrives; the listed-then-gone test stays green.

Not changed

free_port() still has its bind race, which can fail other e2e tests the same way. Fixing it means handing the daemon a listener or retrying on a bind failure, which is a harness or product change of its own.

Checks

The local CI-equivalent gate (fmt, clippy, workspace tests, frontend, e2e, GUI, MSRV, install verify) is green on macOS. The Linux and Windows legs run in CI only; the listed-then-gone test stays #[cfg(unix)], as before.

HackPoint and others added 2 commits October 12, 2026 14:37
…the faults they assert on

They took the first ingested turn as proof that the notify watcher was registered, made one
directory, and read the first fault in the whole spool. Under load that directory could land
before the watcher existed and never be reported, and a ws_restart from a lost port race could
be spooled ahead of it.

The tests now make a fresh directory per check until one is reported, filter the spool to
ingest_failed faults for the paths they made, assert that no such fault names a vanished
path, and wait on the turn rows they assert on instead of a running total.
The three comments the review found longer than the repository's one-line rule are cut to one line each; no code changes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant