Tests: stop handing core ports that were just released

[fylorn]

The failure

CI failed once on Linux in tw-control's remote-port tests (job, first attempt on #191):

narrowing_allow_from_closes_the_connections_it_no_longer_lets_in
the remote port never got there: Listening { addr: None, error: Some(Msg { code: "gw.listen.port_taken", args: {"addr": "127.0.0.1:34393"}, ... }) }

free_port() bound 127.0.0.1:0, read the port and closed the socket. The test then wrote that number into the config for core to bind. A released port goes back to the system's ephemeral range, and parallel tests draw from that range whenever they bind port 0 or open a connection. Linux picks those ports at random, so now and then another test took the number before core bound it.

The same "bind 0, read the port, release it" step was in 38 places that start a gateway with tw_gateway::serve, and in the hot-reload, PUT /listen and remote-port tests.

The change

• tw_gateway::serve binds before it returns and hands back the address it bound. The 38 call sites pass 127.0.0.1:0 and use the returned address, so the system picks the port and nothing can take it in between. serve is only used by tests (twcore serve --port goes through serve_at), and neither the desktop app nor ThinkWatch Enterprise depends on tw-gateway.

• Where the number has to exist before anything binds it, tests take it from spare_port():

  • the remote control port. The config refuses port 0 on purpose (config.remote_port_zero), so core cannot pick it the way --port 0 lets it pick the gateway port;
  • the port PUT /listen saves (it refuses 0 too);
  • the hot-reload tests that move the gateway between two ports, or keep one port across all and a single address.

  spare_port() picks at random from REMOTE_PORT_RANGE (20000–32000), below the ephemeral ranges of Linux (32768–60999), macOS and Windows (49152–65535), so the system never hands these ports out itself. It holds each number with a UDP socket for the life of the test process. UDP and TCP ports are separate, so core still binds the TCP port, but another test, or another test process running at the same time (two worktrees running the suite), cannot get the same number. An earlier version that only kept a list inside the process collided about once in 120 runs with 12 copies of the remote tests at once. With the UDP hold, 16 copies over 20 rounds had no failures (0 of 320).

• Ports that should have nobody listening (an unreachable upstream or proxy in passthrough, endings, ws, live_state and the l1 unit tests) were also released ephemeral ports, which a parallel test could start listening on. They come from the same range now, held the same way. (A socket that is bound but not listening does not work for this: macOS drops the SYN instead of refusing it, and the connection times out.)

No assertion changed. The remote-port tests still check the exact port in the status, the hot-reload tests still move between two concrete ports, and the old port is still expected to refuse connections after a move.

Reproducing it

On macOS the race does not show by itself: ephemeral TCP ports are handed out in sequence (net.inet.tcp.randomize_ports: 0), so a released port is not reused until the counter wraps. To stand in for Linux, a background process walks the ephemeral range with bind(127.0.0.1:0) and holds the last 4000 sockets. All 20 affected test binaries then run at the same time, once per round:

Build	Rounds	Failed runs	Failing binaries
main (8095405)	6	12 of 120	remote 6, hotreload 4, listen 2
main (8095405)	6	13 of 120	remote 5, hotreload 5, listen 3
this branch	6	0 of 120
this branch	6	0 of 120
this branch	6	0 of 120

The main failures in the remote tests are the same gw.listen.port_taken as on CI. In listen and hotreload they are the gateway failing to bind its configured port.

Checks

cargo fmt --all, cargo clippy --workspace --all-targets -- -D warnings, cargo clippy --workspace --lib -- -D warnings, and cargo test --workspace with the proxy variables unset: 1581 passed, 0 failed, 4 ignored.

🤖 Generated with Claude Code