Fixed #535: ThreadSensitiveContext.__aexit__ blocking the event loop.

[helgefmi]

Fixes #535.

ThreadSensitiveContext.__aexit__ is async, but it calls executor.shutdown(), which blocks the thread (it's a Thread.join). Running a blocking call straight on the event loop can deadlock the whole loop, in this case when the executor's worker thread is itself waiting on that loop. When it happens the server stops responding to everything.

The fix runs the shutdown off the loop with await loop.run_in_executor(None, executor.shutdown), so the loop stays free. This is what asyncio itself does in loop.shutdown_default_executor(). I also moved the contextvar reset to before the await.

My test deadlocks on current main and passes with the change. It runs in a separate process because the hung thread would otherwise hang the test suite.

AI disclosure: I used Claude Opus 4.8 to help track down the deadlock, write the fix and the test.

[carltongibson]

Thanks for this @helgefmi. Just about to disappear on holiday, so will resolve this when I return 🏖️

[Arfey]

Nice, the new unit test makes the error obvious 👍

[Arfey]

Still working on the code review. Investigating a cancellation problem with nested async_to_sync

[Arfey]

Finished my investigation 😅 First, thanks again for the test case, it really helped.

Running executor.shutdown() in a separate thread works, but I think it treats the symptom rather than the cause. Let me walk through what I found.

We run sync code through sync_to_async, which schedules it on the executor via loop.run_in_executor. While the sync function is running, the task gets cancelled, so run_in_executor raises CancelledError on the loop side. But the sync function itself keeps running. It then calls async_to_sync, which does loop.call_soon_threadsafe(loop.create_task, ...) and blocks the executor thread waiting for that coroutine to complete.

Now the parent reaches __aexit__, calls executor.shutdown(), which is a Thread.join, and waits for the executor thread. But that thread is parked waiting on the event loop, and the loop is the very thing being blocked by shutdown(). Deadlock.

So the real issue: we "cancel" the sync function, but it doesn't actually stop, it runs on into async_to_sync and parks. await loop.run_in_executor(None, executor.shutdown) unblocks the loop so the queued create_task callback can drain and the thread eventually unparks. That fixes the hang.

I want to float a different angle: fix the cause instead. We can't truly cancel running sync code anyway, so instead of cancelling mid-flight, we let it run until it finishes or reaches the async_to_sync handoff, and only then propagate the CancelledError. I put a rough illustration in #567.

I don't have a strong opinion on which approach is better. await loop.run_in_executor(None, executor.shutdown) has a cost, but it's simple and keeps the existing cancellation behaviour. The second one fixes the root of the problem and is closer to how async should behave, but it changes the existing cancellation behaviour. 🤷‍♂️

Thoughts?