Skip to content

fix: Run LiteLLM/WML async inference without relying on an implicit event loop - #1982

Open
wak327 wants to merge 1 commit into
IBM:mainfrom
wak327:fix/litellm-event-loop
Open

wak327 wants to merge 1 commit into
IBM:mainfrom
wak327:fix/litellm-event-loop

Conversation

@wak327

@wak327 wak327 commented Oct 3, 2026

Copy link
Copy Markdown
Contributor

Summary

LiteLLMInferenceEngine._infer and WMLInferenceEngineChat (async path) drive their coroutines with asyncio.get_event_loop().run_until_complete(...). That call raises RuntimeError: There is no current event loop in thread ... whenever the thread has no current loop, which happens:

  • after any asyncio.run() in the main thread (unitxt itself calls asyncio.run in metrics.py), on every supported Python version;
  • in worker threads (e.g. ThreadPoolExecutor, asyncio.to_thread);
  • on Python 3.14+, where get_event_loop() no longer creates a loop implicitly.

Inside an already running loop (Jupyter, FastAPI, ...) it fails with the bare RuntimeError: This event loop is already running, which is what #1647 asked to turn into an actionable message.

Changes

  • Add get_or_create_event_loop() / run_coroutine_synchronously() in inference.py: reuse the thread's current loop, or create and set one if there is none or it is closed. When called inside a running loop that was not patched by nest_asyncio, close the coroutine and raise a UnitxtError that suggests asyncio.to_thread(engine.infer, dataset) or nest_asyncio.apply().
  • Use it in LiteLLMInferenceEngine._infer and WMLInferenceEngineChat.
  • LiteLLMInferenceEngine's asyncio.Semaphore and AsyncTokenBucket lock get bound to the first loop they are contended in, so reusing the engine in another loop failed with ... is bound to a different event loop (this also affected nest_asyncio users whose engine had already run outside the notebook loop). They are now recreated when _infer_async runs in a different loop. The loop is tracked by id() instead of a reference so the engine stays picklable; a primitive bound to a loop keeps that loop alive, so its id cannot be reused.

Before / after

Same LiteLLMInferenceEngine instance with _completion stubbed out, Python 3.12:

scenario main this PR
first call OK OK
after asyncio.run(...) RuntimeError: There is no current event loop in thread 'MainThread'. OK
from a worker thread RuntimeError: There is no current event loop in thread 'Thread-1'. OK
inside a running loop RuntimeError: This event loop is already running UnitxtError with instructions
inside a running loop + nest_asyncio.apply() RuntimeError: <Semaphore> is bound to a different event loop OK

Fixes #1647

Test plan

  • New tests/library/test_inference_utils.py covers: plain call, after asyncio.run, worker thread, loop reuse with a contended semaphore across calls, and the error inside a running loop (and that the coroutine is closed, not left un-awaited).
  • Scenario table above, run end to end against LiteLLMInferenceEngine.
  • pre-commit run on the changed files.

…vent loop

asyncio.get_event_loop() raises when the thread has no current event loop:
in worker threads, after asyncio.run() returned, and on Python 3.14+.
Reuse or create the thread's loop instead, raise an actionable UnitxtError
when called inside a running loop (IBM#1647), and recreate the LiteLLM
semaphore/rate limiter when inference runs in a different loop.

Signed-off-by: Waleed Khalid <wak327@gmail.com>
@wak327

wak327 commented Oct 3, 2026

Copy link
Copy Markdown
Contributor Author

@elronbandel @yoavkatz @martinscooper could you take a look when you have a moment? This fixes the nested event loop error from #1647, plus the related There is no current event loop failures that show up after asyncio.run(), in worker threads, and on Python 3.14. The before/after table in the description shows each case.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

'This event loop is already running' error when using LiteLLM inside another event loop

1 participant